SAFE Network Dev Update - August 6, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • We have built a testnet automation tool that will fully automate the creation of droplets, deployment of vaults and introduction of churn on the network.
  • Work has continued on design and implementation of a prototype to demonstrate the feasibility of using the crdt-tree datatype as the basis for a locally mountable filesystem (via FUSE).
  • Community member @happybeing forked the Rust polyfuse project and created a FUSE wrapper that can serve as a starting point for calling into our Rust filesystem code when it is ready.

Thanks to @Sascha for providing this week’s cover photo!

Vaults Phase 2

Project plan

Over the past week, we have built a testnet automation tool that will fully automate the creation of droplets, deployment of vaults and introduction of churn on the network. We will be using this tool for testing any new features that we implement in vaults before community testnets.

The next step in vaults will be to handle data operations on section splits and handling of message routing across multiple sections. With a significant amount of work being done on farming already, we are syncing up with the features already implemented and basing all work on top of the farming branch. We are debugging some issues seen during integration, with those resolved these features will be underway.

CRDT

safe-fs

Work has continued on design and implementation of a prototype to demonstrate the feasibility of using the crdt-tree datatype as the basis for a locally mountable filesystem (via FUSE).

A particular design challenge centered around how to support hard-links, a key feature of Unix filesystems which allow multiple filenames to reference a single file inode (metadata and content). A solution was proposed here.

We then discovered that the ino (u64) do not need to persist between mounts, so they can be kept in a local data structure outside the (shared) crdt-tree.

With such issues worked through, an experimental (quick/dirty) prototype filesystem class was built that implements the main features of a filesystem, and thus demonstrates that the design works in practice.

Community member @happybeing forked the Rust polyfuse project and created a FUSE wrapper that can serve as a starting point for calling into our Rust filesystem code when it is ready.

Going forward, our next high-level goal is to create a locally mountable filesystem using the crdt-tree for storage. This will be a single replica without any networking or policy code. With this in place, we can actually mount the filesystem and then run benchmarking tests, identify any problem areas, and hopefully demonstrate that design and performance is adequate to justify further development. The nice thing is that the CRDT properties mean that once networked together, the replicas will always converge without conflict.

Reaching this goal will entail implementing both the crdt-tree and filesystem API in Rust, then integrating with the FUSE wrapper.

Farming

SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan

SAFE Client Libs and Vault integration

We’ve been steadily progressing the integration here with the latest farming changes and tweaks. We’ve updated both vaults and SCL with the latest changes from master branches, which introduced some issues as we moved to the new XorName crate. But we tracked that down and after another wee to do with AT2 transaction history retrieval for clients, we have an almost complete message flow between the two.

Right now we’re looking into some issues client-side, where the client hangs while waiting for TransferValidation responses. It looks like this is related to our use of quic-p2p, which is slightly outdated in Rust terms, so we’ve also been upgrading this lib to be fully async, which should hopefully unblock us in the client, and also simplify some of the safe_core code significantly.

Store cost and Rewards

We have so far investigated a farming algo with adaptive store cost, essentially based on network size and amount of money in circulation, and a rate at which network owned money is transferred to farmers, a rate that was steadily declining until all network money had been transferred.

For the first testnets with farming, we’re going to use a much simpler approach, where there is a simple static store cost (1 byte = 1 nanosafecoin), and a simple age-based reward, paid out at every relocation (2^age safecoins).

The work now continues with ironing out the initial few moments of a network, and the solidity of the distributed AT2 actor, also known as the section funds.

Routing

Project Plan

First of all this update will be to celebrate the good news that Adam managed to spare some time (usually a half-day) every day to get back to work. Welcome back to the team, Adam! (he even apparently got some sleep)
So with Adam back, we are first reviewing the work during his leave. There are a couple of improvements spotted, and we have merged the work of only send promotion to correct new elders, sending DKG message to the participants directly, and Remove Genesis event handling. And the on-going work of remove bounced_unknown_messages cache
shall be merged soon.

We also started investigating the last remaining feature refactoring work (Remove relocate_queue and make relocation independent on the order of churn events) before moving forward to PARSEC removal. There was a fruitful debate on the potential approach on relocation and we got that documented and updated.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together!

:love:

73 Likes

First :slight_smile: an now to read it

Sounds like @happybeing is playing a blinder – Gawn yersel Marky-boay!!!

“Add strength and lightness” Colin Chapman

Good … KISS to get us running and testing, then we can work later towards the more theoretically “correct” method to try to suit the purists – I hope that does not sound too dismissive, I’m very keen myself that we have a long term model that is fair to all participants. - later.

Excellent progress everyone, good to hear Adam is back as well – Thanks to you all.

27 Likes

Can’t wait for the next community testnet! Great job team!

22 Likes

#MeToo - my hammer is getting rusty

12 Likes

Thanks. I’m doing my best but @danda is doing all the hard bits and has made great strides. He’s been tearing through tricky code and come up with some very neat solutions to hard problems such as how to implement hard-links within the TreeCDRT.

Most of my time has been spent learning and discussing all the different parts with him, and hopefully assisting by sharing and discussing. I’m still a Rust virgin, so not ready to write the code for this which is quite advanced. That’s frustrating but I hope to find some parts I can get my teeth into once there’s a framework in place, or if I can pull code from other areas and fit it into safe-fs.

31 Likes

BEA-utiful! gogogo team! we are getting close!

20 Likes

Man, @Sascha is so into chicken. And this project is great. Adam, welcome back.

20 Likes

Keep on coding, thanks team, great job as always.

17 Likes

Thx for the update Maidsafe devs

Welcome back Adam

Would be great to see this in action :beers: really so super exited for the next testnet… :muscle:

Keep hacking super ants, this puzzle is coming together nicely :stuck_out_tongue:

Nice picture @sasha :chicken: :deer: :deer:

20 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! And thanks to @happybeing for your contribution too. :racehorse:

17 Likes

Keep it up team can’t be long at this pace :clap:t2:

17 Likes

Wow.

We are already looking at test farming!

:tada:

14 Likes

The progress is absolutely staggering as of late. It feels like a straight road to Fleming now. :+1::+1::+1:

14 Likes

Welcome back, Adam! Thank you to the whole MaidSafe crew and @happybeing for continuing to push the work forward!! Looking forward to the next community testnets as well :smile:

17 Likes

It has been some time when I’ve been reading the last updates. Looking good though. Still good progress being made.

Back in the beginning 2017 I’ve been reading the whole maidsafe wiki fundamentals but there have been a lot of changes in even the core of how maidsafe is working if I’m correct. And now I’m looking to update my understanding of the basic principles of maidsafe and I was looking at this primer but I see that the latest update is of november 2019 and parsec still seems to be a large part of it and if I am correct it shouldn’t be? https://primer.safenetwork.org/

Can someone explain to me how to get back up to speed and if it is still worthwhile to read the primer this weekend or if there is another source that might be better suited for this?

4 Likes

Oh yes - this next testnet will be worth waiting for - even if only for the large array of features I can attempt to break :slight_smile:

8 Likes

Since the team got slimmed down to 99% engineers there has unfortunately been less resource available to update documentation. Yes read the primer but the unfortunately I think the best way to get up to speed is to trawl through the last few moonths archives here and on https://forum.safedev.org/ - If you can read backwards that might help too :slight_smile:
PARSEC got dumped because while it proved an invaluable concept, it was slow, pedantic, unnecessarily complex and was very “chatty”. Fine as a Proof of Concept - not so fine for a production solution.
But it was 100% necessary to climb the PARSEC mountain to see the better alternative routes in the valleys below .

12 Likes

This is awesome! It will make the network very real when this sort of feature starts slotting in.

Great work team!

10 Likes

I found the video in this post useful: Conflict-free replicated data type explained

11 Likes

The pace is crazy! Everyone @maidsafe we truly appreciate you and the progress you bring us each week to get us closer to privacy, security, and freedom. Following along has been so interesting and it’s so neat to have been around as long as I have and to be able to grow with the project and it’s community die hard devs. Best of the best.

@danda and @happybeing I’ve enjoyed reading your progress over on the dev forum.

Welcome back @adam

@Sascha I like your picture a lot, looks very peaceful.

Can’t believe farming is coming! I’m sure many here felt like it would never come but yet, here we are! Get your straw caps, overhauls, and hard drives ready! :man_farmer:

23 Likes