SAFE Network Dev Update :safe: June 11, 2020

Vaults Phase 2

Project plan

We did promise you a new toy and today it has been delivered :tada:

We have been testing and iterating furiously all week to perfect the previous months of work that went into producing a testnet which supports vaults from home. The full details of this release, along with instructions on how you can participate, can be found in this post.

This new testnet includes some exciting features that we have been working on so please head on over and give it a shot either as a member with a vault of your own, or even as a client creating data and performing transactions that we can see on our vaults.

We will now be fully focussing on the next steps which primarily include section splits, data organising and client request processing across multiple sections.

Another milestone reached :rocket:

SAFE API

Project plan

Work continues on symlinks. The final step is to enable resolution of symlinks in a SafeUrl path. Most programming languages have a realpath() function (or equivalent) that strips out ./ and replaces ../ and each symlink in the path as appropriate to generate a final real aka canonical path. This week we have written a realpath() function that understands the metadata in a FileContainer and is called within the Safe::fetch() API so it will work for all SafeUrl resolution lookups. Some improvements and testing remain, but hopefully a PR can be ready early next week with full symlink support.

CRDT

This past week we focused on the client-side of the Sequence CRDT, starting with creating some initial E2E tests on the safe-api crate, and running them against a Baby Fleming section. We were able to have very basic scenarios working for creating Sequence content, appending and retrieving data from it. We’ve also migrated the FilesContainers and NRS Containers to make use of Sequence CRDT (instead of the AppendOnlyData type) and all E2E tests are also passing against a Baby Fleming section.

We also worked on adding an LRU (Least Recently Used) cache on the client side which is where the local CRDT replica is being held. We will have to research more about what kind of strategies we can use in terms of updating the cached content, e.g. if an application is using some content shared with other clients it may want to refresh every so often, even though its own mutations will always be merged on the network, but we’ll be researching more about the possibilities related to this after we have all basic scenarios for a single active client fully functional end-to-end.

Lastly, we started converting the other parts of the Sequence data type to also become CRDT, i.e. the list of permissions and owners that is kept within any Sequence instance. This will allow us to then test some more complex scenarios, such as two different authorised clients making mutations on the same content, and all the CRDT magic happening when concurrent appends/mutations are merged on the network with no conflicts.

Transfers

SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan

SAFE Client Libs integration has been progressing well during the last week. We’ve updated the core library for the new AT2 style transactions and have started with the basic integration of a TransferActor. This will be a wrapper struct abstracting away a lot of the request logic and management that were previously handled in the client layer APIs, giving us a bit more modularity there. We’ve started unit testing this, and are progressing with updating tests across the core lib in its entirety.

It has been a pen and paper week for AT2 safe-vault and safe-transfers. We are now designing and setting up a mechanism (SectionActor) which takes responsibility for the Money that is paid by the clients for the data they upload. A Section in the SAFE Network would have its own account which is maintained by this distributed SectionActor, situated at the Elders of that section. When a client pays for uploading data, the money gets credited to the data-storing section’s account, and the instances of the SectionActor are now responsible for distributing the rewards to the nodes that have done the work (of storing). This work ties in with the farming, where we are currently testing granular rewards with batched payouts. We are improving and refining this flow as we proceed ahead with the implementation simultaneously for this all across the board.

Routing

Project Plan

Parsec removal work is going ahead at full speed. Last week we mentioned we are making all the data in the shared state signed with the BLS signatures to make them self-validatable. This work hit some unexpected complications earlier this week and so is taking longer than we wanted. But the hardest issues seem to be solved now and what remains is relatively straightforward. Expect a PR soon.

We are also almost ready to replace the DKG (distributed key generation) module with the BLS-DKG crate. DKG wasn’t originally meant to be part of Parsec (there is no mention of it in the whitepaper) but it was tacked onto it because it was the simplest way to implement it at that time. It has some serious limitations though, so this replacement is definitely a big improvement, regardless of the Parsec removal itself. The PR is currently undergoing review and there are still some minor bugs to squash, but we expect it will be merged soon.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

85 Likes

thanks for sharing!

25 Likes

Hey - nice pic!

My favorite mountain!


Albert Bierstadt_Sunrise on the Matterhorn

Now to the words…

25 Likes

Awesome work and virtual :beer: for all of you!
(hopefully one day it will be a real one)

25 Likes

Wow you kids are so quick!

Thx Maidsafe devs now read :crazy_face:

Jippy finally again vaults from home :kissing_heart:

28 Likes

These words are so exciting to read! Thanks once again for incredible work team!

31 Likes

Fantastic work team Maidsafe. Vaults from home testing is such a big step, but I see lots of other progress too. CRDT and Sequence particularly exciting to see being rolled out so fast.

Thank you all. I won’t be able to run my vault continually as it isn’t sunny enough to leave power on overnight, but it is doing its stuff for the next few hours!

I shall have a go with creating an account too.

23 Likes

Awesome work. Vaults from home is a major achievement and the pipeline is still full and moving quickly. Can’t wait to fire one up.

19 Likes

Does this mean there is now a plan to reward the first vaults that store a chunk?

Does the new network work with the browsers? I don’t seem to be able to connect to the default network in the Android browser anymore.

It’s great to see these pieces fall into place, big thank you to everyone involved for all your work! :smiley:

19 Likes

Woohoo!!! :tada::star_struck:

Such fantastic news today!! Super exciting to see the new testnet!! Keep up the great work!!! Such amazing stuff :smile::smile::smile:

29 Likes

Thanks so much to the entire Maidsafe team for all of your hard work.

17 Likes

Vaults from home!?

image

Finally, I can move my stash online!

24 Likes

Hey, very nice work, feels so much more real now. Thank you!

16 Likes

Thank you.

image

19 Likes

Note that the browser on both desktop and mobile has not been configured to work with the new section yet, something I’m sure we’ll be looking at in due course :slight_smile:

18 Likes

ah ha!.. that answers a question before it’s asked. :+1:

Just wondering that I could download with CLI and not the browser.

So, I’m thinking the network has been rebooted of content, which is great… always like a clean slate a key points.

13 Likes

Yep, we’re using fresh new nodes for this testnet, so completely clean

14 Likes

A competition between devs to see which browser can be made available first…

9 Likes

Beautiful job guys and gals - some amazing work. :tada:

On an unrelated note, I really like the intro images w/ the SAFE Network + logo. It would be great to see this concept expand outwards into a broader marketing push, whenever the time’s right for that. But I find the “everywhere-ness” of these images apropos and compelling. My 2c …

13 Likes

Not just the first but all the nodes that store chunks in that section(this is just a gist of the upcoming plan). It all begins here! :wink:

21 Likes