SAFE Network Dev Update - July 16, 2020

Vaults Phase 2

Project plan

The past week we’ve been testing and finalising the next iteration of the vaults from home testnet. After a few rounds of testing and bug fixing we are happy to announce the next iteration of the vaults from home testnet that has been released today :tada:.

Head over to the release post to play your part, all are welcome.

In parallel, we have been working on some of the next steps, which are primarily focussed on multiple sections. We have also been refactoring our test suite with multiple improvements which include using real Routing for the tests on our continuous integration platforms.

SAFE API

Project plan

Today we released new versions of safe-api, safe-cli, safe-authd, safe-ffi and jsonrpc-quic, all to support today’s new vaults from home testnet iteration :tada:

See the changelog for highlights of the new features and bug fixes which are in the new releases, and of course see our detailed CLI User Guide for step by step instructions on how to use all new and existing functionality.

We have begun to explore possible design improvements for the SAFE File API.

In particular, a recent research paper (and video see 35:30) caught our attention that could dramatically improve performance, help simplify our API, and solve collaborative editing challenges. The paper is titled A highly-available move operation for replicated trees and distributed filesystems and it details a formally proven CRDT Tree data type that is suitable for filesystem usage and, being eventually consistent, solves a longstanding move operation problem that affects even Dropbox and Google Drive when dealing with concurrent updates. It is also suitable for working offline.

The core design idea is to add Tree as a SAFE Data type, and implement a FileTree as a specialisation, just as FileContainer has been a specialisation of Sequence. The important difference is that each Directory and File is stored individually in the Tree, instead of all serialised together as a flat JSON map in a single Sequence entry. This makes updates and retrievals much more efficient for any non-trivial directory structure.

Going further, we would like to incorporate elements of Safe.NetworkDrive for fast local access and offline usage, as well as native file system integration via FUSE.

To be clear, the idea is that SAFE apps could access the File APIs directly if they wish, but non SAFE apps could still interact with SAFE files via a local mount. This would mean that tools such as rsync could be used easily, file explorers, etc.

For the moment, this remains brainstorming and high level design. There’s a lot to solve. A first challenge would be to implement the CRDT-Tree algorithm in Rust. It presently exists only in the form of Isabelle/HOL formal logic. If any community member is proficient with Isabelle/HOL and could help translate it to Rust, or even pseudo-code, that could help speed things up.

In parallel, we have been trying to expose the Authenticator FFI APIs from the safe-ffi crate. This will unify our APIs on all platforms. During the last couple of weeks, we were facing an issue with FFI APIs when trying to connect with the real network while the mock APIs were working fine. This week we fixed this issue and we can now continue with the final round of testing and review.

SAFE App C#

Project plan

As we mentioned in the SAFE API section, resolving the connection issues in the safe-ffi has made it possible to test the new Authenticator APIs in the C# APIs. This means in the coming weeks we will be releasing a new version of the NuGet package which will contain all the new APIs, including the Sequence Data APIs and Authenticator APIs. Once this package is released it will be easy to update the Authenticator and SAFE Mobile Browser apps to support the latest Vault.

These recent changes have already proven very helpful and for the very first time we can easily set up our CI to run the tests against a real network :tada:. Until now we would have to run these tests manually before a new package release, while on CI only mock tests were able to run.

SAFE Browser / SAFE Network App

The Browser and the SAFE Network App have been somewhat neglected of late with the focus being on the various testnets and underlying AT2 work. But we’re at a good place now with the latest vaults and underlying SAFE API changes so we’re working on a new release to be compatible with the latest Vault and the test network.

This won’t be arriving today, but we’re building and testing things locally, so these should be with us in the not too distant future.

CRDT

This last week we continued with our research around Access Control on our CRDT data types. We are mainly looking at two different approaches at the moment, a first one which allows operations to be applied even if they depend on an older version of the Policy, and a second approach where such operations are not accepted, or even undone if a new Policy makes it invalid.

Both approaches have their own complexities and implications which we are still trying to fully understand, not only from an implementation point of view, but also from a user experience point of view.

In both cases we need to make sure we keep track of this dependency (causality) information between data and Policy operations, and that’s why we started a small refactoring in our current Sequence CRDT implementation to make sure it can accommodate the upcoming changes, regardless the approach is taken.

Transfers

SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan

This week has seen our network messaging refactor settle down somewhat, after an additional iteration. With the major changes now into safe-nd and applied in vaults, we’ve been bringing safe-client-libs back inline and making some improvements to message handling there now that we’ve increased specificity of the network messaging. Additionally, this has allowed for a more or less complete flow of farming rewards to be implemented in vaults. We are currently proceeding with cleaning up some internal messaging patterns there.

In SCL we’ve set things up to allow proper handling of network events (such as validation of an AT2 payment request), the full query flow (which all our messages were going through previously, even if that wasn’t necessary), and finally put in place the basics of receiving errors from the network. Actually handling such errors will likely be more of an app-level decision for developers to take.

Now the focus for SCL + vaults will be testing testing testing these interactions.

Routing

Project Plan

As part of the work to improve the code quality, we started to split out some Routing code blocks into separate crates. The first crate we split out this week is safe-network-signature-accumulator, which is dedicated for generic BLS signature aggregation for the SAFE Network. There will be more code blocks split out when this becomes stable, to reduce the complexity of the Routing logic. Tidy up like this is something we feel is essential for launch as each small module can be thoroughly tested and documented with greater clarity than when mixed with other code.

As mentioned within last week’s dev update, the first task towards removing the parsec usage, ensure mutations to SharedState are approved by the section, was merged this week. This enables us to go further with the task of overhauling the node promotion. However, due to Adam’s parental leave (huge congratulations to him :tada:), the work is replaced by PR 2160, which was merged today. We are now a step closer to implementing message-based voting, which replaces the current parsec-based voting.

Useful Links


Feel free to send us translations of this dev update and we’ll list them here:

:germany: German ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together!

:love:

77 Likes

Bäm! First in a long time!

28 Likes

seconded !!!

22 Likes

Third :nerd_face:

21 Likes

If your router does not (correctly) support the IGD protocol, you will not be able to take part in this testnet iteration.

I will have to wait again for the vault, as that sounds the same as last test for who can start vaults.

but still…

you do not need to run a vault in order to participate in the shared section testing. Whether you are running a vault or not, you can still connect to the network as a client and do everything you could do on the previous single section networks that you may have ran from home or that MaidSafe hosted and you connected to. See the CLI User Guide for a comprehensive list of available commands and instructions on how to use them.

:+1:

17 Likes

Thanks Team SAFE

16 Likes

Nothing but progress! Amazing efforts by the team. The researching the team has been doing to find existing solutions that fit has been extremely beneficial. Great work @maidsafe and congrats on the second vaults at home release!

13 Likes

This sounds fantastic! :partying_face:

As you know I’ve also been looking at implementing a FUSE drive in Rust, based on Syncer which I now understanding well enough to pull to bits and adapt to the SAFE APIs.

One of the things which I noticed was that it would be much cleaner to implement directories, files, links, symlinks etc as nodes as Syncer does, so the FileTree approach could give us a much more conventional filesystem which would be excellent. The Syncer code seems well thought out and clear, and might well be a good starting point for your own Rust FUSE. I can share my notes if it would help you review the code.

I’d reached the stage where I was ready to have a go at a FUSE mount of the current SAFE FilesContainer, by using about half of the Syncer code, bypassing its disk caching and plugging it directly into a SAFE backend. However, I think I should put this on hold!

I had also started to look at how the SAFE API and core libraries work in order to implement caching for speed, and streaming read/write which are important for many applications and POSIX compliance.

I’ll keep poking around to improve my understanding, but am now wondering if there is much point in me continuing the work I had planned?

I’m happy to collaborate or to take on a sub-project, or equally might go back to some one of my other projects that I set aside when I stumbled on Syncer. I’m new to Rust, but do feel ready to have a go at something reasonably complex, having read an awful lot of Rust code in the last couple of weeks and made lots of notes to understand how it all works.

So @Maidsafe, please have a think and let me know what options seem sensible. I’m quit happy to step aside as I have lots of other stuff to do, but I’m also quite keen to try doing something in Rust while I’m up to speed. So I might have a go at something regardless.

30 Likes

This sounds huge! Also sounds like it would solve one of the problems I was pondering over on the search app thread.

Thanks again for all the hard work everyone. So many steady weeks and then it seems like everything happens all at once!

15 Likes

gogogo team!!

14 Likes

That’s what we always want. Perhaps @danda can get together for a quick brief? Thanks Mark.

20 Likes

Thanks so much to the entire Maidsafe team for all of your hard work. :racehorse:

15 Likes

:tada: indeed! Must have been somewhat nerve wrecking pregnancy with all the covid-19 situations etc. Is this your first one? If so, you are in for a wonderful journey! (My daughter is a bit over a year now.) BTW Netflix has good documentary series about the babys first year.

… and thanks for the team for all the work done! Incredible progress.

16 Likes

I thought you might like it. :slight_smile: me too.

Yes, a goal here is that each FileSystem object is represented uniquely all the way through the API and also in the data storage. This makes update and read operations much more efficient, and the resulting API more flexible.

Sure, if you can send me a pointer, that would be great. I’ve taken a brief look at syncer, as well as surveying various other fuse based systems, rust and beyond. Initially though, the focus will really be on getting the CRDT data type working.

Until we get the Tree CRDT type translated to rust and are able to test it out, this is all just theory. Even then, there could very well be some showstopper. So in that sense, you might want to continue your efforts against the present File API. However, if all goes as we hope it will, that API would be replaced. (Though a thin File API compat layer could possibly live on, not my call.)

That’s great. Ok, so here’s a couple things I could use some assistance with:

  1. Translate Isabelle/HOL formalism to Rust, or even to pseudo-code. There is also a Scala implementation that was machine generated from Isabelle/HOL but it is (a) inefficient and (b) extremely obfuscated, so I think not a good starting point. If you or anyone here can assist with this, it would be amazing.
  2. I’m still trying to decide if its better to use the Fuse low-level (inode based) or high-level (path based) API. Keeping in mind of course that we would like this to work on Windows also. Path based seems easier, but has limited support in rust/unix… ie only fuse-mt. I’d be curious to hear your thoughts on that, and how an inode u64 might be used/mapped if going that route.

In any event, the idea will be that we have a FileSystem API/crate that can be called directly by SAFE apps, and is also used/wrapped by FUSE daemons (for unix/mac/win). So we will need these wrapper daemons, and that may be something you can get started on…

Hopefully the above gives you some ideas. I will share some design docs as things flesh out a bit more.

edit: @happybeing if you’d like to create a new thread for this, we can continue discussions there.

16 Likes

Sounds good @danda, I’m distracted atm but will get back to you. Very exciting :smile:

13 Likes

Brilliant update.
Brilliant test network.

You can clearly see that we put blockchian tech to shame.

In zero time… tonight I

  1. Installed the safenet-cli and connected to the network, didn’t need to sync 100GB blockchain.
  2. I was able to create an account, and fill it with test coins.
  3. I was able to send test coins to another person instantly.
  4. I was able to upload a file to the network, immediately.
  5. I was able to retrieve a file from the safe network, immediately.
  6. I was able to create a website, and link it to the file container containing my file, immediately.

Block chain, bitcoin… feel like stone age technology compared to the experience I had tonight.

:rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl:

I just want to state for the record books,

Craig ‘Faketoshi’ Wright did not create the SAFENetwork or SAFECoin.

Bitcoin is about to become irrelevant.

36 Likes

9 posts were split to a new topic: What do you think about using a SAFE female sticker?

This is super exciting news!!! Broadcasting the news here: https://twitter.com/sotros25/status/1283914731410149376?s=21

More grease to your elbow, MaidSafe Team!! Also, congrats to Adam! :smile::blush:

9 Likes

Finally, someone who speaks English. Thanks for the explanation. I am so excited to understand what this means

11 Likes

Hello everyone, here you can find a German translation of this weeks dev update. Enjoy

12 Likes