SAFE Network Dev Update - May 7, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • We ran our first internal testnet with Vaults running from home connecting to a single section and are seeing good results.
  • RFC-0052 was updated to make the RFC and the code match, and make some areas clearer for other implementers in the future.
  • We updated the SAFE App C# CI to auto-publish the API docs to GitHub Pages. The latest docs are hosted here.
  • Our AT2 implementation (which is looking like a solid replacement for the current Safecoin implementation) now has tests passing across SAFE Client Libs, Vaults and safe-nd.
  • We’ve converted the safe_core library to use futures and async/await, and are now well underway with safe_authenticator.

Vaults Phase 2

Project plan

We took a small detour to try something new this week. With the new features being added in quic-p2p, it can now fetch the public IP address of a local area network and forward a specific port of a computer to the entire Internet via a router. I’m sure you’ve already guessed what’s cooking. It’s Vaults from home. We’ve been integrating these new features of quic-p2p with the SAFE Vault and with some small fixes and refactors we ran our first internal testnet today with Vaults running from home. :tada: We are seeing some good results there and we’ve observed that some improvements can be made. Some dead ends as well, for example, routers not supporting IGD, but hey, that’s expected. We’ll be working on some improvements on this front in the coming week.

In parallel, we’ve also been continuing with the design of the multi-section network. Some interesting ideas are blooming which will improve the data handling between Elders and Adults and this will tie in nicely with Vaults running from home. The next testnet will be Vaults running from home and these Vaults will be given minimal data handling responsibilities like storing Immutable Data chunks. For this, it’s important to consider that Vaults may go offline so we are thinking of ways to handle this as well. Stay tuned!

SAFE API

Project plan

This week a large refactor of the XorUrlEncoder was merged. XorUrlEncoder has been renamed to SafeUrl, as it now encapsulates both NRS and XOR type URLs. Also, RFC-0052 was updated to clarify some vague areas, provide examples, and adjust nomenclature so that the code and RFC will be in sync.

The API and CLI are also being updated to support recursive resolution of NRS URLs for all operations.

Work has begun to support uploading and retrieving empty directories, relative symlinks, hidden files, and original timestamps of local files. The goal of this effort is to make a more robust round-trip experience and make the SAFE Network a solid option for archiving files. This effort is just beginning and no pull-requests have been made yet. More on that next week.

SAFE App C#

Project plan

This week we fixed the API documentation generation issue and updated the CI to auto-publish the updated docs to GitHub Pages. The latest API docs are hosted here. We have updated the README file to point to the latest docs.

Based on recent changes in the safe-api, we are working on a couple of improvement and refactoring changes in the SafeUrl/XorUrlEncoder APIs. The changes include exposing more APIs from the safe-api and simplifying the existing APIs for C# devs. We will update the Inspect API as well to return all the steps taken to get some data from the network.

CRDT

The base AT2 implementation update to our Safecoin implementation is looking good, with tests passing across SAFE Client Libs, Vaults and safe-nd. Now we’re moving on to speccing out new APIs that should enable Clients to formulate their transactions (including any necessary dependencies to that transaction), and punt this to our Elders for validation. We’re looking at the Client then receiving a BLS sign-share from each elder (if they validate the transaction), which should then be all the client needs to fire off a transfer request.

There are some dependencies here in terms of getting section keys, for example, but they shouldn’t block progress on the whole.

On the Sequence CRDT implementation front, we’ve been progressing with it on our core libraries. We were able to get a couple of basic operations supported in the whole lib stack, from Vaults to CLI, so we can already send requests from the client to create a Sequence in the vaults. At this point we see the requests arriving but we still have some work to finalise in the vault so the operation is applied, which is what we are working on right now.

Additionally, we will be trying to add support for all type requests a Sequence can support, so we can then work on implementing the logic on the vaults for those operations to be effectively applied.

Futures in Rust

Last week we unblocked SCL progress, with Self-Encryption updates. This week we’re happy to report that we’ve converted the safe_core library to use futures and async/await, and are now well underway with safe_authenticator. It’s not a quick job, nor an exciting one, but it’s one that’s getting us much cleaner code, as well as potential benefits from using the Rust native futures.

Routing

Project Plan

This week we merged the large refactoring PR we mentioned previously. In the resulting simpler code, some inefficiencies and redundancies that were previously hidden became apparent and we cleaned them up in a followup PR. We followed that with yet another refactoring PR in order to simplify the codebase. The goal was to remove the concept of the section version which is a number that gets incremented by one on every change to the section elders. This number together with the section prefix serves as a unique identifier of a section at some given point in time. But it turns out we already had such an identifier - the section key! So this unnecessary version is now gone and the codebase is ever so slightly simpler.

After that, we started working on improving the mechanism by which nodes keep their routing tables up to date. Currently the way this works is that after every change to the section elders, the section pushes an update to the neighbouring sections. This works, but is somehow fragile and at the same time inefficient. We are working on changing that to a pull model where the sections will request such updates only when they need it. This work is currently in progress and should be finalised soon.

BLS - Distributed Key Generation

Project plan

As we head towards the completion of the DKG crate, we are putting it through thorough and exhaustive testing, both with and without the transport layer (mock-network), before we can start putting it to use in the live network. Last week we were into testing the process with network-level errors like disconnected nodes, delay in messaging, churn happening, etc. We are now taking this to the next step by adding malicious nodes into the testing routine and making them send false values during a session. DKG should be able to handle this effectively by chucking out these malicious actors as we would be forcing 100% of the participants in the m of n to be non-malicious and responsive during the session.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

88 Likes

Woah, first - in a long time! And not a bad one to read early:

And this…

Things are cooking :fire:

And I love this… including the illusive and essential “empty directories”. Thank you folks!

37 Likes

Thanks so much to the entire Maidsafe team for all of your hard work!

30 Likes

Third!!! Now to read!!! :smile::blush::smile:

Lots of fantastic news this week. Super excited to hear vaults from home are imminent.

With not just vaults from home but also multi-sections on the near horizon, you can certainly feel things coming together with increasing speed!

Then you add routing and Safecoin to boot, and how close we just might be really comes into focus :smile::smile:

32 Likes

that sounds really really exciting!!! good progress

24 Likes

Great stuff!

24 Likes

:partying_face:

short bursts of unresponsive I worry about on my connection here. good to see this in the mix then and perhaps more importantly the idea of recovering what is useful from a node as it rejoins rather than risking defaults to square one each time.

Don’t forget its the weekend! … take a break… a well deserved one! :+1:

23 Likes

:star_struck:

21 Likes

Keep cooking, I know its going to be delicious! Exciting news once again. We are getting closer the vaults from home. Great work team!

21 Likes

Tenth is the new first !

Well worth reading through before posting :slight_smile:
Excellent progress on several fronts.

Well done to all involved . Getting really excited about Vaults from Home

Tiny nitpick - should we not also have a link to Terms Used on the Forum in addition to the Useful Links that is posted with every Thursday update?

including VfH ( Vaults from Home) :slight_smile:

24 Likes

Awesome work SAFE. These updates are really something, all progress and focus. Glued to the updates once again, I can indeed smell something cooking.

21 Likes

Nice to see progress. I’m starting to hope again that you will reach the ultimate goal.

19 Likes

VAULTS FROM HOME, yeah baby!!!
So many exciting things getting done but Vaults from home is the life of this party :tada:
(At least my party)

27 Likes

Wooooo! Vaults from home! :star_struck: :star_struck: :star_struck: :tada: :tada: :tada:

If I have to stay at home, so should the vaults. :stuck_out_tongue_winking_eye:

30 Likes

Great work ! :slight_smile:

14 Likes

You’re trolling me aren’t you?
Edit: or is it a fence-post error?

5 Likes

Sweetly sixteenth… Trolls are getting trolled by the team, luvit.

9 Likes

I can not believe it you are so close Maidsafe! :heart_eyes: :heart_eyes:

9 Likes

Troll? Moi?

mwaaaahh!!!

7 Likes

Great stuff, team!

I’m trying to keep Terms used on the Forum up-to-date. What does SCL stand for above?

6 Likes