SAFE Network Dev Update - May 21, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • quic-p2p v0.6.0, which includes IGD, was released today. :tada:
  • The new, work in progress, safe-transfers repo has been made public.
  • Support for PUT/GET of empty directories has been added to the CLI.
  • @jimcollinson provides an update on the progress on the SAFE Network App UX.
  • CRDT progress continues - we are now ready to start adding the actual CRDT logic to the client.
  • We merged the pull model PR into Routing, making section updates more robust and efficient.

Vaults Phase 2

Project plan

Over the past week, we’ve focussed mainly on implementing the way Vaults handle data when Adults join and leave the network. This is shaping up nicely and about 80% of the implementation is done. There are just a few more things to consider and of course a lot of testing before the changes are pushed upstream. This will fit in nicely with the latest changes in quic-p2p that introduce port forwarding using IGD. A new version of quic-p2p was released today with these new features. We have already completed integration testing with Routing and Vaults in the last couple of internal testnets and we look forward to the next one, which will include the data handling as well. We are looking at a community testnet in the future with these features included so stick around!

SAFE API

Project plan

Support for empty directories was merged this week after fixing some test-case issues on the Windows platform. Also, a pull request has been raised to include hidden files (beginning with .) in PUTs. Previously they were excluded with no option to override.

Meanwhile, a fix for NRS paths in files ls (issue 510) is currently being tested locally.

Finally, a small refactor was made in the foreign function interface (FFI) to represent FilesMap as a native data type rather than a JSON string.

SAFE App C#

Project plan

As mentioned in the SAFE API section above, this week we refactored the Files APIs to return a FilesMap instance instead of a JSON string. This simplifies the usage of the Files APIs as devs can directly query the FilesMap data instead of going through the JSON operations.

In parallel, we have made some progress on exposing the authenticator APIs using the bindings from the safe-ffi library. We still have to expose some more APIs and add a couple of unit tests to ensure the integrity of them. Once completed, the Authenticator app can be refactored to use the APIs from the package, meaning we wouldn’t need to maintain the bindings in two different repos. This will also simplify the development of the SAFE Network App in the future.

SAFE Network App UX

Slightly truncated sprint this week due to lockdown home-schooling shift patterns, but progress nonetheless. Not much razzle-dazzle, but working through accessibility tasks relating to the design system, as well as continuing through the UI layers on top of this all.

We’re majoring on the functional here, so while we’re aiming to add just a smidgen of ‘SAFE’ personality and branding into the UX, this should never be at the expense of usability, and function.

We’ve gone around the houses somewhat on that, and while you’ll only see some subtle and incremental changes in the screens below, it’s all tending towards something that will hopefully be highly useful, and highly used. And of course not forgetting all the componentisation behind the scenes that will make things easier to improve and build upon!

CRDT

This past week we continued with the implementation of the Sequence data type as a CRDT. As mentioned previously, the first steps have been about having all the core libraries support its type of requests. We’ve now added the same to our safe_client_libs mock-vault implementation which helps us validate all client-side logic for any type of request.

In addition to that, we migrated the NRSContainer and FilesContainers (in safe-api) to start using the Sequence data type, and we were able to verify this is all working seamlessly as before, with all our CLI E2E tests passing when using the mock-network.

We also tried to run these E2E tests with a Baby Fleming local network, and we are at the same stage as with older data types, which means we are now ready to start adding the actual CRDT logic to the client so these issues of content not being found in some unsynced vaults are resolved.

Transfers

Following up on the past few weeks of research into AT2, we put up the new safe-transfers repo, a work in progress that captures the essential properties of AT2, applied in the SAFE Network setting.

A short description

We have Actors and Replicas to carry out the different steps of the AT2 algorithm. An Actor instance is run by a client, and it initiates transfers from its key, to some other key. The Replicas are run by the network nodes - more specifically the Elders of a section - and will validate the request and use multi-sig to prove their agreement on its validity.
A section will run a single Actor to manage the section balance, and farming payouts, as well as receive payments for data uploads.

Next

Work will now continue with testing and ironing out details around section keys handling for their Actor instance.

Links

Futures in Rust

The last week has seen more intense Futures conversion work. We’ve chucked out the internal event loop, which was causing no end of trouble. This can easily be replaced in lib consumers by more specialised runtimes. And now we have safe_core compiling (after fixing some underlying issues in the self_encryption multi-thread conversion) and all tests are now passing too :tada: .

safe_authenticator and safe_app both also compile now, with conversion of the test suites underway, and we’re in the home stretch with the conversion of safe_authenticator_ffi now too.

It’s been a big job (a lot of code to go over), but as things are coming together it’s looking tighter, cleaner and leaner than ever!

Routing

Project Plan

This week we finally merged the pull model PR we talked about in the previous dev updates. We then enhanced the API by adding a method to retrieve the current section BLS public key set and added an event which is raised whenever the section elders change. This was another feature request by the Vault team. We followed that with a small refactoring to further simplify the code and remove unneeded bits.

After that, we started working on improving the way nodes handle messages whose trust they can’t establish. Currently, this is handled rather crudely - the message is bounced back to the sender who then waits a short while before resending the message again, hoping that the node caught up and is now able to process the message. If not, the process repeats. The problem is that it’s generally impossible to determine how long the “short while” should be - either it is too short and the node still can’t handle the message, or too long which slows down the message flow and hurts performance. This is a general problem whenever we introduce time into our logic and is why we usually try to steer away from doing so. The new way will get rid of the delay and instead we will resend the message immediately, appending any additional information the recipient needs to be able to process it.

BLS - Distributed Key Generation

Project plan

Last week we wrapped up with our testing spree on the crate. With the feedback gathered, we are now working on implementing some tweaks to the mechanism that makes the user of the DKG no longer need to handle specifics. As it stands, the user has to call the APIs manually to reply to messages and force transition to other phases. We’re updating them such that the user doesn’t have to do anything manually with them.

This led to us using internal timers for DKG to complete a session without any user intervention. By default, users/nodes who initiate a DKG session have to participate in six DKG phases, in between each of which messages/replies would be multicasted. In a happy case, the user would not have to intervene with the process as once all six rounds of messaging are done, nodes can conclude the session on their own without a problem. Whereas with unhappy scenarios (real-world problems), the six rounds of messaging might not get completed (due to, for example, dropping of nodes or messages) and would require the user to intervene with the process to complete the session.

Timers are being introduced to mitigate this intervention where we set a specific timeout to wait for all the inflight messages to be handled. On timeout, the DKG can automatically check the readiness of the Generator. If the six phases are not completed and the Generator is not ready, the mechanism would force itself to continue the session making it run the remaining phases, which can either be successful if the threshold had been reached, or fail if not. Note that this timeout value needs to be set based on the number of nodes in a group, as the amount of time required to complete the messaging rounds would increase if there are a higher number of nodes in the session.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

75 Likes

first - and now to read it…

Looks like a lot of hard work has been put in since last week - All this is inexorably moving in the right direction.

Thanks again to David and all the team.

25 Likes

secondo read now

Thanks Maidsafe devs for all your hard work.

Huh the Snapp UX images links are broken

Would it be an idea to have the people who wrote the 82 whitepaper look at the code? :stuck_out_tongue:

Really good progress devs

24 Likes

third :slight_smile: long time

17 Likes

That’s a nice change to see…

I understand git-hub words but unclear the reality for accessing that difference…

Will CLI be released as v12.1 perhaps or new tag that sees it downloadable without some need to build from scratch?

:+1:

14 Likes

Very exciting :grinning: anticipation is palpable!

Going to shamelessly plug a post I made in off-topic for help with Filezilla here
If anyone can tell me where it is that I am being dense it would be hugely appreciated.

20 Likes

This is a fantastic update. Many thanks to the entire Maidsafe team for all of your hard work.

16 Likes

Thanks for all the hard work again everyone.

Exciting to hear about the successful integration of new methods of data handling etc. Not super familiar with other projects for comparison, but sounds like the whole package will be really something…

16 Likes

It seems things are nicely fitting together as expected. I am really excited! Thank you team, great work!

16 Likes

@JimCollinson for some reason I can’t view the images or links in the SAFE Network app UX section. Anyone else having this issue?

This kind of work is under appreciated but we all appreciate it! Every little bit that makes SAFE simpler and faster is essential to being a long term success.

Maybe an interesting way to think about this is a section is made of many nodes but in terms of the section actors, they are seen almost as a single node amongst other section actors. Kind of reminds me of fractals and the building blocks of nature.

Can’t wait to see these routing upgrades, CRDT, BLS progress, and vault progress all come together. This is going to be epic!

14 Likes

Don’t get too excited :joy: Here they are. Perhaps a mod could drop them in to the main post again:

25 Likes

Great UI @JimCollinson, definitely my taste - hope others like it too!

19 Likes

You sure make the SAFE Network look good sir :vulcan_salute:

16 Likes

I’ve been doing some mock ups in figma (which is strangely addicting) and what I’m mocking up will have heavy use of SAFE ID’s so I’m really excited for them!

18 Likes

Ooo, exciting!!! Really great pace and progress :smile:

19 Likes

These changes will be in the next release of safe-cli. (The hidden files PR has been merged today btw). I do not have any date for next release.

The only way to access the functionality right now is to build safe-cli oneself.

18 Likes

Couldn’t find anything entertaining online to watch, so I ate my large snack while reading the update.
Enthralling climax: “If the six phases are not completed and the Generator is not ready, the mechanism would force itself to continue the session making it run the remaining phases, which can either be successful if the threshold had been reached, or fail if not.” Can’t wait to tell my friends about this show!

8 Likes

This update feels akin to reaching a summit and realizing the gradient ahead has changed.

9 Likes

Good work team!

7 Likes

For all us non techies, is the gradient getting steeper or lesser?