SAFE Network Dev Update - June 4, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • We released new versions of the API (v0.13.0), CLI (v0.13.0) and safe-authd (v0.9.0) which bring several enhancements and fixes, e.g. the files put command now supports uploading empty directories and hidden files.
  • SAFE Client Libs is now fully set up for Rust async/await.
  • We’ve collated and answered a few of the most common questions surrounding last week’s announcement of the removal (or significant reduction) of Parsec.
  • Work is now underway to remove Parsec, with the first step being to attach a BLS signature to all shared state information.
  • The Parsec crate itself is not being completely abandoned! We just released a new patch version which updates dependencies.
  • The BLS-DKG crate is now considered complete and work is underway to integrate it into routing.

Vaults Phase 2

Project plan

Over the past week, we implemented a number of improvements in the way Adults and Elders communicate with each other within a section to handle immutable data requests. This includes the messages exchanged to maintain the required number of copies for every chunk. A couple of internal testnets helped us identify some bugs in the implementation and the PR is now approved. We are now running what we hope to be a final iteration of the internal testnet to make sure everything is healthy before releasing it into the wild. Assuming internal testing goes without any major hitches, you’ll have a new toy to play with soon!

We have some ideas in mind for the next steps going forward, for example, distributing data management responsibilities to new Elders when a section splits, and handling the event of an Elder leaving the network. We will be starting implementation in the days to come.

SAFE API

Project plan

This week we released new versions of the CLI (v0.13.0) and safe-authd (v0.9.0) which bring several enhancements and fixes, along with some new features for the files commands. E.g. the files put command now supports uploading empty directories and hidden files. As usual, you can use CLI to update both the CLI (with $ safe update) and safe-authd (with $ safe auth update), or follow the instructions in the CLI User Guide for installing them for the first time, and/or any other instructions.

A new version of the safe-api (v0.13.0) is also part of this week’s release. This now makes use of the very latest version of the safe-client-libs, which exposes its API as async. With this we finished the migration of all our crates in the safe-api and safe-client-libs repositories to use the Rust async/await syntax, which will also allow us to make many more enhancements and simplifications to the safe-client-libs codebase as we move forward.

Meanwhile, work on supporting symlinks continues. In the latest development code (no pull-request yet), it is possible to PUT and GET both relative and absolute-path symlinks, and also they display correctly in files tree and files ls. Symlinks may be disabled using a new --follow-links flag to files put, resulting in the old behavior.

One speed-bump encountered: on Windows, writing symlinks to disk requires certain permissions (depends on the exact OS version), so files get may skip the symlink and issue a warning in such case. Git also has this problem, and typically writes a small text file containing the link-target. We may go with that approach also. As a Windows user, symlink support is enabled by choosing run as administrator when starting a command terminal, or one may enable Developer Mode in Windows 10+.

The next development work will be to resolve relative symlinks when used within a Safe-URL path.

CRDT

After the new release of safe-client-libs early this week, we resumed our task with the Sequence CRDT implementation. We’ve done so by adapting the client so it keeps a local replica of the CRDT content (including its causality information) in order to be able to apply any mutations locally.

What this means is that the client would be making the mutations in its own local replica of the CRDT content, and once that succeeds, it will simply broadcast the operation to the corresponding section on the network for persistency.

Note that once a mutation has been made successfully on the local replica, it means it will be merged on the network’s replicas without conflicts, therefore the client won’t need to wait for any type of consensus response from the network, it will only need to make sure the request was received by the network.

As part of this effort, we adapted the Sequence “append” requests to contain the causality information which is needed to merge the operations in the network’s replicas. We are now looking at ways to make sure that whenever we read the content we get enough copies from the network to be able to get the most recent version of it.

Transfers

Project plan

Now that we have async/await merged in safe-client-libs, we’re updating our SCL AT2 implementation for this setup, and updating it to meet the latest changes in safe-transfers itself. At the moment that’s involving setting up a local Actor for managing/merging validation proofs from elders, and effectively wrapping all our Money requests to the network. This should hopefully nicely encapsulate that logic client side and make some of our client APIs that wee bit clearer too.

As to initiate an Actor, the Replicas need to expose an API that lets a client query for validation proofs of an account, from which an Actor instance can be initiated. Additionally, this supports ongoing synchronisation from Replicas. We are currently getting that in place.

At the other end of the spectrum, we have begun to upgrade safe-vault to accommodate the AT2 safe-transfers framework, which will be handling the flow of Money in the SAFE Network - in the CRDT way. Replica(s), just as the name says, are replicas of the client’s Account and their Money balances in the network. The responsibility of managing Replicas stays only with the Elders as they are the most trustworthy in the network. As churn happens in the network and as Elders join and leave, it is crucial to maintain the Replicas by generating and updating them respectively. We have started working on these mechanisms first and will be progressing to implement the entire end-to-end flow in the coming weeks. Stay tuned!

Futures in Rust

Have we mentioned that SAFE Client Libs is now fully set up for Rust async/await! The last week was spent testing against safe-api, the CLI and vaults, investigating some issues and updating safe-api to initiate its own runtimes for consuming these newly async APIs. Now, with all tests passing there, we’ve happily made a new API/CLI release against the updated client libs. We’re still seeing some issues with the FFI libs, which may need further tweaking for the new async/await setup.

Finally, we’ve seen some issues in the SCL tests when running against a local network, though our CLI tests are all passing with the SAFE Client Libs branch. We’re now looking to ensure that SCL tests run against a local network as part of CI, and are looking into those failing tests.

Routing

Project Plan

Last week’s announcement of Parsec removal triggered some interesting discussion and also raised some questions amongst the community. Let’s summarise the main questions raised and hopefully clear them up in one place:

  1. Are we back to the drawing board?

    Not at all. Removing Parsec doesn’t mean we are now looking for another consensus algorithm. We realised that Parsec was simply doing too much and what we need is much simpler. We’ve been analysing a solution for some time using CRDTs, AT2 and BLS where appropriate to achieve only what we need. You would have read about our research and work on implementing these for several weeks and months now, and we believe that they are more than adequate to push forward with, without Parsec.

  2. Is this going to delay alpha/fleming/baby fleming/… ?

    No. The work on Parsec removal goes on in parallel to any other work that is needed for release. Requests from the vault team are now highest priority and are quickly addressed. Also, routing as it currently is does almost everything we need to, just possibly not in the most efficient way. So in the worst case we can always release alpha with it as is and swap it out for the Parsec-less version later.

  3. Are we definitely going to remove Parsec?

    Most likely, yes. But even if things go horribly wrong and it turns out to be much more difficult than we thought, we can always fall back to keeping Parsec around in some reduced fashion. But we don’t think it will come to that.

  4. Was Parsec a waste of time & resource?

    Absolutely not. Parsec is still a fantastic achievement in ABFT, groundbreaking in its own right. It was justifiably the path we chose to go down in 2018 and undoubtedly helped get us to where we are today. Sometimes the only way to tell if something is suitable for your evolving needs is to try it out. Over time we became more aware of drawbacks and limitations for our particular needs, while alternatives became clearer and more mature. We’ll always do what’s right for the network and if that means scaling back the use of Parsec then that is the course of action we follow to reach our end goals.

We hope this helps summarise the answers we mostly provided in the thread of last week’s update, of course if you need any further clarifications then we’re here to help!

Here’s this week’s progress in routing:

We are working on the first step towards Parsec removal. Currently, the nodes in a section keep all the information about their section and other sections in a data structure called SharedState. Any modification to this structure must be agreed on by a quorum of the elders which currently means it must go through Parsec. The nodes that are actively participating in Parsec can trust this info because they saw the Parsec consensus on it. But what about new nodes? The shared state isn’t verifiable by itself, so if a new node becomes an elder, all the existing elders must vote on the current shared state so the new node sees the consensus on it. This is quite expensive. So to improve this, we will make it so that every single piece of information stored in the shared state will have its BLS signature attached. That way the shared state, or any part of it, becomes self-verifiable. The mechanism for this is already in place, we just haven’t been using it as much as we should. Expect a PR soon.

Although we are removing Parsec, it doesn’t mean we are completely abandoning it. We just released a new patch version which updates all the dependencies to their latest versions and removes the deprecated maidsafe_utilities crate, clearing up a security audit flag.

BLS - Distributed Key Generation

Project plan

This week we completed the review process for PR 14 and merged it into the master branch :tada: This of course added the transport layer (quic-p2p) into the crate, ensuring the users do not have to worry about the exchanging of messages & completion of the BLS session. With this final PR merged, rendering the crate complete, BLS-DKG is now ready to be put to use in the network. The first goal is to make routing use the BLS-DKG crate directly, instead of the current DKG generator from Parsec. The work of integrating BLS-DKG with routing is well underway.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

68 Likes

First I think

17 Likes

Nah it can’t be me again! Close though :smile:

How about testing before the get and asking if the user wants to continue, in case they want to stop and change their admin status/permissions? Maybe also offering to permanently disable to warning in a config. Might be nicer than finding out later

16 Likes

Made it to the podium at least, now to read…

15 Likes

8mins late and I missed out… but then I did read it first! :smiley:

Noting the changes to CLI dog have been very interesting for the insight they give to how urls resolve. Has kept me amused all week!

:+1:

18 Likes

Yay a toy! :gift:

17 Likes

Thanks so much to the entire team for all of your hard work. The PARSEC discussion was very insightful.

18 Likes

I love new toys! Thanks team, always great progress!

14 Likes

Great job. Just a little worried about last news regarding PARSEC. I hope it won’t make the safenetwork less promising that it was…

I hope I am wrong but weeks spend and still no MVP release date. I wouldn’t want to wait another few years to see this safenetwork online.

4 Likes

Excellent news!! Thanks as always to the MaidSafe crew for pressing forward :smile:

22 Likes

The fact that the client will see their changes seemingly immediately because it’s operating as a CRDT and even at2 is supposed to function in the same way, I just can’t wait to see this all in action. It just gets better and better and that includes the removal of PARSEC. It’s a journey of self discovery and that is not a linear path.

I don’t want to press but it would be nice to know what the new toy will be shipping with to test.
Will it have sequence data, will it have the at2 implementation? It sounds like the biggest thing we are testing is adult handling of immutable data, deduplication, quic with IGD, and these are huge new improvements/developments in their own right. But it would be awesome to just see the list of new achievements getting ticked on this upcoming test net. Really looking forward to it and beyond! The @maidsafe team as @Dimitar says is a constant inspiration to us as individuals in a shared community, so thank you!

26 Likes

Thx for the update Maidsafe devs.

Bring it on :baby:

In the meanwhile I’ll try out these new toys v0.13.0…

I can’t wait for the day when you super ants can take a break from coding up the network… release it in the wild and the world attacks/enjoys it to make it stronger/better/faster… :wink:

13 Likes

Oh goody something new to try to break with my new shiny hammer.
I had been intending to keep it for a later release but I can’t resist.

Well done to everyone for all their hard work as always.

11 Likes

Great suggestion, and I might implement it, though there are tradeoffs to consider.

Consider that many/most file get operations that a windows user performs will probably not have any symlinks at all, at least when dealing with their own files. It would typically occur when accessing someone else’s public files. So if we do an “are symlinks permitted” check and warn before the GET then we are warning/prompting about something that might not even be a problem and may just confuse the user. If we perform the GET first, then we have a choice: loop through all the files first, before doing anything else to check for presence of symlinks, or just process each one as usual and warn/prompt on first occurrence. The former introduces additional performance overhead. The latter is OK in theory, but the progress indicator doesn’t really play well with prompts, which create visual glitches. (Actually, I notice it has problems on windows in general). Plus, we have non-interactive (not a TTY) mode to consider.

All of that to say that I’m still considering what is the cleanest/simplest/best approach.

It’s really quite frustrating that Windows technically supports symlinks since ages ago but effectively turns them off by default. Hopefully that may change in the future.

14 Likes

Kill your darlings, is what they say when writing. It means cut out all the Unnecessary eloquence, to leave just pure narrative. This makes for a much better story. MAIDSAFE, willing to kill their darlings, proves their professional integrity of the network.

17 Likes

Such an important point needs repeating… the robust grows naturally out of challenge. The opposite error is all to common - small evils for the greater good. Cutting corners and short-cutting does not provide success in the long term. I do wish more people understood how important it is to get the basics right… (apologies) so much of politics :face_vomiting: is exactly this problem. Takes society a long while to understand the basics and projects like this push hard find better solutions for it.

15 Likes

Could this be the play thing?

Great work once again!

8 Likes

Hi, nice update but I am bit confused TBH.

I am not really aware of all the technical details but in this blog post PARSEC looks to be the most viable solution for the Safe Network and the other alternatives are described as “centralised”, “limited”, etc. But now PARSEC is given up. How this will impact the network in term of decentralisation, security, privacy and scalability?

Vlad Zamfir was not totally wrong at the end concerning the hype.

1 Like

Zamfir’s criticisms have nothing to do with Maidsafe’s use or not of Parsec.

Parsec is a, more o less, “classic” Byzantine consensus algorithm, with its own characteristics. The problem for Safe is that this kind of consensus are, computationally, very expensive mainly because it gets a total order that is not necessary for the network.

The solution, in which devs work, is to use CRDT (data types that do not need total order), consensusless transfer with AT2 and verification through BLS signatures.

The basic idea behind everything is less computational cost, because much of the work will be done by the client instead of the network, more network capacity, faster and more TIPS (transactions of information per second).

12 Likes

“Zamfir’s criticisms have nothing to do with Maidsafe’s use or not of Parsec.” I know but he was right by saying it was overhyped. You are yourself describing PARSEC which WAS suppose to be revolutionary as a “more o less, “classic” Byzantine consensus algorithm”.

But why it’s a “more o less, “classic” Byzantine consensus algorithm” today but 2 years ago it was marketed as a new revolution.

And I just want to know how this will impact the network in term of decentralisation, security, privacy and scalability?

Trying to do my part by looking online but I feel like it PARSEC was an important component of the network and I don’t really understand how it can be replaced without repercussions.

The basic idea behind everything is less computational cost, because much of the work will be done by the client instead of the network, more network capacity, faster and more TIPS (transactions of information per second).

So there are absolutely no downsides? I mean I am reading this: https://techcrunch.com/2018/06/02/not-just-another-decentralized-web-whitepaper/ and I see

MaidSafe’s network is not blockchain-based. It’s engineered to function with asynchronous voting of nodes, rather than synchronous voting, which should avoid the bottleneck problems associated with blockchain. But it’s still decentralized. So it needs a consensus mechanism to enable operations and transactions to be carried out autonomously and robustly. That’s where Parsec is intended to slot in.

What Parsec does is it can reach consensus even with malicious nodes. And up to a third of the nodes being malicious is what the maths proofs suggest

And so many other things that PARSEC was suppose to “improve” or “do better” than Blockchain or DAG based techs. I am really confused.

2 Likes