Summary
Here are some of the main things to highlight since the last dev update:
- We released new versions of the API (v0.13.0), CLI (v0.13.0) and safe-authd (v0.9.0) which bring several enhancements and fixes, e.g. the
files putcommand now supports uploading empty directories and hidden files. - SAFE Client Libs is now fully set up for Rust
async/await. - We’ve collated and answered a few of the most common questions surrounding last week’s announcement of the removal (or significant reduction) of Parsec.
- Work is now underway to remove Parsec, with the first step being to attach a BLS signature to all shared state information.
- The Parsec crate itself is not being completely abandoned! We just released a new patch version which updates dependencies.
- The BLS-DKG crate is now considered complete and work is underway to integrate it into routing.
Vaults Phase 2
Over the past week, we implemented a number of improvements in the way Adults and Elders communicate with each other within a section to handle immutable data requests. This includes the messages exchanged to maintain the required number of copies for every chunk. A couple of internal testnets helped us identify some bugs in the implementation and the PR is now approved. We are now running what we hope to be a final iteration of the internal testnet to make sure everything is healthy before releasing it into the wild. Assuming internal testing goes without any major hitches, you’ll have a new toy to play with soon!
We have some ideas in mind for the next steps going forward, for example, distributing data management responsibilities to new Elders when a section splits, and handling the event of an Elder leaving the network. We will be starting implementation in the days to come.
SAFE API
This week we released new versions of the CLI (v0.13.0) and safe-authd (v0.9.0) which bring several enhancements and fixes, along with some new features for the files commands. E.g. the files put command now supports uploading empty directories and hidden files. As usual, you can use CLI to update both the CLI (with $ safe update) and safe-authd (with $ safe auth update), or follow the instructions in the CLI User Guide for installing them for the first time, and/or any other instructions.
A new version of the safe-api (v0.13.0) is also part of this week’s release. This now makes use of the very latest version of the safe-client-libs, which exposes its API as async. With this we finished the migration of all our crates in the safe-api and safe-client-libs repositories to use the Rust async/await syntax, which will also allow us to make many more enhancements and simplifications to the safe-client-libs codebase as we move forward.
Meanwhile, work on supporting symlinks continues. In the latest development code (no pull-request yet), it is possible to PUT and GET both relative and absolute-path symlinks, and also they display correctly in files tree and files ls. Symlinks may be disabled using a new --follow-links flag to files put, resulting in the old behavior.
One speed-bump encountered: on Windows, writing symlinks to disk requires certain permissions (depends on the exact OS version), so files get may skip the symlink and issue a warning in such case. Git also has this problem, and typically writes a small text file containing the link-target. We may go with that approach also. As a Windows user, symlink support is enabled by choosing run as administrator when starting a command terminal, or one may enable Developer Mode in Windows 10+.
The next development work will be to resolve relative symlinks when used within a Safe-URL path.
CRDT
After the new release of safe-client-libs early this week, we resumed our task with the Sequence CRDT implementation. We’ve done so by adapting the client so it keeps a local replica of the CRDT content (including its causality information) in order to be able to apply any mutations locally.
What this means is that the client would be making the mutations in its own local replica of the CRDT content, and once that succeeds, it will simply broadcast the operation to the corresponding section on the network for persistency.
Note that once a mutation has been made successfully on the local replica, it means it will be merged on the network’s replicas without conflicts, therefore the client won’t need to wait for any type of consensus response from the network, it will only need to make sure the request was received by the network.
As part of this effort, we adapted the Sequence “append” requests to contain the causality information which is needed to merge the operations in the network’s replicas. We are now looking at ways to make sure that whenever we read the content we get enough copies from the network to be able to get the most recent version of it.
Transfers
Now that we have async/await merged in safe-client-libs, we’re updating our SCL AT2 implementation for this setup, and updating it to meet the latest changes in safe-transfers itself. At the moment that’s involving setting up a local Actor for managing/merging validation proofs from elders, and effectively wrapping all our Money requests to the network. This should hopefully nicely encapsulate that logic client side and make some of our client APIs that wee bit clearer too.
As to initiate an Actor, the Replicas need to expose an API that lets a client query for validation proofs of an account, from which an Actor instance can be initiated. Additionally, this supports ongoing synchronisation from Replicas. We are currently getting that in place.
At the other end of the spectrum, we have begun to upgrade safe-vault to accommodate the AT2 safe-transfers framework, which will be handling the flow of Money in the SAFE Network - in the CRDT way. Replica(s), just as the name says, are replicas of the client’s Account and their Money balances in the network. The responsibility of managing Replicas stays only with the Elders as they are the most trustworthy in the network. As churn happens in the network and as Elders join and leave, it is crucial to maintain the Replicas by generating and updating them respectively. We have started working on these mechanisms first and will be progressing to implement the entire end-to-end flow in the coming weeks. Stay tuned!
Futures in Rust
Have we mentioned that SAFE Client Libs is now fully set up for Rust async/await! The last week was spent testing against safe-api, the CLI and vaults, investigating some issues and updating safe-api to initiate its own runtimes for consuming these newly async APIs. Now, with all tests passing there, we’ve happily made a new API/CLI release against the updated client libs. We’re still seeing some issues with the FFI libs, which may need further tweaking for the new async/await setup.
Finally, we’ve seen some issues in the SCL tests when running against a local network, though our CLI tests are all passing with the SAFE Client Libs branch. We’re now looking to ensure that SCL tests run against a local network as part of CI, and are looking into those failing tests.
Routing
Last week’s announcement of Parsec removal triggered some interesting discussion and also raised some questions amongst the community. Let’s summarise the main questions raised and hopefully clear them up in one place:
-
Are we back to the drawing board?
Not at all. Removing Parsec doesn’t mean we are now looking for another consensus algorithm. We realised that Parsec was simply doing too much and what we need is much simpler. We’ve been analysing a solution for some time using CRDTs, AT2 and BLS where appropriate to achieve only what we need. You would have read about our research and work on implementing these for several weeks and months now, and we believe that they are more than adequate to push forward with, without Parsec.
-
Is this going to delay alpha/fleming/baby fleming/… ?
No. The work on Parsec removal goes on in parallel to any other work that is needed for release. Requests from the vault team are now highest priority and are quickly addressed. Also, routing as it currently is does almost everything we need to, just possibly not in the most efficient way. So in the worst case we can always release alpha with it as is and swap it out for the Parsec-less version later.
-
Are we definitely going to remove Parsec?
Most likely, yes. But even if things go horribly wrong and it turns out to be much more difficult than we thought, we can always fall back to keeping Parsec around in some reduced fashion. But we don’t think it will come to that.
-
Was Parsec a waste of time & resource?
Absolutely not. Parsec is still a fantastic achievement in ABFT, groundbreaking in its own right. It was justifiably the path we chose to go down in 2018 and undoubtedly helped get us to where we are today. Sometimes the only way to tell if something is suitable for your evolving needs is to try it out. Over time we became more aware of drawbacks and limitations for our particular needs, while alternatives became clearer and more mature. We’ll always do what’s right for the network and if that means scaling back the use of Parsec then that is the course of action we follow to reach our end goals.
We hope this helps summarise the answers we mostly provided in the thread of last week’s update, of course if you need any further clarifications then we’re here to help!
Here’s this week’s progress in routing:
We are working on the first step towards Parsec removal. Currently, the nodes in a section keep all the information about their section and other sections in a data structure called SharedState. Any modification to this structure must be agreed on by a quorum of the elders which currently means it must go through Parsec. The nodes that are actively participating in Parsec can trust this info because they saw the Parsec consensus on it. But what about new nodes? The shared state isn’t verifiable by itself, so if a new node becomes an elder, all the existing elders must vote on the current shared state so the new node sees the consensus on it. This is quite expensive. So to improve this, we will make it so that every single piece of information stored in the shared state will have its BLS signature attached. That way the shared state, or any part of it, becomes self-verifiable. The mechanism for this is already in place, we just haven’t been using it as much as we should. Expect a PR soon.
Although we are removing Parsec, it doesn’t mean we are completely abandoning it. We just released a new patch version which updates all the dependencies to their latest versions and removes the deprecated maidsafe_utilities crate, clearing up a security audit flag.
BLS - Distributed Key Generation
This week we completed the review process for PR 14 and merged it into the master branch 
This of course added the transport layer (quic-p2p) into the crate, ensuring the users do not have to worry about the exchanging of messages & completion of the BLS session. With this final PR merged, rendering the crate complete, BLS-DKG is now ready to be put to use in the network. The first goal is to make routing use the BLS-DKG crate directly, instead of the current DKG generator from Parsec. The work of integrating BLS-DKG with routing is well underway.
Useful Links
Feel free to send us translations of this Dev Update and we’ll list them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together
is exactly this problem. Takes society a long while to understand the basics and projects like this push hard find better solutions for it.