Here are some of the main things to highlight since the last dev update:
- We are currently looking at removing (or at least significantly reducing) usage of Parsec in the Routing library.
- We’re almost done with migrating the SAFE Client Libs to switch to
asyncfunctions and exposing its API also as
- We implemented an approach to supporting symlinks that is pretty much the same as what
gitdoes. We’re also adapting the code in safe-api which invokes the underlying SAFE Client Libs API. We should then be able to release a new SAFE CLI version
- The expected final BLS-DKG PR is now raised and under review, with the crate considered ready for use in the network once this is approved and merged.
You may have noticed that the SAFE Network Forum was inaccessible for the majority of Tuesday and early on Wednesday of this week. This was down to the domain name expiring, and with it being registered to a now unmonitored MaidSafe Foundation account we weren’t directly alerted that it was due for renewal. On Wednesday morning (UK) the domain was transferred to a MaidSafe account and renewed (with auto-renew enabled )
It can take time for these changes to propagate through, but everyone should be able to access it again by now.
Vaults Phase 2
This week we completed the implementation of the data handling flows when an Adult leaves the network. Adults have the responsibility of holding immutable data chunks so after it leaves we should replace the lost copies of the chunks in order to maintain the required number of copies of the data to ensure data availability. The initial implementation works as expected and the test results are looking good. The current implementation has a bit of network overhead that we are aware of. Thanks to features in Routing that optimise the communications within a section, we can overcome this with a few adjustments that are in progress. With this optimisation in place, we will be in a position for a testnet that includes Adults from home with the responsibility of holding data. The next steps would then be handling Elders leaving the network, section splits and so on which we will be working on in the days to come.
Support for symlinks is coming
This week work has begun on supporting storage and retrieval of symlinks that exist in the local filesystem.
The initial approach was to:
- allow relative links inside of the directory tree being uploaded
- convert absolute links to relative links inside of the upload tree.
- convert links outside of the upload tree to real files.
However, the implementation of this was somewhat complex and it also modifies the user’s data, such that symlinks may be different on disk after a put/get cycle. That would be surprising to many users and violates the principle of least astonishment.
So after some thought, the very simplest scheme is being implemented, which is actually pretty much the same as what
git does. In this scheme, all symlinks are uploaded exactly as they are, even absolute paths to other locations on the user’s filesystem and broken links. If the user is uploading this data for archival/backup purposes, then it can be very useful to have those symlinks restored correctly upon download. Otherwise, if the user is uploading the files for a website or other public usage, then it is up to the user to ensure there are no broken links.
As of this writing, symlink upload and retrieval are working on Unix platforms. Windows behaves a little differently, so will take a bit of extra work in the coming week.
Under the hood
We’ve also worked on adapting the code which invokes the underlying SAFE Client Libs API as it’s being migrated to be exposing
async APIs. We are aiming at getting this finalised this week and we will therefore be ready for a new release of the SAFE CLI which brings several enhancements and new features for all of you to try out.
This past week we’ve been working on some refactoring in SAFE Client Libs to finally switch to
async functions and exposing its API also as
async. This has taken priority as it’s crucial for getting our CRDT implementation on the client side properly. The refactoring in SCL is making the code a lot simpler and is also accommodating our needs of adding these new types of requests for the
Fortunately we are finishing such a refactoring effort today/tomorrow, and we will be able to then start adding the logic on the client side for CRDT, i.e. start caching a replica of the data so it can be mutated locally by the client and broadcasting the operation to the data’s Elders section for storage on the network. This will give us our first PoC and E2E implementation of the
We’ve cleared up some questions with regards to the repo’s responsibility. Basically, membership is a responsibility for the upper layer, and adding and removing replicas to the internally handled sets is the result of Elders being added or removed in a section.
An extension of this problem was the verification of results from a remote group of replicas (in higher layers known as a remote section). It turns out to also be a membership question, although not replica/section membership, but system membership. Therefore this is also functionality provided by the upper layer.
Alongside this, some general naming and code clean up has been ongoing and will continue to be, and there’s been work on tests and integration testing of the components, with additional work on finding testability/modularity pain points. Basic integration tests are passing and we are now setting up property based testing.
Futures in Rust
The last week has seen a bit of a flurry of activity as things come together in async land. We have the whole of SCL compiling nicely, and CI tests are now passing .
We have updates to self-encryption merged into master and a new version building there (which SAFE Client Libs depends upon).
safe-api has been updated for the new SCL version. This highlighted some other issues in the code, including some tricky blocking behaviour with some of our
await code for messages, but we’ve got that licked now and are moving through some final tests with the APIs before getting this all merged in.
It’s been a big job, so it’s great to see this coming together (and to say goodbye to around 2000 lines of code, too! )
In the last dev update, we talked about how we are improving the way we handle messages that are either not trusted or not processable by the node in its current state. That work has been finished and the PR is now being reviewed.
We are currently looking at another large change for Routing - the removal of Parsec.
This might sound strange, because Parsec is the algorithm we developed ourselves specifically for the needs of Routing. It has worked very well and has passed many rounds of very extensive testing.
So what’s the problem with it? Performance. It can take in the order of tens of seconds to reach consensus on a single observation which is a rate that simply does not scale to our needs. That’s why we decided some time ago to stop using it for data requests (which we are transitioning to CRDTs), but that’s still not enough. We also want to remove (or at least significantly reduce) its usage in Routing itself. This is actually possible without sacrificing any of the goals we set for Routing because it turns out that Parsec is doing quite a bit more than we need. For example, we don’t really need the total order of observations. All we need is that actions are agreed by the quorum of the section elders and that they are consistent. So this is what we will be working on in the next few weeks.
BLS - Distributed Key Generation
This week, we focused on pushing all the tweaks that we made from observing our real-network simulations and tests to the master branch with PR 14. This PR adds the transport layer (quic-p2p) and so completes the crate. Being a generic crate, it targets users of all types (Nodes and Clients) therefore the integrated transport layer makes it easier for the users as they do not have to worry about the exchanging of messages & completion of the BLS session - as discussed in last week’s update, the crate takes responsibility for these using timers. With this PR currently going through the review process and close to being ready to be merged, the next step will be to begin our integration of BLS-DKG with the other SAFE Network parts, another step closer to our end goal!
Feel free to send us translations of this Dev Update and we’ll list them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together