Here are some of the main things to highlight since the last dev update:
- The main parsec removal refactoring PR has been merged into the upstream routing master branch.
- The safe_fs FUSE integration is progressing well with us now having implemented a proof-of-concept in-memory (local) filesystem utilising crdt_tree that supports regular directory and file operations, plus symlinks and hard links.
- The safe_vault crate switch over to follow the async/await paradigm is well underway, with the porting pretty much done, and integration with the other async crates progressing well.
Over the last week, the team did some additional review of the crdt_tree code and we made some minor improvements, such as inlining functions, logging, and comments.
Perhaps more exciting, work is progressing on the safe_fs FUSE integration. We were able to implement a proof-of-concept in-memory (local) filesystem utilising crdt_tree that supports regular directory and file operations, plus symlinks and hard links. Much work remains, but it is a nice milestone to be able to actually mount the filesystem and interact with it.
Safe Client Libs and qp2p
This last week has seen further progress in quic-p2p (now:
qp2p), with streams being exposed to enable listening to network events. The reuse of streams for multiple messages that was recently added required the length of the message to be known ahead of time. To facilitate this requirement, along with other metadata such as message flags, we have now formalised a message header which will be sent over the wire before sending the message itself. It is of the following format:
|Version||Message length||User message flag||Reserved|
|2 bytes||4 bytes||1 byte||2 bytes|
This standardises how messages are sent over the wire, while also allowing us to introduce backward compatibility with the help of the version flag.
The process of integrating this into routing, vaults and clients is well underway. We now have small sections working again, are able to bootstrap clients to them and perform queries once more. So as we get event listening set up in clients we’re getting closer to a holistic network with all the changes in.
In parallel to this, we’ve tied up some loose ends in terms of Continuous Delivery, catching a Changelog generation issue in Safe-ND (now:
sn_data_types), and fixing the GitHub release generation. That more or less completes the first CD process for one of our Rust repos so we’ll be looking to roll this out to others as it really does simplify things there.
With the first iteration of async routing just about wrapped up last week, up next is the safe_vault crate. We are working to make it follow the async/await paradigm and with the porting pretty much done, we are now integrating and debugging all of the async crates in unison to spin-up a section and start running real-network e2e tests (no mocks). This will help us identify the real-world connectivity issues and hiccups, and have them resolved before we put them out to the community.
This week we merged the main parsec removal refactoring work into the master branch
There is still some ongoing work to resolve a few remaining parsec removal issues but we are confident that these issues are a formality. Meanwhile, Adam is also working on improvements to the DKG process which should also resolve some issues we are seeing after removing parsec. To summarise, there is no need to ever have more than one DKG session per node, there should only ever be the most current one (corresponding to the most recent churn). More details to follow in the PR which should be raised in the coming days.
In parallel, the work to expose an
async API is now based on top of the latest master branch. Once some final minor issues are addressed, this PR will be reviewed and merged. It is our intention that any further failing routing test investigation/resolution work will be based on the parsec removal and async API work. This new
async API is already being used for testing vaults and clients, proving to us we are on the right track.
We have also discussed and started to investigate using real routing subcomponents for some network tests. Expanding the current minimal example will probably be a good starting point.
Adapting routing automated tests to the new
async API is also a task we’ve just started this week. We will be focused on trying to split existing tests into two different categories, one suite for the most basic and general use cases, and a second suite where edge cases and more difficult to reproduce scenarios will be tested with the help of mocks and/or simulators for some subcomponents.
We mentioned a couple of weeks ago that we were undertaking some repository and crate renaming tasks in order to standardise. Anyone keeping an eye on our GitHub activity will have noticed this has taken a big step forward over the last week with the majority of our repositories and Rust crates now updated to snake_case, prefixed with
sn_, if Safe Network specific, and a couple (so far) renamed to be more accurate (safe-network-signature-aggregator --> bls_signature_aggregator and safe-nd --> sn_data_types).
The remaining repositories and crates will be updated in the coming days, assuming we can agree on what to name them
Feel free to reply below with links to translations of this dev update and moderators will add them here.
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!