Summary
Here are some of the main things to highlight since the last dev update:
- We were able to
git clone
the crdt_tree repo and build it withcargo build
in the safe_fs filesystem prototype, another encouraging stage reached in its evolution. - Our newly async components are interacting more each day, with Safe Client now able to receive events and errors from the network, and vaults now serving events.
- A fully asynchronous vault binary took a huge step closer this week with this PR being raised and now going through our review process.
- A significant routing bug fix PR was raised today, resolving almost all of the parsec removal bugs.
CRDT
safe_fs
Work has progressed on the (local only) safe_fs filesystem prototype this week to make it more usable. Support for OS specific (POSIX, Windows) inode attributes were added, e.g. uid, gid, mode on POSIX and hidden, readonly, compressed, encrypted on Windows. These should make safe_fs more useful as a backup/restore solution for local files.
We set a goal this week to be able to git clone
the crdt_tree repo and build it with cargo build
. This exercises the filesystem substantially. Initially, memory usage would increase rapidly due to storage of file contents inside the inode (in crdt_tree metadata) and further it was unusably slow due to use of large sparse-files being repeatedly copied in memory. We switched to a pass-through type of system, where file content (only) is stored in the underlying disk filesystem. This performs well enough that we were finally able to complete a full build, which is encouraging. We also started to run some filesystem testing tools against it to identify any further problem areas.
Safe Client Libs, Vaults and qp2p
Safe Network Transfers Project plan
Safe Client Libs Project plan
Safe Vault Project plan
The Safe Client can now listen to the network using the qp2p stream that’s initialised on network bootstrap. Network queries will still use their own streams to keep response handling nice and clean. But this gives us the ability to receive events (e.g. TransactionValidated
) or errors (e.g. InvalidTransaction
) from the network.
Vaults are also now set up to be serving events. There’s some rebasing to be done and a bit of refactoring to get this into master, but with the listener and event sending we’re in a very good position with regards to getting the full AT2-ready client going and Safe Client Libs tests back up and running.
Async vaults has also progressed well alongside the other modules undergoing the async revolution this week, with this PR being a leap towards a fully asynchronous vault binary. The introduction of async/await across all modules in the vault has also allowed us to refactor the vault test environment so we can set up and test a complete network with the asynchronous vaults. Fully-fledged sections are getting more and more stable as we integrate, debug and fix them with the new async routing module. With the listeners/responders set up at SCL and vaults, client tests are the last stop in the pipeline for us to finish building a prototype farming network.
Routing
This week, alongside the manual testing we are doing where we run vaults and clients using the new async API across the crates, we also started creating a new test suite in routing to validate some of these scenarios. This new test suite doesn’t make use of any mock or fake components, instead launching real routing nodes on localhost and verifying the routing nodes can bootstrap, send messages to each other, and of course report the events as expected on its API, which is what the vault consumes.
We are also doing some research around how to make the data kept by a routing node more CRDT-compliant, which will allow us to test routing data mutation in isolation to verify it can handle network messages properly, regardless of the order they arrive. Additionally, we expect this to give us more confidence in the correctness of the main business logic of a routing node.
We created a PR fixing some of the bugs that remained after the removal of parsec. It overhauls the DKG module, fixing some edge cases and making it simpler, while also fixing some issues around relocations. The test failures went down quite a bit with this PR, but they are not at zero yet. There is more work to do but it seems to tie nicely into the CRDT work we started.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!