Summary
Here are some of the main things to highlight since the last dev update:
- The Safe Network filesystem POC took a major step closer to being made public, with it now being merged to master and going through the final stage of testing.
- We renamed the
Safe Client Libs
repository /safe_core
crate tosn_client
. - We’ve laid the foundations for chaos testing in
sn_node
. - Props to @Scorch for 2 async contributions to
sn_node
, PR1105 and PR1090, in the last week. - A Continuous Delivery Pipeline has now been rolled out to just about all of our Rust crates, meaning more frequent releases for you!
CRDT
sn_fs
This week a pull request was raised and merged to push the local filesystem prototype into the new sn_fs
repo. Once the rest of the team has had a chance to kick the tires a little, we plan to make the repo public so that community members running Linux can try it out, keeping in mind that this is only a proof-of-concept. Details should be forthcoming next week.
Safe Client (previously Safe Client Libs/safe_core), Nodes and qp2p
Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan
Another repository and crate name change to announce this week - Safe Client Libs has now been renamed to sn_client
. These libs
have gone through a huge transformation over the past months with a massive refactoring and simplification push by the team, as documented in these updates, and we’re down to only the safe_core
crate remaining. We’ve renamed both the repo and the crate to sn_client
to bring it in line with what it now represents, and with the other renamed repos and crates. With this, the renaming task is now more or less complete, with the only outstanding actions being to publish a few of the renamed crates, namely sn_routing
, sn_client
and sn_node
, which we’re holding off on until they are considered stable enough for a new version release.
We are progressing more on the Node/Client integration this week, taking down bugs as we re-enable e2e tests one by one to increase the code coverage. We found a couple of blockers at Replica Management that were hindering us from getting clients to pay for data writes. Therefore we set out to streamline the AT2 Transfer tests, focusing mainly on the Transfer Operations and removing all and any discrepancies among the section. Once we clear that, the next steps would be to cover the data layer operations and introduce chaos testing to strengthen the overall integrity. We’ve laid the foundations for the same in PR#1124 which introduces a new feature-flag chaos
in the crate to enable chaos testing. Once enabled, it reads the chaos level from the SAFE_CHAOS_LEVEL
environment variable which dictates the probability at which chaos is to be induced in the network by randomly dropping messages/not performing operations.
On another front, we have been working on adding some features to qp2p following the async/await paradigm. During the migration towards using async/await across all our crates, we only migrated the modules that were immediately required. On completion, we saw good results and it integrated well with the other layers. Now as we have already started testing the end-to-end system again, we will soon need features like usage of the IGD protocol and echo service for automatic port forwarding, which is an important component when running testnets. These features have been implemented and their integration is currently being tested with the other layers.
Shoutout to @Scorch for PR1105 which not only optimises but also makes the chunk store module follow the async paradigm in sn_node
. Not content with this, @Scorch also had a PR merged this week making the statedb file I/O async
It’s always a huge shot in the arm for team morale when we see contributions like this and realise how much this project means to us all
Routing
In Routing this week, we have merged the work tackling message resend and lost peer detection. This was part of the work to get routing working under the new async architecture. There has also been related work to remove message accumulator, which we expect will be merged soon.
Alongside the above, there is also ongoing work to simplify the codebase by removing the id.rs file completely, and to make our nodes keep a Keypair and SocketAddr for peers. The first stage of this was to get rid of the id traits within the bls_dkg
crate - this has now been merged. The routing part of this refactoring is ongoing with a PR expected to be raised soon.
The effort to move to CRDT continued in parallel to all this, reaching a state where we have a very basic and isolated PoC implementation. In this first step we are trying to validate if such an implementation can successfully work as the container for a node to store all information it keeps in sync with other peers in its section, e.g. a list of Adults and Elders, where these sync up operations are treated as CRDT operations, thus nodes don’t need to worry about the order they are received as they all converge to the same state. We are now working to advance in creating a good set of tests which can prove this approach is solid, before we start trying to integrate it into sn_routing
.
Continuous Delivery
Back at the end of July we implemented a Continuous Delivery (CD) pipeline in one of our Rust repos for the first time. After some initial tweaking, we were satisfied that the pipeline was solid and of great benefit to the project. This week we completed the CD rollout to all but one of our Rust repositories
Wondering what this CD pipeline process means? When any pull request is merged to master in a repository with CD enabled, our CD pipeline automation kicks in. The CD generates a changelog containing all the valid updates added since the last release, works out what version the crate should be updated to, and generates a PR updating to that version. It auto-merges that PR then creates a matching GitHub tag and Release. Finally, the crate itself is published to crates.io. All automated with no human intervention required. This means that you will begin to see much more frequent releases, bringing that feedback loop with the community down as much as is possible.
sn_api
is the only exception for now due to the complication of multiple crates in the workspace, we’ll come back to that down the line.
Note that we have not merged the CD pipeline PRs for sn_routing
, sn_client
and sn_node
just yet, preferring to hold these back until these repositories, which are currently in a period of flux, are considered stable and able to begin automated releases.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!