Safe Network Dev Update - February 18, 2021

Summary

Here are some of the main things to highlight since the last dev update:

  • A new set of Infrastructure messages which sn_routing can use to better handle and return errors during section splits and churn has been added.
  • Work to handle invalid SignatureShares by bad actors in transfers has begun, with penalties for those bad actors now being drawn up and agreed upon.
  • Messaging flow work is nearing its completion, meaning significantly less code and complexity to analyze incoming messages in sn_node, and clearing the way for the integration of rewards.
  • New versions of the CLI (v0.19.0) and authd (v0.1.1) have been released, making them compatible with the latest sn_node (v0.26.16), as well as including several other improvements.
  • The very first example app is now available in sn_api crate, demonstrating how to upload a file to the network and then fetch it using its XOR-URL.
  • We’ve been making steps towards integrating sn_fs and BRB so as to create a P2P byzantine-fault-tolerant distributed filesystem prototype.
  • sn_routing section chain modifications to resolve forks are mostly in place now, with our testing indicating quite a bit of improvement to the stability of the network, especially around splits.
  • Yet more fantastic community contributions to the code base :heartbeat:

Testnet status - on hold

As discussed last week, we decided to switch our full efforts to include rewards in the next testnet. Rewards itself has been blocked by related work to adjust the message flows after we made the switch to only do message accumulation in sn_routing. We believe we are in the final stages of the message flow work now, with what we believe are some final kinks being ironed out (more details below). Once done, this clears the way for the full integration of rewards to proceed.

As with all groundbreaking progress, we expect that there will be issues found during the integration and testing stages, so we don’t want to put a timescale on that and put additional pressure on the team. As always, we will keep you up to date with precise progress in these weekly updates.

Thank you for your patience!

Safe Client, Nodes and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan

Lazy messaging

To better handle issues at section split, and during churn, we’ve been adding a new set of Infrastructure messages which sn_routing can use to return errors such as DKGINprocess when we attempt to message elders during such times. In order to do this, clients/nodes will supply the current PublicKey for a section, as far as they know, and we can take action if that is, for example, out of date. These changes have been added into sn_messaging, sn_client, and sn_routing, and we’re looking at some small refactors to simplify things with nodes.

On the back of this, we’ve been further solidifying client handling of such infrastructure changes, so a change in the pipeline will allow clients to re-bootstrap any amount of times as is necessary. Alongside this, we are also squashing bugs that are arising with the message accumulation refactor done over the previous weeks.

Penalising bad actors

Handling invalid SignatureShares by bad actors in transfers is also in the works. Up until now we have been approving a transfer whenever valid threshold+1 shares were aggregated successfully, ignoring any faulty shares that aren’t necessary for that aggregation. We’ll now be introducing penalising mechanisms in the network to punish nodes accordingly after verifying. We are beginning our refactor from sn_transfers, as that is where accumulations happen for payment transfers, then working our way up through the layers.

Messaging flow

When moving over to accumulation in sn_routing only, we needed to remove unique info in messages, so that Elders would sign the same message.
With this the source and destination pattern had to be changed and while at that we made it simpler, by slimming the messaging interface between sn_node and sn_routing, and using the same primitives over the boundary.

As client bootstrapping is now taking place in the sn_routing layer, these location primitives had to be extended with a client variant. A registry of connected clients was implemented in sn_routing together with providing public key and signature over socket address at the client bootstrapping. There is now significantly less code and complexity to analyze incoming messages in sn_node. So far a total of about 1000 lines have been removed.

API and CLI

New versions of the CLI (v0.19.0) and authd (v0.1.1) have been released this week, which includes all the fixes, improvements and new features implemented since the previous release. As usual, they can be upgraded with $ safe update and $ safe auth update respectively. These applications are compatible with the latest sn_node version v0.26.16, so for those upgrading please also make sure you upgrade your sn_node binary as well ($ safe node install or $ safe node update).

@scorch has spotted that the very latest version of clippy was complaining on Windows for the CLI and authd codebase, and submitted a fix for it. This led us to improve the clippy checks in our CI scripts to run not only on Linux but also on Windows, which should prevent this from slipping through again.

With the addition (by @mav) of a new API in our CRDT Sequence data type to fetch an entry specifying a single index rather than a range, @mav now changed our sn_api to make use of this new API for retrieving a specific version of a Sequence.

A couple of enhancements have been also made to this new CLI (v0.19.0) which allows the user to set up bootstrapping information for a network using an IP (v4 or v6) and port number, for more details of this new command please refer to the corresponding section in the CLI User Guide.

For those curious to see how the Rust API is evolving and what it will look like to build an application with it, a very first example app is now available in sn_api crate which demonstrates how to upload a file to the network and then fetch it using its XOR-URL. Hopefully this will inspire others to create other simple example applications for our Rust Safe API, find out improvements opportunities as we start using them, as well as be a good resource for new devs willing to write Safe applications.

CRDT

David Rusu, author of rust-crdt as well as our BRB implementation, has been reviewing the LSeq code after @mav reported issues with it when many inserts are performed. He replaced the ID implementation that was based on the original LSeq paper with a rational number from the num crate. This makes the LSeq code much simpler and also results in a better (more even) distribution, resolving the problem and allowing the insertion of an arbitrary number of items. LSeq has also been renamed to List. Pull Request.

BRB - Byzantine Reliable Broadcast

Remember sn_fs, our prototype of a filesystem using the tree CRDT? Well this week we’ve been making steps towards integrating sn_fs and BRB so as to create a P2P byzantine-fault-tolerant distributed filesystem prototype.

First we forked brb_node_qp2p and created brb_node_tree, which demonstrates sending crdt_tree operations via brb. Then we modified the sn_fs crate and turned it into a library. Most recently we created a brb_node_snfs crate in which we bring everything together. As of today, we have been able to bring up two (or more) nodes and issue directory operations such as mkdir, rmdir. The operations are mirrored in the mounted filesystem of each peer. File operations are not yet working, but should be soon.

It should be stressed that this is bare-bones proof-of-concept work. There are many details to be worked out before it could be put into real usage.

Routing

Project Plan

The section chain modifications in order to resolve forks, as discussed in last week’s update, are mostly in place now. We are currently running internal tests using the stress-test tool to identify any remaining issues. It seems the stability of the network has improved quite a bit, especially around splits. Some issues still remain however and we are busy debugging them, but overall we are optimistic about this approach.

We also merged a PR to fix a message explosion, this happened when an elder went offline but the remaining elders would still attempt to send them the Offline vote, which would of course fail to be delivered and so would then trigger further Offline votes.

We merged another PR to respond with error to a client request if there is a DKG ongoing. An ongoing DKG (Distributed Key Generation) means that the section is about to transition to a new set of elders and so things might not be entirely stable or consistent at this time, therefore it’s better to let the client know so they back off and try again a short time later.

[UPDATE 22nd Feb] - Safe Browser

Something we originally forgot to add to this update :man_facepalming:

@bzee has created a PR to get sn_nodejs up to speed with the latest API changes, which is needed for the Safe Browser to be updated to be compatible with the rest of the Safe landscape again. He has also been taking a look into the limitations of the current lib, and seeing opportunities to improve things!.

Thanks @bzee! :tada: and apologies for originally missing your contributions in the OP.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

73 Likes

First :stuck_out_tongue: and a very nice update again, thank you everyone working on this. Things are still flying and I’m happy that the next testnet will include rewards. Even as a first cut that is a very exciting prospect :clap:

I’m particularly happy to see work being picked up on sn_fs as this is something that I’m relying on for Git Portal…

Can someone say a bit about the sn_fs details needing to be worked on here?

And now back to the studio for a…

Git Portal Micro-Update

Lots of activity but the best part is that Git Portal had two new contributors! :tada:

By @dredd :pray:

  • Windows support added (for development)
  • build and test commands tidied up (renamed and rationalised)

By @happybeing :pray:

  • added CONTRIBUTING.md for anyone wishing to contribute. Contains guidelines for non-coders, as well as for front-end and Rust subsystem covering documentation, testing, error handling etc.
  • added Windows build to github actions (CI) so everything is built, and tests run on Linux and Windows when code is pushed or merged
  • added Rust checks to CI and Rust source code is checked with clippy (lint) and rustfmt (layout)
  • added Semgrep to CI to scan code and dependencies for security issues (untested, so :crossed_fingers:). Thank you to @StephenC who made this trivial by doing the hard work and leaving me with a copy/paste job.

By @janriemer :pray:

  • proof read and updated CONTRIBUTING.md with useful links plus fixes to spelling and grammar

Help Welcome!

Ad hoc contributions are welcome so don’t think you need to make a big commitment to help out. If you are interested in any area, coding (Svelte, JavaScript or Rust) or related to documentation, testing or feedback get in touch. For specifics, check out CONTRIBUTING.md.

40 Likes

First to it?
Edit yeeeee
Also super excited each week to read these, they are always showing the massive commitment and ongoing movement forward. Go Maidsafe :slight_smile:

25 Likes

Second? Thank you so much @happybeing for the mentions. Glad, I could help and I hope I can contribute some more in the future. :heart:

23 Likes

Great to hear this! Amazing job! Slow and steady, and getting it right. Thanks team

17 Likes

More great progress and work being accomplished. As a side note, many of us have posted on social media about an imminent test net. As in the past, perhaps it’s again better to not be promoting something we do not have yet. The team will move at the pace they need but we have again prematurely said “test net with a week or two” and that is not happening. Time to dial back until it’s here IMO. Very happy with the continued progress!

21 Likes

That is a pretty picture!

Sometimes I forget it’s Thursday… and it’s a thrill to see an update on twitter… don’t know who posts that but perfect timing :smiley:

While testnets are on hold… what is possible atm… it there option to run a local something… minded to have a look if there’s something to see the new CLI against.

Keep at it …

15 Likes

It’s taking shape. Keep up the great work team!

Hey, it’s my 7th anniversary on this forum! Time flies. :sweat_smile:

21 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! Keep it going. :racehorse:

17 Likes

In my book, they just skipped having to do days/weeks of excess/redundant work by – at the last second before an otherwise release of a test net – deciding to include rewards in the future test net. This is because it would take the length of those days/weeks to realize, during it: “Yes, good progress. But now we need to focus on including rewards AND keep an eye on interesting developments in this Test Net Without Rewards.” Now they don’t have to worry about that extra workload—until the best possible test net, that can be both a test net and a more exclusive focus towards a test net.

Maybe they’ll realize they can add Upgrades and Network Restarts next, and continue delaying test nets :stuck_out_tongue: (I am kidding).

Hopefully the continued integration of all components becomes less overwhelmingly unknown, as they branch together to form visible certainties! The evolution beyond PARSEC continues, and it’s damn impressive that the horizon is already being said to become the savanna in just over half a year; it really should come as no surprise that there is still work to be done. But not for long, as far as being able to begin colonization. Some of the travelers (mav, etc.) are already marching with the team onward to that destination, providing much needed water for their philanthropic thirst.

14 Likes

This Testnet looks like, we will release It only when this become full release lol

3 Likes

Just amazing how you guys keep going. Your persistence is baffling! :+1:t2::+1:t2::+1:t2:

13 Likes

A truly impressive mixing of metaphors there :slight_smile:

And all true - loved that post - one like is not enough :slight_smile:

6 Likes

It won’t be far off but it still needs to actually be tested, especially the tokenomics. It needs to be proven stable and safe in the wild, discover bugs or blockers and then iterate until it’s polished enough to be a beta. Part of the process.

14 Likes

We won’t have upgrades etc. but apart from that feature complete. Lots of small stuff, browser upgrade API etc. but upgrades should be the only missing feature.

However, test and debug will not be 1 or 2 weeks, I expect that to be at least 6 weeks (could be 6 months, just in case I am quoted :wink: ) and maybe more. That will take us to Beta I hope. Let’s see though.

30 Likes

That’s super :ok_hand: this next testnet sounds like it will be the real deal. Take your time and get it right. Super update :+1:keep it up :point_up_2::point_up_2:

17 Likes

Happy to hear this! Also, when hearing already yesterday that the testnet is not going to happen today, it is now natural for me to focus on what has been done, instead of getting caught in thinking what is not done - yet.

P2P byzantine-fault-tolerant distributed filesystem prototype is quite a mouthful, but sounds incredibly cool!

It’s nice that Davis Rusu’s name is finally out, because I have wanted to recommend checking out his site https://davidrusu.github.io/ … and I would like to know if there is a name for a site like that?

In general all the progress in CRDT front sounds to me like the professionals in that area should soon become very interested in Safe Network project.

All in all, cutting the cutting edge, all the time. Good job! :beers:

17 Likes

I’m enjoying the small community vibe we have here whilst it lasts. Make the most of it folks. We will be grains of sand in the dessert when we release. We will look back on the early days with a sense of nostalgia. Like the band you loved that made it big.

Keep on trucking Devs.

33 Likes

Great to hear all about the progress. Great work guys!

11 Likes