Update February 16th, 2023

Time for a quick review of where we’re at with the various moving parts that make up the Safe machine. As usual, it’s a mixture of rapid acceleration in some areas, with others waiting for progress elsewhere before they can get moving.

Overall, we’re excited by the simplification possibilities offered by the stable set idea, very happy about the establishment of the Foundation, and pleased with progress on consensus. It’s also great to have community testnets back up and running, and to have folks leaping into action with their own GUIs :tada: .

General progress:

We’re always looking at ways to get the client to do more work and @oetyng is investigating how it can keep the elders in sync with what’s happening in the section (along with AE, gossip etc).

@anselme has been refactoring DBCs and is integrating his changes into the main code, and @Chriso is finishing off the monitoring stack implementation.

@roland has refactored the relocation process into having the relocated node drive the process.

Mostafa and @davidrusu are getting stuck into the stable set, working on a handover protocol that does not require DKG. Mostafa is also looking into the binary compilation process to try to speed it up.

Also on the stable set, @joshuef is confident we have a path to tiered storage with a few extra changes.

Qp2p is still undergoing surgery with @bochaco testing for anomalies around reading and writing streams, while @bzee has moved onto analysing logs in relation to delays on the network when nodes are stressed.

Finally, @JimCollinson has been working on various legal bits and bobs for the board, and also considering launch strategy.

Progress report

Stable set

Treating the elders and the oldest adults as more trustworthy than younger nodes, not requiring them to relocate for node ageing, and introducing tiered storage. Could well remove the need for DKG and relocation, which are both difficult.
Progress: Design stage, testing assumptions, still experimental.
Todo: Complete tests, if successful, work on implementation.

Many processes can be handled by AE and eventual consistency (CRDTs) but in some cases, some local ordering is likely required as forks could make double spend possible while we wait for them to resolve. https://github.com/maidsafe/sn_consensus
Progress: Multi Values Byzantine Agreement (MVBA) algorithm has been developed and tested.
To do: Apply to membership algorithms. Trials in live environment.

Qp2p is a networking library based on QUIC that enables peers on a P2P network to communicate securely. https://github.com/maidsafe/qp2p
Progress : qp2p is being refactored with the aim of bringing most of its functionality into sn_node where it will be easier to work with. Quic is a black box.
Todo: This work is ongoing.

Messages and memory spikes
We were seeing a flurry of messages and surge in memory use particularly when a new node was contacting elders and asking to join. https://github.com/maidsafe/safe_network/tree/main/sn_comms
Progress: We’re mostly not seeing memory spikes any more. Messages are a central part of the way the network works so we’re constantly refining things there. We’re working on better backpressure leveraging Quinn’s congestion control more reliably.
Todo: Squashing bugs when we see them, continued simplification.

Monitoring and observability
We have moved from the ELK setup we were using previously to OpenSearch (a more modern open source fork of ElasticSearch), OpenTelemetry and a Data Prepper, because we can host those services on ECS rather than using EC2 instances. https://github.com/maidsafe/terraform-testnet-infra
Progress: This is pretty much there now, although we’re still looking at the best configuration.
Todo: Finish up, then open it to the community, where feasible.

Data handling
Data handling, redistributing chunks when an adult goes offline, and gracefully handling splits.
Progress: Again, this is a core part of the functionality and work is ongoing, see stable sets above.
Todo: Establish tiered data. Groundwork has been laid, we’ll need to test things out atop of covering relocation and fine tune.

If an elder notices that the current elders are not the seven oldest nodes, then it sparks a vote on promoting the oldest adult(s) and demoting the youngest elder(s) to make way. Handover is the algorithm that controls this process.
Progress: Completed to plan, although it is being refined with gossip, consensus (see below).
Todo: Introduce gossip to speed communication between elders, and look at consensus to ensure processes terminate.

This is what allows elders to keep control over the adults in their section.
Progress: Membership is integrated into sn_node but remains the hardest area to fix in ensuring each elder converges on the same view. It’s a coordination problem.
Todo: Remove the dual view of membership we have between SectionPeers and Membership. Work has started here to reduce this down to SectionPeers and is progressing well.

Gossip is a type of p2p protocol where a node periodically broadcasts knowledge to another node, often selected at random, and that node passes the message to another node, and so on until it becomes ‘common knowledge’.
Progress: We are introducing gossip at various places where the processes can sometimes get stuck, such as DKG rounds, membership and network splits. In the case of the latter, gossip also allows us to reconstruct the network after a catastrophic failure.
Todo: Look for further opportunities where gossip may help us.

Distributed key generation is used when elders need to make decisions about membership.
Progress: DKG has been refactored but still occasionally suffers timeouts.
Todo: Looking to reduce the need for DKG as in decision making.

Payment flow
When the client pays for an upload it gets a receipt and the elders and adults in the sections that store the data receive some part of that payment. Payment flow is about how that distribution works.
Progress: Basic design is in place.
Todo: Detailed design to cover security, UX, economic incentives. Awaiting other developments.

The Safe Network Foundation is a not-for-profit in Switzerland to facilitate, support, fund and promote research and development, as well as enabling the ongoing delivery, maintenance and adoption of the Network and its technologies.
Progress: The Safe Network Foundation has been established!
Todo: Put it to work!

Safe Network Token Distribution
After the initial distribution of tokens at the inception of the network (10.6% to MaidSafeCoin holders, 5% to shareholders, 14.4% to network royalties pool), the remaining 70% of the total maximum supply will be created and securely distributed as a consequence of people uploading data over the long term. https://safenetforum.org/t/updated-rfc-0061-safe-network-token-distribution/37883
Progress: Initial proposal drafted and submitted to FINMA for approval. The mechanics of which node operators will be paid during a particular transaction are still being worked out.
Todo: Design and test node payments.

Digital bearer certificates are the vehicle for transacting tokens on the Safe Network.
Progress: DBCs have been refactored removing a superfluous element supposed to obfuscate transactions but with dubious capability to add to the security, and improving performance. This work is nearing completion.
Todo: Improve APIs and the UX around this. Work the DBC APIs into the payment flows and testing there.

Node refactor
sn_node has been refactored to remove as many instances of multithreading as possible in order to simplify the code and remove unwanted locks. Elders now store data so they can estimate storage levels without polling adults, and we’ve been looking at having more smaller nodes in a section, which has been successfully trialled in testnets.
Progress: Done
Todo: Keep monitoring.

Section chain refactor
Section chain, the secure linked list that connects section keys all the way back to Genesis is being refactored as a Merkle DAG. It’s now called SectionTree.
Progress: Done.
Todo: Nothing.

Fault Detection (previously dysfunction)
Regularly testing that nodes are online and taking action if they are not.
Progress: Refactored with its own test suite, awaiting progress in other areas.
Todo: Improve testing with live networks and ensure things are working as expected. Add more fault detection routes.

Name Resolution System and domain names
NRS maps are multimaps (registers under the hood) that contain the mapping of all the subnames to URLs for a specific topname. Each topname has its own multimap.
Progress: No movement on this.
Todo: Changes to NRS ordering to be implemented.

Client / UX
sn_client is the set of APIs and the command line interface (CLI) that allows devices to connect to and interact with the network. https://github.com/maidsafe/safe_network/tree/main/sn_client
Progress : sn_client has been refactored - fixing messaging which shows users whether they are connected or not, especially during bootstrapping.
Todo: Continuous improvement of the APIs and CLI as we move forward.

Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!


Keep on trucking, good work team!


ffffffffffffffffirst ?, btw wow we post in same minute T824601 :smiley:


Nice work!


The Network Primer is great to see refreshed

and the suggestion there of what SN will do includes

  1. Allow real-time upgrades in a secure manner.
    The Network will refuse upgrades that could break it.

and I wondered would be a similar confirmation by some stable set of nodes to prove the upgrade. Perhaps also disrupts what is an elder if there’s any risk in what upgrades do.



Fantastic stuff this is shaping up to be the year we have all been waiting for.

I noticed on the node binary there is a flag for - - wallet-Id

Does that mean that we will be able to receive test SNT in test nets already for providing a node ?


I thought he said “lunch strategy” and immediately wanted to offer practical help.

Perhaps the Yadgar for some tandoori nibbles, stroll along to the Allison to alleviate thirst then a bus into town and into the Horseshoe for a pie and a pint?

Its worked well for me before. Sets you up well in the town for an afternoons serious jollification.


That certainly is my area of expertise


On that we can agree :slight_smile:

Oro in Shawlands for a celebratory launch lunch?

cos we’all be rich, unimaginably rich, I tell you - or at least flush enough to afford a pizza out of Oro


Lunchmunch fantasies aside, that is an excellent comprehensive update. Thank you to all involved who move this project forward.

Looking forward to testing out the various components as soon as its feasible.


Very nice update team. Maybe can recycle the format for all-libraries updates in the future.


Cheers and thanks for the hard efforts :wink:


Thank you team :clap:t3: and I very much like the [content warning for Willie] bite size summary format with Progress and Todo sections. Looks fairly easy to maintain/update week by week too.

Moooaaaarrr test networks please. :partying_face:


Thanks so much to the entire Maidsafe team for all of your hard work! :horse_racing:


this is shaping up! and update on all fronts! lucky us!


Fully ready for the next one !


I was shopping and had to buy these distinctly non- plebian fish fingers in your honour.

Shhh nobody tell him I was in Lidl


Biggest problem will be to “prove the upgrade”, hard to do that. I can imagine the update could go like this:

  1. Randomly select little % of nodes and suggest them to update (with rollback if they get kicked out).
  2. Watch what is happening and how many of the selected nodes remains on the network with the new version.
  3. Either scratch the update and stop signaling it or start signaling it to the rest of the network.

Would be nice if there was an element of survival of the fittest but getting the feedback loop on what is better would be a challenge, where that is not a normal metric. Depends very much what the fix is, the options for seeing it adopted.