Safe Network Dev Update - March 11, 2021

Summary

Here are some of the main things to highlight since the last dev update:

  • After working through some minor refactors and fixes, we are now in a position to update all our Rust crates to Tokio v1.
  • We’ve revisited sn_routing this week to PR changes to the way agreement is reached to now require a supermajority (more than 2/3) instead of a simple majority (more than 1/2). We believe this is necessary to make the network resistant to certain types of attacks.
  • We’ve decided to implement Lazy Messaging in sn_node, with work already underway. This was originally planned for post-testnet but we’ve deemed it worthwhile to bring forward.
  • @jimcollinson kicked of an AMA on Reddit, and right here on the forum, last week - there’s still time to get your questions in!
  • @jimcollinson has created the first in what we believe will be a series of YouTube video responses to the bigger questions received in the AMA - you can watch it here. :movie_camera:
  • @dimitar has been working behind the scenes to help increase Safe Network awareness in India with a Facebook and Twitter ad campaign. :heart:
  • Keep a regular eye on the Like This Tweet thread on the forum for some excellent guidance on how to help promote the Safe Network, and surrounding components, with a simple button click! :bird:

Safe Client, Nodes, Routing and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan

Last week a long awaited release of Quinn finally arrived with an important upgrade: Tokio v1. Up until now, the usage of an older version of Tokio in Quinn prevented us from updating all our crates to Tokio v1 due to incompatible runtime versions, so we are in the process of upgrading all our crates to Tokio v1. With some minor refactors and fixes we have been able to get all tests passing with the new Tokio version. This update also helped us identify a previously undiscovered issue that left streams open ultimately stalling network communications once the upper limit was reached. The Quinn team promptly assisted us and the issue is now fixed in qp2p. sn_routing's communications are working flawlessly again! We expect all our crates to be updated in the next few days.

This week we also added more examples to the qp2p crate to better demonstrate the use of the API, and for stress testing qp2p both locally and on Digital Ocean.

In sn_routing this week we decided to change the way agreement is reached to now require a supermajority (more than 2/3) instead of a simple majority (more than 1/2). This was necessary to make the network resistant to certain types of attacks. We also increased the number of elders in a section from 5 to 7 which means a section can still lose up to two elders and remain functioning. These changes are currently undergoing review and testing, and we expect them to be merged soon.

With the need to enable lazy messaging (see subsection below), we’ve been looking at how best to achieve this in sn_node. We may be able to shim some small parts of this in, but we’re also having a look at a larger refactor here to simplify things. It’s looking like with some of the sn_node code gone (essentially removing Duties) and directly parsing messages, we’ll end up with something that we can error out of quite readily, while also probably dropping a lot of sn_node's complexity. Initial efforts in this area have been pretty positive. We’re hopefully this won’t be that big of a task, as the underlying logic should be staying the same, but regardless we’re approaching this in parallel to more lightweight solutions so we’re hopefully not blocking anything.

sn_node Lazy Messaging

Lazy messaging in sn_node would work by slightly increasing the size of messages sent between nodes so they include some extra information on the Network’s current state as seen from different observers in space and time. The alternative to this approach would be to poll continuously for changes - we firmly believe that the cost of extra data per message is more than offset with the reduction in overall traffic when compared to constant polling. With polling, even when the Network is going through a quiet period the polling would need to continue furiously in the background. Other approaches to halt/pause parts of the Network to get to agreement are fraught with many side effects and complex code, therefore those are off the table.

As a very brief overview, with lazy messaging if a node receives a message and it realises that the network state details in the incoming message differs to what it believed was the state, then it communicates further with the sending node to bring itself, or the sender, up to date with the correct network state, then the original message can be processed accordingly.

Lazy messaging has been implemented in sn_routing for some time now, proving to be effective and reliable. In sn_node we’ve faced some challenges over the last couple of weeks trying to bring nodes up-to-date with the latest network state changes (after churn, promotion, demotion, etc). For example, when a section splits, we’ve faced challenges getting the new Elders of the respective new sibling sections up to speed with the parent section Elders, while still being able to cope with the typical expected traffic and Network events.

We’ve had lazy messaging marked as the end goal for a while, but until now considered it as a post-testnet optimisation. However, it’s felt like an uphill struggle with the challenges we’ve faced over the last couple of weeks, and that we are putting a seemingly never ending amount of temporary plaster over cracks that we know are comprehensively resolved by lazy messaging. So, we’ve decided to waste no more time here and move to using this pattern in sn_node to update the network state on each node over time, as and when required.

As mentioned, this is extra work that we didn’t think would be required pre-testnet, but we don’t consider it so large that it delays a testnet too significantly. And it’s another item checked off our to-do list on the road to beta & launch.

CRDT

Work has begun on a bounded counter type that will allow us to cap operations on a data type. This is a valuable component which allows mutable data to continually grow, while bounding data into compartments. This means we can pay to upload the data type, then freely add to it up to a bounded point, at which time the user would pay again and get another set of operations space freed up. Splitting up growable data in this way spreads the load across the network. This is an optimisation, but an important one that is required for launch, but not for Fleming. It’s nice to see these nips and tucks in the pipeline now.

We’ve also been making changes to the Sequence data type since we are migrating to use the new MerkleReg CRDT, which will allow us to keep all history of appends, along with allowing forks of data if the end user or application generates them either on purpose or not. This is impacting how our API shall be presented to the end user, as depending on the use case there could be forks/branches of appends the user may want to not only traverse, but also resolve. Therefore, we are also working on adjusting our API on the client side to present all the capabilities to the user, without making it more complicated, and without removing the flexibility and power this new CRDT data type brings in for applications and developers.

Community & Marketing

@jimcollinson kicked of an AMA on Reddit, and right here on the forum, last week.

The original aim was to round up a brace of juicy questions and answer them in a Q&A session on YouTube. There’s been a splendid response, with the answers to many questions turning out to be far more detailed than a single Q&A session could possibly handle. So here is the first video response in what will likely become a series:

Please do like, share, disperse, distribute, disseminate, discuss and enjoy in whichever way you see fit. And feel free to keep the questions coming.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

82 Likes

First :wink:

23 Likes

Second…

22 Likes

That’s fantastic! Looking forward to all of this work culminating in a launched testnet in the next week or two🤞🏾

Here’s a tweet in support of this week’s dev update. Be sure to like, retweet & comment about what excites you most about the upcoming testnet! :grinning_face_with_smiling_eyes:

24 Likes

Where’s “Quantum Messaging” when you need it?
Spooky messaging at a distance.

Great update @maidsafe! Keep up the great progress

16 Likes

Great update! Thanks for the effort!

16 Likes

No stress! It is time to launch that matters. Great work @maidsafe! @JimCollinson keep the videos coming.

14 Likes

I almost replied before you sent that :wink:

18 Likes

Yes, please. Keep it up, y’all.

13 Likes

Last! Then next to last, and next to next to last, and…

Very interesting update. I was curious about a couple of things, noticed elsewhere, which have been answered today. Nice to see, and thank you everyone working away. Great job.

17 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

16 Likes

Music too my eyeballs!!!

The team of volunteers coming together is great to see!

Thanks everybody!

16 Likes

Thank you team for all your hard work.

Repeat after me

“What do we want?”
“Testnet with no feature creep!!!”

“When do we want it?”
“Just as soon as you can add that Quantum Messaging!!!”

20 Likes

Great work team! I’ve been following the PRs closer, now I’m starting to get to know the codebase. I should have done this sooner, as it gives me so much more confidence in what you guys are achieving. Can’t wait for test net so you guys can take some credit for it all!

23 Likes

After a bit of a hiatus, dev updates now going out weekly on Blockfolio too.

We’ve got 17,000+ followers, so worthwhile, especially as we are on the cusp of the test net.

45 Likes

Excellent!! :smile:

25 Likes

I was meaning to ask - is there any limit to the number of entries in a sequence type?

Also, I noticed that internally in the sn_client that the whole sequence appears to be retrieved, even when only a range or the last item is needed. Will this be optimised and will these changes improve this area too? Thanks!

11 Likes

I see testnet coming on a radar :slight_smile:

What kind of risk you found to make this change ?

5 Likes

We are now working on having limits to all “mutable” data types so the user would pay once when storing it and all updates (appends in this case) will be for “free”.

Yes we are considering having special type of network messages to retrieve only part of the content, what you see is the initial minimum required to have this functionality working on the client side, but we will eventually should be able to have query content based on some filters. It’s too early to clearly see in detail how those messages, filters and/or API will look like and that’s the reason we want to have the basic ones in place to then see what’s the best way to do it based on the challenges we face when adapting our client side apps like the CLI and browser.

13 Likes