Safe Network Dev Update - December 10, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • sn_client v0.44.0 has been released this week, the client’s first release since July.
  • We have completed and merged the implementation of the duplication of Blob chunks when an Adult leaves the network - now release in sn_node v0.25.9.
  • Our consultant has come up with a coded, tested and working algorithm for dynamic membership.

General Update

A quick general update of our progress towards the upcoming testnet.

We are all heads down pulling all the pieces together for the upcoming testnet. As expected, there is some bug hunting and some last minute additions. One thing that we won’t have time to include in this testnet is the new BRB (Byzantine Reliable Broadcast) membership. BRB membership is a really important step, but for now we are looking to test so many other moving parts that we’ve agreed the focus should be to launch this phase I ASAP, then we can fully focus on BRB.

Safe Client, Nodes and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan

This week saw the first release and publish of sn_client since July. This repository previously consisted of various crates together under the Safe Client Libs banner, but as regular readers here will no doubt be aware, we heavily refactored which led to us stripping out many thousands of lines of code that we felt was inefficient, buggy or unnecessary. This is now a single sn_client crate, released under this name for the first time. This joins sn_node and sn_routing in new significant crate releases over the last 2 weeks, and so we hope demonstrates the progress we are making as the major pieces begin to fall in place. This release doesn’t mean we are finished with sn_client, it just means that we consider it stable enough to introduce a Continuous Delivery pipeline. Development continues at pace, and now each merged PR will result in a further automatically generated new release.

This week we’ve been trying to track down the source of a stack overflow issue which has surfaced insn_node. It initially seems there is no specific source, but that stack usage has simply grown over time. This was originally only happening on Windows, where we discovered the default stack size is 1 megabyte, which is low when compared to Ubuntu, which has 8 MB. Lowering the stack size on Ubuntu does allow us to replicate this, which is good. So with that, we’ve been investigating this further and options to prevent this from happening.

We have completed the implementation of the duplication of Blob chunks when an Adult leaves the network. This feature, which was temporarily disabled during some of the refactors in the sn_node repo to bring in farming and rewards, is now compatible with the new structure of the code, tested and merged. This implementation also lays the foundation for the next tasks on our cards - distribution of data on Elder churn, promotion events and network splits. We have a few ideas in mind already and will start implementation soon.

BRB - Byzantine Reliable Broadcast

Good news! Our consultant has come up with a working algorithm for dynamic membership. This is original work that utilises a Generation Clock with one join or leave operation permitted per actor, per generation. This algorithm has already been coded up along with a test suite, and all tests are passing.

The next step will be to integrate it with the existing DSB implementation. So in the end we will have a protocol for negotiating dynamic membership that is distinct but complementary to the DSB protocol for BFT transmission of regular operations/data.

Routing

Project Plan

This week, to bolster our logging and aid with debugging, we decided to switch routing to use tracing, instead of the log crate. This enables us to use structured logging and spans. The PR for this switch has now been merged. Our intention is to switch to use tracing in other crates over the coming weeks.

After further discussion on how and when resource proofing is used, we decided that relocated nodes should be trusted and so there would be no need for them to undergo the resource proofing procedure again. The PR Do not require resource proof from relocated nodes has now been merged to refactor such behaviour. In the same PR, we also fixed a couple of bugs and test failures which we tracked down thanks to the improved logging provided by the aforementioned tracing crate.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian; :germany: German; :spain: Spanish; :france: French; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

76 Likes

First! My thanks to everyone at @Maidsafe, its wonderful to see the code being streamlined and coming together so fast. :partying_face:

Quick question: do you plan to maintain logs for ‘lowest common denominator’ analysis as supported by tracing (described here)? I assume this will still be useful, but will also affect vdash which currently relies on watching and parsing vault logfiles to gather live vault metrics. Any other comments on that?

Thanks for the blistering pace and good luck squashing the last few bugs :bug: for the testnet! Ants away… :ant:

Safe Git Portal Progress

In other news I made lots of progress with the Safe Git Portal proof-of-concept (status update here) and have made a request for help on the Golang forum for help with Svelte and Golang.

Please share the following post on social media etc as it is a good summary for anyone learning of the git portal for the first time. Maybe you’d like to have a go yourself?

Emoji Treasure Hunt

I also came up with a task anyone can help with in a spare five minutes:

Many thanks to those already sharing and supporting me here and on social media. It is really awesome to receive so much help and encouragement :pray:

Have fun!

40 Likes

Second my friend

20 Likes

Treasure chest!! Third again, haha. Now to read :grinning_face_with_smiling_eyes:

Excellent news, and the priorities make total sense. :grinning_face_with_smiling_eyes:

Woot!! :tada::tada::tada:

Awesome! Wonderful to see the boxes getting checked on such sizable and pivotal bits of work. More grease to your elbow, MaidSafe Crew!!!

Also, everyone, please like and retweet:

27 Likes

Give that man a bells! :partying_face:

20 Likes

Give that consultant something extra in his Christmas stocking please. Great update, great progress, thanks team!

13 Likes

You want to poison the poor bugger with cheap cooking whisky?!!?! Bells is for tourists and steak sauces, no for drinking.

Give him a good malt. I will refrain from suggesting any particular malt seeing as how I am already accused today of starting an OS Holy War :slight_smile:

Well done and grateful thanks to all involved.

Fine update BTW, as expected steady progress on the way to a test net that we can all join in with

14 Likes

Haha I was expecting a retort, not sure why that old ad came to mind but yes, I agree! :grinning:

8 Likes

Well done guys! The pieces are all clicking together now, by the sounds of it.

Very cool! :sunglasses:

14 Likes

Now there’s the real source of market price upwards pressure.

:slight_smile: :tada: :tada: :tada: :tada:

14 Likes

So has this update just put Parsec to bed? Great stuff all.

8 Likes

Very good news! Congrats to all those involved.:clap::+1::+1::star::star::star:

9 Likes

A two hearted! :drooling_face:

6 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

28 Likes

Maidsafe making the net work.

:+1:

16 Likes

Great work Maidsafe team as per usual! Also thanks to the super contractor! I’m going to add some pressure though as I found the “Next” Generation Clock:

image

lol … looking forward to this next testnet … then onwards to Fleming!

cheers

19 Likes

@dirvine

Generation clock?

This sounds very new. Would love to hear more about how it is kept and how it works.

Oh and great news on progress (as usual)

15 Likes

I’ll shed a tear at 600 :yum:

2 Likes

7 Likes

A bit confusing, CRDTs will be replaced with BRB?

2 Likes