Safe Network Dev Update - November 19, 2020

Summary

Here are some of the main things to highlight since the last dev update:

Safe Client, Nodes and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan

First off, this week in qp2p we implemented connection pooling. This means that if a node wants to connect to a peer and the connection has been opened before (and is still open), we will reuse the existing connection instead of establishing a new one. This improves performance because establishing a connection is expensive (it involves a TLS handshake, among other things). This also improves ergonomics because the users of qp2p don’t need to worry about connection caching anymore. We also implemented connection deduplication which means that multiple concurrent connection attempts to the same peer will all resolve to the same connection instead of opening a separate connection each. This again improves performance.

We’ve been getting back at chunk replication of Blob data at sn_node. Starting out with a 4x replication factor, the network’s adults will be primarily responsible for storing them. We are porting the older implementation from pre-async vaults to the new codebase, adapting it to the latest changes of storing, querying, etc. We have also been doing a bit of maintenance all across the board this week by hunting down sneaky unwraps, expects and panics from our production code and tests, essentially stabilising our codebase and catching all exceptions. This is best practice and something we’ve been putting off for too long. Look out for additional CI checks being added in the next few days across our crates ensuring these don’t sneak back into our code.

We invested a bit of time in researching and thinking about how the APIs will eventually need to evolve to support signing requests from the client using multiple key pairs rather than just one. For example, a client may want to store a file that would be owned by a public key whilst the payment for such operation would be made using a second public key that owns the funds, and perhaps a third keypair may be used by the client for encrypting the file’s content. This is not something we are considering high priority at the moment, more of a PoC to help us identify the challenges and realise how to eventually evolve our client APIs.

CRDTs

Work continues on dynamic membership in DSB and this week our consultant has written a test case that demonstrates a concurrency problem with data operations while a member is leaving the group. A correct implementation of dynamic membership should always pass the test while our current naive implementation fails, so we now have something concrete to measure against.

To that end, we are excited about a recent academic paper by the AT2 authors titled Dynamic Byzantine Reliable Broadcast. It provides quote: “the first specification of a dynamic Byzantine reliable broadcast (dbrb) primitive that is amenable to an asynchronous implementation”. In other words, this paper provides a formally proven solution for exactly our problem.

Our consultant is presently reviewing this paper as well as another possible solution using something called a Generation Clock that might not require as much network communication.

Routing

Project Plan

As mentioned in last week’s update, the work to allow a node to rejoin with the same name was this week approved and merged. This means any rejoining node would be immediately relocated with half its age, as long as the halved age is greater than the MIN_AGE (currently 4). This is designed to discourage malicious restarts.

We had observed during internal testing that the genesis node was sometimes being demoted too quickly, this was down to a recent change where we now make nodes with a random range of ages during the startup phase. To resolve this we decided to Start the first node with a higher age, currently set to 32. This has now been merged to master. This ensures the genesis node stays stable as an elder for a sufficiently long time, which eases many things for testing and testnet setup.

The ongoing work to improve lost peer detection is progressing well. We have already taken advantage of the new connection pooling feature in qp2p, which allowed us to simplify the code. Some initial integration tests show the refactoring works well. This PR is now going through some final review and testing and so will hopefully be merged soon.

To ensure that when a node’s resources are close to being used up there will be new nodes flowing in to share the workload, we are going to allow nodes to tell routing to accept new nodes or not. This restriction on when the network accepts new nodes will also help prevent sybil attacks by not allowing potential attackers to add unlimited nodes at will.

In addition to this we’re also looking to make some changes to assist in making us aware of routing’s exact status during network startup & beyond, we have two ongoing PRs Indication for section start-up and firing PromotedToAdult event and notify when key is changed during relocation. These in progress changes will help us ensure that routing is behaving as intended, and should be completed soon, once the API designs are agreed between nodes and routing.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:bulgaria: Bulgarian; :russia: Russian; :germany: German; :spain: Spanish; :france: French

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

66 Likes

Bingo first! And I’ve nearly finished the first stage of the p2p git portal proof of concept. :grin:

See:

34 Likes

I got you this time @StephenC! :slight_smile:

22 Likes

Literally shaking my fist at the screen when I was beaten to it :joy:

22 Likes

Pieces falling into place… :+1:

16 Likes

Nice pic - another one from Nepal?

9 Likes

Is all this work required for a test net?

1 Like

Nice to see that other parts are working well.

So how long you can be disconnected now to hold your age?

And after rejoing with same name means to relocate back to original place so you can use same chunks as before?

What stage is API development so will be worth for developers to built own apps?

6 Likes

This made me laugh out of sheer appreciation. It’s a very unscientific feeling and I could be wrong, but after all the dead ends, rabbit holes, whatever, one just gets the sense that the stars are aligning.

Last week’s UX/terminology changes had a similar effect too.

Thanks for all the continued efforts…

15 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

4b19o5

What is the exact blue color for the Maidsafe logo? I might want to have text that is exactly that color in a later meme.

11 Likes

HSV 211:55:80
RGB 91:145:204 == 5B91CC

10 Likes

Yes, we have a dynamic list in house of these things that are left.

17 Likes

Yea, it’s fun reading papers that say what you should do and we have plus some more :wink: all good and great to see so many communities and researchers entering the space (at last).

16 Likes

The academic paper was written by a team at EPFL in Lausanne. It’s 30mn drive from Geneva, one of the best university in Europe.

I think I ll contact the guys privately to meet them and ask what they think of the project. They probably never heard of it. Could be an interesting conversation

30 Likes

Absolutely, the more the merrier. We have some advances on their paper to help transition to permissionless sharded network. So extend the research there a good bit. There is another advancement we have not put in place yet and that is bulletproofs to mask amounts held in any balance or transactions, some decent MIT videos explain that part though.

23 Likes

I mean, what are the chances? :joy::joy::joy:

10 Likes

Thank you for hard work dev team!

10 Likes

Relevant xkcd?

16 Likes

Thanks @davidpbrown ! This works great. :racehorse:

4n2hvb

Edit: I made another one with dollar bill green, 85bb65. here it is.

20201119_145715

8 Likes