Update 22 September, 2022

Nothing major to report this week on the tech front, so we’ll keep it to a general progress report rather than the usual deep dive. More to report on the team front, however, where we’re happy to welcome a new team member. Mostafa is an experienced blockchain dev who hails from Malaysia.

Welcome to the team, Mostafa!

Hi. This is Mostafa. I am a software engineer. Nice to meet you.

Once when I was a student, I decided to visit an ancient fire temple (a building more than 2000 years old) which was located in a wild place. Finding the fire temple was tough, so I asked some local people to help me. Interestingly, they called it the “Devil House” and when I asked them why they said, “because humans can’t build such a thing, it must be the devil”. This was in my mind recently when I read the story of devil’s bridges in Europe. The bridges that were built centuries ago and still function. People call them devil’s bridges, because they can’t believe a human built it, it must be the devil.

No one knows who built the devil’s bridge or the fire temple, but they are still standing. Let’s build a devil network!

General progress

The SectionTree work is now done :muscle:
A reminder from @roland why we re-implemented some parts of the SectionTree

The SectionTree (previously NetworkPrefixMap) is a data structure that encapsulates our current knowledge about the network. It can be thought of as a tree where each node is a SectionKey signed by its parent SectionKey, except for the root node (genesis_key). The leaves of the tree also hold the section’s SAP while discarding the SAPs of the non-leaves.

Since every Client/Node can have a different network view, the SectionTree can vary vastly between the participants. Moreover, it is necessary to send parts of the tree out (SectionTreeUpdate, which bundles the proof chain + SAP) to anyone that requests it and be confident that they do not mess up their tree while trying to update it.

The SecuredLinkedList was previously used to construct the SectionTree, but it had a couple of issues that led to incorrect insertions, and it was not CRDT compliant. Hence it was re-implemented as a CRDT (specifically MerkleRegister), and the old SecuredLinkedList was replaced with the new SectionsDAG. Now we can be sure that it behaves as expected no matter the order or length of the update!

@roland has now moved onto distributed key generation (DKG) testing with @anselme. As explained over the last couple of weeks, there have been issues with the DKG process not always terminating. After some intensive testing, it’s looking pretty stable now, and @anselme is writing some documentation for the test process.
A note from Anselme on the upcoming DKG…

The upcoming DKG, more resilient, and without timers

The DKG process is currently an active process using timers that in cases of heavy network activity will often fail because of timeouts. We’ve recently been working on a new DKG that doesn’t make use of timers, instead allowing messages to be delayed as well as supporting heavy packet drops. When nodes don’t receive messages for a while, they gossip their knowledge to the others in the hope of getting them up to speed, or even getting updates from them if the sent information is out of date. This way, eventually every node will reach DKG termination.

We now also embrace concurrent DKGs with this new implementation. It’s now a race between DKGs and whichever set of candidates reaches the end before the others gets a chance at becoming the new set of elders. Some DKG sessions might be stalled or finish late, but we don’t consider them to be failures, they participated in the race and simply lost. We want the next set of elders to be as reliable as possible, so choosing the winner in this DKG race is also a way for us to pick the best nodes. Eventually, after section churn, these DKG sessions become outdated and as the candidates realise they lost the race, they know they can stop.

Another dynamic duo @bochaco and @chriso have been debugging the DBC tests. They’ve found a glitch whereby the SAP (list of current elders) is being removed by mistake during the client testing process, which destabilises the network.

Meanwhile @bochaco continues to work on the other major activity stream at the moment, streamlining the messaging processes and using single threads where possible.

@joshuef and @davidrusu are also working on aspects of message rationalisation, Josh is investigating extracting the comms module out of node code, to see if that might lend itself more neatly to multi-threading, while David is decoupling some of the code shared by the sn_node and network_knowledge crates, which will hopefully eliminate some of the node joining errors we’ve been seeing.


Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

49 Likes

One two three first back on form. Now to read :wink:

17 Likes

Its Thursday, how in the hell. Didn’t see this coming, must me a devils network, I like your thinking Mostafa :slight_smile:

14 Likes

Nice update Maid team! Seems like things are coming along as usual.

Does multi-threading in comms mean being able to communicate with different nodes in parallel?

Welcome to the team Mostafa - are you a general purpose engineer or are you a specialist brought in for a specific function/task?

Keep up the great work devil-ants, don’t sell your souls though!

Cheers

10 Likes

Thx 4 the update Maidsafe devs

Wow just wow another box checked :partying_face:

Welcome Mostafa

Hihi do you mean clearnet?
netflix’s lucifer, almighty GOD :exploding_head:

Keep hacking super ants

6 Likes

Welcome Mostafa and wishing you great success with David and the SAFE team! Being a Christian fellow, here’s to hoping I’m not invested in what’s to become the “devil’s network.” :sunglasses:

10 Likes

Thanks to all for the update and a big welcome to Mostafa. I hope your time with us will be productive and fulfilling

Good to see Section Trees ticked off and hopefully DKGs about to join them. Hope we can get a stable comnet up soon and we can put DBCs through the wringer to validate the work by @chriso and @bochaco.

Too late, plans are well advanced to rename statemap to satansmap…
The signs have been there for a long time though. For many years our COO was “Auld Nick” Lambert.

9 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

8 Likes

Thanks for the hardwork, MaidSafe team!

9 Likes

Seems still like a in-depth update haha! I’m wondering if there are more (Sequence, Flow, ER, etc.) UML diagrams available about current state and flow of the different components that MaidSafe is working on?

Something like this how the different components come together and interact?

The images in the primer are more about specific components like how elders work and last update was on November 2021. For people who want to join and understand the code base it would be really beneficial.

Even for people who don’t code I think these visual diagrams are the face of understanding the network and so much easier to grasp, especially when development updates are released and they have no clue of what it talked about or related to other developments.

11 Likes

I like Mostafa already!

Hoping to see some network stability soon.
The tidying continues!

11 Likes

Welcome Mostafa! In Bulgaria we also have devil bridges, here is one for you:

Thank you for the heavy work team MaidSafe! I add the translations in the first post :dragon:


Privacy. Security. Freedom

13 Likes

Welcome Mostafa

7 Likes