Update 21 July, 2022

Something of a prequel this week as we look at how SectionChains and SAPs fit in with the PrefixMap changes announced last week.

General progress

No big reveals this week, more of the steady testing, optimisation and debugging work that marks this phase of the development.

@davidrusu has been looking at ways to monitor the network and administer Node Age, including how adults can prove to elders that they are alive and kicking, even if there is not much network activity.

@bzee continues to look into OpenTelemetry the open source monitoring tool, with some refactors already in place to enable logging over the OTLP protocol. This should hopefully get us logging to the elastic server, where we can more easily get an overview of the state of testnets.

And @joshuef continues to tinker with memory optimisation. We’ve removed some overzealous retries which were potentially keeping connections alive, and pending messages in memory while the send was retried and retried and retried (both at the qp2p layer, and in the node). He’s also been making other smaller tweaks which have all helped to bring the memory under load down considerably.

@yogesh has a PR in place to enable traceroute info to be returned to the client. This will list all nodes involved in the messaging flow for a given query/cmd, which should help with debugging.

SectionChains and SAPs

SectionChains provide a record of changes of elders within a section, acting as proof that the current set of elders is valid. We can’t take an individual elder’s word for it that they are who they say they are, nor can we trust the current group - that would be an obvious attack vector. Instead we use the wonders of cryptography to create a SectionChain, a secure linked list that proves the elders are who they say they are, by linking the current section key all the way back to the genesis key.

The Genesis key can be thought of as ‘proof of network’. It is the very first key created by the very first node in the very first section – the section with the prefix 0. As soon as that first lonely node is joined by another node (trying to resist calling it Eve as that will swiftly get confusing), a new section key B is created, which is signed by the genesis key. When a new node C joins, again a new section key is created, this time signed by B, and so on.

The Genesis key’s signing days are over as soon as it signs key B. However, it remains extremely useful as proof that the current section key and all preceding it are valid, not sneaked in by an attacker, because it’s cryptographically very easy to prove that any section’s current key is ultimately linked to Genesis, however long the SectionChain becomes.

Signing

But what do we mean by signing? Signing simply means using our secret key to add some unique bits to a message or a file. Anyone who has our public key can verify that it was us who signed it (and in most cases that the file has not changed after signing).

To use the classic example, Alice sends a message to Bob and signs it with her secret key, meaning she creates a unique ‘signature’ from a combination of the message contents and her secret key. Bob can be sure it was signed by Alice because he can verify the signature with her public key. Eavesdropper Eve also has Alice’s public key, but she is powerless to trick Bob by altering Alice’s message as it would no longer carry a valid signature.

Rather than individuals like Alice and Bob, in a section we are dealing with group decisions that must be approved by a supermajority of elders. To be Byzantine fault-tolerant (meaning that a third of the nodes can be dysfunctional without causing failure), five out of seven elders must sign a message. Rather than have each elder go about a process like Alice and Bob above, a much more efficient way to do this is through distributed key generation (DKG), where each elder contributes a key share. As soon as we have five of these key shares - any five - we create a valid signing key and the decision is validated. This is the equivalent of the section secret key signing the message, even though with BLS a secret key does not exist.

That’s all fine and dandy so long as we only have one section, but eventually as the network grows the section will split to create new sections 01 and 11 with new elders and new SectionKeys signed by that of the parent section. These new sections will experience elder churn over time, on each occasion creating a new DKG SectionKey. Eventually a tree structure is formed with genesis as the root and each branch spawning two new branches as the network expands.

The important thing is that every current section key in every section has a path down the branches to the genesis key. That is, we can prove we are on the right network and that the process of welcoming new nodes has been valid at each step, at least as far as the DKG is concerned. But as soon as there’s more than one section, each section has a different path back down to the Genesis, at which point the section chain becomes a DAG (directed acyclic graph) rather than a linear chain.

The Section Authority Provider (SAP)

The section chain is a simple but vital tool. Every node and client holds a SectionChain. As a client or a node, it tells us that the current section we are talking to is cryptographically valid, and that we’re in the right network (as proved by Genesis key), rather than some fork, but that’s about it.

When a node or client joins the network it is given the SectionChain and the SAP. The SAP tells us about the current elders. It is a list of the current elders (each elder has a unique ID, an IP and a port) signed by one of the keys in the SectionChain. This means when we connect to a node in this list, we can be sure it is valid, because, once again, the signature can be traced all the way back to Genesis. The SAP also contains the current SectionKey.

As explained last week, the SAP is included in the PrefixMap, which maps each section prefix to all the keys in the history of that section. As well as being able to check that any data we are handling is correct (it will have the same prefix as the section), this also means that when we move between sections we don’t have to grab the entire section chain again. Instead, we can just go down the tree as far as we need to, until we hit a section that already exists in our section chain and take it from there, which is obviously more efficient.

Next week, more about how it all fits together.


Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

53 Likes

First bwahahahaha.

Now that I’ve claimed first I get the privilege to read!

20 Likes

Not first!

18 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

16 Likes

Is it out early this week or just my timing?

I expect that more and more as the parts get nailed together and the more obvious bugs get hammered out, the weekly updates are going to be more like this one – which is good!

Keep hammering ants!

10 Likes

Yeah me, 5th place Participation Trophy today. Reaction time is waning.

10 Likes

nice work on the section security design, growth and adjustment as elders come and go… :slight_smile:

11 Likes

Nice to read about the Section Chain mechanism. I have a couple of thoughts related to calling it a chain. Firstly isn’t it more like a tree than a chain? And might that be a better name than a chain which makes it sound like a blockchain? I think it would be good to avoid that if possible - Safe Network: decentralisation un-chained! :wink:

Also, you didn’t mention the Sled DB removal, or am I wrong in thinking it has been completed?
EDIT: Joshuef confirms this is done :clap:

There hasn’t been an sn_node release for nine days. Can we expect one soon for a new community network?

EDIT: release process is being fixed so…
https://forum.autonomi.community/t/pre-comnet-thread/36423/290?u=happybeing

Thanks very much team - things are moving fast towards launch and it’s a privilege to watch.

24 Likes

Thinking the same :slight_smile:

Also regarding membership, do we need the single thread work to be completed before the membership PR can go in?

12 Likes

I agree with you here, HappyBeing. “Chain” does carry some baggage and limitations. :sunglasses: It has always torqued me that on the WorldCoinIndex.com site the definition at the bottom for MaidSafeCoin starts with this sentence:

SAFE(MAID) is a blockchain-based resource trading platform which combines multiple popular services into one network. The SAFE network allows for the hosting and renting of CPU power, bandwidth, and data storage space. The cost of storing and renting is decided by a decentralized, autonomous smart contract which sets prices according to the ratio of available storage space versus customer demand.

12 Likes

To be exact the very first section has an empty prefix which is usually represented by () and when it splits that creates sections (0) and (1).

17 Likes

My bad, I was to check that and missed that point. You are right, though, the first prefix is the null prefix. Good catch @tfa

17 Likes

Being on night shift has really hampered stalking the forum for the coveted spots on the podium.

Well done to all the team for good steady progress.

Also fingers crossed for a fresh release tomorow so we can get to try out the progress in a Comnet :slight_smile:

15 Likes

Is there a regular time for comnets to go live?

3 Likes

Usually Friday evening UK time ish every second week is the consensus at the moment but nothing set in stone

4 Likes

Ah no, it is gone indeed! Oversight when compiling the update there :man_facepalming:


No, the main large changesets are in, so the codebase has stopped moving as much, so it’s being worked on again now.

15 Likes

Thx 4 all your hard work Maidsafe devs.

Optimisation and debugging is a great place to be

Can’t wait 4 another comnet with all the changes

Keep hacking super ants :crazy_face:

15 Likes

Its beautiful update! Thanks for hard work!

10 Likes

Thank you for the heavy work team MaidSafe! I add the translations in the first post :dragon:


Privacy. Security. Freedom

7 Likes