Upcoming Testnet New/Improved Features Summary

To be clear, this is not a public testnet announcement post!

Work continues in a couple of areas to link the features in this document together, and so a testnet announcement will not be this week.

We hope to have a public testnet ready for you to try out within the coming weeks. Thank you for your patience while we work through the remaining issues.

As always, a full update on progress will be provided in Thursday’s Dev Update post.


With a new public testnet inching ever closer, we thought it would be good to post a summary of some of the main innovations we’ve made recently to get us to this point. These details are mostly available via our usual weekly dev updates, but have been spread out over several months now, so we bring them together here in one post.

Will this be the Fleming release?

Not just yet. We view this as a testnet for the Fleming release.

As well as there being a couple of known gaps in some of the areas that are described below, for example not all data is split between sections yet, we fully expect to have to make tweaks to algorithms and address bugs or shortcomings identified through the much more realistic, varied, long term and intense testing that a public testnet provides when compared to our internal testnets.

This will be the first time many of these new features, and even refactored existing features, have been used in the wild - theory on paper and in code doesn’t always match up with the real world! We therefore expect that there will be several iterations of this testnet before we consider it feature complete and stable enough to officially say that the Fleming goals have been achieved.

See the Safe Fleming Network target feature set on our roadmap page for an overview of what the Fleming release will achieve, originally published in 2019. This is what we have been working towards all this time, with so many related and supporting features and updates required to achieve this.

What will be new/improved in this testnet?

There will be many new, improved or refactored parts to the upcoming testnet, probably the most notable being section splits, rewards, AT2 payments, accepting new nodes only when resources are required, and the section chain being fork resistant.

Note that we are close with two other features - lazy messaging in sn_node and one single (CRDT) data type with immutable policy, but we are not 100% sure at this point whether they will make it into the initial testnet. Perhaps a partial implementation of lazy messaging will be sufficient for the first iteration, while the data type change will only be included if it is complete in time and does not cause any adverse effects which would delay testnet release.

Some further details on all of these below.

Section Splits

Section splits mean that the Network can grow naturally.

As the section size grows and there are enough nodes to separate into two new sections, a split happens. This is quite tricky as the section chain splits into two, while the data held also splits with approximately 50% moving to the new sections.

As we balance nodes’ addresses — note that we still have more work to do here in a future testnet iteration — then the data should already be on the correct nodes in the vast majority of cases. However, some promotion will happen. This means the requirement of switching real chunk copies among nodes is reduced to a minimum. Mostly, it will only be the indexing map of the chunks, held among the elder nodes, which needs to be split and transferred among the now two elder groups.

To ensure the two split-out sections both contain enough capable nodes to handle requests and provide storage space, we enforce that both of the after-split sections satisfy the RECOMMENDED_SECTION_SIZE, and only then shall the split happen.

Not all data is split at the moment, we have not yet implemented the splitting of Map, Sequence and chunk metadata. These will follow in future testnets.

Wallets should be split, however we are still to see how this fares with higher traffic.

Rewards/Farming

Individuals who choose to supply the resources that the Network requires will now be rewarded with Test Safe Network Tokens in the upcoming testnet. In testnet iteration 1, these rewards will be paid to a node (by its new section) after each relocation of it. In an upcoming testnet iteration, the plan is to add a (smaller) payout on split to Elders only.

As this is a testnet, please keep in mind that these are only Test tokens, they will not be able to be traded for real Safe Network Tokens on launch of the real network.

Token Payments: The AT2 way

We’ve had Safe Money and its transfers, or payments if you will, integrated into our testnets for a while. We recently revamped the whole implementation with the AT2 protocol, which gives cryptographically verified transfers at theoretically higher speeds than blockchain-based technologies.

The introduction of shared signing of Elders makes these payments historically verifiable, and together with clients keeping their replicas, this would also allow for re-uploading and verifying a wallet even without it existing in the network.

This lives entirely apart from the consensus mechanism of the Network and is done one-on-one with the users, hence reducing the stress on the network.

It’s worth noting, that the client driven process can be frozen if there’s an attempt to make an out of order payment. The balance will currently thereafter be unusable. We’ll be addressing this in future testnet iterations.

Accepting New Nodes Only When Resources Are Required

A recent improvement to the Network, to be showcased for the first time in the upcoming public testnet, is that existing nodes now decide when they want new nodes to join the network.

This improvement is designed to help prevent Sybil attacks by virtue of the fact that we do not allow unlimited nodes to join the network at will; the danger of an unchecked number of nodes being that potential attackers could easily flood the Network, taking it over with hostile nodes.

Routing assumes the elder-nodes will track all the adult-nodes in the section, and when they detect the average storage capacity (or some other resource) becomes too low, they will flip a flag so the section starts accepting new nodes. The section elder-nodes should detect this at more or less the same time, so consensus can be reached.

Fork Resistant Section Chain

In order to prove that a piece of data was signed by a group of nodes that were once valid members of a section, even after those nodes are long gone, we have a “section chain” which is a list of section keys linked together with signatures. Previously, this chain required that the section agreed on which key to append to it next. If there was a disagreement on that, the chain could fork into two (or more) mutually incompatible chains which could break the section. This could happen, for example, at times of intense churn.

This problem has now been resolved by accepting section chain forks internally, but always presenting a unique and deterministic order of the chain blocks to the outside.

A new implementation of the section chain data-structure, which is now a proper CRDT, guarantees (eventual) consistency regardless of in what order the operations are applied, how they are grouped, or even duplicated. Even if multiple distinct keys are inserted into it, everybody will eventually agree on which one is the most recent one and thus who the current elders are (because each section key is uniquely associated with a single group of elders). And all of this is achieved without involving any complicated consensus mechanism.

Lazy Messaging

Note - Lazy Messaging across the board is not guaranteed for the upcoming testnet

A major hurdle in any decentralised network which will experience churn, such as ours, is keeping the nodes on the network up to date with the network’s current state. For example, who has joined and who has left? Or who are the current Elders in any given section? We have implemented lazy messaging to resolve this issue.

The lazy messaging pattern has existed for some time in sn_routing, we’ve now updated sn_client's infrastructure communications to the pattern. Which means we’ll now get updates to section info as soon as we attempt to message an incorrect, i.e. old, section.

Lazy messaging works by slightly increasing the size of messages sent between nodes so they include some extra information on the Network’s current state as seen from different observers in space and time. If a node receives a message and it realises that the network state details in the incoming message differs to what it believed was the state, then it communicates further with the sending node to bring itself, or the sender, up to date with the correct network state, then the original message can be processed accordingly.

Lazy messaging is currently being implemented at the sn_node level, and will feature in a future testnet.

One single (CRDT) data type with immutable Policy

Note - Not guaranteed for the upcoming testnet, a nice to have which is very close to completion

Up until now we’ve been exposing two different Safe native data types, a Sequence (already being CRDT) and a Map (which we haven’t yet converted into CRDT). They both used to allow mutations to the Policy set for them, i.e. the permissions to access and mutate them.

After more research and tests made on the Sequence data type, we came to the conclusion that being able to handle all different types of scenarios of concurrent Policy mutations, combined with concurrent mutation of the data itself, was making the logic and code very complex. We also realised we were trying to go against the CRDT nature itself. This is why we moved away from that path and we now have a Register CRDT data type which doesn’t allow mutations on the Policy - once the content is stored on the network the Policy is enforced on each message and while the policy cannot be mutated.

This all means that all our client side abstractions, e.g. FilesContainers and NRS containers, are now based on this single CRDT data type.

54 Likes

First hahahah

12 Likes

Well, it is…without any dates though :stuck_out_tongue:

Thanks for all the details !

10 Likes

Wow, great read! phenomenal work! Really really good stuff :grinning: While its not ready yet, it seems like it’ll make a big impression when it does arrive :slightly_smiling_face:

12 Likes

Thanks StephenC, this is mouth watering and helpful.

Any chance of a short post listing the fundamental types and their key characteristics, and then the network based types built upon them? Either matching the next testnet or describing what is being aimed for in Fleming.

12 Likes

We will defiantly get to this. Documentation is a big issue in house right now and we will reach out wider for help there. I am not keen on another data dump and would really prefer we formalise all algorithms, including some mathematically formalised. I would like these to be easily auditable and describe the whole network, barring any implementation detail. The same for the API.

I am of the opinion we can document the whole algorithmic network (including data types) in 4 pages at most. These will be very math dense pages, but should rarely or perhaps never change.

Then with that we can expand with descriptive text, examples etc. in a manner similar to what @Scorch is looking at. i.e. it’s time to cement all this in place in a provable manner now. It feels great to be in this position now.

24 Likes

I was thinking of a short summary to give an overview of the fundamental types we now have or are aiming for, with very basic details of what they do. Then a list of the types (wallet etc) which are built on top, and which fundamental types they incorporate.

It doesn’t need (or necessarily help) for this to be detailed or technical but a lot has changed since the data types were last enumerated and described.

5 Likes

Ah, yes, our fundamental type now is going to be a CRDT Register type, then potentially a gset. We are currently looking at all data being built on top of that, wallets, NRS, sequence data (for type and fork resistant). This is not a bad write up https://bartoszsypytkowski.com/operation-based-crdts-registers-and-sets/ However we are testing and cleaning this as we speak :wink: So data to follow is perhaps the best thing to say right now. We are hoping to get this cracked in days though. (also keep the API we have as much as possible)

11 Likes

Setting the stage though :grin: 2 weeks out is Thursday April 1st so let’s write that off… three weeks out is too far away for setting the stage.
That leaves… only one other option, right?

I was thinking a post like this was sorely need just this morning, thank you!!
I have been feeling a bit lost with all the changes.

Do we have a nice catch phrase for this?

This is going to cause a race to first like no other.
If I dont get a node in… expect punches to fly. :facepunch: :wink:

8 Likes

Agree like Ants do! :smiley:

11 Likes

What happens with the data that is not split?


What does this mean?


Is lazy messaging essentially a hopefully more efficient way of doing gossip?

Does this current testnet address the issue where some people couldn’t establish connections through their routers (aka has the hole-punching, UPNP, etc. been addressed?)

3 Likes

I hadn’t realised Register was the name for a kind of CRDT so the link is very helpful. With my meagre understanding I’ll attempt to summarise :wink::

  • Register is a simple and fundamental CRDT which holds a single value.
  • GSet is a “grow only” collection of key/value pairs. So you can add a key/value and update it, but you can’t remove a key/value (so history is retained).

There’s no mention of the CRDT Tree, so I’m guessing that’s going to need looking at later, maybe refactored on top of a GSet, or is no longer planned for Fleming?

Fair enough, looking forward to it!

6 Likes

Good question. We have made changes in this area that we “think” should make a big difference for many who had the IGD/UPnP issues that were seen in testnets of the past. I know from testing internally I was never able to connect from home to previous public testnets, but I now can. It’s the same story for other MaidSafe staff. We’ve held back on claiming that this issue is resolved as we have such a small sample to base that on, so we’ll see once the public testnet is up.

19 Likes

We actually think we may handle all fs calls with a register, more to come.

5 Likes

Intriguing!

4 Likes

Having seen some of the team talking about limits to entries and mutations for mutable data types, one thing I’m curious about is what happens when these are reached?

For a map, could the list of key/value pairs, or number of mutations just be extended by an option? Or will the network handle somehow splitting the data into some new structure that can be extended? Or will it be the app developer that has cater for this eventuality? Or is it not decided yet?

Possibly a question for elsewhere though, so I’ll hold onto it if no answers are forthcoming just now!

2 Likes

A new container is spawned from the first one. It’s ongoing work right now as ensuring integrity under high levels of concurrency has to be proven.

7 Likes

Every credit and debit is signed individually. These are kept at Replicas in the network (essentially being Elders in the section matching the wallet xorname).
A client runs an Actor which communicates with these Replicas according to the AT2 protocol, to carry out the transfers. So, the Actor and the Replicas will eventually have the same set of credits and debits - all signed by section keys that are verifiable by the network.
When we say that the wallet could be wiped from the network, and the Actor could upload it sometime later, it is because of these signatures by section keys. The network knows of previous section keys and can therefore say that the sigs and thus the transfers are valid.

The wipe-and-reupload isn’t implemented yet, and there is currently one caveat to that approach: When wiping, we would need to leave a last-tx trace of some sort, as to be able to accept the upload. This is because there is currently no way to prove that there was no debit after the last credit. And if the Actor doesn’t upload debits following on a credit, then it would seem the wallet has a balance which it actually does not… But there could be other approaches.

8 Likes

This sounds promising, I hope I will be able to join this time. Thanks for this post. Its great to see everything laid out for the public to see. The excitement is building. My intuition says the testnet iterations toward Fleming will be short, and we will be heading toward beta fast.

8 Likes