A little taster of things to come this week. As you will know, consensus mechanisms are at the core of almost all decentralised networks, including bitcoin. In fact, the chief innovation of bitcoin is arguably the blockchain’s consensus mechanism. But there’s always a tradeoff. In bitcoin and other blockchains it’s the fork-handling mechanism, which means that two versions can exist for a long time (in theory forever) and have to be resolved via PoW, which is bitcoin’s Achilles Heel.
As such, we’ve been keen to avoid consensus algorithms (as they are generally understood) where possible - do ants use them? But there are some occasions where with computer networks they are probably necessary. This is what Mostafa has been looking into, and we’ll dig deeper into that in the coming weeks.
General progress
Right now we’re revamping the CI machine process to be a bit leaner and more secure. This is very much @chriso’s baby and he’s been working to get all the elements in place including the APIs, EC2 instances, serverless functions and DDoS protection. Nearly there now.
We’re working on removing as many read/write locks as possible during data replication. This is mostly @joshuef’s domain. Josh and @bochaco have also pretty much completed the sn_node refactor to reintroduce bidirectional communications into the code. This is something we had before but it got very complex. Simplifying the codebase means we can bring it back. Still testing, but we’re already seeing stability improvements.
@roland is removing the traceroute feature. It causes bloat, it’s not proved useful (and even less so with bi-directional streams), so we’re getting rid.
Consensus mechanisms
There are many consensus mechanisms in computer science. Some are simple binary affairs, others are complex multi-variant mechanisms. The important thing to understand, though, is that they are all just tools for a job, and nothing more. That job is ensuring agreement between entities for a certain amount of time. But just as people need to agree on certain rules to get along, such as which side of the road to drive on, they don’t need to agree on everything to live side by side. Unlike blockchains, people don’t need system-wide ordering, and nor does Safe. So we need to pick the right tool for the job, do so only when necessary, and importantly make sure we are not hemmed in by that choice.
Safe is an asynchronous decentralised network. Things are changing all the time; “state”, meaning a snapshot of the current understanding we have of the world, is almost always different from the point of view of every individual node. And yet for a short time at least, these nodes need to agree.
Consensus is basically a word to say that at some point in time, everyone that matters needs to see the same state in order for an action to happen.
So how do nodes reach agreement about state? Well, that depends on the circumstance, the action and the scope.
Often on Safe, we can reach agreement through anti-entropy (AE). This is a simple exchange of information to make sure we’re talking to the current elders in a section. As such, AE could be seen as a simple, binary, local consensus mechanism, although we just think of it as a check that parties that need to be in sync are all on the same page. Certainly, it’s a far cry from bitcoin’s total system-wide order.
When elders form an agreement about a course of action, we use distributed key generation (DKG) to ensure a supermajority of elders agree. That’s a type of local, binary (yes/no) consensus too, although we don’t tend to use that word.
Then we have conflict-free replicated data types (CRDTs), which we use for mutable data. These have a logical clock, allow for concurrent access and eventually resolve forks to one state. This all happens without any consensus. Forks are fine and self resolving.
But then there are other situations where we can’t tolerate a fork, even for a nanosecond, and that’s where, as an asynchronous decentralised network, we definitely need consensus.
And there are grey areas where we can tolerate forks for a certain time, but eventually need to resolve them. These may need a formal consensus mechanism.
So where might the ants need some technical help? One is recruiting new nodes from a list of perhaps hundreds of candidates, all with a node age of zero - which one do we pick? Here we may need to rely on higher intelligence that the group can provide.
Another is Membership - managing which nodes are in a section. DKG is a blunt instrument and doesn’t handle edge cases like multiple simultaneous joins and leaves very well.
And we may need a higher level of consensus to manage DBCs across sections to prevent double spend.
This is what the team - particularly Mostafa and @davidrusu - are focussed on at the moment: implementing the right protocol in the right place at the right time. More soon folks!
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
At least, I think some of it will be surprising for many. It’s great to dig in to the guts of things like this. Helps us all.
by David’s write up, I’ll need decades more to understand it