We’ve all agreed that this week would be a good time to look at what Safe brings to the table in terms of distributed consensus. Actually, a couple of us thought it was a rubbish idea, but a supermajority voted to press ahead, and that’s all that matters 
. So, this week we explain how Safe bridges the gap between blockchain and the protocols that underlie distributed databases like Cassandra and AWS DynamoDB.
General progress
@ChrisO has completed the task of continuous release for sn and sn_api in the safe_network repo. This means we’re back in a place where we can freely merge commits and be confident in the subsequent release. With this in place, @chrisO is moving on to looking at the CLI and stabilising that before including it in releases.
David Rusu has been putting together a demo on ring signatures and Safe. Expect an update soon (warning: it will be ‘heavy on the math’). 
@qi_ma , @yogesh and @lionel.faber have been refactoring the DKG crate and safe_network (our main current source of head-scratchers as elder promotions are stopping at earlier than seven) to reduce the number of messages to be broadcast and increase stability of section startup and split. Work continues there, but we’re definitely reaching the underlying issues.
There’s also been some work on enabling message handling priority for nodes, though this is still being evaluated. This should help us ensure that larger changes are handled more keenly than less important messages. This blocking of low priority messages was unintentionally present in the less concurrent codebase, but was lost as we freed up nodes to take advantage of more of the CPU. So this should hopefully bring a bit of order back to node’s operations.
Distributed consensus and how Safe bridges the gap
The core of Safe Network, and what makes it uniquely useful for a wide variety of purposes, is how it achieves agreement between distributed nodes that may be unreliable.
In the 1980s, decentralised agreement without reference to a central oracle was thought to be impossible, but then along came Leslie Lamport to prove everyone wrong. In fact, he was actually trying to prove that it was in fact impossible to have consistency and fault tolerance, but then he stumbled upon an answer, which was to promote temporary leaders and ensure system-wide order of operations.
The resulting algorithm was Paxos, for which Lamport eventually won a Turing Prize. Paxos guarantees eventual consistency in a distributed network where some machines may be unreliable. It means networks can avoid having a single point of failure – any node, even a leader, can drop out and the system will still work.
Paxos has proved massively successful and spawned hundreds of imitators and variants. It is mostly used for reliably replicating data between distributed machines and is the basis for cloud databases like AWS DynamoDB and Google Chubby.
It does this by trying to decide a total order of operations. There are two stages in a Paxos operation - promise and commit. A promise is an agreement to do something and a commit is the action of doing it. A majority of nodes need to agree before making a promise to mutate state or change permissions. A majority is also needed to move from a promise to a commit, for example a write or a change of ownership.
Because distributed networks are asynchronous, order is maintained by giving each operation a numerical ID and increasing the IDs over time. In the case of a clash over a promise, newest (highest ID) will generally win.
In Paxos there is always a leader node. In the case of a commit, the leader appends each op to its log and asks the other servers to do the same. Once it has heard back from a majority of the servers in its cluster it commits the change.
A new leader is elected when the current one fails to respond within a certain time. When this happens any node may put themselves forward. To become leader, a new candidate must receive a majority of votes. It must then update its logs by synching with other nodes. The voting nodes send their logs with their votes to make this easier, but it may still take a long time to update, during which time the cluster is inactive, which can be problematic.
Raft (2014), which is based on Paxos (or more accurately Multi-Paxos, the updated version) simplifies the voting process and other operations, but is essentially a modification of Paxos. Raft has now overtaken Paxos in terms of project activity.
So far so good, but while Paxos/Raft are elegant solutions to distributed consensus, their implementation requires a lot of extras if they are to work as planned.
Cloudflare is based on Raft and in 2020 it suffered a major outage due to the leader election process going into an endless loop. To guarantee liveness, PreVote and CheckQuorum optimisations are also required, plus plenty more others beside. Not so simple now.
Extending the functionality of Paxos and Raft and other variants makes them complex, which in the tech industry means they become proprietary. If only Amazon knows how to reliably set up DynamoDB, Amazon isn’t going to give that secret away in a hurry.
But the major drawback to these algorithms is their assumption that nodes do not collude, lie, or otherwise attempt to subvert the protocol i.e. Byzantine failures don’t occur. If you are Amazon you might be able to guarantee that since you control your own hardware and software, but for mixed public networks, there’s no way you can do this. Which is why you don’t hear much about Paxos et al in the crypto world even though it underpins the tech Goliaths.
Enter Satoshi
Byzantine fault-tolerant (BFT), decentralised algorithms are dominated by blockchain, which solved this hard problem thanks to the genius of Satoshi. Blockchains are highly ordered and BFT. But total order is essential and that blocks or slows down multi-access and concurrency. Blockchains are not general purpose systems, but ones designed specifically for value exchange. Blockchains cannot replace Paxos and its derivatives because they are optimised for securing and sharing transactions, not data operations.
Step aside Satoshi, Safe Network’s here
Safe is different. It bridges the gap between the store-forever transactions of blockchains and the distributed data management of Paxos and Raft.
Data is stored forever without heavyweight PoW.
There is no overall leader to attack, not even a temporary one (like the PoW winner in a blockchain). Instead, accountability is distributed between section supermajorities and these consist of constantly changing (but agreed) groups.
Then we have the issue of fault tolerance. Paxos and Raft use the concept of ‘crash’ fault-tolerance (CFT). But public networks require Byzantine fault-tolerance. The difference seems subtle but it’s not. Crash fault tolerance covers how many nodes can fail between maintenance runs, whereas Byzantine fault tolerance is how many nodes can go rogue at any point in time. CFT is calculable, BFT is not.
Safe is BFT and designed to be ready for real-world conditions where things fail and ill-wishers roam. Node age ensures Byzantine nodes are quickly disempowered. By contrast, Paxos/Raft are easily attacked by promoting a leader that doesn’t reply or replies too fast. They also rely to an extent on server clocks to manage consensus. They seem to be simple (as do Kademlia networks) but they need a lot of pre-conditions to make them work, including trusted hardware and software, no NAT traversal etc.
Safe is fully decentralised. AE, BRB and CRDTs do away with the requirement for network wide agreement to achieve consistency, and there is no need for total order. Even Amazon admits that for large distributed systems, the game is eventual consistency not total order (speed of light is a hard limit). Trying to achieve total order at massive scale inevitably leads to low performance.
So, Safe crosses many boundaries: Eventual strong concurrency on a public network, general purpose BFT, and permanent data.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
Will users be able to login two Safe accounts aside each other?.. or will that be limited initially to one account at a time… rather hoping two active accounts will be possible.
