This week we look at changes to data types as a result of bringing in conflict-free data types (CRDTs).
For those in need of a quick refresher, CRDTs are data structures with some special mathematical properties. When multiple versions of data exist, such as copies of a document being edited offline on different machines, eventually, when the machines are reconnected, they will always converge onto one ‘true’ version.
For the Safe Network, CRDTs are one of the key ways of ensuring consensus without requiring the type of heavy-duty, system-wide ordering of events as with blockchains. It’s a beautifully elegant concept, although the apparent simplicity hides the fact that under the hood some seriously clever stuff is going on. Even Google couldn’t crack the issue of concurrent writes when they created Google Docs - they had to resort to an ugly kludge. Beating Google in the concurrency game is an opportunity too good to miss!
To take advantage of the CRDT wizardry, we are reconfiguring our mutable data types - more on that below.
General progress
No testnet this week, but it shouldn’t be too far away now. 
A few niggly issues to get through first.
@chriso has been working away in the background on adding support for AArch64, or ARM64 if you prefer. You will see in recent releases of sn_cli and safe_network that ARM, ARMv7 and AArch64 binaries have been produced. Chris has been testing the AArch64 binaries and has been able to confirm that these work, there’s a test script we’ve been using to verify that here for anyone interested. We’ve not tested the legacy ARM and ARMv7 binaries, we don’t have the hardware to hand for a start, so use these at your own risk! Do also note that our crates are currently under heavy development as we move towards the next testnet, so the latest binaries are more often than not incompatible with each other on all architectures. We would advise anyone wishing to test to hold off until we announce compatible versions across the board, i.e. the next testnet.
@qi_ma and @chriso have been nailing some bugs in the continuous integration (CI) process. Smooth running CI is essential in being able to merge pull requests quickly, which itself is vital for rapid iteration and testing.
Lots happening with updating the qp2p library, with elements such as XorName being moved out of a slimmed-down sn_routing and into qp2p. Investigations are ongoing into dropped connections, keep-alives, retries and bootstrapping. Too much to go into here, but if you have questions we’ll do our best to answer them in this thread. Safe to say, it’s keeping the team pretty busy!
@oetyng is also deep into batching of chunk and payment flows. Batching allows paying once for any number of operations. Within this is also the possibility to chunk and encrypt offline, and pre-pay.
So, after local encryption and persistence, a user can:
- At any time of choice, connect to get a perpetually valid
quotefor the batch. - With that quote - again, at any time of choice - pay and retrieve a
receipt. - And, finally (you guessed it, at any time(s) of choice) upload arbitrary number of those chunks/ops.
This is all related to ongoing work to shift more work onto the Client. After all, if the Client is making the request it’s only fair it should do the work too; this approach also has anti-spam benefits.
Sequence data type have now been removed completely from `sn_api
by @anselme to be replaced with CRDT-compliant datatypes. More about that below.
Data types
Previously we had the Map data type, which was an editable key:value store used for storing information like account details; the append-only Sequence data type suited to web pages and file containers, with the ability to dial back to previous versions; and the Blob - immutable data.
The updated data types are all based on - you guessed it - CRDTs, which brings some distinct advantages.
A register is a general CRDT data type that holds a value. For Safe, we wanted a register that would also allow us to go back in time and look at previous values, so David Rusu created a new register type based on a Merkle DAG called MerkleReg - a register with history. This was implemented a while ago along with its client side public API.
This new Register makes Sequence obsolete since it can do everything Sequence could do in terms of storing previous versions with the added bonus that it can also handle concurrency. If a client concurrently writes to the Register it leads to a fork which can then be resolved by the client (this was impossible with Sequences). On Safe, we use the Register to store pointers to current and previous versions of the data, such as a text file or web page, which are stored as Blobs.
So long Sequence 
, but what about Map?
Well, Maps have been beefed up to become Multimaps, the ‘multi-’ referring to the fact that each key can now have multiple values. Once again, this allows for concurrency. Multiple clients (e.g. apps) can edit a Multimap at the same time, and the result will be forks. How those forks are resolved will ultimately be up to the app, but in time they will converge in the same way that Git branches do. The network file system NFS and name resolution system NRS are, as of this week, completely implemented as Multimaps, which will be very important down the line.
And Blobs? Well they’re immutable. In a world of constant change, some things must remain the same.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French ; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
