Update August 5th, 2021

This week we look at changes to data types as a result of bringing in conflict-free data types (CRDTs).

For those in need of a quick refresher, CRDTs are data structures with some special mathematical properties. When multiple versions of data exist, such as copies of a document being edited offline on different machines, eventually, when the machines are reconnected, they will always converge onto one ‘true’ version.

For the Safe Network, CRDTs are one of the key ways of ensuring consensus without requiring the type of heavy-duty, system-wide ordering of events as with blockchains. It’s a beautifully elegant concept, although the apparent simplicity hides the fact that under the hood some seriously clever stuff is going on. Even Google couldn’t crack the issue of concurrent writes when they created Google Docs - they had to resort to an ugly kludge. Beating Google in the concurrency game is an opportunity too good to miss!

To take advantage of the CRDT wizardry, we are reconfiguring our mutable data types - more on that below.

General progress

No testnet this week, but it shouldn’t be too far away now. :crossed_fingers: A few niggly issues to get through first.

@chriso has been working away in the background on adding support for AArch64, or ARM64 if you prefer. You will see in recent releases of sn_cli and safe_network that ARM, ARMv7 and AArch64 binaries have been produced. Chris has been testing the AArch64 binaries and has been able to confirm that these work, there’s a test script we’ve been using to verify that here for anyone interested. We’ve not tested the legacy ARM and ARMv7 binaries, we don’t have the hardware to hand for a start, so use these at your own risk! Do also note that our crates are currently under heavy development as we move towards the next testnet, so the latest binaries are more often than not incompatible with each other on all architectures. We would advise anyone wishing to test to hold off until we announce compatible versions across the board, i.e. the next testnet.

@qi_ma and @chriso have been nailing some bugs in the continuous integration (CI) process. Smooth running CI is essential in being able to merge pull requests quickly, which itself is vital for rapid iteration and testing.

Lots happening with updating the qp2p library, with elements such as XorName being moved out of a slimmed-down sn_routing and into qp2p. Investigations are ongoing into dropped connections, keep-alives, retries and bootstrapping. Too much to go into here, but if you have questions we’ll do our best to answer them in this thread. Safe to say, it’s keeping the team pretty busy!

@oetyng is also deep into batching of chunk and payment flows. Batching allows paying once for any number of operations. Within this is also the possibility to chunk and encrypt offline, and pre-pay.

So, after local encryption and persistence, a user can:

  1. At any time of choice, connect to get a perpetually valid quote for the batch.
  2. With that quote - again, at any time of choice - pay and retrieve a receipt.
  3. And, finally (you guessed it, at any time(s) of choice) upload arbitrary number of those chunks/ops.

This is all related to ongoing work to shift more work onto the Client. After all, if the Client is making the request it’s only fair it should do the work too; this approach also has anti-spam benefits.

Sequence data type have now been removed completely from `sn_api
by @anselme to be replaced with CRDT-compliant datatypes. More about that below.

Data types

Previously we had the Map data type, which was an editable key:value store used for storing information like account details; the append-only Sequence data type suited to web pages and file containers, with the ability to dial back to previous versions; and the Blob - immutable data.

The updated data types are all based on - you guessed it - CRDTs, which brings some distinct advantages.

A register is a general CRDT data type that holds a value. For Safe, we wanted a register that would also allow us to go back in time and look at previous values, so David Rusu created a new register type based on a Merkle DAG called MerkleReg - a register with history. This was implemented a while ago along with its client side public API.

This new Register makes Sequence obsolete since it can do everything Sequence could do in terms of storing previous versions with the added bonus that it can also handle concurrency. If a client concurrently writes to the Register it leads to a fork which can then be resolved by the client (this was impossible with Sequences). On Safe, we use the Register to store pointers to current and previous versions of the data, such as a text file or web page, which are stored as Blobs.

So long Sequence :wave:, but what about Map?

Well, Maps have been beefed up to become Multimaps, the ‘multi-’ referring to the fact that each key can now have multiple values. Once again, this allows for concurrency. Multiple clients (e.g. apps) can edit a Multimap at the same time, and the result will be forks. How those forks are resolved will ultimately be up to the app, but in time they will converge in the same way that Git branches do. The network file system NFS and name resolution system NRS are, as of this week, completely implemented as Multimaps, which will be very important down the line.

And Blobs? Well they’re immutable. In a world of constant change, some things must remain the same.

Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!


Tada, first first in a while!

New Data Types

@MaidSafe I’ve been curious to understand the new data types so it’s great to hear they are now in place and being built upon. I’ve a couple of questions.

  • I understand that a Register is a single value with version history, but not sure how forking works when two different values are written concurrently.

When you say a MultiMap has multiple values, I take it that each key in the map can have zero or more values:

  • how are the values ‘indexed’ or referred to?
  • is this type also versioned (seems necessary)?

Network File System

Is there a plan to implement a mountable filesystem, eg using FUSE, on top of the NFS? It seems you’d need an additional type such as a Tree CRDT, or is this catered for by NFS on MultiMap?


Only because I felt sorry for you and hung off…

Second !!!


Haven’t been on in a while. Great to see progress is still steady!


I don’t know how you do it but I’m glad you do…

that’s some high level magic going on there.


Does this mean we will also get a different NRS system for websites (from a functional point of view)?


One of many opportunities for journalists to write juicy headlines when covering the launch of the network.

Woooooo! :partying_face: :partying_face: :partying_face: :partying_face:


No functionality should remain the same for NRS


Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:


What Google can’t crack, Maidsafe will make a feature. Go team, great superhuman work here!


What are these? Why are they important?


I think ensuring that everything works on all platforms/processors is a good way of doing general testing too. And it should be made easy to generate different compilations automatically. I believe CI has something to do with this?


Its the processor architecture used by Raspberry Pis and most other SBCs (Single Board Computers) - also phones.
This will be very important for low-power vaults and for mass adoption of clients on phones.


They are different computer CPU architectures. So building to work on more CPU’s.


To be clear: they are different from ARMv7, but AArch64 and ARM64 refer to the same thing.

Just for the fun, can someone increment nesting level ?


[quote=“tfa, post:16, topic:35311, full:true”]

If you insist…

1 Like

Dont try this at home on an 8Gb card, kids…


OK, now I see… I thought they were only for Raspberry Pi’s, and thus sounded way less important than phones for example.


Thank you very much!