SAFE Network Dev Update - August 20, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • Check out the awesome SAFE Network Wallet Bot developed by @DeusNexus here - Telegram has now been connected to the SAFE CLI! :clap:
  • There has been a huge refactoring and simplification effort in the SAFE Client Libs over the past few weeks, with circa 18,000 lines of code gone, and a simpler, clearer and cleaner codebase in place.
  • Work to expose a Routing async API continues smoothly with the same code simplifications we have seen in other crates over recent months.

CRDT

safe-fs

Work has continued on the crdt-tree implementation. Some refactoring was done to move each data structure into its own file. Also, some additional examples/tests were added that deal with log truncation via causally stable threshold. Finally, some minor optimisations and cleanup were done. The code still needs to be commented and basic unit tests added for each struct, but is nearing completion.

In related news, we received a Ph.D. dissertation authored by a Doctoral candidate who has written a distributed filesystem which implements the crdt-tree algorithm. Even better, the filesystem is implemented in Rust as a FUSE filesystem. We couldn’t ask for a better fit with our efforts.

As such, some time this week was devoted to reading the paper carefully, taking notes, and evaluating vs our own design decisions, as well as trying out the code and reviewing it. We have identified 2-3 ideas that we may be able to incorporate into our design.

We do not presently have permission to share the paper or code with the community, but will do so if that changes.

SAFE Client Libs and Vault

SAFE Client Libs Project plan
SAFE Vault Project plan

SAFE Client Libs and Vault integration

Refactor (with a machete)

This last week in SCL we’ve been working on finalising some larger refactors and simplifications to our codebase on the back of the recent quic-p2p async push.

So on the one hand quic-p2p has enabled us to remove a lot of complexity at the client layer, removing internal event loops and extraneous structs to get towards a much simpler and leaner client.

We’ve also been re-evaluating the role of the SAFE Client Libs more generally. For a while it has been a big library, with code for a core client, applications, authenticators and FFI integrations. But as we’ve been diving around the code more, some things have been looking like they needed an overhaul.

At the highest level, the main thing we have done is strip back what a client is. Previously it was a number of things (app, authenticator… “client”), now it’s just a “client”. A client can connect to the network using a random set of keys (and so not write data), or you can pass a secret key to your client, and if there’s enough money, you’ll be able to do write operations.

This drastically simplifies things. “Apps” just become “clients” (more on how that’ll work below), using secret keys in the same fashion as any other client. We no longer need the ‘authenticator’ layer now that we have no apps. Which again, means less code both in SCL and in vaults (where an app’s permissions would have been checked prior).

So with these (rather sweeping) changes, we have been able to reduce the SCL codebase by something like 18,000 lines of code in the last few weeks. Getting a codebase that is much simpler, clearer, and cleaner. Easier to document, and easier to review as well.

New “apps” and “authentication”

As noted, we’re dropping the “app” distinction in the codebase, but not in general. What we’re looking at is this: Instead of relying on another layer of network code to check applications (with all the complexity and specific network requests that brings), in the coming time we’ll be reworking our Authenticator app, to be just that. An application which works using standard network APIs. And this application will manage any “apps” in much the same way as previous. The main distinction being that each app will have its own keypairs, which the Authenticator will know and control.

This opens up an increased amount of flexibility in controlling app spending and permissions, and it’s these permissions which will now solely rely on the Permission structs at the data. Which means we’ll still be able to fully control what/where an app can write, as well as increasing the control over spending.

It’s been a large refactor. But the benefits are already abundantly clear. With 18,000 lines of technical debt gone, we’re free to move much faster in SAFE Client Libs. We are now continuing with finalising integration with updated Vaults and quic-p2p to get back at working towards testnets.

Vaults async and tests

Integration of async routing has begun also in Vaults, in parallel to the efforts in routing.
The minor (multi-threaded) test framework we mentioned last week is more or less completed in this PR, has already been put to use and is working well.

Alongside the above, we are continuously testing and fixing any issues found in the wake of recent messaging refactors, and the new transfers and farming features.

Routing

Project Plan

The work to remove SendNeighbourInfo votes, as mentioned in the last dev update, has now been merged with all failing tests resolved, along with a PR to (fix incorrect section key assigned to non-elders).

Meanwhile, the implementation of the relocation refactoring work (Remove relocate_queue and make relocation independent on the order of churn events) is progressing well. We also started to use the testing framework from Vaults to verify Routing’s behaviour under real network usage.

The effort related to exposing a Routing async API continues smoothly, and as in previous refactorings made in other crates we are already seeing code being simplified as we introduce async functions in the codebase. So far we were able to expose an async stream which is what the vault will be using to receive each of the events the routing layer produces, and right now we are working on getting vaults integrated with these changes and successfully bootstrapping with other vaults using the new APIs.

Standardisation

We are pushing our standardisation process a bit further forward now. We will switch bincode to MsgPack for serialisation (as @mav pointed out in a post a while back, bincode is Rust only). We will standardise on base32z for human readable encoding.

This builds on the move to QUIC from Crust which gave us a standardised secure transport layer.

We will also have a push in-house to document the message API that exists in Vaults. This represents the most fundamental API and so using only industry recognised formats and types in this area will allow folks to build vault clients in any language, or indeed from any device capable of sending and receiving messages.

More importantly, this should allow us all to see how everything works at a single level, i.e. a vault receives message X, it then does Y to X and sends a message Z to another vault. Or a client sends message A to 7 vaults (its elders) who take A and agree B, then send message C to a remote section to “do D”, and so on.

Lots of fast moving parts right now, but we want to be ready for some quick testnets and to have everyone much more informed about all the moving parts in the network.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together!

:love:

77 Likes

Thanks again for all that you are doing @Maidsafe team!

21 Likes

This sounds very interesting, the ability to control where an app can write data was discussed some time ago because it seemed a much better way to ensure privacy, so I’m very please to see it being implemented.

This alone is great news, but simplifying the client side codebase at the same time is quite something. I look forward to writing more Rust!

Thanks all at @Maidsafe, and well done @DeusNexus for that Telegram bot which is great to see.

30 Likes

Amazing update. To delete is heavenly :slight_smile:

20 Likes

The bot was a nice surprise… is that open source and available somewhere?

Always encourages by refactoring, like new generation and spring cleaning out the clutter.

Look forward to the next iteration.

:+1:

21 Likes

Also, that bot is really cool @DeusNexus

18 Likes

Sigh. A Primer-writer’s work is never done.

I like the sound of all the simplifications, but not sure I understand this. What are ‘Permission structs at the data’? Is this written up anywhere?

…we’ll be reworking our Authenticator app, to be just that . An application which works using standard network APIs. And this application will manage any “apps” in much the same way as previous. The main distinction being that each app will have its own keypairs, which the Authenticator will know and control.

This opens up an increased amount of flexibility in controlling app spending and permissions, and it’s these permissions which will now solely rely on the Permission structs at the data .

25 Likes
16 Likes

Thx for the update Maidsafe devs.

:clap: :clap: :clap: love code reduction and adapting ideas.

Keep hacking super ants :vulcan_salute:

:rofl:

14 Likes

This is part of the CRDT work, a while back signatures and permissions were removed from the actual data and put in the vaults logic. It was not a good move for many reasons, it went against the data chains principles of being able to republish valid network data etc.

Now the data elements themselves contain all of the keys, signatures and permissions. So it’s much better and much simpler to grok. You look at the data, see who can do what (permissions check) then look at the operation supplied (logic check, i.e. is it crdt correct) and then apply the op. The great thing is you can apply the op whenever you want and different replicas can be out of sync, which sounds bad, but it is very natural. They all get there and you can tell them to merge, get all the data elements yourself, and merge locally and put it back up yourself if you want. So much simpler, more correct and just what ants would do :wink:

Mentioned briefly in the update, we are having a big push to document our code in much better ways. So the code and logic will be explained in the code docs. Focus initially on messages and what they do. This will be “live” documentation and any code changes will fail the doc tests which we run in CI.

We have started a conversation internally around the primer and how to link this, but it all in gitbook/GitHub or some form of a live wiki. We have no clear path yet, but the primer you have is a first-class citizen for our docs as far as we are concerned and we are hoping that the current push will enhance the primer, but not replace it. Hope that makes sense?

24 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

4a5qv4

17 Likes

Ah OK - so it’s all there in the metadata basically. Got it.

In gitbook/GitHub or some form of a live wiki.

Sounds like a great idea.

Glad to hear it! Proud moment! I think a middle layer will always be needed between the technical docs and the non-technical readership and I’m happy to work on that. I really enjoy trying to get my head around it all, actually, even if I don’t always fully understand it (understatement).

23 Likes

With all the guys are doing just now, understanding it will get much simpler. We are removing so much complexity and homing in on a much smaller set of rules. It will definitely get simpler and therefor more efficient and secure.

The primer so far is invaluable and having a community that does these things is quite unbelievable at times. Thanks John, it is a huge task you do there.

33 Likes

Perfection. :star::star::star::star::star: Thank you! :+1:

16 Likes

A big thanks to @DeusNexus for developing the wallet bot, and congratulation to the MaidSafe team on cutting 18,000 lines of code!

14 Likes

A lot of positive things to address in this update! Just for now, isn’t the removal of PARSEC some 18,000+ lines of code as well? So we’re talking roughly 36,000 lines of extra complexity removed. Wow.

I can’t wait to see solid docs. If it’s that time and they can stay relevant then they will be a huge aid in outsider development.

Things keep rolling and it’s amazing what this small but tight knit team are able to achieve. You are a constant inspiration team @maidsafe

20 Likes

One of the things that’s so exciting about this is it exemplifies a third party successfully building on and expanding the core of what MaidSafe is developing. Are you on Twitter, @DeusNexus? Would love to tag you in a tweet!! :smile:

Thrilled to see continued progress from MaidSafe as well!

22 Likes

Yes, expect that possibly even next week :slight_smile:

16 Likes

Great update. Just out of curiosity, where are we at with DBCs?

8 Likes

Back-burner for now. We might have some interesting developments with safecoin etc. as it’s an ongoing discussion in house. I must bring DBC back into that convo.

13 Likes