Safe Network Dev Update - August 27, 2020

Development Updates
1

Summary

Here are some of the main things to highlight since the last dev update:

  • The CRDT Tree repo has now been made public, and the crate has been published on crates.io :tada:
  • Refactoring Routing and Vault codebase to make use of async continues to progress well.
  • Routing relocation refactoring work and parsec removal will be combined into a single PR, with this now in the final stages of development.
  • We have a behind the curtain sneak preview of the SAFE Network App ‘Action Menu’ UX.

2594×1344 249 KB

CRDT

safe-fs

This week was spent writing correctness tests for the Tree CRDT data type as well as minor refactoring to make the API cleaner and reduce the number of clone() calls. All aspects of CRDT (idempotency, commutativity, associativity) now pass tests, as well as other Tree specific tests specified in the academic paper. A crdt_tree repo has been created on GitHub and the crate published on crates.io :tada:

With the Tree CRDT basically complete, focus can now turn to the filesystem layer that will make use of it.

SAFE Client Libs and Vault

SAFE Client Libs Project plan
SAFE Vault Project plan

SAFE Client Libs and Vault integration

In the last week, alongside some smaller changes to be ready for the new async quic-p2p, and some tidying up of repo CI, we’ve been working on improving message handling in the client. The basic refactor work got us to a point where we could receive query responses from the whole section, though this presented issues if a response from one elder didn’t come through (we’d never get any response… despite actually having six). As such we’ve used some of Rust’s future goodness to check which responses we have as they come in, and then choose the most appropriate responses from those we get from elders. This gets us closer to a ‘quorum’ version of the response, and means we can return this as soon as we have it, without having to wait for all 7 elders to respond.

Another change is coming in the way of how we handle Notifications from the network at Client-side. Previously, we had only one quic-p2p connection from the clients to the vaults by which all means of communications happen (Requests/Responses and Notifications). But now with quic-p2p being updated to async and allowing us to create multiple connections to a node, we can make use of this advantage by having two connections, a primary one that will be used for sending Requests then awaiting Responses, and a secondary connection on which we only listen in for Notifications. Notifications are something that we might receive even without getting involved. For example, somebody crediting our wallet with coins is not something of our doing, but we would still receive the notification. Therefore it should be a connection that must be open all the time while we are connected, and we cannot use the primary connection for the same reason as it’d be pretty active awaiting its own Responses.

Vaults async and tests

Refactoring Routing and Vault codebase to make use of async has been progressing well. We were able to start using the quic-p2p new async API and we have Routing also already exposing an async API the Vault consumes to receive network messages. We are currently trying to finish hooking up the Routing message handling logic into this new implementation. We have so far been able to get clients and vaults to talk to each other and are almost done with re-enabling the bootstrapping process, which is what we’ll be working on in the next few days.

Since we are working in parallel on the whole stack, from quic-p2p, through Routing layer, and Vault, as we progress and test all these changes we make adjustments to any of those layers as necessary, which is what we’ve recently done to the new quic-p2p async API when testing the vaults trying to bootstrap.

Routing

Project Plan

First of all, the fleming Routing branch has now been merged into the master branch, and master branch set as the default repository branch. We created a pre-fleming tag as a reference point for the previous master branch code base. We believe it is time to take this step as we are getting closer to our fleming goals in Routing, and to be consistent with other crates to avoid any confusion.

During last week’s work of using the Vault’s testing framework to verify Routing’s behaviour under real network usage, we spotted some issues. The first was related to dkg_voter, which the PR to record completed DKG section_key_index has resolved. Another issue is that we are resetting parsec on the consensus of SectionInfo and OurKey, but those are not parsec events anymore, which means they are not strongly ordered relative to other events. This shall be resolved once we have parsec removed.

We have also been working to improve the DKGVoter’s resistance against the malice attack in the Accumulate DKGOldElders messages PR, which was merged today.

Relocation refactoring work continues, with Adam making great progress there. While resolving failing tests, it was realised it would make sense to remove parsec at the same time, in the same PR, as splitting the work turns out to be particularly tricky and a waste of resource trying to make it work with parsec still in place. So, the final PR, not quite ready to officially raise yet, will contain both the work of relocation refactoring PLUS parsec removal. Adam will be taking next week off, so this work will be handed over to Qi to finalise.

SAFE Network App UX

Feature Tracker / Screens & Flows

This week we take you behind the curtain on a feature that is midway through the design process. You won’t find it in the Screens and Flows link above, but it ticks quite a number of boxes on the todo list on the way to the Minimum Viable Experience (MVE) we are building, and could prove to be quite a useful design pattern, especially when we start the jump into desktop UIs.

It’s what we are calling the ‘Action Menu’. A navigation system for the SAFE Network App that gives both access to some of the outliers in the information architecture and allows quicker access to regularly repeated actions, while also providing a mildly text command driven interface to boot. Let’s take a look.

The Action Menu is accessed from the big fat button bottom left…

993×1501 646 KB

The items in this menu can be clicked/tapped through to underlying menu items, in the way you might expect…

993×1501 145 KB

But here’s where it starts to get fun, that input bar at the top allows me to take a more text driven approach, searching through menu items and commands

993×1501 41.8 KB

And I can take that even further by typing a more specific natural language command. In this example, I’m transferring some funds from my wallet to a friend:

993×1501 40.7 KB

Which would then skip me right to the end of a payment flow, with an amendable summary, ready to send

993×1501 82 KB

Plus, I can get a history of what I’ve searched for here for quick access to recent actions

993×1501 43.5 KB

On top of all that, I can do things like pin actions to the menu homescreen, for even more rapid access to commands I want readily at hand

2727×1501 449 KB

Also, as a wee extra you might notice with the action menu open, is that the close button at the bottom left can be dragged/wiped to log the user out of their session.

The idea is to extend this gesture to the action menu button itself, allowing the user to hold and swipe it to rapidly log out from almost anywhere in the app.

993×1501 662 KB

So there we are, I hope you like the sneaky peak. It’s still pretty raw, but will be refined and iterated over the coming weeks. Let us know what you think!

Standardisation

You will have read about ongoing standardisation work in last week’s update, this week some further work has commenced around repository and crate naming conventions. We have toed and froed over the years between different case conventions and ended up with somewhat of a mix between snake_case and kebab-case. It was previously our intention to standardise on kebab-case, however this led to complications in Rust, which converts - to _ in some cases, and so most crate names remained with snake_case. We concluded this week that we should switch all crate and repository names to snake_case to finally standardise this across the board.

In addition to this, we also decided that our crate and repository names were, in some cases, inconsistent, no longer accurate, or unclear given the changes in our code structure over the years, particularly recently. We will be going through each crate and repository and renaming where appropriate. Any which are SAFE Network specific will be prefixed with sn_.

These changes will be made in the background over the coming days/weeks.

Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together!

:love:

83 Likes
2

first hasdofjasoidfjsadfsad

Edit - AH HA, idk if I have ever snagged it, but somehow my random browse time today caught it 1m in :laughing: .

Thanks for the hard work MaidSafe team!

25 Likes
3

Frsecond 20 Charachters!

19 Likes
4

1st to have read it?.. :partying_face:

I wonder the Balance symbol looks like a dollar $ and the other might not be ideal.

I wonder a symbol that is not a currency already and easily available might work better… eg section symbol § looking like a couple of S from a larger network of Ss.

Anyhow looks good… I keep wondering at TypeDart but I guess that’s just an example.

:+1:

20 Likes
5

UI/UX is looking very sleek!! :sunglasses:

The streamlining/standardization efforts across the board are great, and will likely make it easier for people to track along and maybe even pitch in too :smile:

Here’s to smooth sailing!

:sotros25:

25 Likes
7

Looks like @qi_ma and Adam in particular are super busy tying up lots of loose ends. More power to your elbows chaps!

21 Likes
8

Can someone explain the benefits that CRDT Tree will bring to SAFE? It’s about guaranteeing consistency when you move tree-type structures such as folders within folders within folders, is that the basics?

14 Likes
9

Yeah, this comes up a bit WRT these mockups and prototypes: it is actually the dollar symbol as a time saving substitute/placeholder for a glyph of the Safecoin symbol.

19 Likes
10

Great progress is being made!!

13 Likes
11

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

4b98zv
4b98zv500×651 82.2 KB

20 Likes
12

yes, that’s pretty much the basics. It ensures eventual consistency of tree modifications by multiple independent actors (replicas).

There are two primary use cases:

  1. As a building block for developing a distributed filesystem.
  2. As a general purpose Tree data type for use by Safe Apps.

The motivating factor for implementation of the Tree data type at this time has been (1). The phase 1 (prototype) goal is to build a local filesystem that uses the Tree internally and can be locally mounted and used by regular applications. The phase 2 goal is to network these together via SAFE for a true network file system. The CRDT properties of the Tree data type ensure that when different replicas merge writes together any conflicts can be resolved automatically and in the same way on every replica.

For details of the problems that the CRDT Tree solves (and how it solves them), please see the paper:
A highly-available move operation for replicated trees and distributed filesystems

29 Likes
13

Great - thanks for the clear answer @danda

9 Likes
14

@JimCollinson :drooling_face: UI/UX pron… love the simplicity of Pay @JimCollinson $1M for his great work. :stuck_out_tongue:

Thanks Maidsafe devs for a great update :+1:

21 Likes
15

I’m sure we could make that specific command even easier. I’ll speak to the guys about hard coding it :smile:

20 Likes
16

Thanks as always for all the hard work, everyone @maidsafe. Such a marathon to be running… an ultra-marathon!

A basic question for front end development:

Would I be understanding right to assume that this means that all elements of a web page will load sequentially, so instead of my loading waterfall looking something like this (when simulating a slow network (to make things clearer))

eg-website-load

It would look more like this:

eg-async-website-load

If so, is there any likelihood of multiple connections in the future?

Just wondering cos I think it’s always worth adapting design to what is there and what might be there soon rather than some imaginary perfect scenario.

9 Likes
17

almost, well certainly more so.

Already there are multiple connections and each chunk will be from a differnt connection (likely).

12 Likes
18

Was hoping also to read more on CRDT and how it helps in the context of “blockchain” type networks(I know we are not blockchain but we still desire some similar features, distributed secure consensus etc.) for a faster/better consensus and there seems to be a paper hidden here:

Dunno if anyone has read up on it but just dropping it as a helpful idea if it sparks any thoughts with MaidSafe folk. I need the ELI5 to grasp complex consensus that is fast enough for a large usable network and how we can achieve a distributed consensus with fairly minimal latency via CRDT and how they will tie into SAFE concepts of elders and sections etc. (well hopefully low enough latency the network will be usable to the common man. Not as slow as tor or onion routers for instance).

Edit - AH HA found it without the lame signup/member crap: https://epezhman.github.io/pubs/fabricCRDT.pdf

3 Likes
19

Trick is make it much less complex. So SAFE for instance the network is there and section know each other, Elders approve “stuff”, so some agreement happens with over 50% of them and that is consensus.

Then consensus == 1 is when a user signs an operation that can be provably correct, i.e. the next version in a list. So that works to. The huge difference from blockchain to BFT algorithms though is with CRDT data, each data item is it’s own wee “blockchain” if you like.

So then to ties that together, the client signs his operation and changes a data item “state”, the network section also signs that and we have provably correct data (client signed it) and network valid data (section signed it).

I don’t know if that helps, but instead of a god algorithm that does strict ordered consensus for “everything” having to go through a single “thing” CRDT sectioned signed data means each data item can update at once and even better each data item can update concurrently. So simpler, less code and more features. It is really the way these things should work.

Quick caveat though is that the data items themselves need carefully thought out. So it’s not any data item can work, you really need to consider provable updates (technically the types need to be a semi-lattice to provably merge).

Also quick clarification, both CRDT and BFT, even blockchain are all provably strongly ordered, many folk still think crdt is weak order or fuzzy other, they are strictly ordered (via causal order). This is something I See a lot and it’s quite wrong.

13 Likes
20

So from this:

image

To this:
image

I suppose geniuses have a different conception of “less complex”!!

lol

Keep hammering Maidsafe! love the progress :wink:

11 Likes
21

Not sure if @DeusNexus served as inspiration for this with his Telegram Tip Bot but in both cases it is beautiful to have such an easy way to pay using natural language.

The action tool looks very nice, quick and handy Jim!

Just a thought that had popped in my head regarding labels when uploading. Should a user be able to add an app label (assuming apps have there own specific label) when uploading or if an app has capabilities for certain labels that the app would handle then this would be unnecessary? I would assume the latter makes the most sense. Guess I’m curious which route Maidsafe would take and if adding labels will be included in the upload UI/UX.

The ability for any app to use tree directories will be insanely useful and even better to have safe-fs to recursively upload whole file directories for the user and to share with an app to use and add to its own file system! This is huge imo and make things so easy for app developers. Seems like it’s a good setup for SAFE OS as well? :thinking:

PARSEC going by the wayside and nothing but real network tests sounds very promising. I have this sense when I’m reading these things that the devs are all grinning together as they see all of their hard work and recent shifts to new novel approaches really paying off. I think many here and far beyond will be all smiles when they can get their hands on the goods.

Keep up the amazing progress and pace @maidsafe
We’re all rooting for ya.

15 Likes