Safe Network Dev Update - January 21, 2021

Summary

Here are some of the main things to highlight since the last dev update:

  • We’ve found and merged a workaround for a lag issue that had been doing its best to remain unsolved.
  • A draft PR for the rewards flow has been raised, and is nearing completion.
  • Some broken client network tests, which have prevented us from progressing with testing the fix for a KeySection and DataSection messaging issue (the final issue blocking a public testnet), should now be resolved and testing will now resume.
  • We merged the CI/CD PRs in each of our BRB crates, which resulted in automated test coverage, versioning and our first releases of the BRB crates on crates.io.
  • @JimCollinson provides another captivating update on the progress of the Safe Network App & UX.

Safe Client, Nodes and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan

Solving of lag issues

There’s been a lot of deep diving in the last week attempting to find the source of the significant lag we’ve been experiencing while running internal testnets. And while we’re not wholly ‘fixed’ we do have a much better idea here, and a simple enough workaround has now been merged.

The crux of this issue was something blocking a node’s handling of client requests. This may be related to some sub-optimal node concurrency (which we’re also looking at fixing), but the root seems to stem from dropped connections blocking the ability of node’s to respond, and so they’ve been effectively waiting on the connection to be fully abandoned by the lower layers - this wait is the lag that we’ve been seeing.

It seems that a simple tweak to a node’s idle-timeout (lowering it from 30s to ~5s…we’re experimenting to find optimal numbers now) will get the nodes sufficiently unblocked that we can again get client tests running nicely. So while we’re digging in to see if there’s an issue in qp2p’s connection pool, and refactoring sn_node somewhat, we’re able to move forwards now with testing everything once again.

Rewards flow

We’ve previously mentioned the ongoing work on the rewards flow. The rewards are paid by the Elders in a section, by them using a distributed AT2 actor - (i.e. the section actor) and aggregated signatures. Every time there is a change in the Elder constellation, the section key changes, and the distributed actor transitions to the new key by transferring all the tokens to it. There’s a different initial path on the transfer flow for the section actor (and any multi-owner actor), compared to single owner actors, where validation of a transfer is proposed by a majority, before there is a validation at the replicas.
The transition from genesis node as well as tests for this are completed and confirmed, now we are working on the subsequent transitions.

While working on the above, we found that the Elder state was not consistent between the dependent operations in such a transition, and thus this week we refined the representation and handling of this state. A draft PR for the rewards flow was raised, containing all of the above mentioned work, as it is now nearing completion.

KeySection and DataSection messaging issue

In a previous dev update, we mentioned that we were investigating removing the possible duplicated signature-aggregate work carried out by sn_node during exchanging messages among KeySection and DataSection. There is a WIP PR removal signature aggregate raised for this purpose. This PR passes sn_node unit tests, and we had been using sn_client network tests to carry out further verification. However, due to other refactoring work in sn_node and sn_client, carried out in parallel, the sn_client network tests have been broken for other reasons, so the verification work has had to be put on hold for a short while.

We believe that this duplication removal work will address a KeySection and DataSection messaging issue which we observed during our internal testnet runs at the end of last year, something we should be able to verify via the sn_client network test. This is currently the expected final remaining blocker to being able to host a public testnet, so we haven’t been able to progress with internal testing here in the meantime.

However, with the recent merge of some refactoring work in both sn_node and sn_client, we are now back onto this as a priority, and hopefully able to work through the test cases without complications.

API and CLI

After the introduction of the sn_messaging crate, where all client message types are now defined, we were working on making the necessary changes in sn_api and CLI to be all compatible again with sn_node and sn_routing messaging. This is now complete and all changes have been merged to the main branches of all these crates. We’ve also made some progress in trying to move the sn_routing message definitions to the sn_messaging crate, although this is still an ongoing task with a lower priority than most of the other things we are working on.

We’ve also merged some API improvements made by @Scorch onto the qjsonrpc crate which makes path parameters generic. Thus we’d like to thank him for his contribution. :raised_hands:

BRB - Byzantine Reliable Broadcast

Tuesday marked another milestone as we merged the CI/CD PRs which resulted in automated test coverage, versioning and our first releases of the brb crates on crates.io.

A major improvement since last week’s update that made it into this release is support for generic Actor types via rust traits. This means that the code is now flexible enough to support most/any public key cryptography algorithm or library without further changes to the core libraries. Support for ed25519 is already included but we will be looking at adding BLS support. Even hardware signing devices could be added in the future.

Additional improvements include:

  • The brb_node_qp2p CLI has some display fixes and a new retry command in case of any dropped packets.
  • logging. All println calls have been changed to appropriate logging functions.

Work in progress:

  • test cases are being improved to return results and remove any calls that would panic()
  • doc comments are being added to make the code more usable by other developers

Finally, some more great news! Our CRDT specialist, and the big brains behind the brb crates, has agreed to extend his contract by another 3 months to assist with further feature and integration work. :tada:

Routing

Project Plan

This week we merged another batch of fixes for issues uncovered with the stress test tool. With these in, we seem to be in a much better place. We still have problems with forks (divergences in the section chain) which sometimes happen during very heavy churn. This problem will be solved by integration of the new membership algorithm, work that is already in the planning stage.

We also merged a couple of minor changes. First we made sure nodes never create connections to clients - instead when a node needs to send something to a client, it reuses the connection that the client created previously. If no such connection exists, it reports an error. This by itself doesn’t really fix anything, but it makes certain failures manifest themselves in a more obvious way, which simplifies debugging. In a similar vein, we improved error handling (PR under review) so that errors are now more specific and contain more information, again to simplify debugging.

Finally, we fixed a long standing issue with our continuous integration pipeline where one task would keep consistently failing. Even though it wasn’t critical, it still prevented us from seeing that satisfactory green check mark :heavy_check_mark: before merging a PR. Turns out we were just a bit too strict in the way we treat compiler warnings. This fix was also rolled out to several of our other crates.

Safe Network App & UX

Feature Tracker / Screens & Flows

If you recall the UX update in November, we took you through proposed updates to the Safe Network lexicon, introducing a new metaphor of a Safe to replace Account, tokens rather than Safecoin, and deprecating the use of the term Vault.

Well, these changes have started to make their way through to candidate UI designs where they can start to prove themselves in the user experience pixels, rather than just on paper.

If you jump into the Figma file for the Safe Network App designs, you’ll start to see these in action, as well as a few other refinements.

It’s a big ‘ol file though with hundreds of different screens and flows, so it’s not always easy to take in everything that’s going on. So here are a few highlights:

Unlocking Your Safe

Here we see the metaphor of a Safe for the user’s data being introduced—locking and unlocking it—with some visual cues that hopefully re-enforce that too.

You’ll also note that the Action Menu has been promoted to the top app bar, via a traditional hamburger. And that it’s now accessible in the locked state, too. But more on that later.

Working with Tokens

Along with ushering the end of the term of Safecoin, changes to the wallet screen include an overarching Tokens section, a bit of a UI spruce up, and a more multifunction New Transaction action button. It’s a way to kick off a new transaction—skipping some of the slightly longer flow we had previously—and also another entry point to start earning Tokens.

Earning Tokens

With the deprecation of the term Vault, here we can see an exploration of how we can favour a verb such as Earn to start flows for offering resources to the Network in return for Safe Network Tokens.

This flow can be started from a number of points, including from the Earnings tab on the Token screen, from the home or apps screen, and from the utility itself (as well as from the Action Menu or a typed command.)

A couple of steps to configure things, and we are off and rolling.

The Action Menu

You’ve seen this previewed before, but here’s the Action Menu, so named because you can use it to search and access all functions of the app, as well as directly kick off commands.

It’s now accessible top left of the app, and here you can see it in both locked and unlocked states.

A wee reminder of how commands can be searched and kicked off too. Handy.

Feel free to dig into the Figma file to see all the details on this, with a multitude of individual flows explored and documented.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

78 Likes

That’ll be first :sunglasses: Super update, thank you @Maidsafe.

Mini Git Portal Update

I have an exciting (to me anyway :grimacing:) update on my research for the Safe Git Portal. I’ve been researching options to build apps using both JavaScript and Rust Web Assembly in the browser while sharing a common filesystem and today I cracked it. :tada:

First I looked at Emscripten, and as part of that added a feature to wasm-git (which is libgit2 compiled to WASM). The feature (async http transports) was needed in order to make it usable in a Safe web app. That fixed, it looked hard to get Rust and Emscripten to play so I moved to look at using WasmerJS/WASI instead of Emscripten.

As of today I have a working Svelte WasmerJS/WASI app with the ability to use Rust WASM compiled for WASI (target wasm32-wasi) and using wasm-bindgen to make it easy to pass variables between JavaScript and Rust.

Or to put it in ‘human’: a really cool way to build stuff in the browser (and ideal for fast, web apps with Rust/Web Assembly on Safe). It’s taken me days to get this working and today it flies!

Now published here: GitHub - happybeing/svelte-wasi-with-rust: Svelte Rust/WASI in Browser Example

56 Likes

2nd !!! yaaay !!!

20 Likes

Top five I think? I’ll take it!

23 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

This meme has 3 versions. I can’t decide which one is the best.

19 Likes

WOW! This update was packed full of gems! Thanks @Maidafe!!!

24 Likes

Couldn’t even be early for this one!!! Super excited to read :blush::smile::blush:

This is great news!

I’m also looking forward to seeing the UX work integrated into future testnets :smile:

Thanks to @Scorch for pitching in too!!

Help spread the Safe word on Twitter!

https://twitter.com/safenetworktech/status/1352339852222722048?s=21

23 Likes

Is this the only place you create a connection?

Do you not need to set a set_write_timeout with TcpStream in std::net - Rust true?

5 Likes

It was quic connections that were the issue. sn_client and sn_node create connections and that is where we have been setting some values to idle timeout (and keep alive).

13 Likes

The app really looks so natural to use, can’t wait to use it. Also looking forward to the testnet. Good luck taking the last hurdles, we are close!

15 Likes

3rd one is best…

4 Likes

Yes No3 by quite a long way.

2 Likes

This image could be confusing.

I can read “store your data, Create a Safe” at the bottom, and there’s something called a “Safe Browser” at the top. The connection my brain made was that the Safe Browser would be used to browse the contents of my Safe.

One suggestion to solve this is to name the browser Safe Web Browser to make it clear it’s not for browsing the contents of a Safe, but for browsing the contents of the Safe Web.

18 Likes

Or just ‘Browser’ - easy fix though when the time comes.

10 Likes

In one word revolution :heart_eyes:

4 Likes

Just browser feels better, yeah.

And, you know, the time is now as this sort of stuff is currently being worked on. :slight_smile:

7 Likes

I have one question in mind where we can change maidsafecoin into safecoin? Any option in app Or web

1 Like

Yay - now with Brave browser instead of Firefox I can see the figma files for those designs.

I hesitate critiquing “UX”/?UI as it’s apparent a lot of thought and work has gone in and there’s always option to do it different.

Still… are these for all devices or just mobiles?

  • How much of the UI content is liable to be cached on device?.. there’s thumbnails and perhaps other data showing - does that matter.

  • Perhaps there’s something missing as a toggle for app being visible and closed that makes apparent that it’s “safe”… one problem I have still is understanding which mobile apps are active… can’t get M$Teams to stop pinging work chatter!

  • Conversely, with the app “open”, can the user dump media off the device - is there are route out of the Safe environment?

  • Is this to be a layer of UI on top some interface, that would allow then skins?.. I wonder that’s a magnet for getting noob graphic hackers excited about what they can do.

  • Will there be option to switch language?.. one quarter of those who speak English as their first language and so many others… if the graphics are a pool and option to switch lang, that might be a big plus.

  • Do the invites grab at phone contacts?.. I don’t know if that’s a good idea or not… I don’t like apps that do but then it’s a quick amplifier and no reason to be shy but better perhaps at a point of their being a core app that is about relationships and communication?

  • What is the invite to?.. is that into a group by default?.. might be useful as that reason to grab contacts… though option to shuffle and class them afterwards. Degrees of bacon.

  • Paye Pay once

  • Wallet everywhere needs a ?currency, otherwise will be mistaken for $£etc… talk of tokens perhaps just a “99tkn” or “99_sn”

  • Identities has the display_name and @ but I wonder user might want to recall which is which… [Anonymous] is one; [Public; close circle; friends; family; work; gym; 1degree; 2degree; etc]

  • The blue is icky but perhaps that’s just me and distinctive is ok… and a UI with hooks for getting users to create skins might be a good idea. (Safe logo blue rulz!)

  • Identity has a blob, I don’t know if that would be better with also a banner in the way Twitter etc do; so, other apps could adopt the banner and not just the avatar circle blob.

  • SafeID is taken - perhaps add what symbols are possible, as that might make a simple positive option for fix?

  • “Get the Browser” - I wonder that will need a warning on mobiles about the size that tempts… which kind of brings me back again to a thought that Create a Safe is more important than Browser, though I can see why it’s that way round… a new Safe, is cheaper more accessible option cf 100MB+ data file downloading for a mobile user?

Overall there is much to like about these designs…
:+1:

14 Likes

I just want to say how much I appreciate these weekly updates. Great work!

:sascha:

11 Likes