SAFE Network Dev Update - November 28, 2019

Summary

Here are some of the main things to highlight this week:

  • @jimcollinson made a video intro to the work we’ve been doing on the UX of Vaults.
  • We finalised the implementation of JSON-RPC in the authd communication protocol.
  • Community member @danda submitted another feature implementation to the SAFE CLI, this time to support two additional formats for the CLI output (YAML and pretty-printed JSON) :tada:

Vaults Phase 2

Project plan

We are continuing to streamline and enhance the handshake process in Vaults, making it closer to how it should work in the real network supporting multiple sections. It closely follows the existing process, expanding it with new requests and responses returning elders’ connection information. We have made changes in safe-nd, our types library, and in the Vault code.

With that part getting close to completion, we’re starting to discuss the multi-section design and requirements. With more people switching to the Vault team, we should be able to start implementing it soon, gradually moving to Phase 2b.

SAFE API

Project plan

This last week we were able to finalise the implementation of JSON-RPC in the authd communication protocol. We’ve created a new child jsonrpc-quic crate within the safe-api repo which exposes a minimised set of functions that are used by other crates to implement the Authenticator daemon communication protocol. On one hand the safe-api makes use of it to be able to send JSON-RPC messages to authd over QUIC, and on the other hand authd makes use of it for accepting those requests from clients, generating a JSON-RPC response, and sending it back over QUIC. Please refer to the safe-authd README to see some examples of these types of requests/responses.

Another feature implementation was submitted by @danda to the safe-cli, this time to support two additional formats for the CLI output. In addition to the JSON format already supported (now renamed to jsoncompact), the CLI can now generate its output in YAML format (yaml), or in a pretty-printed JSON format (json). These new formats can be activated using the --output flag on any CLI command, e.g.:

$ safe dog --output json safe://hbhyrydt5b95dmumcm8yig4u1keuuh8hgsr5yx39xn4mqikp91sbdhbpwp
[
  "safe://hbhyrydt5b95dmumcm8yig4u1keuuh8hgsr5yx39xn4mqikp91sbdhbpwp",
  {
    "data_type": "PublishedImmutableData",
    "media_type": "text/x-markdown",
    "xorname": "c761fec6b9ad8b382a6d4e4a44e7c3f0d626c0fcfde2d2dd5537f2b047c0b68d"
  }
]

Another aspect which was in our TODO list for a long time, and that we finally were able to put up is to have an e2e test suite which can be run as part of our CI process. By e2e we mean running our front end tests against a real vault. We were able to set this up on our new GitHub Actions setup, running all our safe-api and safe-cli tests automatically on CI. This will for sure give us some peace of mind whenever we are upgrading SAFE libraries in safe-api, as well as when releasing new versions of vaults, we will be verifying that many use cases are still all working fine throughout all the layers of the stack, i.e. from safe-cli, through safe-api, safe-client-libs, routing, and finally the vault/s. Moreover, we detected already an issue when running all these tests for the first time on CI which we are investigating.

A new release of safe-api and safe-cli with the new safe-authd is imminent now, and we are just finalising the required changes to our CI scripts to automatically generate the packages and publish them as usual, so we are aiming at having this new release ready in the next few days.

Now that we have the first release of safe-authd almost ready, we will be able to focus more on the design and implementation of the new data types APIs, as well as on some new ideas around the named containers (a.k.a. default containers in Alpha 2) and how they will fit in the new APIs. We plan to get you all involved in the discussion as usual very soon, so be prepared for some interesting technical discussions (…we know you always are though :slight_smile: ).

SAFE Client Libs

Project plan

Over the last week, the team has been working on removing the rust_sodium dependency which has been causing some issues with regard to maintenance for quite some time now. We took this up in a divide and conquer approach and that’s helped us make good progress so far. The dependency will be removed from self_encryption in this PR. In SAFE Client Libs, we have migrated signing, symmetric encryption and key derivation in PRs #1093 and #1096. What’s left to be migrated is asymmetric and public key encryption, both of which are already in progress. We’re confident that we’ll be wrapping this up soon and moving on to supporting development of the next Vault phase.

We have also started on improving the connections manager. This is a module which handles actual QUIC network connections and data transfer between clients and Vaults, and so far it has been simple, supporting only communication with a single Vault. This is not like the real network would behave though, so now we are focused on implementing it in a proper way, making clients bootstrap to their section and maintaining connections to multiple elders. This would allow us to finally start running the SAFE Client Libs test suite in integration with multiple Vaults.

SAFE Network App UX — Vaults

@jimcollinson has made a video intro to the work we’ve been doing on the UX of Vaults.

Check it out for a rundown on how you’ll be able to use one to create an account, or offer up your spare resources to earn Safecoin:

SAFE Network App (desktop)

Project plan

We’ve fixed a couple of small bugs in the SAFE Network App, including one that was preventing account creation via the UI. There’ll be an alpha channel update to this effect coming soon, though we’ve just noticed an issue with auto-updating in that channel so this update may need a fresh download of the SAFE Network Alpha app. We’ll update here on the forum once we have this solidified.

With that update in, we’ll be moving on to a beta release of the authenticator functionality asap. The only blocker we have here, which has been a major focus during the last week has been prepping authd application builds and the CI pipeline. We’ve successfully set up the authd build process using GitHub actions, and have only to get the code signing and notarization set up there before we’ll be in a good place for more regular updates to both authd and safe-cli (which will unblock us with SAFE Network App releases too!)

SAFE Network App (mobile)

Project plan

Previously we started to work on the screens (LaunchPage and CreateAccountOnBoarding) for the SAFE Network App on mobile, however, due to many small refactoring requirements, we couldn’t merge those. This week we revived those older PRs, refactored them and merged them into the master branch. With these PRs merged we have some of the initial pages for the app. In the coming weeks when we will be focusing more on the SAFE Network App for mobile, we will start with fixing some UI styling issues on these pages.

SAFE Browser (desktop)

Project Plan

Our intention was to release an alpha version of the SAFE Browser at the end of last week, however, we decided that it made sense to hold off until the updated versions of the SAFE Network App, SAFE CLI and APIs were all released as there’ll be breaking changes therein which will mean to maintain compatibility between all the apps, updates will be released across the board. Once everything lines up with these other repositories we will release an alpha version and as always, let you know via the forum.

SAFE Authenticator (mobile)

Project plan

We updated the native libs to the latest safe_authenticator for the mobile app to support all the new API changes. Since the new APIs don’t use invitations and we have moved to test coins instead of using mutations, we removed all the APIs related to the invitations and account info. The app UI was also updated to reflect the same changes.

We also updated the auth request dialog to show the requested test coin permissions and the PR is currently under review and testing.

The apps and their files are sandboxed on the mobile phones and that’s why it’s not easy to update or replace the vault_connection_info.config files for any app. This file contains the information required to connect the client with a vault.

To tackle this problem on mobile devices, we are working on a small feature for the authenticator app to help you manage vault connection files. This week we implemented and tested this feature as a new section in the authenticator app settings. With this available in the next release, you can have multiple connection files in the authenticator app and easily switch between those to connect to different running vaults for testing purposes. Some final touches may be required to make it more user-friendly and easy to understand.

SAFE App C#

Project plan

This week during testing we noticed that the native iOS libs generated from the safe-ffi which we ship as part of the NuGet package, aren’t working as expected. The lib files are showing some segment fault issues and do not allow the devs to run the code on the simulator or physical devices. After many trial and error attempts, it seems the issue is related to the latest SDKs. Currently, we aren’t sure what’s the best way to resolve this but as soon as we figure this out, we will be releasing another preview NuGet package.

Node Ageing

Project plan

We are still making progress on Promotion of Adults to Elder and Demotion of Elder to Adults. We are now down to a few remaining failing tests, and have been able to extract parts of it and merge it early so we could simplify the final integration.

For handling disconnects via tracking unresponsiveness, we resolved the bulk polling issue due to the new testing pattern. However, we decided to delay the merging of the work a little bit. This is to allow other cleanup works to be merged first and enables this critical change to be fully tested and verified.

Additionally, we also progressed our clean up of handling active connections and we’ll be able to complete it once the Promotion item is complete.

BLS

Project plan

We fixed issues and rough edges that were identified, including this issue which was causing us trouble in tests. We are working through the remaining planned tasks, and remaining bug fixes, but are focussing on Node Ageing first.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

73 Likes

First? …

19 Likes

I am second…T.T

14 Likes

Thanks to the entire Maidsafe team for all of their hard work!

Happy Thanksgiving everybody!:turkey:

23 Likes

Thank for another great video @JimCollinson

21 Likes

It’s my pleasure!

24 Likes

UX looks great in the video. simple and well thought through. very much looking fw to actually running a vault soon :slight_smile:

22 Likes

Great video! Very exciting to see just how easy it is to run a vault. This is exactly how it should be for mass adoption. Great work team!

25 Likes

The exciting news. Looking forward to more achievement.
:grinning:

17 Likes

giphy

@danda you are a hero.

26 Likes

Go Team! Not entirely sure what you mean by phase 2b. But that sounds like closer to Fleming to me :slight_smile: Sounds like the easier things are pretty much solved and now everyone is ganging up on the few remaining big problems that need to be solved to have a Fleming.

19 Likes

It looks fantastic Jim. Seemingly idiot proof the most important aspect of all.

22 Likes

Thanks team really good to see things progressing and again starting to fit together, and again great to see your input @danda!

Very nice video @JimCollinson, looks nice. You are getting into the swing of these… more, they’re great!

20 Likes

You’re too kind. Rushed this one out a bit (was a busy day all in!) but glad you like it. More on the way for sure.

18 Likes

Great to hear @JimCollinson - and thanks for this one! I’ve been enjoying these videos.

13 Likes

The UX is just FANTASTIC! :heart_eyes:

A grandma could farm with her phone with such simplicity!!

Just wonderful.

23 Likes

Watching all the github commits flying in on BLS this week. Great to see!

Keep up the fantastic work. Fleming is getting closer by the day! :sunglasses:

22 Likes

Great work to all the Devs. The UX stuff is fantastic. Even I understood it. Keep it going team.

15 Likes

Thx for the kind words @zoki. Really though I’ve just been trying to polish a rough edge or two in the CLI as I learn rust. The maidsafe devs are doing the tricky important stuff. Also @bochaco has been especially great to collaborate with and patiently show me the right way to do things.

26 Likes

Don’t underestimate the value of your contribution @danda, it’s not just the code and features you bring but the example you set, as well as the learning you are doing that may help others in various ways.

I look forward to trying out the things you have added, and all the new CLI features that are coming.

19 Likes