SAFE Network Dev Update - December 12, 2019

Summary

Here are some of the main things to highlight this week:

  • The Shared Vault has been updated to the latest version and therefore requires a new config file.
  • We’ve been taking a more detailed look at how we could technically implement Labelled Data.
  • Some minor enhancements were implemented in the CLI, like being able to use the CLI in read-only mode when querying public data.
  • Community member @danda submitted another feature implementation to allow the files add command to read the content of the file being added from stdin (instead of from a local path).

Shared Vault update

Today we have updated the Shared Vault from v0.19.2 to v0.20.1. As always, any restart of the vault means a new vault config file so we have uploaded the latest config file to S3 here. If you want to use the Shared Vault then please download this config and save it in the appropriate location on your machine. Note that we decided to take the vault config file out of GitHub and host it on S3 so it did not relate to any one specific GitHub release.

You can read a reminder on how to connect to the Shared Vault, where to store this config file, or how to run your own local vault here.

Important note - SAFE Vault v0.20.x data format is incompatible with v0.19.x data format, therefore the new Shared Vault we have started is clean with no data on it. If you have a local vault running v0.19.x and upgrade to 0.20.x you will find that data on that vault does not display as expected. We recommend re-uploading any data/sites, etc. to the new vault, local or shared.

Vaults Phase 2

Project plan

We have made good progress in making vault integration tests use real Routing (with mock parsec). The client requests can now be handled by the virtual vault network established by Routing, and we can start testing different scenarios related to having a network of many vaults. The pull request with the changes is currently being reviewed and is expected to be merged after we resolve an issue with the random generator. A seeded RNG is important for predictable tests, but we’ve found that it behaves inconsistently between the Routing and Vault crates.

The updated handshake process for clients bootstrapping to vaults is progressing further, both on the client and vault side, and the pull request is currently being reviewed. It seems to be working fine, but we need to run some more tests before we can merge in the code.

SAFE API

Project plan

After last week’s release, we’ve focused these past few days on some technical debt we had on several parts of the safe-api repo. For example, we updated the release scripts to publish only binaries for connecting to a real vault as we won’t be publishing binaries for using the mock network anymore in our safe-api releases. Using the mock network is still possible but developers willing to do so will need to build the artifacts themselves following the instructions from the corresponding README files.

We were able to upgrade to the latest commit of the safe_client_libs repo which is allowing us to re-enable our end-to-end (e2e) tests in our CI process. The safe-cli automated tests will now be run on GitHub Actions (GHA) for every PR like it was before on Jenkins. This first step of e2e tests use the mock network still, but we already have everything ready to also run them with a local vault instance on GHA as well, as soon as an issue discovered (apparently in the vault) is resolved.

Some minor enhancements were also implemented in the CLI, like being able to use the CLI in read-only mode when querying public data, e.g. using the cat or dog command on a public website won’t require the CLI to be authorised. This is a nice step in terms of UX, as a user may want to query public data from the network using the CLI and won’t need to log in and/or authorise the CLI for doing so, which makes it as simple as when accessing public data from the SAFE Browser.

Another feature implementation submitted by @danda in the last few days was to allow the files add command to read the content of the file being added from stdin (instead of from a local path). This opens up the possibility of using the files add command in a piped chain of commands, which is something @danda has also been researching to define a proper strategy across all commands in the CLI.

Lastly, we started looking at adding a feature suggested by the community to be able to obtain the XOR-URLs of files without uploading them onto the network, i.e. expand the --dry-run feature to also work with files and not just with their containers when doing a files put or files sync. We are working on adding such a possibility in the SAFE Client Libs first, so we can then properly expose it from safe-api and CLI, along with a new top-level safe xorurl command which can provide such information directly on local files/folders.

SAFE Client Libs

Project plan

The team had begun working on bringing back a bunch of tests that were disabled during the migration from mock routing to the connection manager. These tests are primarily meant for testing the mock-network setup in the SAFE Client Libs without considering any type of client or account. With PRs #1106 and #1109 merged, the tests are now re-enabled.

Meanwhile, we’ve also been helping out the rest of the team with the Labelled Data RFC. We are still in the initial discussion phases and a lot of comments have come in already. Join us on the linked thread, if you haven’t already.

On the CI/CD front, we have completely ported our setup to GitHub Actions and have released the latest versions of our libraries. Note that the new versions of the crates have not been published yet. Publishing new versions of all three crates (SAFE Core, Authenticator and App) is pretty straightforward. We’re looking for an approach where we could selectively upgrade specific crates. That’s the last step towards initial migration to GitHub Actions. We’ll be looking into ways to improve our build speeds using caching etc. as well.

We are continuing to work out the details and APIs for the new connection manager. Now that we’ve implemented the new handshake process for the clients, there are some more details missing: some of the APIs related to disconnection and to notifying of connection status changes need to be rehashed in terms of the new approach of talking to the vaults. Once we merge the new handshake protocol implementation in Vaults, we can finally start testing the client with multiple vaults running within a section.

Labelled Data and Sharing

After a lot of discussion in the pre-RFC thread on labelled data, we’ve been taking a more detailed look at how we could technically implement this and what work would be involved.

Discussions are still ongoing, but we’re coalescing around an approach that uses BLS key shares for handling and verifying labels on data. This approach will make use of our current permission scheme on data, which keeps things flexible there but also gives us an added boon: The ability to share private data.

Using BLS key shares, applications will be provided with their own sign keys which the DataHandler will be able to check and verify a label’s presence on a given data. Which means sharing data will be possible by generating a new KeyShare to be given to another account, and as long as the corresponding label is present on the data, the data handler will permit the operation. (Leaving the ClientHandler to manage apps and verify things on the account level.)

So now we’re getting into the nitty-gritty of the proposal, checking data structure and looking for edge cases. All of which we’ll be working into a fully-fledged RFC soon.

SAFE Network App (mobile)

Project plan

This week we solely focused on the SAFE Network App on mobile and made significant progress when it comes to adding new pages and improvements. We added the create account, app details and settings page. Now only two pages are left to be added in the app from the designs.

We added a check in the app to verify if the SAFE Mobile Browser app is already installed on the device or not. If the app is installed then the launch page will show a banner from where user can directly switch from the SAFE Network App to the browser, otherwise we will provide a link from which you can download the Mobile Browser app.

We improved the navigation and gesture logic so now we can easily navigate between all the pages. These improvements also involved adding navigation gestures for multiple carousel views implemented on different pages and closing/opening of the popup pages.

Since we skipped the styling tasks for all pages initially, this week we started to port colors, themes, and styles from the designs to the code, and update the pages to use the same.

SAFE Authenticator (mobile)

Project plan

We made some minor refactoring changes in the mobile authenticator app to improve the user experience around choosing different vaults to connect to, a PR for the same is under review and will be merged soon.

Node Ageing

Project plan

We are still progressing the work on promotions (adults to elders) and demotions (elders to adults) (PR 1929). We have merged a number of PRs that we extracted as stand-alone parts needed for that work to be completed. We are now working through the more rare test failures. The main remaining difficulty is ensuring in all cases that in case of elder changing or splitting, all the needed elders receive and vote for the events that need consensus.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

73 Likes

First time I’m first !!!

32 Likes

Good news. Looking forward to vault phase 2. Great work once again!

19 Likes

Nice work again team. Fingers crossed that testing the vault with real networks goes smoothly. Fantastic progress in all areas.

16 Likes

The labelled data idea sounds like a really nice improvement. Turning the file system into more of a database … I’ve been wanting that sort of thing in my local file system for a long time - so this is a big leap technically and may allow a lot of new functionality down the track.

Very Excited!

Keep it cranking Safer’s :wink:

22 Likes

Looks like you have all had another busy week , thanks for all your hard work team.

16 Likes

Exciting to see vaults connecting with real routing.
I hope for nice Christmas gift.

14 Likes

First time I’m 7th.

All I want for Xmas is time to play with some CLI and push the limits of what I can do!

Big year 2020 :smiley:

18 Likes

Thanks so much to the entire Maidsafe team for all of your hard work!

16 Likes

So much cool stuff happening! Excited to see the API’s mature even more and see the amazing looking front end and back end being tied together more and more.

Labels seem like a great idea and even better with BLS

Great work to everyone @maidsafe!

16 Likes

7th is the new first :slight_smile:
As always, thank you to all on the team for your hard work and excellent communication.

EDIT: echoing Nigel and others Labels look like a great feature and the shared private data too

14 Likes

I keep seeing all the commits flooding in and it is great to see! Keep the heads held high - just one last push before Christmas! :grinning:

Great to see the MVE plan earlier in the week too. It all looks like it is coming together nicely!

16 Likes

Great to see vaults+routing coming together—beautiful job guys and gals! :star2:

10 Likes

told ya routing seems stuck on Elder promotion demotion didn’t I. Can someone make a poll when do you think Fleming will be released? I would do it if I knew how to…

Well lets just click our fingers and get it done for you.

Its a software project and some areas take longer than others, who would have thunk. Wow colour me pink and call me Sherlock.

Starsmick grow up and if you just asked nicely you might have got a positive response rather than sounding like a trader trying to suppress the project.

All you had to ask was.

Can one of the dev team please help me understand where Elder promotion is at and how much work there might be ahead for it?

See that would get a positive response in future. How about you try that and no one will complain about your entitled attitude

And no I will not make a poll for you

15 Likes

I recall you lying about your aunt working on the project!
Nothing better to do?

Edit. Plus a quick glance through your posts appears to show this to be a lie also.
Not that I will waste any more time on it.

8 Likes

In the spirit of giving.
Merry Christmas @anon62744983

  • Less than 3 months
  • 3 to 6 months
  • 6 months or more
0 voters
14 Likes

Which version of fleming?
Because the release being worked on is far more substantial than fleming was ever going to be.

10 Likes

Exactly - the version being built now is being built with beta version modules. That is final versions before beta testings

Whereas previous it was a “just working” versions of the modules which may need further work/additions to bring them up to beta versions.

15 Likes

thanks @Josh for the poll, Merry Christmas

4 Likes