Summary
Here are some of the main things to highlight this week:
- The Shared Vault has been updated to the latest version and therefore requires a new config file.
- We’ve been taking a more detailed look at how we could technically implement Labelled Data.
- Some minor enhancements were implemented in the CLI, like being able to use the CLI in read-only mode when querying public data.
- Community member @danda submitted another feature implementation to allow the
files add
command to read the content of the file being added from stdin (instead of from a local path).
Shared Vault update
Today we have updated the Shared Vault from v0.19.2 to v0.20.1. As always, any restart of the vault means a new vault config file so we have uploaded the latest config file to S3 here. If you want to use the Shared Vault then please download this config and save it in the appropriate location on your machine. Note that we decided to take the vault config file out of GitHub and host it on S3 so it did not relate to any one specific GitHub release.
You can read a reminder on how to connect to the Shared Vault, where to store this config file, or how to run your own local vault here.
Important note - SAFE Vault v0.20.x data format is incompatible with v0.19.x data format, therefore the new Shared Vault we have started is clean with no data on it. If you have a local vault running v0.19.x and upgrade to 0.20.x you will find that data on that vault does not display as expected. We recommend re-uploading any data/sites, etc. to the new vault, local or shared.
Vaults Phase 2
We have made good progress in making vault integration tests use real Routing (with mock parsec). The client requests can now be handled by the virtual vault network established by Routing, and we can start testing different scenarios related to having a network of many vaults. The pull request with the changes is currently being reviewed and is expected to be merged after we resolve an issue with the random generator. A seeded RNG is important for predictable tests, but we’ve found that it behaves inconsistently between the Routing and Vault crates.
The updated handshake process for clients bootstrapping to vaults is progressing further, both on the client and vault side, and the pull request is currently being reviewed. It seems to be working fine, but we need to run some more tests before we can merge in the code.
SAFE API
After last week’s release, we’ve focused these past few days on some technical debt we had on several parts of the safe-api repo. For example, we updated the release scripts to publish only binaries for connecting to a real vault as we won’t be publishing binaries for using the mock network anymore in our safe-api releases. Using the mock network is still possible but developers willing to do so will need to build the artifacts themselves following the instructions from the corresponding README files.
We were able to upgrade to the latest commit of the safe_client_libs repo which is allowing us to re-enable our end-to-end (e2e) tests in our CI process. The safe-cli automated tests will now be run on GitHub Actions (GHA) for every PR like it was before on Jenkins. This first step of e2e tests use the mock network still, but we already have everything ready to also run them with a local vault instance on GHA as well, as soon as an issue discovered (apparently in the vault) is resolved.
Some minor enhancements were also implemented in the CLI, like being able to use the CLI in read-only mode when querying public data, e.g. using the cat
or dog
command on a public website won’t require the CLI to be authorised. This is a nice step in terms of UX, as a user may want to query public data from the network using the CLI and won’t need to log in and/or authorise the CLI for doing so, which makes it as simple as when accessing public data from the SAFE Browser.
Another feature implementation submitted by @danda in the last few days was to allow the files add
command to read the content of the file being added from stdin (instead of from a local path). This opens up the possibility of using the files add
command in a piped chain of commands, which is something @danda has also been researching to define a proper strategy across all commands in the CLI.
Lastly, we started looking at adding a feature suggested by the community to be able to obtain the XOR-URLs of files without uploading them onto the network, i.e. expand the --dry-run
feature to also work with files and not just with their containers when doing a files put
or files sync
. We are working on adding such a possibility in the SAFE Client Libs first, so we can then properly expose it from safe-api and CLI, along with a new top-level safe xorurl
command which can provide such information directly on local files/folders.
SAFE Client Libs
The team had begun working on bringing back a bunch of tests that were disabled during the migration from mock routing to the connection manager. These tests are primarily meant for testing the mock-network setup in the SAFE Client Libs without considering any type of client or account. With PRs #1106 and #1109 merged, the tests are now re-enabled.
Meanwhile, we’ve also been helping out the rest of the team with the Labelled Data RFC. We are still in the initial discussion phases and a lot of comments have come in already. Join us on the linked thread, if you haven’t already.
On the CI/CD front, we have completely ported our setup to GitHub Actions and have released the latest versions of our libraries. Note that the new versions of the crates have not been published yet. Publishing new versions of all three crates (SAFE Core, Authenticator and App) is pretty straightforward. We’re looking for an approach where we could selectively upgrade specific crates. That’s the last step towards initial migration to GitHub Actions. We’ll be looking into ways to improve our build speeds using caching etc. as well.
We are continuing to work out the details and APIs for the new connection manager. Now that we’ve implemented the new handshake process for the clients, there are some more details missing: some of the APIs related to disconnection and to notifying of connection status changes need to be rehashed in terms of the new approach of talking to the vaults. Once we merge the new handshake protocol implementation in Vaults, we can finally start testing the client with multiple vaults running within a section.
Labelled Data and Sharing
After a lot of discussion in the pre-RFC thread on labelled data, we’ve been taking a more detailed look at how we could technically implement this and what work would be involved.
Discussions are still ongoing, but we’re coalescing around an approach that uses BLS key shares for handling and verifying labels on data. This approach will make use of our current permission scheme on data, which keeps things flexible there but also gives us an added boon: The ability to share private data.
Using BLS key shares, applications will be provided with their own sign keys which the DataHandler
will be able to check and verify a label’s presence on a given data. Which means sharing data will be possible by generating a new KeyShare
to be given to another account, and as long as the corresponding label is present on the data, the data handler will permit the operation. (Leaving the ClientHandler
to manage apps and verify things on the account level.)
So now we’re getting into the nitty-gritty of the proposal, checking data structure and looking for edge cases. All of which we’ll be working into a fully-fledged RFC soon.
SAFE Network App (mobile)
This week we solely focused on the SAFE Network App on mobile and made significant progress when it comes to adding new pages and improvements. We added the create account, app details and settings page. Now only two pages are left to be added in the app from the designs.
We added a check in the app to verify if the SAFE Mobile Browser app is already installed on the device or not. If the app is installed then the launch page will show a banner from where user can directly switch from the SAFE Network App to the browser, otherwise we will provide a link from which you can download the Mobile Browser app.
We improved the navigation and gesture logic so now we can easily navigate between all the pages. These improvements also involved adding navigation gestures for multiple carousel views implemented on different pages and closing/opening of the popup pages.
Since we skipped the styling tasks for all pages initially, this week we started to port colors, themes, and styles from the designs to the code, and update the pages to use the same.
SAFE Authenticator (mobile)
We made some minor refactoring changes in the mobile authenticator app to improve the user experience around choosing different vaults to connect to, a PR for the same is under review and will be merged soon.
Node Ageing
We are still progressing the work on promotions (adults to elders) and demotions (elders to adults) (PR 1929). We have merged a number of PRs that we extracted as stand-alone parts needed for that work to be completed. We are now working through the more rare test failures. The main remaining difficulty is ensuring in all cases that in case of elder changing or splitting, all the needed elders receive and vote for the events that need consensus.
Useful Links
Feel free to send us translations of this Dev Update and we’ll list them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together