SAFE Network Dev Update - March 12, 2020

Summary

Here are some of the main things to highlight this week:

Vaults Phase 2

Project plan

After last week’s Baby Fleming release, we’ve received a lot of feedback and test results from the community. First of all, thanks for all your contributions. We greatly value these, especially during testnet releases. Keep 'em coming!

As you may have seen, most of these issues were due to some delays and high memory usage at the Elders (Client Handlers). This is because all client requests that involved spending of safecoin were sent through PARSEC to reach consensus before processing the request further. This could take some time depending on the size of the request, the number of requests being processed, etc. We are aware of this and will be working to improve this in the upcoming weeks. That being said, in the interest of helping test other features (API, CLI commands, etc.) we will be removing the PARSEC step from the client request-processing flow and releasing a new version of the vault. We are testing this internally and once ready, we will release the next iteration of the Phase 2a testnet. With faster response times, we should be able to connect to vaults hosted on cloud machines and this means shared section :wink: Stay tuned!

SAFE API

Project plan

During the last couple of weeks, we were able to implement a few more features for our CLI. One of them is the addition of a files rm command which can be used to remove a file or an entire path from an existing FilesContainer. This was finalised just in time to be part of last week’s Baby Fleming release, so you should be able to use it already with SAFE CLI v0.9.0.

Right after the Baby Fleming release, we continued with another feature which gives us the possibility to list files in a tree form with the addition of a files tree command, which can be really helpful for users trying to understand the hierarchy of the files in a FilesContainer, e.g.:

$ safe files tree safe://hnyynysurwzoiudugs47d7fjx7hwg3i7ch1ryowdm7gpgj4538n1mwqukgbnc
Files of FilesContainer (version 0) at "safe://hnyynysurwzoiudugs47d7fjx7hwg3i7ch1ryowdm7gpgj4538n1mwqukgbnc"
SIZE  CREATED               MODIFIED              NAME 
                                                  /
6     2020-03-03T22:31:21Z  2020-03-03T22:31:21Z  ├── another.md
0     2020-03-03T22:31:21Z  2020-03-03T22:31:21Z  ├── noextension 
                                                  ├── subfolder
4     2020-03-03T22:31:21Z  2020-03-03T22:31:21Z  │   ├── sub2.md
23    2020-03-03T22:31:21Z  2020-03-03T22:31:21Z  │   └── subexists.md
12    2020-03-03T22:31:21Z  2020-03-03T22:31:21Z  └── test.md

Another new feature added since Baby Fleming is to support fetching a range of bytes from a file, which will enable streaming for SAFE apps. We’ve changed the fetch API to accept a second (optional) argument which is expected to be the start and end position of the bytes to be retrieved from the file targeted by the URL provided.

As a parallel task, and as anticipated in previous updates, we continued with our migration to use Rust async/await in the safe-api repo, a task we started back in Nov 2019, and finally we have that merged into the master branch, which migrates the whole safe-api codebase to use async/await. Since we are trying to do this migration without affecting other components, in this first step we are exposing the Rust SAFE APIs as async already, but internally they are still locking when calling the underlying safe_client_libs API. Our next step is to try to migrate not only safe_client_libs to expose async APIs, but also the safe-nodejs APIs so they can return Promises.

After all the hard work from folk on the forum (over in the test thread), we’re looking to set up some proper benchmarking in the CLI using the Baby Fleming network. We will hopefully be able to do this in a way that gives those who are already making excellent test scripts a place to contribute directly to benchmarking tests for the network. These tests will run on CI, but should also aim to be easy enough to use/modify so that we can use them to test locally/reproducibly on any system.

Lastly, we were also able to create an installation script for the SAFE CLI. This script not only downloads the latest SAFE CLI available, but it also unpacks the CLI binary onto the ~/.safe/cli/ folder (C:\Users\<user>\.safe\cli\ in Windows), as well as set it in the PATH, so you can run the safe binary from any location when opening a console. We have updated our User Guide with instructions for how to use this CLI installation method. It’s already possible to install SAFE CLI v0.9.0 using this script.

Note that there have been reports of the install script not adding to the PATH for zsh and fish shells, you may need to add to your .bashrc equivalent manually if the script does not add to your shell’s PATH.

SAFE Network App

Design work continues on the user interfaces and flows for file permissions, and sharing.

We made the call this week to separate out what was initially a combined feed, into separate manifest lists for App Access and Sharing. This makes the slightly different flows for each a touch clearer, and also avoids overly anthropomorphising apps unnecessarily; App Access is predominantly about managing your access to your own data, rather than conferring permissions to other users.

It helps keep the divide between when a file or folder is accessible to you only, or when it has been shared with others—or published—a little more crisp too.

We’ve also now got complete layouts for file, folder, and label detail screens, and controls for their navigation and manipulation.

On top of that, we are now working through the flows for publishing and sharing files via the SAFE Network App.

On the development side, with all our dependencies updated to remove the painful openssl dependency (and use the native rust rustls), we’ve made a start on updating the SAFE Network App, to once again, pass CI, but also, get some compatibility with everyone’s favourite :baby:

SAFE Browser (desktop)

Project Plan

This week we finally released SAFE Browser version 0.16.0, which has a few tweaks: Permissive access control headers to simplify web development, enables the fetch API and updates safe-nodejs to 0.8.0. But beware! This is not :baby: compatible.

For those intrepid folk who want the ability to communicate with their own wee one, we offer v0.17.0-alpha.0. This brings in Baby Fleming compatibility, as well as clipping the wings of the safe-nodejs lib to prevent file system access without going through the browser APIs (thanks for the spot there @shane). This isn’t feature-complete yet. We’re planning to enable the APIs in browser to work directly with File objects, to make everyone’s lives easier. But for now this at least improves security, and gets us towards more finalised APIs.

SAFE Browser / SAFE Authenticator (mobile)

Since last week’s Baby Fleming release, we have been in the process of updating all required repos (quic-p2p, safe_client_libs, safe-api) to be ready to support the same set of features on mobile devices, as well as updating our NuGet package. We are using a separate baby-fleming-mobile branch in all these repos to track the required fixes and changes. Once we get the builds working on the CI and get the working set of native libraries, we will test the C# APIs and mobile apps with the :baby: Fleming and will release the new versions for the mobile Authenticator and Browser.

Routing and quic-p2p

Routing Project Plan

With the Routing team having been on leave there is only a little progress to report on this week. One ongoing task which is nearing completion is the improvement of message delivery in the presence of lagging nodes (PR 2068). We are also looking into improving our test setup to make it more closely reflect reality, which should make it more reliable and possibly even faster (in planning phase, so no PR yet). We will give more information about these changes when they are closer to being done.

We also started working on a group signature scheme using distributed key generation (BLS-DKG, without central dealer/manager) to be used for achieving consensus when clients query data from the network (though this is potentially aimed to be used for much more in the future), without having to always go through PARSEC. This work is at the initial phase at the moment. As always, we will keep the community informed as we progress.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here.

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

70 Likes

Great work and keep going towards Fleming!

Big thanks to all guys which helped to fixed baby fleming issue.

20 Likes

2nd
+some chars

12 Likes

Yee ha !!! Fan dabi dozi !!! *

Dunno if you planned it this way but this is superb example of finding an advantage in a crisis.

*Other utterances of joy are available

16 Likes

Wahey! Great update! Just to clarify, are you saying that version 0.16.0 is incompatible with baby fleming routing? Will it continue to work on the shared network?

And another question, since you’ve mentioned updates to the permissions flow here: Is it going to be possible in future to give public write access to an object you own?

(To prevent any confusion / miscommunication here, I’ll give the technical description to the question I’m actually asking: Will it be possible for a published FilesContainer with read-only access to give public write access to specific XorURLs stored inside of that FilesContainer?) In this question, the word public means, specifically, access to absolutely every user of the SAFE network, rather than specific users.

16 Likes

I am just writing up some examples on this exact thing. so Sets and Graphs (linked lists with forks). They fit well with CRDT types and as initial examples of this I selected these. It allows, collecting “likes” on posts etc. and the latter on allowing folk to reply to posts and reply to replies and so on.

Debating internally on user uploads the data type and pays X for it, then others can write to it and fill it up (so uploader has paid for the empty container to be filed). It’s a relatively simple thing and saves a ton of network work to not charge for every mutation, but just for the initial creation of the type.
Down sides as @joshuef pointed are extending the container, but we have had a few discussions about that. It could be interesting.
I think these types of containers may be flexible for devs as well as they can easy create twitter/discord and so on and not have to force users to pay. Perhaps the users pays the OP and so on, so a lot of nice twists in usage.
Anyway it’s on the drawing board so far and no further. But that does not mean it’s a ton of work either :wink:

18 Likes

Anyone wanting to continue using the old single Shared Vault should be using:

Please note that our intention is to take down the Shared Vault at the time when we put up a Shared Baby Fleming section.

Anyone wanting to try out Baby Fleming should be using:

SAFE Browser Alpha v0.17.0 has the same features as the v0.16.0 release, with the addition of our 1st attempt at making it work with Baby Fleming. It is not backwards compatible though, so v0.17.0-alpha.0 will not work with the single Shared Vault, or CLI v0.8.1.

16 Likes

That sounds really good to me, I like the idea that the option for the app account to pay for the data storage is there, it simplifies a lot of things which would otherwise result in lots of obnoxious SAFEcoin transfer interfaces in SNAPP.

Do you plan to release any sort of extensive document / vlog / podcast covering your plans here? I would be really interested in exploring it in nauseating depth once you have something workable to present.

Please note that our intention is to take down the Shared Vault at the time when we put up a Shared Baby Fleming section.

Yep, that’s totally fine, I was just checking compatibility because I’ll be fiddling around on the shared network tonight and tomorrow. Cheers!

13 Likes

I have internal docs, we will look to make them public soon. We have started ( thanks @frabrunelle ) a safe network hackmd space and a maidsafe one. The maidsafe one is internal and where we get crazy , intention is to work internal and switch to SAFE asap with most of these docs.
[EDIT Be aware I am typo manic in internal docs, you think I am bad here, well man :smiley: ]

19 Likes

Not quite as seismic as separating out the SNAPP from the browser, but sounds like one of those decisions that seems obvious only afterwards, but then makes a huge difference to simplifying everything.

As someone constantly trying and failing to organise my computer space, and then getting confused, I really like the sound of this!

It’s all coming together!

14 Likes

I’ll be keeping an eye out for what transpires from this :slight_smile:

3 Likes

Thanks so much to the entire team for all of your hard work. The progress is accelerating. A new dawn is coming. :sunny: :sunny: :sunny:

12 Likes

@dirvine Les Dawson was great at striking all the wrong notes on the piano for laughs, it’s a bit like that but with a keyboard. :slightly_smiling_face:


Thanks for all the great work everyone @maidsafe. and the community
members.
15 Likes

Looking forward to testing wee Fleming on mobile. Great work as always!

9 Likes

Great update once again!! Thrilled to see the quick diagnostics and push to iterate on Baby Fleming. As always, it’s wonderful to see the focus on effective UXD.

More grease to your elbow, MaidSafe! Thank you to all the more technical community members who also pitched in to help test Baby Fleming :smile:

15 Likes

Especially @mav and @Southside who are close to being salaried test engineers at this point! All hail our fearless alpha testers!

18 Likes

Thanks for the hard work and great update Maidsafe devs…

:clap: much respect to the brave CLI super ants helping with testing :vulcan_salute:

It’s fun to read the feedback of the testers (even if I don’t completely understand it) thanks for helping :baby:

15 Likes

Hey it was fun to do the testing (even if I didnt completely understand quite large bits of it)
but I know more than I did this time last week so it wasnt totally wasted time :slight_smile:

17 Likes

When I’m testing, and all the parts tally and roll like well greased cogs, and at last it’s working; and then I put a image file, and its gone, I have to wonder what just happened. I sit back and realize that what just happened has never happened before. Some magical Algorithm has dissolved that image file into a cyber ether of which dreams are made of, cheers!

11 Likes

Great update maidsafe team! Have not posted for a while, but read every update. Great to see the passionate community feedback and help.

18 Likes