SAFE Network Dev Update - January 16, 2020

Summary

Here are some of the main things to highlight this week:

  • With the next release of safe-cli, you’ll be able to install authd by simply invoking $ safe auth install, and you’ll also be able to update authd with $ safe auth update.
  • Various suggestions from the community during the past week have been included in the most recent update of the Data Types Refinement RFC.
  • We closed the BLS board as well as the Secure Message Delivery board that was only missing its BLS dependency, and updated the roadmap :tada:

Vaults Phase 2

Project plan

We announced last week that we were able to spawn a number of Vaults that connect to each other and form a section with the help of the Routing API. We ran the Client tests in SAFE Client Libs against this section and the initial results were promising. This meant that the integration between the different components such as Vault, Routing, PARSEC and quic-p2p was working as expected. However, there were some failures as well. Some of the tests were failing with RequestTimeout since the section failed to respond to some of the requests. We’ve been looking into this and with some debugging, we have more information. The vaults were run on DigitalOcean droplets and while processing a number of concurrent requests the Vault binary is sometimes killed by the OOM Killer due to unusual memory usage. We took a closer look into this and with the help of a few memory profiling tools we’ve identified that the quic-p2p thread seems to be using a lot of memory. It might be because it’s holding on to the pending messages in the message queue before they are successfully sent via the network. This is just one of our strong leads that we will continue looking into along with more debugging and performance analysis. Stay tuned for more updates!

SAFE API

Project plan

We finished with adding support for downloading new releases from Amazon S3 which was needed for safe-authd assets. safe-authd is now able to update itself by checking its current version against the latest version available in S3. This is ready to be merged and released, we are just waiting for a new release of the dependency we use (self_update) for such updates, which should be available soon.

Being able to download safe-authd releases from S3 allowed us to add a couple of commands to the safe-cli for installing and updating safe-authd, which simplifies the process a lot for users. With the next release of safe-cli, you’ll be able to install authd by simply invoking $ safe auth install, and you’ll also be able to update authd with $ safe auth update.

In the last couple of days, we resumed a task we started at the end of last year which was about trying to expose the APIs in safe-api as async functions, plus trying to migrate the authd implementation to use async/await internally rather than using futures. We made some progress with having the APIs exposed as async functions (even though internally they are not yet fully async), and we’ll try to understand if the safe-nodejs bindings can be modified to now return promises when calling these async APIs. At this point, we see this as a parallel task we want to advance but we’re not treating it with really high priority, we’re just trying to learn the upcoming challenges for such a migration.

Labelled Data, Indexing and Token Authorisation

RFC, Project plan

This last week has seen some refactoring around the initial rough token implementation. Moving some SAFE Client Libs functionality, along with the token, to safe-nd, where it can be properly used across our libraries, opening the door for token verification by vaults.

We’ve also increased the test coverage of token creation and simple verification, getting us something much more useful.

The plan going forward will be to finalise basic token verification in the vault ‘client handling’ code, so we have a good view of the whole process across our libraries, before then beginning to make the token actually useful, adding and verifying tokens such that we can start to deprecate the existing authentication setup.

Data Types Refinement

RFC, Project plan

Various suggestions from the community during the past week have been included in the most recent update of the RFC. Thanks to @david-beinn, @tfa, @JPL, @dask, and @jlpell for the included contributions, as well as to many others for participating in the discussions.

SAFE Network App

This week the SAFE Network App Feature Tracker pinged blue as we worked on the UX designs for individual file and app data permissions.

It’s one of those areas that starts off feeling quite simple—“Yeah I’ll rattle this in a day or two”—and then the complexity rapidly unfolds—“Oh my” :open_mouth:—and after slogging through the iterations it wraps round to simple again—“It took you how long?” :sweat_smile:.

It aims to provide nicely for what we need for the MVE, yet building out patterns that will be useful for more nuanced future use-cases, and some of the controls that will be helpful for the recipes that Data Labels can accommodate.

The efficacy of the approach can only really be proven through testing and understanding the possibilities that the system allows—which are expansive—so we’ll take the wraps off next week to get your thoughts, start simple, and build from there.

SAFE App C#

Project plan

This week we made some improvements in our CI setup. These changes enable the CI to automatically create a draft GitHub release from the build pipeline when a version change PR is raised. Using the same setup, we released a new RC (Release Candidate) of the MaidSafe.SafeApp NuGet package.

We added a new API (AppConfigDirPathAsync) that developers can use to get the path from where the different configuration files are read. You can start using this API by updating your apps to the latest RC package. See the changelog file for other improvements.

We had different tools and checks in place to enforce the coding standards in the safe_app_csharp code which seemed redundant and were making code more complicated to write and understand. So, we refactored the code and made it more simple and clean.

To test real-world scenarios with the C# API and provide more confidence when using the package to develop different types of mobile and desktop apps, we are working on a setup to run the tests against a locally running vault. Some of the required work is already done in this PR. The PR contains the changes required to run the desktop tests against a local vault. Next, we will figure out the best way to do the same for mobile platforms.

Safecoin Farming

RFC

This week @danda has written a small simulation of the farming reward flow in a mock multi-section, multi-vault network starting with a single client’s PUT request. The simulation is written in an interpreted language as a quick/dirty tool for rapid prototyping and learning with no plans for public release. Observations and lessons learned from this are going into notes/discussion and ultimately will go into design docs. He has also begun diagraming the rewards flow and is starting a new RFC document specific to farming rewards that aims to provide more specifics and clarity.

@oetyng has also found time for some prototyping and discussion with other team members particularly around PARSEC integration details, and is advancing the rewards flow document, which will be incorporated into the new RFC.

Node Ageing

Project plan

We progressed further implementing Elder promotion/demotion, addressing 2 more items in the list, validating trust when Adults receive updates from elders, and cleaning up unnecessary events.

We also cleaned up more of the Routing library, improved documentation and clarified names to ease its use.

BLS

Project plan

As part of addressing unnecessary serialization when Routing messages are sent over the network, we addressed the last remaining issue for this project regarding incorrect message filtering.

We have now closed this project board. Any performance issues or clean up will be addressed as part of other projects if and when needed.

With that, we also closed the Secure Message Delivery board that was only missing its BLS dependency, and updated the roadmap :tada:

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

75 Likes

First… Second whatever

Ok, guys I never expected to be first, but as a matter of fact I just opened the page and posted this. After years of silently stalking I eventually managed to be first. YAY! (now I read)

36 Likes

Second…:slight_smile:but thanks for sharing!

“This meant that the integration between the different components such as Vault, Routing, PARSEC and quic-p2p was working as expected.”

Oooooooh, shit! It is working. Now, the old man’s dream is not just dreams.

27 Likes

Third…treasure chest! Now to read; the early update has me thinking there are plenty of goodies inside :smile:

Great to hear everything’s stitching together!

The puzzle pieces are falling in place. Every day it becomes clearer that Fleming will be robust enough to showcase differential value, and that will continue to distinguish SAFE. We’re almost there…more grease to Maidsafe’s elbow :smile:

22 Likes

This is awesome news!

Thanks @maidsafe! Y’all the best! :wink:

25 Likes

It’s exciting to see al these components come together. The last updates really start to feel like little sparks and with every spark, the Network feels more alive. Looking forward to farming and seeing it in action!

15 Likes

Woo, who thinks we will we see the safenetwork test net…

  • Next Week
  • 2 Weeks
  • 1 Month
  • 3 Months

0 voters

6 Likes

Is it intended for that reliance to be replaced with your own CDN? I believe it would be preferable to some users to not allow network connections to Big Tech ASNs.

Thanks so much for all of the hard work of the Maidsafe team! The progress is accelerating!

8 Likes

I’m not sure about our own CDN, although I heard some people are creating a decentralised network where data can be published for perpetuity and for everyone :wink: :smiley:

18 Likes

@bochaco

I’m referring to user bootstrapping from clearnet.

2 Likes

Oh, I know that I remember they are named “impossible network” by Daniel Larimer. :slight_smile: :wink:

14 Likes

Not sure I follow, but what we have now in next release is to be able to download the authd binary from S3 (that’s where we currently publish authd releases) and install it in your system/PC, all in clearnet. Downloading it from SAFE should be possible too using its protocol.

12 Likes

@bochaco

I recommend creating https://cdn.maidsafe.net rather than requiring network connections to Amazon S3 or any other Big Tech network service.

12 Likes

me guessing 12 months will be considered fud, but 1 week is kind of delusional.

4 Likes

They just implemented amazon S3 standard and are temporarilly using it now during development of the network. Once the network is ready and working, they will of course switch to the safenet. It is very important that they imlemented amazon S3 standard. All storage frameworks support it, so it will be super easy to use safenetwork as storage backend for existing software.

15 Likes

Until OSs integrate support for SAFE, there would still need to be the first binary delivered to users and that would come via the legacy internet.

4 Likes

If government can change files on amazon S3, than it can do it on any clearnet domain. They can simply confiscate the domain, or do DNS attack. If you are in controll of DNS servers you can do whatever you want. You will never be able to trust the clearnet if your opponent is almighty government. Even creator of such private CDN can’t be sure that users are downloading his version.

6 Likes

@Antifragile

The value would be to reduce the clearnet spying footprint of SAFE users and prospective users.

1 Like

I get you, however we will also encourage users to share with each other, give the binaries etc. Just need to make sure security is easy (web based sig to check against etc.) The binary we would want to share should be a very simple SAFE down-loader. so all folks would know is you might have downloaded something from SAFE but not what/when. Not a great answer, but you see the route.
There will be a clearnet bootstrap for many though.

18 Likes