SAFE Network Dev Update - November 14, 2019

Summary

Here are some of the main things to highlight this week:

  • We are just about ready to release v0.0.4 of the SAFE Network App. The major change is the addition of app updates, which means we can update apps such as the SAFE Browser through the SAFE Network App itself.
    EDIT: SAFE Network App v0.0.4 is now available!
  • We are also almost ready to release a small browser update which fixes a couple more pWeb resolution issues, some SAFE Network App interaction changes and enables background updates to be triggered from the SAFE Network App itself.
    EDIT: The browser update is now available too.
  • The first RC version of the MaidSafe.SafeApp NuGet package is now available :tada:

Staff

We were delighted this week to be able to secure Qi Ma’s role as an Engineer in the team. Qi has been around for a great deal of the journey and he is delighted to be around all the way to launch. As we are moving forward we do intend that there will be more additions to the team when it is feasible and prudent. Qi will be continuing in the backend and expect to see a lot more of him as safecoin farming comes into play.

Vaults Phase 2

Project plan

We are continuing to make some good progress with the multi-vault tests. This week, we have been designing and implementing a new RPC (Remote Procedure Call) mechanism in Vaults to support connection information requests. This is one of the important steps to allow clients to bootstrap to the network without prior knowledge of IP addresses and public keys of all nodes within their section. It should be sufficient to know the connection info of at least one node in the network, which can be used to discover other nodes managing a particular client. Having this mechanism in place would also be very helpful for connecting to the multi-section network (Phase 2b). We have added the prerequisite API to Routing.

Apart from that, we are continuing to improve the Vault testing framework. We discussed and designed a way to improve it by removing some duplicate code, and, more importantly, by utilising mock PARSEC for testing. This would allow us to automate Vault tests without relying on the ‘real’ network infrastructure: in certain contexts it’s more valuable to test Vaults & Routing integration in isolation, taking the network side effects out of the equation. With that, we should have 3 testing modes for different contexts and scenarios: real network (with real clients), mock PARSEC (Vault with real Routing, but with a simulated network), and mock Routing (Vaults with simulated Routing).

SAFE API

Project plan

Given last week’s new release of Rust (1.39), we already started migrating safe-api to have those APIs which are asynchronous (e.g. those which need to communicate with the network) be exposed as async functions. We were able to already do this in the APIs and also make adjustments to safe-cli to await when calling these async functions. We still have to make the corresponding adjustments in the safe-ffi layer, as well as in the implementation of safe-api where we call the safe_client_libs APIs. We see some minor challenges there but overall this process has been going smoothly so far. Once we have this complete, we are aiming at being able to have the corresponding Node.js bindings APIs return Promises.

A couple of additional features were added to the safe-cli authenticator commands. When using the CLI to either create an account, or log in using an existing one, the account credentials are being prompted to the user, thus we added the same functionality that the safe_auth CLI used to have making it possible to provide the account creation/login details via environment variables (SAFE_AUTH_PASSPHRASE and SAFE_AUTH_PASSWORD) or through a config file with --config <filepath> argument. This is still in progress as we are trying to also adapt our safe-cli integration tests in CI to run against the safe-authd all in the same PR.

In addition to that, the auth login subcommand now accepts a --self-auth flag which allows the user to automatically self-authorise the CLI application on the account that is being logged in with. We believed this was necessary in this particular case, now that we have the SAFE Authenticator daemon, if the user is using only the CLI and no other Authenticator UI, the user would otherwise need to open a second console to authorise the CLI from, which makes it not only confusing but impractical.

Finally, we started adding more documentation to the safe-api repo, trying to detail the main purpose of each of the crates it contains. We started already by creating a top-level README which provides a high-level overview and acts as an index to the other crates. This is still in progress but you can already look at the progress we’ve made with the top-level README. We added a couple of diagrams which we hope will be helpful to understand how each of these components fit together.

SAFE Client Libs

Project plan

Last week we announced the introduction of named containers. We’ve been chipping away at this and we now have the initial implementation in place as part of this pull request. After some review, we will be passing this on to the teams working on the language API and CLI to see if it satisfies the requirements. We’ve also wrapped up the deprecation of the maidsafe_utilities dependency.

We received some useful thoughts about the refactoring of the account containers. Although we are aiming for a basic feature set for the initial release, thoughts and ideas are well appreciated so don’t hesitate to shoot them across on GitHub or on the forums.

SAFE Network App (desktop)

Project Plan

This week we’re all but ready to release an update to the SAFE Network App which enables update management for its applications. So you can easily see what applications are in need of an update, and opt to install those updates ahead of time via the SAFE Network App.

It’s a small feature, but it’s involved a lot of testing and tweaking to get this working across platforms. We’re waiting on one last piece of the deploy puzzle (more notarization fun) before we can hit the big ‘DEPLOY’ button on @StephenC’s desk. But once we do, we’ll post an announcement here on the forum, or the SAFE Network App should offer you an update notification if you already have the application running.

Once you do that, you should hopefully see an update is available for the SAFE Browser too (if you have v0.15.2 installed via the SAFE Network App prior) as we intend to release an updated version of that at the same time (see SAFE Browser section below!).

With this just about done, we’ll soon be now fully focussing on the authentication functionality. With working initial versions on the go, we’re improving the aesthetics and our testing and CI setup.

EDIT: SAFE Network App v0.0.4 is now available!
Note that if you are using v0.15.2 of the SAFE Browser you will NOT be able to update via the SAFE Network App. You’ll either need to uninstall / install via the SAFE Network App, or use the SAFE Browser’s baked in updater. Attempting to update via the SAFE Network App will fail as v0.15.2 isn’t capable of background updates.

SAFE Browser (desktop)

Project plan

A small browser update is coming, just after our deploy pipeline is fixed (see above). This update fixes a couple more pWeb resolution issues, some SAFE Network App interaction changes and enables background updates to be triggered from the SAFE Network App itself. If you already have this installed, on release you would be able to try updating from the SAFE Network App, to get a taste of how application management could work in the future!

Otherwise, we’ve been further looking to streamline deployments, builds, and how we can more frequently release the browser, and other desktop applications to folks. We’re settling on using release channels, so this should be making its way into the browser in the near future. We’ll have an alpha channel for in-house testing (or for those who want to live on the cutting/probably-broken edge), a beta channel, where we can release more stable code for those out there who want early access and don’t mind filing issues / helping debug, and then a latest (stable) channel. This should make updates smoother and more frequent for the community, while also heavily streamlining how we go about testing releases/updates and the applications themselves. We hope this upcoming move will also encourage even more community involvement.

EDIT: SAFE Browser v0.15.3 is now available!
Note that if you are using v0.15.2 of the SAFE Browser you will NOT be able to update via the SAFE Network App. You’ll either need to uninstall / install via the SAFE Network App, or use the SAFE Browser’s baked in updater. Attempting to update via the SAFE Network App will fail as v0.15.2 isn’t capable of background updates.

SAFE App C#

Project plan

We are really excited to announce that the first RC version of the MaidSafe.SafeApp NuGet package is now available :tada:. The package exposes all the new APIs available in safe-api and you can use it to perform any operations on the SAFE Network possible using safe-cli.

It’ll be wonderful to see what you develop using this updated package. If you find any issue please raise an issue on GitHub and if you need any help with the APIs/app development using C#, please start a thread on the Dev Forum and ping @ravinderjangra.

We have also updated the safe_app_csharp README file to reflect the updated supported desktop and mobile platform versions.

In parallel, we are trying to expand our build and test setup to run the tests against a local running vault. This will help us to validate the proper working of production libraries on all the platforms.

Node Ageing

Project plan

The week we had a very large and complex PR merged that deals with triggering relocation based on churn events as well as fixing flaws in the previous relocation logic. This is a significant step towards implementing node ageing and went through intense soak testing to iron out any subtle issues. Investigating soak test failures for some of the more complex cases can be a very time consuming and difficult task so everyone involved in that deserves a beverage of choice this weekend!

Another crucial step in implementing node ageing is elder promotion and demotion. We’ve made some good progress and are in the testing phase. As usual when working with the routing crate and the very async nature of the problems it solves, once things are implemented and you start running real system tests you discover all sorts of corner cases that need handling. So this week saw additional improvements to the testing framework to help us there.

We also merged the next step in the Connection handling simplification with one PR related to tracking connection info for elders merged. There are two more PRs in the pipeline to complete this task, with one of them in the soak test stage and almost ready to be merged as well.

To facilitate integrating with safe_vault, the API was extended to give access to section elders.

Work on PARSEC pruning has been mostly completed and is currently in the merging queue. Handling disconnects via tracking unresponsiveness is now being continued and is expected to resolve the bulk polling issue due to the new testing pattern.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

73 Likes

First 20 charachters!!!

24 Likes

Another great update. Congratulations to Qi Ma! Also to me for finally getting the silver trophy for 2nd place commenter!

22 Likes

Great update. Welcome back @qi_ma, happy to have you back :slight_smile:

38 Likes

Thanks for sharing it

12 Likes

Nice update - can’t believe it’s Thursday already!!

:+1:

20 Likes

Thanks so much to the entire Maidsafe team for all of your hard work!

15 Likes

This really seems to be shaping up and I start to hope for Fleming before Christmas again :thinking::face_with_monocle:

Glad @qi_ma is back on board again ofc! Nice to see the wiggle room seems to be a bit larger after the maid lending announcement :ok_hand:

21 Likes

Absolutely, although the team will be pretty lean and use all kinds of tricks, like rotating team leads, task leads and more. The key we are looking for is flexibility, equality and every single person having the back of every other person in terms of the project. Nobody needs to be friends with everyone in the team, but we all have a common vision and determination to succeed.
We won’t be the best team or have most experts, but from here we will act as a single force and hopefully we all enjoy the challenge much more. Not behind the cosh as much or worrying about time time time, but instead every week being proud of what our teammates have done. It’s time IMO to show what motivation and a real team can achieve. I will be the casting vote/blamehound, but these Engineers with Sharon backing them up in admin/finance are gonna be running their own show.

Now as a community what I would love is to get to a fully integrated system of paying other Engineers for their help in code etc. Then we will expand to marketing/pr/support etc. If we can work out a way as a community to all share in the rewards for our efforts as we move on then decentralising maidsafe will be easy. It should be natural.
I do dream :wink: but that gives us hope I recon and with grit it also gives us options.

36 Likes

That looks very interesting.

And I hope that ongoing testing of new features will go smoothly.

10 Likes

Just curious how you pronounce the name Qi? Is it like “key?” In that case this has to be the perfect person to work on the coin side of things lol. In any case looking forward to seeing his work! Traders really want to see what the test-coin economy looks like. At least I do. I think it will make pricing MAID much less speculative when we start to see where all the dials are tuned.

4 Likes

More like Chee …

10 Likes

Allright I will be sure to protect my private Chees on the SAFE network! Always amusing to take a foreign name like that and see what English speaker algorithm hashes out of it:P

6 Likes

I’d say this looks like a pretty meaty update on the quiet, or not so quiet. Thanks everyone, can see all the strands coming together!

16 Likes

Well done to all involved, another solid update, genuine progress being made and excellent news about Qi Ma.
Now will you all keep the price of MAID lower for another couple of weeks until I get paid, please?

13 Likes

Looks like roadmap page has had a bit of re-modeling?
:+1::+1::+1::+1:

23 Likes

I am right there with you on this one.
Just 4 more tasks on the roadmap, and if you look at the detailed tasks in github a lot of boxes are being ticked off.

Fleming does massively de-risk the project. Who knows how the wider crypto community will react,…
I just want one more payday buyin before Fleming.

8 Likes

Thank you @maidsafe for another great weekly update, I always enjoy reading it even if I don’t understand everything.

Now, I have a question, would it be possible to unmask the vault IPs at large scale just spam joining the network with many clients?

Also I wanted to know if the people that participated in the loan agreement will get an confirmation email to make sure the tx arrived and everything is OK.

I don’t want to give work and delay things, but I was left wondering if my MAID arrived correctly or not. I can look into the block explorer but it would be great to know if it arrived to your wallet.

If it is silly just say it.

Again, thank you for the hard work.

9 Likes

thanks for the update Maidsafe devs,

:partying_face: @qi_ma developers developers developers developers devloprrs devloprrss… (Steve Balmer legend mode)

:stuck_out_tongue:

9 Likes

Yep, 4 biggies to go. Slowly but surely. Boom

4 Likes