SAFE Network Dev Update - February 13, 2020

Summary

Here are some of the main things to highlight this week:

Staff

Jean-Philippe will be taking some downtime for the next 2 months or so. It’s been a busy period recently with many fast-moving parts and JP feels he has to take a bit better care of himself. That will mean some rest and getting away from the computer for a while. So we wish him well and hope he feels refreshed very soon.

Vaults Phase 2

Project plan

After making some observations from internal testing, we took a swim in the different moving parts (Routing, quic-p2p, Vaults & SAFE Client Libs) to make some adjustments, attempting to resolve some of the issues we’ve been facing. Quic-p2p now has separate channels for messages from Clients & Nodes in the network. It has also removed the use of certificates which simplifies our setup for testing and the network itself. We also identified that the routing nodes were sending gossip messages at regular timed intervals which ultimately meant that this interval would determine the speed of the network itself. We tweaked around with this duration between gossips and observed that with lower gossip intervals the network was faster. So we decided to set it to zero :smile:. Yes, that’s right. In other words, we are removing the gossip interval completely and moving towards an event-based approach so that the gossip will be triggered by client and network events. This improvement is in progress and we will be continuing with such end-to-end testing to identify other areas of improvement as well.

As mentioned in previous updates, we’ve started using the CLI and safe-api tests suite against a local single-section network, and we are very happy to see functional tests passing! However, as said above, we still need to make some improvements to the network response time before considering it ready.

As vaults development progresses towards having a first Phase-2a testnet, this week we are starting with the development of a basic network launch tool that will allow users to launch a local single-section network. We are still thinking of the implementation details, which will soon be reflected in the tool’s project board.

There are also some ideas around making the CLI capable of using this tool, such as running and using a local Phase-2a network can be all done from the CLI. But this is just a rough plan for the moment, we’ll see how things evolve in the next few days on all fronts of development. As always, we aim at making things as simple as possible from a UX perspective.

SAFE API

Project plan

Hot on the heels of some SAFE Network App updates, which involve using the app itself to manage CLI and AuthD installs, we’ve once again come up against libso issues on Linux, which have been blocking the CI servers there. As a consequence, we’ve taken a look into what options we have to deal with this, which has resulted in a PR to the self_update crate (the crate which is actually bringing the libso/openssl dependencies) to use a native rust implementation. At the moment we’re awaiting some feedback on that PR, and once we have that (and some related quic-p2p changes), we’ll be able to update the libs and then use these for the SAFE Network App.

Upgrading quic-p2p in all crates, including safe_client_libs, would allow us to upgrade quinn in safe-authd and safe-api, and this is what we started working on as a preparation for our next release of safe-api and CLI, which is planned to be for the first Phase-2a testnet.

Since the focus now is on the Phase-2a testnet, we also started working on a new set of CLI subcommands which would allow users to install the vault binary in their local system, e.g. $ safe vault install. The plan is that the CLI could be used to download and launch the safe_vault to create a local single-section (Phase-2a) network. This command is planned to be a companion of the network launch tool we’ll be implementing in the next few days. There are no more details than this for the moment, but you’ll see some activity in these repos in the next few days.

Labelled Data, Indexing and Token Authorisation

RFC, Project Plan

We’ve finished integration of mock tests for tokens now, with the full suite passing nicely, which means we’ll be able to move on to the vault implementation of the same and have tokens working across the board. In the meantime @yogesh has continued with work on storing/retrieving the tokens to enable token updates and permission changes from the Authenticator, without having to update the app itself directly.

We’ve also posted a “new” Token RFC, which isn’t actually much new, but it is hopefully clearer on the scope of work involved in Tokens without all the extra concepts from the previous labels/indexing/authentication-changes-catchall-rfc. There are some small tweaks in there for those interested in such things.

SAFE Network App (desktop)

Project Plan

This week saw a bit of work here, with an initial attempt to work around our CLI/Authd dependency, but in the end we’ve decided to keep the CI of SAFE Network App using real libraries to ensure that everything will be behaving as expected. This gives us more confidence not only in the SAFE Network App itself, but will also highlight any issues in related network components too.

SAFE Browser / SAFE Authenticator (mobile)

Browser Project Plan, Authenticator Project Plan

For the last couple of months, we have been working on multiple independent components to bring some new features and experiences on mobile devices. After multiple development and testing milestones in different repos, this week we released new versions of both the mobile Authenticator and Browser apps :tada:.

The new releases include single vault network support for both apps, and pWeb capabilities on the mobile Browser - you can now scroll through the history of a safe site :exploding_head:

If you haven’t already started using the new apps, check the release post to learn more about all the new changes and how you can try them out on your mobile devices (Android & iOS). We are excited to see how both apps improve the SAFE user experience on mobile devices.

Thank you to everyone who has already given the apps a spin and provided us with invaluable feedback. Please keep the feedback, feature requests, and even bug reports coming :smile: so we can start planning the next iteration.

Safecoin / Farming

RFC

CRDT data types are typically used in a trusted environment, e.g. between data centres. This week we have been exploring the idea of adding signature aggregation (multisig) to CRDT set types such as orswot to provide cryptographic proof that a quorum of elders agrees on a given piece of data. Once a quorum is reached, it can theoretically provide the redundancy/availability needed by the bounded counter CRDT we looked at last week.

To this end, we implemented a small proof-of-concept in Rust that demonstrates 7 elders signing a piece of data, adding it to an orswot and then validating signatures of the other elders to ensure a quorum has been reached before the data can be used. This work is still in a very early/rough state and is ongoing, but results seem promising so far and could eventually have uses/implications in areas beyond Safecoin.

To be clear the aim is highly concurrent mutations to the network under very high loads, and for clients that will mean offline working. In terms of mass partitioning then we also get great benefit if we can merge both partitions back to one.

There are many long term goals here, but short term we feel the need for highly concurrent mutations will be required for many apps (i.e. social feeds, likes, comments, etc.)

Routing and quic-p2p

Routing Project Plan

This week we completed the work of removing the use of quic certificates, simplifying our code further.

We also completed work to allow vaults to process client messages directly without needing routing to act as a go-between.

We are improving the way we detect lost nodes. Currently we declare a node as lost the moment we lose connection with them, but that might be premature because the node might just be restarting or experiencing temporary connectivity issues. The work to change this is currently under review.

As mentioned in the Vault Phase 2 section above, during internal discussions we realised we were accidentally hurting the network performance by not sending the Parsec gossip messages as fast as we can. These messages are essential for the Parsec consensus algorithm and the faster they are exchanged, the faster the algorithm reaches consensus, and the better the whole network performs. Work to remove this accidental pessimisation is underway.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here.

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

77 Likes

Thanks for the update Maidsafe devs,

@Jean-Philippe have a great time-out.

:sweat_smile: I backtrack on my SAFE browser comment a while back, I was basically drooling yesterday using it on mobile. Set up was pretty easy.

Love that Maidsafe is still not afraid to just experiment/tweak/remove features to improve the Network.

:exploding_head:Yeah double that :exploding_head:

Thanks again for all your hard work :stuck_out_tongue:

23 Likes

Is this holidays @Jean-Philippe is taking or something else Is going on? Im a bit confused.

2 Likes

It means holidays yes.

22 Likes

Most probably Coronavirus … 1% of the humanity is quarantined… If I were you I would not waste my time wondering what other people were doing and I would look for masks and stock up on food…

11 Likes

Great update! Thanks so much for all your hard work everyone - enjoy your break @Jean-Philippe!

It was great to use the mobile browser again, especially as it feels fast and sharp. Test net phase 2a sounds close and it will be fantastic to give it a whirl and prove that the tech is coming together nicely!

19 Likes

Anyone dare to summarise whats left for Fleming, that is not clear for me any some others I think. I understand we will in a ,week or so have testnet of vault 2a phase, will there also be testnets of vault 2b, 3 and 4 in the future? How much coding is there between those phases or is it mostly testing. Phase 1 and 2a took more than 6 months so this remaining phases might take more right?.Is this something that anyone can guess? When i read some updates I feel there is still alot work for Fleming (many months), but some updates indicated we are much closer. Confusing stuff or Is it me?

4 Likes

Divide by zero… sounds like a great idea :+1:

12 Likes

don’t panic! :smiley: :smiley:

14 Likes

Burnout is so common amongst developers. We pour our hearts in to our work, every minor bug we come up against, every time an issue goes over a little over time, over time they build up in our minds to be much larger than they really are. Sometimes, with so much project ahead of us we can feel like we haven’t really accomplished anything at all.

It must be so hard working at such an open company, where all of your work is out in the open for critique and you’re pressed at an individual level for constant updates. The mood of the forum really lives or dies based not on whether there is progress, but on whether there is the appearance of progress. That level of pressure must be painful to shoulder at times, and I understand why someone might need to take time out to blow off some steam.

Especially for Jean Phillipe this must be difficult, being in a new country and away from family. At times when it’s all feeling too much it’s important to take a step back with the people you love and immerse yourself in recovering from the burnout. It sounds silly but at the times when I felt the lowest, something simple like walking my dog was cathartic and really helped to ground me.

39 Likes

Will other users be able to access these with a shared config?

9 Likes

I think if it has a public IP it should be possible, just like the MaidSafe hosted single-section network will be accessible. I’m hoping to be corrected by others if I’m wrong.

15 Likes

A very exciting update indeed, best wishes to @Jean-Philippe and well done to you and Maidsafe for choosing self care. Learning to do this is one of the most valuable skills in life, but increasingly difficult over time IMO as work and society have changed.

25 Likes

Great to see such a progress.

I actually thought that Vault 2a test network will enable to connect all Vaults, but there will be just one huge section :slight_smile:

@Jean-Philippe take the time-out thoroughly and responsibly!

7 Likes

Maybe this is just the start, I thought so too. But David mentioned they were looking at ways to push out tests as quickly as possible to us hungry monkeys.

9 Likes

This sounds interesting, I have been thinking of if nodes can rejoin after a minor lost connection due to updates, ISP infrastructure maintenance, power outage or other similar situations. Are sections capable of going into a Initiate/wait mode if a major power outage, in for example country, would happen? I have been thinking if there could be a similar initiate/wait mode for sections when splits/merges happens that would go on until a stable section structure have been reached. I know the merge function seems to have been dropped but just some thoughts.

5 Likes

It’s not relevant any more. Major partitions though are something we are aiming to handle. (section merges would require losing over 90% of nodes and all data, so no section merges ever needed).

11 Likes

Great update, great work. The updates described for Vaults Phase 2 are impressive.
Thank you team MaidSafe!

No, not possible to give a prediction that has any validity.

It’s not confusing. This was a great update. Many months is not a long time… " Time flies like an arrow; fruit flies like a banana"

10 Likes

I agree great update indeed, apart from that routing development will slow down due to one less active developer for some time.
@maidsafe what features will the testnet 2a provide to the user?

1 Like

It’s stage 1 for stage 2. If they work then light the cigars