Safe Network Dev Update - January 28, 2021

Summary

Here are some of the main things to highlight since the last dev update:

  • Today is Data Privacy Day (also known as Data Protection Day in some parts). We encourage everyone in our community, as well as everyone associated with other projects with similar goals, to push harder than ever to reach those shared goals. Every day we all move a step closer. :raised_hands:
  • The rewards PR is now going through peer review and testing - watching coins being earned is very close!
  • We finally nailed down and fixed the connections hanging bug that had been evading us for a few weeks.
  • Debugging the connections hanging issue has highlighted some areas for improvement to make it a much simpler process if anything similar rises again and so work is underway to update qp2p and simplify its API.
  • With this PR now merged, the CLI is now capable of checking the balance of its own SafeKey.
  • New versions of sn_authd (v0.0.15), CLI (v0.17.2) and sn_node (v0.25.39) have been published, with the CLI and authd now built using MUSL for Linux.

No Public Testnet Just Yet…

We were really on the fence today as we considered whether to continue fixing messaging, rewards and make some qp2p enhancements (all outlined in more detail further below), or put up a public testnet with everything we had so far. We decided that the right decision was to continue with the fixing and enhancements that we are currently in the middle of, and so holding off a public testnet for what we think will be a few more days. We decided there would be too many distractions for the team in setting up & supporting the testnet, plus the releases “could” have been less stable and harder to debug than we would like if the messaging & qp2p fixes and enhancements were missing. Delaying it also gives us a chance to finish and include the rewards flow work detailed below (earning coins :wink: ) in any public testnet.

We are working on a fix/improve path here and it’s working well with the team on a roll on a one track mindset, so putting the release back for now and taking a few more days to tidy some more parts feels right. There are some great commits and community participation and it all helps, there is a definite drive in the right direction.

Some additional general news this week - we have engaged an intern from the Indian Institutes of Technology who has a focus on cryptography. The intention is that he will start next week and immediately start auditing, poking and generally checking security in as much of the codebase as possible. We will increase that effort substantially over the next short while as we get into full launch mode.

Safe Client, Nodes and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan

Rewards

The work with rewards has continued with a simplification of section startup in sn_node, that now requires the minimum number of Elders to exist in the section, before instantiating the distributed actor - aka section actor - responsible for the section funds.

With that in place, we have now reached initiation of reward payouts. The draft PR has been converted to a proper PR and is being tested from a client P.O.V. before merging this work into master once tests are passing.
Following up will be the completion of the reward payout, at which point the farming feature is more or less ready to be included in a testnet.

Community PRs

We’ve had a couple of PRs merged this week, @mav has sent one which adds some missing client functionality, and another improving efficiency of the sequence.in_range call.

Connection hangs

We’ve continued debugging our connection hangs this last week, and have finally found the root cause there. Qp2p was erroneously dropping connections which had returned an error. In this case the error was valid, and didn’t warrant dropping the connection from the pool. With this fix now merged we’re no longer seeing long timeouts due to lost connections, which feels great :relieved: . The rumour is that @joshuef had a full head of hair before this bug arose!

Master node/client are much more stable now. We’re digging in to some more occasional bugs arising as part of storing blobs, where it looks like we’re failing to store the blob and/or metadata related to the blob.

Simpler qp2p API

The recent issue with the connection hangs in qp2p revealed to us that the user of the API shouldered a lot of responsibility such as maintaining connections and holding on to listeners and streams on which messages are inbound. This made it relatively complex to debug since connection handling was part of the routing and client crates which are feature-packed in their own way. By moving the maintenance of the networking components to the qp2p crate we can isolate and test multiple scenarios and ensure that all connection and message handling is done correctly. This ultimately simplifies the qp2p API as well so it’s a win-win. We have started this refactor in qp2p as a proof of concept to provide a simple async API that routing and clients can use for p2p communications.

API and CLI

The sn_api repo has seen a lot of activity from community members this week, which is excellent news, especially because these applications are all user facing and therefore the more input we obtain from real end users the better we can improve them, even better if the input is actual code!

There is a new feature being worked on by @latch to introduce a reset command in the CLI, which can allow us to clean up all data and config files that the CLI, sn_authd and sn_node applications store locally, keeping the actual application binaries untouched. Anyone interested in helping please join the discussions and review on: WIP: feat(cli): implements 'safe reset' subcommand - #6 by bochaco - Core Libraries - Safe Dev Forum.

Some improvements submitted by @mav to our API have been also merged, there were some typos in error messages that have been fixed, as well as adding validations for NRS names to reject slash characters.

Some example qjsonrpc applications are being developed by @scorch (feat: qjsonrpc Async Server Example by Scorch-Dev · Pull Request #690 · maidsafe/sn_api · GitHub) which will help users understand how the qjsonrpc API can be used and how applications needing JSON-RPC over QUIC protocol can be created. Some background and context information is very nicely detailed in this post for anyone interested in understanding more about this effort.

With this PR now merged, the CLI is now also capable of checking the balance of its own SafeKey, i.e. the balance that was transferred to it when authorised with authd, which is the one the CLI uses to pay for the costs of all operation made by/with it. Thus by invoking $ safe keys balance without providing any secret key, it now defaults to check the current balance of the CLI’s keypair/SafeKey.

New versions of sn_authd (v0.0.15), CLI (v0.17.2) and sn_node (v0.25.39) have been published, for those who have been playing with them locally. Since the CLI and authd binaries are now built using MUSL for Linux platforms, if you are on Linux please make sure you install them rather than trying to update them using the CLI, you can find full instructions to install them in the CLI User Guide.

BRB - Byzantine Reliable Broadcast

This week we did some polishing of validation errors returned by BRB DataTypes and also added thorough doc comments to all BRB crates. These changes should make it easier for anyone that wishes to use the libraries.

There was quite a bit of design discussion around integrating BLS Distributed Key Generation (DKG) into BRB. The hope was that DKG could sort of piggy-back on BRB dynamic membership (join/leave) operations. However ultimately we determined that this does not work as DKG requires 100% participation and must take place as a separate process after the BRB voting group is established.

Routing

Project Plan

This week was mostly quiet in terms of actual code changes in routing because one part of the team was busy discussing the BRB/DKG algorithms and the other one was helping to track down the issues with client connection hangs. We also put the error handling improvement work on hold as we decided there are more pressing issues to tackle first.

We did however merge a small PR that improves the event handling. We had separate events for when the section elders change and for when the current node is promoted or demoted. Turns out these things happen at the same time and handling the events separately was causing some confusion in the upper layers about in what order they should be handled. We resolved it by merging these events into a single event, thus eliminating the confusion.

We also started work to unify bootstrap handling across nodes and clients. Turns out the logic to bootstrap them is exactly the same, but it was implemented in two slightly different ways in two different places. This work will remove this duplication resulting in simpler code. We expect to finish it early next week.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

85 Likes

First! :sunglasses:
Because I’m worth it…

28 Likes

Great news. Have to admit I was a bit worried by this one.

Hope his beard is still intact!

29 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

30 Likes

Third, technically! :smile:

Keeping my fingers crossed :crossed_fingers:t5: :grinning_face_with_smiling_eyes:

This is amazing, and one of the best “side effects” of the wait. Seeing technical community members get involved is fabulous! Thank you, @Scorch, @mav, & @latch :smile:

And this makes it feel all the more real!! Here’s to smooth sailing

:sotros25:

Here’s the Twitter update, so you can like, retweet, and/or comment!

28 Likes

Ooooh shiny things

Seriously though excellent update - thank you to all involved and full marks for holding off (just a few days please) on the test net. I agree with the reasons for not getting this out right now even if it was technically possible. We must all recognise the constraints the team is working under and their ability to support the tsunami of questions which will inevitably arise has to be considered.

I think I’m going to have to dust off my agogometer for next week - I think it will be a biggie. - wonder if this will be reflected on Bittrex etc?

Thanks again to you all

21 Likes

Sounds very near!

I am not expecting this in the nearest testnet, but it being there would be awesome!

Full steem ahead! I feel hyped again - I gave a notice to some old friends to focus. Good job!

20 Likes

It sounds very close now. Can’t wait! :sweat_smile:

22 Likes

Testnet next week, that’s for sure !

31 Likes

Can never 100% guarantee as bugs like the connections hanging one could show up in the coming days in existing code or the new code we’re adding, and once we commit to doing something, i.e. merge work to master, it gets complicated and not worthwhile stripping out just to get “something” out, but it’s certainly our aim :smiley:

30 Likes

Rip joshuef’s hair… :grinning_face_with_smiling_eyes: and be at SAFE

11 Likes

Ask them which languages they can translate the weekly updates into and make that part of their duties :slight_smile:
Best of luck to the guy/gal and I hope their time with Maidsafe is mutually rewarding.

Closer to home, are there any opportunities for some of Prof Bill’s people from Napier?

9 Likes

… a few days … waiting for quality …
Haha this one’s easy. No worries team, we’ve been here for years. Git 'er done right, and thank you for your unceasing dedication.

16 Likes

There would be for sure.

12 Likes

In a world short on good news, that is a nice summary to be reading.
The best news is hearing about bugs fixed.

Always worth putting forward the best possible stable - hunt the hard bugs is more fun and having feedback as rewards, will be very useful.

:+1:

17 Likes

Hahaha, sorta reminds me of how I learned to cook! Got tired of waiting for food, so I figured it would go faster if I just went back into the kitchen and helped :laughing:

But yea, community engagement seems to be accelerating a bit on the technical side which is exciting!

28 Likes

Great update. The reaction to the wallstreetbets/GME saga suggests the Safe network is needed now more than ever.
Thanks for the hard work. Am quite excited about the future.

14 Likes

Love From india … Always…

India have best :+1::100: programmers and have medicine hub :panda_face:

15 Likes

Oh wow! Does rewards mean farming? I sometimes find this new terminology confusing. If it is farming, then except for stability/performance this testnet is equal to Fleming right?

2 Likes