Safe Network Dev Update - March 25, 2021

Summary

Here are some of the main things to highlight since the last dev update:

  • The call for volunteer committee members for the BambooGarden Fund has now closed! We have 8 respected and experienced committee members from the community in place.
  • With the BambooGarden Fund committee now confirmed, we will move on to setting up communication channels and being able to invite the first fund applications.
  • We’ve made multiple relatively small changes to the client code base over the last week which has resulted in us now seeing the majority of the client tests passing consistently in a multi-section network.
  • With the client stabilising, we’ve been able to re-enable reward payouts, and are now working through issues and optimisations.
  • The implementation of lazy messaging continues across the board.
  • More questions roll in to the AMA, and another answer too see @jimcollinson’s latest video here
  • We posted a feature summary for the upcoming testnet earlier in the week.
  • Keep a regular eye on the Like This Tweet thread on the forum for some excellent guidance on how to help promote the Safe Network, and surrounding components, with a simple button click! :bird:

BambooGarden Fund Update

A big thank you to each of the volunteers for the BambooGarden Fund committee. We’ve had 8 volunteers offer their time and so we feel this is a good point to close the committee, for now, and keep it at a manageable number.

We were initially intending to have 3 community committee members at any one time, but decided to change this to have all 8 of these volunteers on the committee. This change has several benefits, such as allowing us to make the most of the varied experience and expertise they offer. It’s also inevitable that there will be times when some committee members have other commitments, so this larger number means we can continue seamlessly regardless. Members will be able to abstain when they can’t make it, or even if they don’t think they’re qualified to judge an application, without having to be concerned that they will be leaving the committee short.

MaidSafe staff will still make up around 3 members of the committee. We’ve also added a condition that MaidSafe will have the right to a “veto” on any applications for funding, i.e. the right to reject proposals that we believe don’t meet the objectives, fundamentals and values of MaidSafe, or that don’t meet the original purpose and goals that this fund was created to achieve, and which MaidSafe has been asked to act as guarantors of. Note that a veto is only for rejecting applications, not for forcing them through when the wider committee has rejected them.

All this has been communicated out to the committee volunteers over the last couple of days, along with some other important decisions. For example, the committee will all be members of a separate discourse forum, where we will be able to discuss and vote on applications. MaidSafe have also proposed some immediate areas which we have identified as urgent in taking steps to meet the number one purpose of this fund, which is “to enhance the pace of deployment of the Safe Network”. These areas are:

  • Formal documentation - we believe the project would benefit hugely from having our algorithms formally documented. Perhaps a technical writer or the like who has experience writing formal algorithms and papers would be able to offer value here. Having these formal documents in place helps onboard developers who are getting involved, while also helping external auditors who we would at some point be asking to check our code for security flaws, bugs, etc.
  • Additional CRDT expertise to help push this to the next level.
  • Additional consensus expertise.
  • Additional networking expertise.

If you have ideas for fund applications in these areas then don’t send them in just yet, but you can start to think them through a bit more in preparation for the application process to open.

We are also proposing that, once the testnet is live, we ask for applications to generally increase our engineering team size so we can move at a quicker pace. The reason why we say after testnet here is that the onboarding of new engineers beforehand would be a distraction from the testnet work.

Over the next few days the committee volunteers will all be invited to join a separate discourse forum where we will formalise some of the committee guidelines together. We will then announce that we are ready to accept our first funding applications, with full details of the specific initial areas, such as those outlined above, that we want the first batch of applications to be restricted to. We will also provide full details on how to submit applications, and what we expect applications to consist of.

Safe Client, Nodes, Routing and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan
Safe Routing Project Plan

Client code enhancements

After getting the new section wallet split changes merged across the repos at the start of the week, we’ve been doing various enhancements to the client code. There has been some small refactors to simplify things, as well as updates to allow for more flexibility with Elder numbers (as they’re increasing with supermajority changes), but also to use this same supermajority calculation for response handling (i.e. do we get the same response to a query from X Elders and can therefore trust it). We have also added a few fixes for issues that prevented the client from bootstrapping to the right section in a multi-section network. With this in place we were able to see the majority of the client tests passing consistently in a multi-section network.

Rewards

After the above mentioned merges, we re-enabled reward payouts and have been inspecting the behaviour over several splits. A couple of small issues were found and addressed, and a couple of others are still being worked on where there’s excessive CPU and memory consumption after a few splits.

Even though the reward payouts do work now, it looks like we could simplify them even more, and so we will be trying to cut away a big piece of logic handling the payout at node relocation, and instead include the reward payouts at the section wallet split.

Lazy messaging

Alongside this, we’ve updated the lazy messaging implementations across the libraries, and have sn_node adapted to the updated messaging code. Updating sn_routing for these changes has been a big task as every routing message will now require a Header of its own that furnishes the destination’s details, which will be verified by the receiver for correctness and used to update the sender or receiver with the current Network status, if they are lagging behind. Core implementations are in place and we’re updating the last tests there. Once we have this in, we’ll be free to create the lazy messaging pattern in sn_node to enable easy handling of bringing nodes up to speed.

Routing neighbours

In routing we removed the neighbours concept. To recall, nodes need to keep information about other sections in the network so they know (among other things) how to send messages to them, and so on. We used to keep only those sections that were “neighbours” of our section. This was defined arbitrarily as those sections whose prefix differs from our prefix by exactly one bit. This meant that if we wanted to send a message to a section that was not our neighbour, we had to relay it through our neighbours. We realised a long time ago that this is unnecessary complexity and we decided to remove it and this week we finally got round to it. So now nodes keep info about all the sections in the network, which means there is no need to relay anything as they always know the final recipients. This simplifies the design and improves performance. One might worry whether this requires a huge amount of memory but it turns out that it doesn’t. We calculated that we can support several hundreds of thousands of sections before memory becomes an issue. We’ll deal with that problem when we get there. As part of this change, we also refactored the lazy messaging logic and extracted it into a separate module, which allowed us to add more unit tests to it, increasing the confidence in the code.

API and CLI

After finalising the changes required for the new Register CRDT data type in the sn_node and sn_client crates, we started adapting the sn_api codebase and its API to support it.

The main aspect of these changes are related to the fact that we would now be exposing to the user of the API the possibility of encountering branches in the data. These branches could have been caused unintentionally by the application when multiple instances are concurrently writing values to the same Register, but they could very well be created intentionally by the application, for example a blockchain application where forks simply happen and are also resolved.

We also started with adapting all our abstractions (NRS, FilesContainers and Wallets) to make use of this data type. This is where we see a bigger challenge in terms of UX, given that reading a FilesContainer could result in seeing more than one state as explained above. Thus, we are now looking at the best way to expose this in our API so the user and/or developer of an application can decide what to do in those situations. For example, the end user may be prompted to decide not only which branch to read, but also perhaps how to merge them into a single branch again. As you can already see there may be several ways the user or developer would like to resolve the occurrence of branches in the data, and this is the main driver now for the new design of these APIs.

When it comes to the Safe CLI, we’ll need to make some decisions on what options to offer to the end user for resolving data branches. An example here being when fetching a FilesContainer with two unresolved states, the CLI can probably present detailed information about them and let the user decide which one to look up.

CRDT

We’ve been continuing research on the Bounded Counter work, and this week we’ve been working to make them Byzantine Fault Tolerant and to implement a PoC. Hopefully in the next week or so we’ll have something concrete to share on this front.

We’ve also fleshed out the design for a new MultiMap CRDT based on the MerkleReg. This will likely be the basis for NRS subdomain data types, as well as a flexible structure for other applications that need a map like data type.

Community & Marketing

More questions roll in to the AMA, and another answer too. This time @jimcollinson tackles one from @dimitar: Is it too late for online privacy? Haven’t they got all our information already?

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

71 Likes

Will this be my first “First”?!

Update: SUCCESS!

Big technical update overall and really excited to see puzzle pieces falling into place. Really been enjoying the AMA as well so far, I think it’s been really beneficial!

31 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

52z5oy

29 Likes

Great news on the BambooGarden fund, and then this is awesome…

Looks like a testnet in the next week! :crossed_fingers:

35 Likes

Will this forum be open to the public so everybody knows what is going on, or is it solely for the members?

Also, how will the progress and doings of the committee be communicated to the community here? It might be a good idea to have a separate section for this in each weekly update.

52m6hr

13 Likes

These discussions will be private and confidential, so not open to the public to view. We want committee members to be able to consider applications and vote completely free from any external pressures.

On the progress update question - yes I agree that a regular section in the dev update would be ideal, we’ll do that :smiley:

27 Likes

Excellent update - it all looks like its going well

Which is nice cos in Scotland right now we are in dire need of a free uncensored internet
https://www.craigmurray.org.uk/ <–and thats just for starters UPDATE Craig was found guilty but will not be sentenced until the day AFTER the vote. - One has to wonder why?

Thank you to ALL the devs, all who voluteered for the Bamboo Garden committee and the community devs like @mav @scorch @happybeing and those I forgot to mention but whose contribution is similarly valued.

We really are at the tickly bit now…

19 Likes

Amazing update, and love Jim’s video. Such a great and clear explanation.

Keep up the good work Maidsafe!

Btw, I’ve posted here:

25 Likes

finally thursday again and great update :slight_smile:

congrats to the very first BGF committee elect and thank you in advance for your time and expertise. to see this all come together so quickly is absolutely impressive :+1:

24 Likes

Very exciting update! There’s a lot cooking right now, and I like it!! :smile:

Here’s the tweet to broadcast this week’s update so you can like, retweet & comment:

24 Likes

Great update! Nice to see the BGF committee getting sworn-in.

13 Likes

Great to hear the fund community is taking shape. Looking forward to the test net too!

14 Likes

Great progress! And these AMA videos are absolutely fabulous!!

Keep it up team!

21 Likes

I’d just like to emphasize how important I think this is.
Cheers!

14 Likes

Another great update Maidsafe devs.

Can we please have weekly @JimCollinson vids?
Not to sound “paranoid android”, but has anyone else noticed that they were unsubscribed from Maidsafe’s Youtube? :scream:

:clap: :vulcan_salute: :stuck_out_tongue_closed_eyes:

7 Likes

Woohoo… Well done all. :grin::grin: Was this algorithm passed to a University as a project? To get some new ideas?

But really great work. Testnet incoming :ok_hand::ok_hand:

8 Likes

I also volunteered (via the BGF email) but heard nothing back.

Would have been nice to get a simple thanks for the offer but you are not qualified for the job back!
Obviously plenty of committee members and no need for more but still a bit of a kick in the nuts to be totally ignored.

Just a bit of friendly feedback to whoever will deal with future offers to volunteer and or fund applications. (Respond to everyone, even if not accepted)

9 Likes

Might be that admin is snowed and will get around to responding in time. There’s a lot to do in setting this up and only one admin person at MaidSafe to do all the donkey work so try not to take it personally.

I didn’t apply and have no inside knowledge, but knowing MaidSafe that seems much more likely than that they don’t care.

5 Likes

We’re definitely not intentionally ignoring any applications, so sorry for that @Josh

I think we must have an issue with the email then. You would have heard something back if the application had been received properly, 100%. (cc @StephenC) . Thanks for bringing this up, and sorry for your nuts. They have not been intentionally kicked.

20 Likes