SAFE Network Dev Update - July 9, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • We have finished work on the vaults from home improvements and have been running internal testnets - results look good so far!
  • Work continues across the board to switch our testing away from running against mocks to running against production code.
  • We are integrating a further refactor to network requests, which will open the door to farming on the network.
  • The Routing DKG replacement PR has been merged.

Vaults Phase 2

Project plan

This week we completed implementing improvements based on the feedback received from the previous testnet. We have been testing these changes in internal testnets and we are seeing very good results. After just a little more testing, we will be ready to announce another iteration of the Vaults from home testnet, showcasing all the improvements we’ve been working on. :wink:

We’ve also been working on making our test suite run against real Routing. The tests take much longer to run, but, this is because we currently spawn a separate network for each test. We are working on some more refactors which will use the same network for all the tests and this will speed things up significantly.

SAFE API

Project plan

A pull request for glob() support was raised this week, however, concerns over efficiency and need for further changes in subcommands for files ls may prevent it from being merged until further work is done. This PR will go through the usual testing and review process and we’ll consider the best course of action for it.

SAFE App C#

Project plan

This week we revived a slightly older pull request which focuses on Authenticator API refactoring and enhancements. In this PR, we are moving away from the safe-client-libs/safe-authenticator-ffi and exposing the Authenticator APIs from the safe-api crate. This will further simplify the application and authentication APIs and make it easier to keep the APIs in sync with safe-cli, safe-api, & safe-nodejs.

CRDT

This week we started to do research around how to properly manage permissions in our CRDT data types, like the new Sequence data type. Having permissions on CRDT data brings up some challenges as any changes to the access rules have to be applied in the same manner on all replicas (i.e. Elders) so we preserve data convergence. As an example, revoking permissions from a user to append items to a Sequence may lead to having some replicas allowing the user to still append items if such operations are seen before the revocation operation, therefore having replicas not converging to the same data state.

We are looking at some available papers where some solutions are proposed, to better understand not only the solutions but the problem itself. For those interested in reading more about the problem, this paper could be of help, at least just to understand the challenge.

Transfers

SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan

After finalising safe_core tests against the AT2 system, we’ve been working on integrating a further refactor to network requests to open the door to farming on the network. As we move towards a pseudo multi-section setup with AT2 and farming, vaults and sections in the network need to exchange messages among themselves to complete a data/payment flow. This raises a need for us to distinguish between client messages and intra-network messages. Therefore an overhaul has been started on the messaging structure which the network will use. Basically, this will align the messaging with CRDT principles and event driven systems.

This has in turn led to further work on the integration with SAFE Client Libs, but things are progressing well here as we uncover bugs and performance issues, and refine the messaging flow there. One more significant change to SAFE Client Libs has been the removal of “ack” type Responses (which essentially tell us nothing), and leaving it up to the client to optionally confirm whether their operation has completed. (This itself should eventually lead us to some try/check/retry flows for certain operations, but it won’t be necessary to do so. Which should give app devs more flexibility in how they proceed, and in turn hopefully make the client itself more responsive).

Routing

Project Plan

This week we finally merged the DKG replacement PR. This enables us to update section keys without using parsec.

To help improve code quality, we also added code coverage statistics into the crate, and enforced the rule that any further PR shall not decrease the coverage rate. As mentioned previously, we are gradually switching away from testing with mocks across all our repositories. This inevitably means that code coverage stats against production code will start off low as this has never previously been our focus, but we will gradually build that up over time.

Our current primary focus is on the tasks to remove the parsec usage. One such task currently going through the PR review process is ensure mutations to SharedState are approved by the section. We also started working on overhauling the node promotion process so that it does not involve parsec as much. That is a prerequisite for another major task, which is to implement message-based voting - with that we will replace the current parsec-based voting, which of course gave us consensus and strong order, with a much simpler mechanism which gives us consensus without order, which we believe is enough.

xor-name

After releasing the first stable version of our xor-name crate last week, we continued to build on that this week by refactoring multiple crates to start using the new xor-name crate.

As it stands, we have what are now redundant XorName type and related functionalities exposed from safe-nd, safe-client-libs and safe-api. To streamline the code in these repos, we are in the process of removing XorName from each, making use of the new dedicated xor-name crate instead. Draft PRs making this switch in each of these repositories can be viewed here, here and here respectively.

We are now waiting on an external crate version to be published to unblock us from proceeding further with this. Once that is available we will continue the refactoring work with safe-vault and other repositories.

Useful Links


Feel free to send us translations of this dev update and we’ll list them here:

:germany: German ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together
:love:

69 Likes

THanks for sharing updates!

22 Likes

Second one.
Go Made go!!!

20 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! The world needs this network more every day. :racehorse:

19 Likes

:godzilla:

27 Likes

= next Thursday?

12 Likes

Asking for a timeline? You new around here? :wink:

20 Likes

gogogo team!

11 Likes

Hope your ready to hold my hand again @Southside :joy:

Well done maidsafe team absolutely flying :ok_hand:t2:

18 Likes

Yes, I will make sure you never walk alone :wink:

16 Likes

I know its not but each time I see this, I look forward to a network that is up and each change is just an update adding features.

Just a little more teasing and we’ll be there… just over the next hill, round the next bend.

:davidpbrown:

15 Likes

Thanks for the update Maidsafe devs and all your hard work.

:clap: :clap: :clap: can’t wait :happyant:

17 Likes

Sounds like heavy work! Thanks for the efforts in all directions everyone. Good luck with the next steps…

15 Likes

:rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl::rofl:

12 Likes

Thank goodness they didnt use “soon” or “around the corner”. The next testnet must be close indeed

13 Likes

Hahaha good one!

9 Likes

Nice update, everything is chugging along so nicely, speeding along is more accurate.

I hope the issue with permissions on CRDT does not throw a spanner in the works now when things appear to be so close to clicking together.

No doubt you will figure it out!

17 Likes

This will probably sound naive because I’m no expert but if an operation has already been seen by a majority of replicas shouldn’t that be good enough? Even if permission was revoked technically before but the replicas already see a majority of the replicas allowing an append operation you could assume which came first (even though the revocation was technically first it was also ‘too late’) with out some the use of some time element. Idk just brain droppings. I’ll leave it to the experts haha.

Wondering what needs to be done in quic-p2p to allow for more IGD support, I remember reading there was an additional little trick that would allow for more router support if their IGD settings were picky. I’ve also been seeing some posts about China’s great firewall and it made me wonder about quic-p2p’s needs to be able to circumvent such a thing. Is udp and/or tcp hole punching or even utp still needing to be added to quic-p2p to achieve such things? Thanks for any answers.

Looking forward to the next test net! Even if I can’t personally participate because of my router (which I might try swapping out for the test) it’s always awesome reading the community comments during and the results during and after.

AT2 progress is stellar. Just hearing the word farming gets me excited! I never thought I would say that phrase :joy:

Looks like PARSEC is being untangled at a good pace as well. Another great week from everyone @maidsafe keep it up! :muscle:
All my support.

13 Likes

I think we agree on that from a conceptual level, the challenge actually resides in making sure all replicas behave in exactly the same way in order to assure convergence. A replica doesn’t ask other replicas when applying an op for anything, they check against their local policies state and act upon it. It seems we need to have a causality relation between the data and the access/policies, so each replica knows what policy version each data operation depends on regardless when it arrived to a replica for applying it.

11 Likes

Sounds kind of like physics or quantum mechanics :smile:
That’s a neat insight. Thanks @bochaco

7 Likes