SAFE Network Dev Update :safe: June 18, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • Thank you to all who contributed to last week’s vaults from home with IGD testnet!
  • We’ve been examining the results of the testnet and are now working on areas identified for improvement.
  • There is a community-run testnet for those who wish to get involved - see forum post here.
  • The safe-cli symlink PR is now going through testing and review.
  • Sequence CRDT progress continues at pace with associated PRs merged in safe-nd and safe-vault.
  • A GitHub issue has been created listing all the steps we think are needed in order to remove parsec from routing.

Vaults Phase 2

Project plan

We’d like to thank everyone for their participation in last week’s testnet. Even though the testnet was limited to users with IGD supported routers, we had a good response with nodes joining the section 59 times. There was a good amount of client participation as well with 2214 immutable data chunks stored in 107 files containers. An important part of the test was to demonstrate data recovery when nodes leave the network and this showed good results. The network executed 4193 successful data recovery operations.

The testnet helped us understand some real-life network scenarios as well. The implementation had already considered some of these scenarios and the testnet helped us identify some areas for improvement that we have already started working on. An interesting example is ahead of time responses. The network needs only quorum number of Elders to approve a request and process it. Once processed the response is sent back to all the Elders for confirmation. While some of these elders will be aware of the request others might not know about it. In such cases, the response is cached until the Elder eventually receives the request. We identified that sometimes the response was sent back twice. The first one was processed and the second one was cached waiting for a request that would never arrive again.

While the double-response is the culprit here, we also identified that caching the responses might not be the best choice for such scenarios. The Elder can verify if the response has a request that was validated by the section and process it accordingly. This would improve both time and memory efficiency. We are also reducing the Elders’ responsibility of taking care of the chunk duplication. Instead, the new holder of the chunk will fetch and store a new copy of the chunk which again removes the need for signature accumulation for data recovery related messages.

We will be including these features (and more) in the testnets to come. In the meantime, you are welcome to join the community-run testnet. Kudos to @tfa, @Traktion, @neo and everyone else who has been involved. Visit this post to be a part of the action.

SAFE API

Project plan

Symlink support is wrapping up this week. Work progressed with test cases, documentation, and fixing some path related issues on Windows that turned up in tests. A large pull request was raised on Wednesday and awaits final testing and review before it can be merged.

CRDT

Good progress has been made in the past week in terms of making the Sequence CRDT ready for a first testnet iteration. We’ve finalised and already merged a PR in the safe-nd crate where this new data type is defined and where the core CRDT logic resides. The AppendOnlyData was also removed from it as the Sequence data type is superseding it.

We were also able to finalise the PR for the safe-vault crate, which was approved and merged today. This means we now have the code ready in the vault to accept requests for storing and mutating Sequence CRDT content. Some of the last things we’ve worked on were to also have the permissions and owner of the Sequence be CRDTs, and the creation of some basic integration tests towards a first iteration and/or testnet. The changes needed to remove the support of AppendOnlyData in safe-vault are now ready for review in a separate PR.

We are now trying to finalise the PRs required to support this new Sequence data type on the client side, which means having the safe-client-libs and safe-api crates expose APIs to be able to send requests to store and mutate Sequence content. The PRs are already functional, there is only a need for some final clean up and the addition of some automated tests before a first testnet using this CRDT data type becomes ready.

Transfers

SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan

We’ve been working hard on transfer actor integration into SAFE Client Libs and Vaults. The former now has our mock tests passing with some placeholder APIs, and we’re now working on integrating with our updated vault code. This has needed the creation of some new flows and messages. We now have some SimulatedPayout request types for mock to load accounts with test safecoins as if they had received a farming payout. This is a small but helpful tweak, which gets us closer to how the network will eventually work.

The AT2 implementation itself necessitates some tweaks to our flow for various other operations. Most notably, CreateLoginPacketFor requests (which allow an account to upload login data for someone else, and preload it with some money) will have to effectively wrap two pre-validated transactions: one for the cost of a put, and another for the transfer of money to the new account. Fortunately, we have the basis for the transaction validation in place so this should not be any great shakes to incorporate into the client flow. (And what’s neat here is that we’re effectively setting up transfer bundling – albeit in a small way – while still keeping the onus for co-ordination of this work on the client).

Routing

Project Plan

The shared state signatures PR was finally raised and is currently being reviewed. The DKG replacement PR is also almost ready apart from some kinks that need to be ironed out first. We created a GitHub issue listing all the steps we think are needed in order to remove parsec from routing. It seems like a lot of work, but we believe most of it is quite straightforward. The whole effort can be summarised roughly like this:

  • We will implement parsec-less voting based on messages and BLS signatures. It will give us consensus / authority but not total order. We believe that is enough for our needs. The resulting mechanism will be much simpler and more performant than parsec.
  • We convert all the data structures that are used to store section-approved information into CRDTs (Conflict-free Replicated DataTypes), which means that all mutations to these structures will be commutative, associative and idempotent. In simpler terms, that means resilient to message reordering and duplication. This will give us eventual consistency for very little cost compared to parsec.
  • Additionally, we only apply a mutation if it was approved through the above-mentioned voting process. This gives us resilience to various kinds of failures and attacks because any mutation must be approved by the quorum of the section elders.

Useful Links


Feel free to send us translations of this dev update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

72 Likes

thanks for sharing it

22 Likes

Excellent update, we know where we are, we know where we are going and we know what is needed to get there. Again it looks like we are adding strength, lightness and simplicity. Taking a very complex notion and reducing it to simple components all working together in harmony, each doing just enough to get by so the whole operates at speed.

What more can you ask for?

Thanks as ever to all concerned for their hard work and also thanks to @tfa for providing a community network so I have something to break until the official one reappears with new shiny bits.

35 Likes

Thanks so much to the entire Maidsafe team for all of your hard work. These updates are excellent and show a lot of progress.

23 Likes

Great work team Maidsafe and congratulations on a first test while still adding in and refining features in several areas. Things are really flying. :rocket:

Getting Ready to Farm

For myself I’m pleased to say my single board computers (5 Watts only) are ready to see if they can operate as vaults. I’ve published a directory which includes the build scripts for anyone building on Linux.

The scripts have been tested with Ubuntu 16.04 LTS and 18.04 LTS on the Odroid-U3+ is ARMv7 processor architecture, but will work on any processor supported by the Rustc compiler. They will likely work with other Linux distros with little or no modification.

I’m happy to add other scripts that are aimed at helping people get farming, find them here:

37 Likes

Fantastic results from the testnet, I’d say! Great progress on integrating CRDT and removing PARSEC too. I’ve also noticed that the updates are increasingly streamlined and concise, like everything’s coming into clearer focus. Awesome work, MaidSafe team, and thanks also to the community members who got the community-run testnet up and running too!! :smile:

29 Likes

When we get parsec out fully (soon I hope) then I will be very interested in this area. So to start raspberry pi etc. but then onto cortex M etc. as well as stl boards and the like. So a few things we are considering/pushing here

  1. Use of no_std libs where possible
  2. RAM usage, so no caching messages, etc. for analysis or to catch up on some syn, but send back to the network with an error to get more info (we call these self-healing messages and a recent big push for us). ofc if there is RAM for data cache we will use it :wink: [ EDIT, this really changes from local RAM use to network use, more traffic, but for small devices this is cool as a tiny device can push loads on the wire, but not necessarily handle huge RAM requirements, apart from that it’s also more natural and correct]
  3. RAM usage, again patterns that simplify the code also reduce RAM considerably (i.e. parsec on latest wanted up to 10Gb storage for the graph and backups, so that’s a no-no).
  4. Cross-compilation and flashing of vault to small devices (rust is a great thing here).
  5. Watching for any gains, so smoltcp as a quic/quinn plugin for userspace (no std) access to the network stack.
  6. For Linux based boards then musl binaries to cross all versions of linux on each platform (Intel 64/arm etc.)
  7. wasm/wasi for even more cross-platform / cross OS capabilities as well as browser-based “stuff”. This is still quite new, but we are watching. For small devices, it may prove useful, well more the wasi part.

So the push for simplicity means more efficient and that I hope means much more cross platform support. Not only vaults, but also network attached sensors and such devices that can feed the network/clients/apps with environmental info and track pollution/animals and all of the good stuff there (no human tracking please :smiley: )

It’s right in our target now Mark and won’t get de-focussed again. Prevalence is easier when we remove barriers.

36 Likes

I switched my home computer for a raspberry pi4 last year, it really fits my needs and does the job for a daily web + still graphics machine. The whole thing for about 25 times less power consumption. Now my display is the one that sucks amperes :slight_smile:

I would love to see if it can run as a vault too . Thank you for the effort and for sharing !

Thank you very much @maidsafe for the great udpate, it is once again exciting to see the pieces of the puzzle coming together. :+1:

24 Likes

I’ve noticed that too! I’ve been catching a lot of talk of the lower levels preparing how to connect to the UI and features that Jim refined to make for a granular and intuitive experience for the average internet user. I think it will be extremely surprising the amount of participation at a time like that when just anyone can try it, and I think they will be surprised enough to stick around. The benefits are undeniable and unmatched.

22 Likes

Which distro are you running - Raspian? The scripts are quite likely to “just work” or maybe need a couple of tweaks so I’d be very interested to see how you get on if you try using the scripts. Also any feedback on making the instructions easier to follow etc.

12 Likes

It is running debian buster yes.
I will test the scripts and will let you know :slight_smile:

8 Likes

Let’s get this up to 159!! (Hopefully when the IGD-less-router test net comes out, or before.)

I like how PARSEC was basically solving what all other/many crypto projects were trying to solve, and then Maidsafe continues boosting off into space beyond a “parsec” – with their own, better solution.

23 Likes

Thx for the update Maidsafe devs and your hard work.

Pretty neat that we’ve just tested vault from home and now
we’re already seeing some hints of farming :kissing_heart:

Keep hacking super ants and community :stuck_out_tongue:

21 Likes

Worth echoing one more time!

Yes, I have so much respect for the endless capability to throw away something that was huge work and an achievment in its own right, but just not fit for the purpose. I guess that’s an expression of clear vision, incredible motivation and percistense, and uncompromising humbleness in the way you look at your own work. It’s an priviledge and inspiration to just follow it from the sidelines.

Thank you once again!

22 Likes

Full steam ahead! I missed out on this one but will try to join the community vaults from home!

12 Likes

First again, two weeks in a row!
AbsoluteMajorAmericancrow-size_restricted

13 Likes

Hey @JimCollinson don’t want to push or spoil any surprises but looking at the Feature Tracker it shows vault setup all ticked and green. Haven’t seen any vault UI yet but does that we will in the near future?

17 Likes

You may have seen a snifter of Vaults UI in the invite/onboarding process videos IIRC, but yeah the MVE essentials area most entirely done. I’m documenting it all and had hoped to get it out for yesterday’s dev update, but some other vital stuff meant it didn’t quite make it out. Stay tuned though!

22 Likes

I thought PARSEC was one of the big breakthroughs. Can someone provide some further insight into this change or point me in the right direction if the topic has been addressed a bit more thoroughly? What was wrong with PARSEC and why are they changing? What are they changing to and how does this help the project? I’m curious and would like to know more - thanks in advance.

5 Likes

See the last point here, friend:

4 Likes