SAFE Network Dev Update - May 28, 2020

Development Updates
1

Summary

Here are some of the main things to highlight since the last dev update:

  • We are currently looking at removing (or at least significantly reducing) usage of Parsec in the Routing library.
  • We’re almost done with migrating the SAFE Client Libs to switch to async functions and exposing its API also as async.
  • We implemented an approach to supporting symlinks that is pretty much the same as what git does. We’re also adapting the code in safe-api which invokes the underlying SAFE Client Libs API. We should then be able to release a new SAFE CLI version :tada:
  • The expected final BLS-DKG PR is now raised and under review, with the crate considered ready for use in the network once this is approved and merged.

4a
4a830×430 290 KB

Forum Downtime

You may have noticed that the SAFE Network Forum was inaccessible for the majority of Tuesday and early on Wednesday of this week. This was down to the domain name expiring, and with it being registered to a now unmonitored MaidSafe Foundation account we weren’t directly alerted that it was due for renewal. On Wednesday morning (UK) the domain was transferred to a MaidSafe account and renewed (with auto-renew enabled :stuck_out_tongue_winking_eye:)

It can take time for these changes to propagate through, but everyone should be able to access it again by now.

Vaults Phase 2

Project plan

This week we completed the implementation of the data handling flows when an Adult leaves the network. Adults have the responsibility of holding immutable data chunks so after it leaves we should replace the lost copies of the chunks in order to maintain the required number of copies of the data to ensure data availability. The initial implementation works as expected and the test results are looking good. The current implementation has a bit of network overhead that we are aware of. Thanks to features in Routing that optimise the communications within a section, we can overcome this with a few adjustments that are in progress. With this optimisation in place, we will be in a position for a testnet that includes Adults from home with the responsibility of holding data. The next steps would then be handling Elders leaving the network, section splits and so on which we will be working on in the days to come.

SAFE API

Project plan

Support for symlinks is coming

This week work has begun on supporting storage and retrieval of symlinks that exist in the local filesystem.

The initial approach was to:

  • allow relative links inside of the directory tree being uploaded
  • convert absolute links to relative links inside of the upload tree.
  • convert links outside of the upload tree to real files.

However, the implementation of this was somewhat complex and it also modifies the user’s data, such that symlinks may be different on disk after a put/get cycle. That would be surprising to many users and violates the principle of least astonishment.

So after some thought, the very simplest scheme is being implemented, which is actually pretty much the same as what git does. In this scheme, all symlinks are uploaded exactly as they are, even absolute paths to other locations on the user’s filesystem and broken links. If the user is uploading this data for archival/backup purposes, then it can be very useful to have those symlinks restored correctly upon download. Otherwise, if the user is uploading the files for a website or other public usage, then it is up to the user to ensure there are no broken links.

As of this writing, symlink upload and retrieval are working on Unix platforms. Windows behaves a little differently, so will take a bit of extra work in the coming week.

Under the hood

We’ve also worked on adapting the code which invokes the underlying SAFE Client Libs API as it’s being migrated to be exposing async APIs. We are aiming at getting this finalised this week and we will therefore be ready for a new release of the SAFE CLI which brings several enhancements and new features for all of you to try out.

Fixes

  • PR 562: fix NRS resolution of nested subnames.
  • PR 558: files ls now resolves NRS URLs correctly.

CRDT

This past week we’ve been working on some refactoring in SAFE Client Libs to finally switch to async functions and exposing its API also as async. This has taken priority as it’s crucial for getting our CRDT implementation on the client side properly. The refactoring in SCL is making the code a lot simpler and is also accommodating our needs of adding these new types of requests for the Sequence CRDT.

Fortunately we are finishing such a refactoring effort today/tomorrow, and we will be able to then start adding the logic on the client side for CRDT, i.e. start caching a replica of the data so it can be mutated locally by the client and broadcasting the operation to the data’s Elders section for storage on the network. This will give us our first PoC and E2E implementation of the Sequence CRDT.

Transfers

Project Plan

We’ve cleared up some questions with regards to the repo’s responsibility. Basically, membership is a responsibility for the upper layer, and adding and removing replicas to the internally handled sets is the result of Elders being added or removed in a section.

An extension of this problem was the verification of results from a remote group of replicas (in higher layers known as a remote section). It turns out to also be a membership question, although not replica/section membership, but system membership. Therefore this is also functionality provided by the upper layer.

Alongside this, some general naming and code clean up has been ongoing and will continue to be, and there’s been work on tests and integration testing of the components, with additional work on finding testability/modularity pain points. Basic integration tests are passing and we are now setting up property based testing.

Futures in Rust

The last week has seen a bit of a flurry of activity as things come together in async land. We have the whole of SCL compiling nicely, and CI tests are now passing :green_heart: .

We have updates to self-encryption merged into master and a new version building there (which SAFE Client Libs depends upon).

safe-api has been updated for the new SCL version. This highlighted some other issues in the code, including some tricky blocking behaviour with some of our await code for messages, but we’ve got that licked now and are moving through some final tests with the APIs before getting this all merged in.

It’s been a big job, so it’s great to see this coming together (and to say goodbye to around 2000 lines of code, too! :tada:)

Routing

Project Plan

In the last dev update, we talked about how we are improving the way we handle messages that are either not trusted or not processable by the node in its current state. That work has been finished and the PR is now being reviewed.

We are currently looking at another large change for Routing - the removal of Parsec.

This might sound strange, because Parsec is the algorithm we developed ourselves specifically for the needs of Routing. It has worked very well and has passed many rounds of very extensive testing.

So what’s the problem with it? Performance. It can take in the order of tens of seconds to reach consensus on a single observation which is a rate that simply does not scale to our needs. That’s why we decided some time ago to stop using it for data requests (which we are transitioning to CRDTs), but that’s still not enough. We also want to remove (or at least significantly reduce) its usage in Routing itself. This is actually possible without sacrificing any of the goals we set for Routing because it turns out that Parsec is doing quite a bit more than we need. For example, we don’t really need the total order of observations. All we need is that actions are agreed by the quorum of the section elders and that they are consistent. So this is what we will be working on in the next few weeks.

BLS - Distributed Key Generation

Project plan

This week, we focused on pushing all the tweaks that we made from observing our real-network simulations and tests to the master branch with PR 14. This PR adds the transport layer (quic-p2p) and so completes the crate. Being a generic crate, it targets users of all types (Nodes and Clients) therefore the integrated transport layer makes it easier for the users as they do not have to worry about the exchanging of messages & completion of the BLS session - as discussed in last week’s update, the crate takes responsibility for these using timers. With this PR currently going through the review process and close to being ready to be merged, the next step will be to begin our integration of BLS-DKG with the other SAFE Network parts, another step closer to our end goal! :smile:

Useful Links

Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

61 Likes
3

The first time :0

We are currently looking at removing (or at least significantly reducing) usage of Parsec in the Routing library.

Great, I look forward to improving performance :slight_smile:

23 Likes
4

Thanks so much to the entire Maidsafe team for all of your hard work. Changing the world is not easy but the rewards will be worth the effort.

21 Likes
5

Great job, keep it up!
Regarding to Parsec, this is the reason that promoting it prematurely is harmful.
Whats going to happen to all the public fanfare regarding to it?
People won’t understand that avoiding sunk cost fallacies is good, it might be the right technical decision but it has bad optics.

24 Likes
6

Well, most people should understand Parsec was a big thing for the project at the stage it was developed, so it got somewhat promoted, but not to a huge extent anyway. And there already were signs Parsec is not the panacea before. So I think most of us expected it.

EDIT: I’m fourth btw. Let a sausage be my consolation prize.

11 Likes
7

I beg to differ, it was promoted as the best thing since the invention of bacon:

“Paradigm shift” is not small potatoes, and for a team that was known for restraint for hyperbole, that was quite an announcement.
And that wasn’t the only announcement, there are plenty of videos on youtube and technical debates defending it.

This U turn will require some damage control.

14 Likes
8

Isn’t Parsec used elsewhere than in routing?

6 Likes
9

So is Parsec deprecated? or will it still be used somewhere in the network? Wasn’t that 18 months of work?

6 Likes
10

I stand somewhat corrected; there was a brief period of high hopes and “the project is literally derisked now” proclamations by the supporters, but it didn’t really last too long. And it’s been two years, almost. Two years? :flushed:

We’ve had enough time to digest the fact the development had a long way to go and Parsec didn’t really solve everything. Well, I don’t feel shocked, and I proudly represent at least a small sample of the backlash you’re having a premonition about.

6 Likes
11

yeah, the whole parsec thing was big at the time and hailed as the last piece of the puzzle that was still missing…
we were all having a party, i remember that.

i guess this speaks to how difficult it is to predict how this network will behave, cause its never been done before…

assuming it is the right decision to abandon parsec, I applaud maidsafe for doing this. it is like sacrificing your queen in order to win the chess game. others would have held on for longer just to realize eventually it was in vain.

if nothing else, it still was a useful academic exercise. who knows, others might use parsec somewhere somehow… how about the blockchain guys? they dont mind waiting for blocks for 10s of seconds right? :rofl:

12 Likes
12

When I said at the time Parsec was like a blockchain I got ridiculed.

2 Likes
13

The problem is not about the trial and error, the problem is the premature promotion.
Some people here were talking about mounting a marketing campaign before the network was ready, but we don’t even have a proper way of describing it yet because we are still experimenting with it.
There is a reason why drugs aren’t promoted in every pre-clinical trial.

9 Likes
14

Can this large change wait? can we launch a slow version of the network?

3 Likes
15

well, you will still be ridiculed, because it is still nothing like a blockchain.

3 Likes
17

I agree with you. hell, i was the first dude to rush and buy more maid at the time i believe :upside_down_face:

thing is parsec was promoted in good faith, not as a pump and dump or similar. there was every reason to believe it was what we were hoping. only time could prove us wrong - and it did.

but not share the excitement would have been kinda lame too. lets hope this does not happen to CRDTs too. or maybe we should hope that it does? cause it means something better will replace them…

13 Likes
18

I can only agree here actually. There was too much promotion and way too much time making an individual crate “production ready”. Parsec is still a great achievement in ABFT. It was believed to be the answer to many things by ordering everything. The problem is centralised data structure to order all other data structures.

In any case we tried it and we tried to make it work with data and it was problematic. I was not a fan of spending more time finding ways of pruning the graph or sorting the malice issue and so on. It’s not required to order as parsec orders as it does now, like template/generic spread through all aspects of the system.

With data it made sense (as it did in 2015 with data chains) to allow data to find it’s own order and with causal order via CRDT types then that is a solved issue. This left parsec handling routing membership and it does that fine, it is right now, but with a caveat, the graph keeps growing and there are naive steps to prune with 10 1GB logs kept, this is out of line with “work in as many machines as possible”. As membership can be handled with group consensus (dkg) and, if required (I think not) a bft crdt orswat (we have POC of that and it will be published soon, but not by us).

All in all I am happier to write all that off and move at the pace we currently are, I cannot afford an 18 month fix period for the order everything approach. It is not how SAFE was envisaged, if it had worked and been a panacea, then great, but it’s just not. Unfortunately a case of tunnel vision there I think and not considering the network as a whole. These days we have meets where all we discuss is the whole network. It’s a different proposition and pace to launch is way beyond any other period in our history.

tl;dr Tried it and really tried to make it work, now back to SAFE and focus on provably consistent highly concurrent data types. It is so much simpler and we will get working on smaller machines as has been my goal for … well forever really :wink:

36 Likes
19

It’s not only slow, it’s a significant memory leak as well. (ever growing data structure)

13 Likes
20

Okay, so I understand the problems aren’t worth the time investment to fix them.

Will the CRDT types approach push the release of Baby Fleming? or Embryo Fleming to next year? or are we thinking months before we see a shared section from home version?

4 Likes
21

Friend, I don’t think anyone knows … You see new problems are emerging. The important thing is that there is a way forward :dragon: Go Team MaidSafe! :bowing_man:

10 Likes
22

I feel like I’ve lost track of what the goal is anymore, or where we are in the development cycle. Working vaults from home is the point that matters. the fundamental building block for the success of the project.

It was said vaults from home testing was going well, for weeks now. Is that with or without Parsec?

4 Likes