Update 19th August, 2021

This week has been a difficult one in many ways in the aftermath of last week’s sad news. Sometimes we don’t realise how close we all are and in all of our different ways, all want the same thing, a better world. People handle news such as this in different ways. The attitude in-house has been to focus on work as best we can and push towards the next testnet.

Recent forum discussions have demonstrated a strong interest in how the Safe economy will work. Inevitably the model has changed a bit with the adoption of DBCs as the vehicle for exchange. As to give everyone a better chance to follow the progress, we’ll be kicking off a multi-part series soon which will start by taking a look at the current token infrastructure changes.

In the meantime…

General progress

@lionel.faber and @chris.connelly have had some productive chats with the quinn team concerning timeouts and error handling. QUIC peer-to-peer (qp2p) uses the quinn library to connect clients to the network and some of the connectivity issues we’ve seen are due to the way the two systems interact. The quinn team is receptive to pull requests and we are making progress. At the same time, we are simplifying qp2p by removing some of the caching behaviour that we no longer need.

@yogesh has also been tracking down timeout and message-dropping bugs from the client-routing angle where lack of CPU cycles seems to be affecting the connection streams of qp2p, eventually leading to dropping of messages, thus affecting the flows. These, along with AE bugs, are the main issues that are holding back the next testnet. Anti-entropy is now mostly implemented for all messages that mutate data, but there are still kinks to be ironed out. With the Client implementing the AE paradigm, SectionInfoMsgs become redundant and only become an overhead to the bootstrap process, therefore it is also now being removed with AE taking responsibility for keeping the Client up to date with the network infrastructure.

On messaging simplification, @anselme has moved PrefixMap, a structure that matches section prefixes, from node functionality to a generic type.

Furthermore, simplification has been achieved in the self-encryption crate as well. @oetyng took a stab at it, having it do no more than necessary, as well as (already mentioned last week) splitting up the work on all available cores. The 459 % write speed improvements mentioned have now increased to 1478 % faster. (Again, this is measured on a 6-core machine. The more cores the higher the percentual improvement.) Reads are “only” 290 % faster.

Otherwise we’re working on a larger feature branch as some folk have noted, taking another step towards merging the node and routing folders in the safe_network repo. This has helped increase concurrency, and we’ve further simplified routing with the AntiEntropy changes.

We’ve reduced memory usage (which spiked again after the refactor and AE) by catching some previously looping messages (we’re down to a few hundred mb at most during heavy test runs). And we’re now looking at how we can reduce CPU usage too!

All of which is coming along well, though we’ve not yet hit our goal of stabilising CI (which requires a ~40 node, two section network passing all client tests on CI). So we’re continuing to merge fixes and debug on this branch before we merge it to main.


Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

74 Likes

thanks for sharing.

16 Likes

second I guess bridge beat me to it. Now read :stuck_out_tongue:

Thx for the update Maidsafe devs

Always great to see the improvements, keep hacking super ants

14 Likes

First to read it all the way through :slight_smile:

I really am an annoying turd, ahmano?

9 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! Remember not to work too hard and take some breaks. :racehorse:

11 Likes

seriously impressive stuff @oetyng. you are optimizing the hell out of this baby-fleming :boom:

24 Likes

Nice to see that the cause of timeout is clearer.

Will uploads ever be in parallel or are those necessarily a serial process?.. Will a large file always take a while, unless the user fragments it themselves?

11 Likes

Thank you to all the devs for moving on and making real progress during this difficult time.
As it looks like a testnet with all the new features may be a week or two down the line, should I devote some time to building a Vagrantfile that would create a local baby–fleming with node v0.9.1?
I am unsure how to most effectively help - would more folk playing with a basic baby-fleming and just gaining basic CLI familiarity be of some use?

8 Likes

Perhaps hold for a little bit on that. It will be helpful for sure, just we would give you headaches as we are merging a ton of changes and breaking APIs right now

12 Likes

I was thinking of choosing a “bulletproof” combo of CLI and node that would Just Work for basic PUT and GET.

I have a simple Vagrantfile working for CLI v0.33.7 and node v0.9.1 PUT and GET work, I have not looked at NRS yet and I just guess at the actual test-coin balance for now, assuming 1100 coins should be enough for now.

vagrant@ubuntu2004:~/Downloads$ safe files get safe://hyryyrbkiudh4xge46ofaz5n1dzfgd18kyenfri91tk9n8pbgajjut4ojpenra ~/retreived/
  [00:00:01] [########################################] 19.77KB/19.77KB (23.35KB/s, 0s)  Transfer
Done. Retrieved 4 files to /home/vagrant/retreived/
vagrant@ubuntu2004:~/Downloads$ safe -V
sn_cli 0.33.7
vagrant@ubuntu2004:~/Downloads$ safe node bin-version
safe_network 0.9.1

this is just to let folks play with what we had a month or so ago and get a little hands-on time.

10 Likes

It is of course one thing having something working on your own box. Trying to share it with others can be another thing…

Some deeper understanding of how vagrant ssh works is required…

1 Like

Send me a DM and we could chat on here or Discord. Would be happy to try and help.

Edit: sorry, that was supposed to be a reply to @Southside regarding the Vagrant setup.

2 Likes

Nice work @oetyng, great work team!

9 Likes

Oh yes, they will be in parallel.

Now there are tons of different things still not optimal. But we are already way ahead of where we were, and will make it better piece by piece.

Larger files should not need to take longer time than anything else, wrt MB/s.

17 Likes

Amazing speed increase, great work devs!

13 Likes

Thank you for the heavy work team MaidSafe! I add the translations in the first post :dragon:


Privacy. Security. Freedom

8 Likes