Safe Network Dev Update - February 4, 2021

Summary

Here are some of the main things to highlight since the last dev update:

  • Today’s expected testnet release has had to be put on hold as we couldn’t quite get rewards in on time, and disabling what we had in so far was still not cutting it.
  • Being able to see network splits occur over recent days, with the latest developments in, has allowed us to find and squash some previously unknown multi-section issues.
  • The qp2p crate API simplification work to manage the connections and streams internally and take some of the pressure off the API consumer, is now going through peer review.
  • A BRB PR to fix random packet loss that was interfering with BRB operations has been raised, tested, and is expected to be merged shortly.
  • Special thanks to community members @bzee, @mav and @scorch for their efforts over the last week with multiple community PRs merged.

Testnet status - on hold

Our goal at the beginning of the week was to include rewards in a public testnet and open it up to the community today, however there were still several required changes to get it running, and we were not able to finish that for today.

Because of that we stripped out the rewards flow to see if we could get a stable enough network, and made some progress with that, but have been hitting bumps along the way which have slowed us down.

Our current status is that we now have a latest version of a testnet up internally but have not had any real time to test it. We expect that it is likely that there will be further minor issues so would like some time to test that out in-house before making it public with confidence.

We’ve yet to fully discuss and decide whether we will release a testnet with rewards excluded, or whether the issues we are coming across mean that we would be better off continuing on with fully implementing rewards, which we expect will take a few more days (plus testing). The stability of the testnet that we have up now, with rewards disabled, should help us decide but either way we don’t envisage putting up a testnet on a Friday, we would prefer to hold off until next week to allow us to fully support it.

The vast majority of our time this week has been focused on the testnet, but here’s a summary of some of the general work we’ve been doing…

Safe Client, Nodes and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan

First off this week, a PR from @bzee which enables programmatic configuration of nodes has been merged to master today. Nice work! :raised_hands:

We’ve created a new Signing trait in sn_data_types to abstract out sign and verification for our various flows in nodes/transfers, and have been testing and bug-fixing some flows centered around the integration of routing’s new infrastructure queries. This has moved the bootstrapping operations into routing itself, as opposed to within nodes. Previously, we were limited to querying only elders (and elders of our own section at that) for network bootstrap info. Moving this to sn_routing means we can query any node, and get back the correct contact info… essentially enabling clients to operate against a multi-section network.

Beyond that, we’ve been debugging various issues with section splits now that we’ve been able to see that in action (albeit limited due to aforementioned issues), and @mav’s node-logger web app has proved to be a great wee tool for quickly seeing cross-node flows and narrowing down the log output of our multi-section network into something a bit more human readable.

We are also working on the data catchup flow that is required for newly promoted Elders in a section, who would need to update themselves with all the data that the rest of the Elders maintain. Since this data would be relatively big in size, we would ideally want to be streaming the data so that the node doesn’t get blocked with that task, therefore we’ll also be tinkering with qp2p a bit to enable streaming data directly to nodes on request.

Over the last week we have made some additions to the qp2p crate to manage the connections and streams internally to take some of the pressure off the API consumer. This work is completed and currently going through review in this PR, with the new API has also being integrated with routing with all the tests passing. We were holding back for a public testnet iteration to be released before merging these, but with the delay we may decide to merge beforehand. Testing has shown that this change has made things considerably simpler and this will make the debugging of any further p2p networking issues much simpler.

API and CLI

Several more community PRs were submitted in this past week, which were merged and are already part of a new version of CLI, just published today (v0.18.0).

Thanks to this PR submitted by @mav, the NRS validation is now excluding a bigger set of characters which would never have any reasonable purpose being included in any URL. Having a limit of total NRS URL length of 255 bytes, where each subname cannot be more than 63 bytes length, is now being enforced with this other PR.

The safe auth CLI commands now give the --config argument priority over the use of environment variables, this is also now in place in the new release thanks to PR #700, also submitted by @mav.

On the qjsonrpc side PR #698 submitted by @Scorch implements a basic client/server example application using the qjsonrpc API. A qjsonrpc client sends a ping message to a qjsonrpc server, and the server responds with an ack message. This example application simply and neatly shows how qjsonrpc can be used to build any application using JSON-RPC over QUIC. There has been a lot more activity and exploration made by @Scorch with more ideas around qjsonrpc - for those interested in more details and/or in participating in this effort, take a look at its own dev thread where he has been sharing all of it. :clap:

BRB - Byzantine Reliable Broadcast

We raised a PR in brb_node_qp2p to fix random packet loss that was interfering with BRB operations. This makes the node stable in case any technical community members would like to interact “hands-on” with BRB in a CLI environment. Basic usage instructions for doing so are here (Note that you must build the code yourself, for now at least).

Design discussions for integrating BRB with BLS Distributed Key Generation (DKG) have continued with several options presented. Agreement appears to be coalescing around one of the options. Details on these plans should be forthcoming in future updates as work commences.

Routing

Project Plan

We’ve made a first step into moving all message definitions out of sn_routing into the sn_messaging crate, where not only the message definitions will reside, but also any logic related to the serialisation of them. The initial iteration of this, along with having a unified type of message for clients and nodes connecting to a section, is now in place in the latest versions of sn_routing and sn_client. Our next step in this regard is to finally move all messages types definitions onto sn_messaging crate, allowing us to decouple the actual sn_routing logic from what would become the API of the network.

Useful Links


Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French ; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

75 Likes

Oh yessss first!

I’ve literally just shared toot and tweet saying how slow I am, but here I am first again! :sunglasses: FYI that was about a mega realisation I had today - how staggeringly big a deal Safe Network is going to be for developers (here).

Bad luck @MaidSafe but thanks to @Savage I’m in a win-win situation and ready to cash out :moneybag:

The news that a testnet is imminent, and might have rewards :pray: is brilliant too. As always, thank you MaidSafe team even though you were trying to make me lose the bet with @Savage. Every incentive helps! :wink:

It’s also awesome to have PRs rolling in from the community. This lifts my spirits and I expect it is great for the team to see too. It is such an important part of the project, so I can imagine David grinning. Thank you @bzee, @mav and @scorch. I think @folaht is doing great work too poking around with running local networks, getting IPv6 working and understanding where the code is at. :clap:

Mini Git Portal Update (4th Feb 2021)

Three really nice things have been added or enhanced:

  • automated tests
  • continuous integration on github
  • improve generation of documentation (for internal APIs)
  • all the above updated in the README

Development and Production Testing

I mentioned previously that I had set up a test framework, but now I’ve supercharged it. What I had was the basics, but now I have it firing up two servers so the tests can run fully automatically.

One server is for the app, which can either be run on a new build (for production testing) or using webpack-dev-middleware for hot reloading during development. :tada:

The second server is a trivial githttp server stolen from wasm-git (which I worked on recently - and had a PR accepted! - while evaluating options for Git Portal). That means I can create more complex internal tests without using a proxy if I want, though the latter will also be an option during development.

It alls sounds a bit geeky to non-developers, but its a big plus for the project and I’m chuffed because I’ve never done any of those things before. All I have to do now is write the tests! And it will make it much easier for others to jump in and help out.

Continuous Integration on GitHub

Encouraged by @StephenC (thanks for offering to help but I managed with a little trial and error) the production tests mentioned above are now run automatically by GitHub whenever code is pushed to main or a PR is submitted. And all tests currently pass!

I’m chuffed as it’s the first time I’ve set up continuous integration on one of my projects and a useful feature that makes it much easier to collaborate with others…so more PRs please!

Improved Documentation Generation (for internal APIs)

To help folks contribute and make the project easier to maintain I wanted the APIs used by the front-end to communicate with the application-logic to be documented as well as tested, and had a basic set-up in place (from the previous update).

But to make it useable I had to do some more tweaking and ended up writing my first JSDoc plugin. It’s simple, but also a new skill that means I can do much more with this when needed.

The main change this time is to shorten the symbol names. Previously JSDoc was prefixing everything with “export.” which was making the menus unusable, and generally polluting the documentation with useless information.

I also learned a bit more about how to use JSDoc and will at some point be writing some guidelines to make it easier for others to jump in and help so…

Help Welcome!

I’m only one person, and a slow one at that :grinning:, trying to create a GitHub alternative and even though there are lots of economies doing this on Safe Network (see below), I won’t be able to complete this all on my own.

So if you want to contribute in any way, don’t be shy. Testing always helps, following the README and finding things that don’t work is invaluable and of course there is code to write in Rust or JavaScript, Svelte stuff to do, front-end UI to envision and build, new post-GitHub features to imagine and much more. All are welcome and I’ll help you skill up wherever you fancy trying something new.

And it is all built on exciting new technology such as [cough] Safe, Rust/WASM, WasmerJS/WASI, my old favourite Svelte and the fancy automated testing and CI I described above.

This is fun, and an exciting valuable project to work on.

47 Likes

Those old fingers are still impressively fast…

31 Likes

THey have to be - I hear he has a new shirt to button up

Oh and 3rd !!!

Well done to all - devs and community for all the hard work - and also well done for the good judgement to hold the test net.

22 Likes

Keep it up can’t wait to try and have a go of the test net :clap:t2::clap:t2::clap:t2:

17 Likes

Post is 4 minutes old and still only 5/6th post. Seems legit :laughing:

Been really interested in some of the BRB stuff lately. It’s so cool to see that stabilizing bit-by-bit, given how cutting edge it is.

Such is life, no need to rush it out the door.

Sometimes I forget this fact and then re-remember it with full force. Really interesting feeling

23 Likes

Great job to the community devs!!! I’d like if rewards were included myself. I say focus on what is closer to the finished product rather than trying to get something lesser than working for us. It’s close enough to make waiting well worth the while! Keep chugging along @maidsafe

28 Likes

My legal counsel went through the fine print and discovered that neither of us set an official date.
So in fairness it should be carried over to next week :grin:

But I’ll concede annihilation :sob:

19 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

4jyh2i

20 Likes

Great effort team MaidSafe. Please dont feel pressured to release any pre-mature testnet. I’d rather wait for a proper one, a testnet where you feel that it is in a good shape.

25 Likes

Thanks for being so thorough team! Drop the main sail, steady as she goes, we’re picking up wind, stay the course, the new world is quick on the horizon. Are you with me crew? Cheers!

16 Likes

Thanks for the update.

I am not sure I fully understand rewards here.
you mean farming rewards ? What is the nature of the issue you spotted on the internal testnet with rewards at this stage ?

13 Likes

:point_up_2: :point_up_2: :point_up_2:

22 Likes

Can’t wait for these testnet-binaries by the way. Very curious to see where things are these days. I do lurk the forum at least on a weekly basis. And I would love to farm some test-coins when I get the chance. Keep up the good work folks :+1:.

27 Likes

They do mean farming rewards here, yes. :slightly_smiling_face:

17 Likes

Like Marius suggested it’s all zen… anticipation is fun too.
Always good to see that bugs are getting squashed.

It’s been a while since we’ve seen polpolrene

:+1:

15 Likes

Well done getting CI in place :tada:
It’s very satisfying getting it set up & see it do it’s thing on every PR :heart_eyes:

22 Likes

Happy to see the progress. No stress, just make it as clean as possible before launching the testnet.

19 Likes

Arrrrrrrrrrrrrrrr
Is it TalkLikeAPirate Day already?
This last year and dragged and whooshed by at the same time

4 Likes

These game-time decisions are always hard. Thanks for continuing to push forward! Also, lots of appreciation to @bzee, @mav, and @scorch! Community dev involvement is mark of a thriving project.

Here’s the tweet for this week’s update, so you can like, retweet and comment! Your support makes a difference. I’m building a dashboard that showcases how impactful your support has been in increasing Safe Network awareness and engagement on Twitter! More to come :smile:

Looking forward to sailing into a testnet next week!

:sotros25:

23 Likes