SAFE Network Dev Update - March 21, 2019

Summary

Here are some of the main things to highlight this week:

  • We released a video with @scoyle as part of our series of interviews with team members.
  • A patch release of the SAFE Authenticator mobile application earlier this week. It includes a fix for the bug regarding support on lower platform versions.
  • Several new pages describing our foreign function interface were added to the Client Libs documentation. We recommend starting with the FFI overview and expanding from there.
  • Community member @riddim made a forum post about the Python interface for the SAFE App API he’s currently working on.
  • The Fleming team published a new forum post about Sybil resilience simulations.

Marketing

First off, we’d like to say thanks to Johnathan & Louise from Cryptonomy for organising another edition of the SAFE Network: Brighton meetup again last week. It was great to meet @john_ordinary, @moose99 and many others for a fun evening of wide-ranging chat about technology and possibilities. Here’s a couple of photos from the evening:

@dgeddes has continued this week exploring feedback on the DevHub and incremental changes will be made to improve the site over the next couple of weeks. We’ve also released a new video today with @scoyle, our Testing & Release Manager. Thanks again for all the great feedback on last week’s video. These videos are very much intended to be quick snapshots that give you some insight into the team and daily goings on with minimal prep time, etc. The more we do them, the better they’ll become so please feel free to suggest any people you’d like to hear from or content that might suit this quick format and style.

SAFE API & Apps

A while ago, community member @ogrebgr suggested a Gradle plugin that could possibly improve the build configuration of the safe_app_java project. We gave this a shot and it helped improve our build script massively :tada:. The changes have been merged to the master branch in this pull request. The build instructions for the Java desktop API have also been updated.

We published a patch release of the SAFE Authenticator mobile application earlier this week. It includes a fix for the bug regarding support on lower platform versions. The issue was with the way the native libraries used by the Authenticator were built from safe_client_libs. We’ve updated the native libraries and fixed a few other minor bugs. For more information have a look at the changelog.

We are still in the internal testing phase for our next release of the SAFE Browser (v0.11.3) and are currently working on the fixes for some issues discovered during this phase which we consider should be part of this release. We also continue with the efforts related to enhancing our browser CI by using TestCafe for our e2e tests. We are still in the early stage of this enhancement but we believe it may bring more stability to our CI testing process, we’ll share more details once we confirm this is effectively the way forward.

We’re incrementally transitioning to writing the browser in TypeScript, with the goal of cutting down on time-consuming bugs that are easily caught with static typing.

Currently, SAFE Browser forwards HTTP requests to the system’s default browser, which will soon no longer be the case, in order to prevent a potential breach of a user’s anonymity. This issue is being used to discuss how we might allow certain links to the clearnet.

As a minor addition to some of our tools/applications, we have updated the toolbox at safe://toolbox.dapp with a new tab titled Generate MD XOR-URL. It allows to obtain the XOR-URL for a MutableData by entering its XOR name (in hex format without 0x prefix, e.g. 8f7fdd831e0ef35eb7f46965b8b0915e636522a0ffda8c73261983b50188289d) and its type tag.

SAFE Client Libs

This week we continued with the Client Libs documentation, releasing several new pages describing our foreign function interface, or FFI for short. We recommend starting with the FFI overview and expanding from there. While working on these pages, we noticed several aspects of our FFI strategy which could be improved, so we are in the process of raising and reviewing a couple of pull requests to address these shortcomings. One of these PRs adds integration tests to FFI Utils, which we also used as simple examples in the docs. All in all, these efforts should make our FFI easier to understand and work with for others, especially since the behavior we expected from the FFI was quite subtle (as most of it is unsafe code and exempt from the usual strictness imposed by Rust). We want to close by recognizing @riddim’s awesome work in creating a Python interface for the SAFE App API. Great job!

Routing

This week the Fleming team published a new forum post about Sybil resilience simulations with thanks to our brilliant marketing team for the help in bringing it about. We are very happy with the valuable feedback the community is already providing and would like to thank you all for your contributions.

We also continued with our focus on node ageing and the impact in the different parts of the system. In particular, we analysed more in-depth the relocation flow, and how to further improve robustness with PARSEC to support node ageing. With so many moving pieces, we are looking at ways to validate the flow and improve it to weed out any undesired situations. We also spent some time to summarise conclusions from the many Sybil discussions that happened on the forum to keep all these insights close to our mind.

On the Parsec team, we ironed out a plan for the remaining tasks that we want to complete for Fleming, and the ones we will postpone. In particular, we decided to postpone improvements to the Parsec documentation, and some test coverage for code we are confident about. In addition, we will postpone some unprovable malice detection and any performance work since we are now comfortable with our results.

We are now focussing on some of the remaining malice detection tasks, and bug fixing. Of particular note, we are progressing on our malice detection protecting the concrete coin. We are also improving our tests and testing helpers for malicious behaviours, then we will verify performance is not impacted by our malice code.

We also implemented a mock-parsec in the fleming branch of our Routing repo. With this, we are getting ready to move more effort to the Fleming work. Parsec, while performant for the system, is not suitable for Routing unit and functional tests. This will allow us to develop the needed code and run soak tests effectively.

Crust

We have now coded almost the entirety of Crust using QUIC from the quinn Rust crate. It was a pretty mad week as we wanted to test as quickly as possible what QUIC brings to the table for us. We know that it should allow us many many connections over a single UDP socket thus not exhausting our open-socket-descriptor limits. This will help the Fleming milestone immensely if it works as it allows us to have big section sizes while not doing the difficult work of dynamic connection management which we would need if we used TCP. Some of the commit messages describe in depth what has been done, for e.g., this and this. The last one is especially useful as it prevents us from handling message chunking for the purpose of interleaving. Using the crate we can now send a large message followed by several smaller ones and we can see that there is no head-of-queue blocking as with single streams. We fire each application-level message on a separate stream and quinn/QUIC ensures that all the streams share the bandwidth fairly. So the smaller messages sent later arrive quicker than the huge message sent earlier. Keepalives are built into the protocol so we don’t need to write user code for that other than setting up the parameters.

However, the biggest benefit would be the industry standard TLS. QUIC works with peer-certificates and we don’t have to do custom crypto handshakes and secure-serialisation any more in Crust. QUIC will take care of it and is also easier to explain to others - all we need to do is point to QUIC and TLS instead of explaining our custom secure-serialisation.

With all that, we will now build a simple application on top of it and spawn 1000s of nodes and check for the stability of the underlying library and monitor parameters important to us. There were a few bugs in the underlying quinn library and issues raised like this one for instance. So far the maintainers are pretty responsive and also quick in merging our PRs to the quinn repo.

83 Likes

first… 20 charachters

edit: will now read update

14 Likes

Second poster? :smiley:

14 Likes

How tolerant is the network design of application-level messages arriving out-of-order?

That’s convenient.

17 Likes

Holy Moly was that QUIC!!! See what I did there? :wink:
This sounds pretty promising and interesting. Is it going to be the long term solution or isTCP dynamic connections something left for further down in SAFE’s maturity?

Great update as usual @maidsafe! Keep up the fascinating achievements!

22 Likes

aah, that’s music to my ears :slight_smile:

15 Likes

jigsaw falling into place

13 Likes

Geat update team! I love this intelligent community with all of it’s great input and ideas, and constructive criticisms.

18 Likes

Thanks so much to all of the team members for all of their hard work!

8 Likes

From the readme:

Are we sure we want this at the CORE tech of MaidSafe if its currently:
“suitable for experimental use” ?

I have heard of the QUIC protocol a bit in the API space and understand its kinda branded as the up and coming HTTP3 proto essentially, I suppose only testing at huge scale with really good metrics capturing will reveal issues. I guess to become one of the best networks we need to push the envelope of bleeding edge :slight_smile: .

3 Likes

Nice video interview @scoyle and @cgray (and well done on the noddy control :grin: ). The setting with the two chairs and the triffid looks much more relaxed I think, and you can’t see people wandering past the window. Good luck with the Jenkins rollout - I understand it’s a bit of a beast.

17 Likes

Great work once again, and nice shoutout for PySafe! We’ve been working hard this week as well on moving rid’s pioneering investigations into a library. We now have a branch in my fork/dev with a proof of concept for a complete abstraction of the library with a working authenticator object that only uses python types. Linux and buggy Windows support are in. SAFE will be on python!

20 Likes

aaaah - haha - didn’t expect the mentioning of pySafe :smiley:
thought i’d highlight that @dask is amazing at structuring the project and with his experience and excellent ideas i’m positive about the future of pysafe :slight_smile:

the structure is already way nicer than in the proof of concept i presented not long ago :smiley:

23 Likes

Great progress Maidsafe devs + marketing team,

Really like the SAFE Buzz :+1:

There might be a link to a malware site :scream:on clearnet as well, beside a potential breach of a user’s anonymity. See it as a link in an email on clearnet people are so addicted to clicking, only way to fix it is to link nothing to clearnet. If it’s not on the SAFE Network it doesn’t exist… Not everything can be a novelty preserving engine.… just my 2cents

:stuck_out_tongue:

1 Like

Definitely would need to gain permission from the user with a suitable warning.

  • Firstly this alerts the user that its not a SAFE site
  • Secondly makes the use of this by scammers less effective and thus discourage them a little or maybe even a lot.

Thus the 2 working together should reduce its use by scammers and discourage SAFE sites from using it too.

8 Likes

It could be useful to have an option for users to add domain names to a whitelist in settings, so these domain names then could be cliked without the warning being displayed. Should have to go to settings to add whitelisted domains though and not just add a checkbox on an alert popup, that would make it too easy to trick users.

10 Likes

Agreed. Another option I’d like to have is the ability to review the clearnet link and either manually copy it or edit it before opening.

That’s already listed as a consideration:image
Great minds and all that!

6 Likes

Great update as usual. I found the crust rewrite quite interesting! I am guessing that Quinn will simplify the crust code a bit - less work for Maidsafe team (long run) in addition to allowing lots of connections. I watched the presentation on Quinn - was interesting and the gentleman behind it seems very dedicated.

The presentation on Quinn (now a bit dated as Quinn has merged with Quicr): RustFest Paris 2018 - A QUIC future in Rust by Dirkjan Ochtman - YouTube

6 Likes

That’s a really good move! It is so much SAFER to have static type checking! :blush:
I also really like the new SAFE Buzz! It gives us really interesting insights into the inner workings of MaidSafe.

Thanks, @maidsafe, for this amazing update! You always achieve so much in one week! I can see Fleming on the horizon! :rainbow:

10 Likes