SAFE Network Dev Update - December 19, 2019

Summary

Here are some of the main things to highlight this week:

Happy new decade!

As we approach the end of 2019 on a high, we at MaidSafe want to wish everyone a happy holiday in the coming weeks. It has been a year of change for us, from the start of the year and all of the crypto price shocks, then of course our own team shocks with some people leaving all of a sudden. The culmination of the changes though has left us probably where we should have been several years ago. We are predominantly Engineers working to release a world-changing network.

Getting back to a single team and more than that, a focused, dedicated team has a lot of very nice benefits. We are more focussed on the whole network, we can speak more equally and openly and most of all we can have a bit of fun in meetings. Less bureaucratic and zero politics means we can just focus 100% on what matters, SAFE. As a goal that is a pretty big effort and that alone is our only target now. Not how we market it, not how we manage departments or have interdepartmental meetings and so on, but how we give the world SAFE.

There is a renewed energy and the team are really making efforts to self manage and that works a treat. The new year should be exciting for our team and we hope that excitement and increased focus continues to spill out into the community.

This is the last dev update of the decade and we will not have another until Jan 9th. So again, have a great new decade and we cannot thank you all enough for the support, it gives us all the energy we relish.

Vaults Phase 2

Project plan

With the new handshake protocol implemented & merged in Vaults today, we have crossed most of the Tā€™s with single-section Vaults, and we are excited to kickstart integration testing of multiple Vaults with clients. We are happy to see some initial promising results, with 3/4 of client tests passing in local dev environments. We are continuing to investigate the failures and we are aiming to begin internal tests at a larger scale soon.

The updating of Vaultsā€™ integration tests using real routing and a vault network has also been merged today. With this update, the code quality can now be further secured by a test framework with more coverage.

At the same time, @lionel.faber and @yogesh have been documenting preliminary designs for Phase 2b (the many-section Vaults project), starting with writing documentation for existing request flows. With that, we should improve our understanding of the scope of work necessary to be done in Vaults to support network communication across multiple sections.

We have also started to look into upgrading quic-p2p to the latest version of Quinn, the underlying QUIC protocol library we use for our networking layer. This upgrade should improve stability, conformity with the latest drafts of the QUIC protocol standard, and importantly it will also bring async/await code to quic-p2p. Async/await is a new syntactic feature that has recently been stabilised in Rust 1.39. It allows us to write fast and efficient code in a streamlined way. Although we are not making a full transition to the new async/await style yet, we have seen some significant improvements on the code clarity in quic-p2p. This shift should result in reducing cognitive complexity, making it easier to contribute code to our projects. Eventually, this change in quic-p2p would also allow us to start propagating this code style to SAFE Client Libs and further.

SAFE API

Project plan

During this last week, there have been several PRs merged to improve and simplify our CI process for the safe-api repo. After our move to GitHub Actions, completely removing Jenkins and Travis, we had to finalise and clean up some of the CI scripts. The most important aspect of these tasks was related to making sure we can still generate safe-ffi libraries with each PR merged as thatā€™s actively used by us for developing the language bindings and mobile applications.

A minor feature was added to the connect_app API in safe-api to allow applications to connect to the network as an unregistered app, i.e. connect with read-only permissions if the authenticator hasnā€™t given it credentials for a registered connection. This is a common scenario in mobile applications now, where the applications always request connection details to the authenticator, for both registered and unregistered connections to the network. Desktop applications, on the contrary, donā€™t need this as they currently obtain the vault connection information from a local path, but in the future some desktop applications may opt-in to requesting the authenticator for read-only connection information as well.

We were able to finalise the implementation for generating XOR-URLs for files without uploading them onto the network, both the implementation in safe_client_libs and also the one to expose such a feature from safe-api and safe-cli. A new $ safe xorurl command is being added to the CLI also, we are just writing a couple of additional unit tests and updating the CLI User Guide before merging the corresponding PRs which will enable this feature.

As for the next steps in the safe-api repo for the coming days, it will all be around making the safe-authd upgradable from the CLI, as you know with $ safe update we can upgrade safe-cli to latest version released, thus weā€™ll be working on exposing a command which allows us to also upgrade safe-authd in an analogous way.

In the next couple of weeks, we may also try to change the way the source and destination arguments are treated by the files put/files sync commands as there has been a lot of feedback and comments around it for some time now, so itā€™s probably a good moment to tackle this down. Thus, please join us in the discussions if you havenā€™t already, to help us find the best approach, either on this GitHub issue or on this forum thread.

Labelled Data, Indexing and Token Authorisation

RFC

Weā€™ve been talking over and around the labels pre-RFC this last week, and have been settling into a BLS-based bearer-token approach inspired by macaroons, though weā€™ll be going off-piste due to limitations in the way macaroons are verified (which is not suitable for our distributed verification steps).

The crux of the idea is that you are provided with a signed token, which can contain various restrictions on its use (for certain labels, during a certain time, for only these URLsā€¦ this gives us a lot of flexibility), and can be validated both at ClientHandlers against the request an application makes, and at the data handlers, against the permissions of the data itself (including labels).

Working with BLS in this way, we can still enable shared data, and are trying to expand the token-based approach so we might be able to pass tokens themselves for data-sharing (and so enable us to say, ā€œyou can access this data, but only for the next 2 daysā€, for example).

For those interested in the technicalities of this, please do head on over to the RFC, read, comment and question! Everything helps :bow: !

SAFE Client Libs

Project plan

Last week we released new versions of all the SCL crates. With that out of the way, weā€™ve been working on some final clean up tasks before heading back to help out with the Vaults project. We will also continue to support any changes required for frontend apps with bug fixes and new features. @marcin worked on an exciting new functionality in SAFE App, adding support for the GET_NEXT_VERSION constant in more places. Details can be found in the corresponding issue, and the PR is under review.

On the infrastructure side of things, weā€™ve ironed out the bugs in our automated releases and deployments, which we added recently to GitHub Actions to expedite the release process. We also added caching to GHA, which in some cases results in a 50% reduction in build times. We are also in the final stages of fully removing Docker from SAFE CLI. Our next goals in GHA are to reduce the verbosity (possibly by implementing our own library of custom actions) and then to roll out the changes to all of our repos, replacing Travis and Appveyor across the board.

SAFE Network App (mobile)

Project plan

This week we covered some more ground on the styling & theming front and improved the UI for both platforms (Android and iOS). We focused on bringing more styles from the design to the app and the UI is shaping up nicely. You can quickly check these two PRs (#29, #28) for a nice preview.

Last week we added a small enhancement to check if the mobile browser is installed or not. This week we built over that logic and now we are storing the result in the app preferences. Using this we can put browser-related conditional controls/UI anywhere in the app. For example, if the browser is installed you can open it from the SAFE Network App, otherwise you can open the download page.

SAFE Browser (desktop)

Thereā€™s been a wee fix added to the browser this last week, updating the SAFE Network App auth flow, which was preventing the browser itself from authenticating on the network (which was causing a failure when trying to create new sites via the browser).

Weā€™ve also fired in with a swathe of dependency updates, some of which made it into a new beta release which you can download now.

Weā€™ve also started the upgrade to Electron 7, which involved updating our TestCafe version (and a wee PR to their repo so we can stay abreast with their latest changes). Though weā€™ve hit a blocker in Electron, where a bug is erroneously throwing an error when non-200 code responses are received (meaning our useful error pages are never being shown). It seems like thereā€™s a fix for that already in the pipeline, just not merged into a new Electron 7.x release, so as soon as we have that, we can get the browser up to the latest release.

SAFE Browser (mobile)

For the last couple of weeks, we have been facing a crash issue on iOS devices, while the Android version was working fine. This crash issue has been resolved this week by updating to the latest preview package. We updated the app to use some new APIs and refactored the code accordingly. With the app debugging crash problem also resolved (more details in the SAFE App C# section below), we can now move forward and continue with the tasks to provide initial pWeb features on mobile.

SAFE App C#

Today we released a new MaidSafe.SafeApp preview NuGet package :tada:. This package includes multiple fixes, new APIs and some refactoring changes.

In the new package, we fixed an issue related to the iOS native libs which was causing an app crash on iPhone simulators and physical devices. Because of this issue we were not able to debug any of our iOS apps. On the refactoring side, we removed all the deprecated SAFE Client Libs APIs from the package as well as removing some unused dependencies.

We also added and tested the new AuthAppAsync API to improve the development experience. Devs can use this API in their C# desktop apps and the users will be able to authenticate the apps with safe-cli and the authenticator daemon. In the coming weeks, we will improve this API to provide the same feature on mobile devices as well.

Node Ageing

Project plan

This week we closed the big PR we had been working on for some time. With that initial work for promotion/demotion now merged, we have the main components of Node Ageing: A fixed size for the section Elders, and remaining nodes as Adults.

Nodes can now be promoted and demoted between Adults and Elders as needed because of Relocation, node dropping, new node joining or a section splitting. This work was essential for a number of reasons. For Reliable/Secure message delivery we now use a constant number of redundant messages. For Parsec, performance is constant. Finally, it is the key to our Sybil resistance since only nodes that have done the most work can participate in decision making.

A significant number of changes had to be merged to support this work over time. We have been further building on this main change to complete the remaining items to finalise promotion and demotion.

Useful Links


Feel free to send us translations of this Dev Update and weā€™ll list them here:

As an open source project, weā€™re always looking for feedback, comments and community contributions - so donā€™t be shy, join in and letā€™s create the SAFE Network together :tada:

82 Likes

first,.

19 Likes

Second. See you all next year

16 Likes

The third one.

14 Likes

Happy Holidays and I hope you all get a decent break. :christmas_tree:

22 Likes

Just finished watching this. Now thinking the SAFE dev team should feel important as R&D4 in the video. Vaults, Routing, Consensus, Economy, UX, and Keepā€™On Troopas! (Theyā€™re the protagonists, right?)

13 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! Make sure to enjoy your holidays and get some rest and have some fun. :beach_umbrella: :desert_island: :tokyo_tower: :roller_coaster: :ferris_wheel: :sunrise_over_mountains: :cityscape:

20 Likes

Wow, congrats on Vaults and Node Aging progress! :+1:

As always, thanks for the update. :clap:

Have a great holiday Maidsafe team! :smiley:

Cheers :beers:

18 Likes

As every week the ants have been working hard! Happy Holidays and enjoy the well deserved rest!
:heart:

14 Likes

:star_struck: Great work! I canā€™t wait to see this on a test net soon!

Thanks everyone for all of their hard work and dedication over a difficult year in the crypto space. Youā€™re delivering something special, which the world desperately needs. It is thoroughly appreciated!

24 Likes

B(e)ith * ā† Ayrshire joke

Thanks for all your hard work, all of the team, past and present and heres to the SAFE decade ā€“ and many more of them

  • base16
16 Likes

:christmas_tree: Merry Xmas :+1:

14 Likes

Thanks for a great year team Maidsafe. :clap:

Enjoy your break and a happy new year to you and everyone in this great community. :champagne: to

Hereā€™s to SAFE Network 2020!

25 Likes

Real innovation isnā€™t easy. Difficulties will swipe at you, naysayers will sneer, and the noise from the myriad me-tooā€™s will threaten to drown you. But keep on fighting anyway. The breakthrough is coming. You know it.

Thanks for all your hard work. Happy holidays to all.

23 Likes

Live reaction to @Southsideā€™s Ayrshire joke from somewhere outside of Ayrshire.

17 Likes

My work is done :slight_smile:

top centre(ish)

EDIT: If you all wait long enough I will find a way to bring Maidens into the sausage network threadā€¦

10 Likes

Thanks @maidsafe for an amazing and eventful year :relaxed:
See yā€™all in 2020

14 Likes

I too wish you all the best in 2020! Thank you team.

And Vaults and Node Aging progress is amazing!

11 Likes

Thank you for Everything Maidsafe Team!

Merry X-:evergreen_tree: and a Happy New Year :star2:

12 Likes

Thank you for your continuing effort and :man_technologist: :arrow_right: :christmas_tree: :partying_face: :spiral_calendar: :checkered_flag:
Ps Rust 1.40 is just out.

16 Likes