SAFE Network Dev Update - February 6, 2020

Summary

Here are some of the main things to highlight this week:

  • We released the first stable MaidSafe.SafeApp NuGet package built using the safe-api library :tada:
  • We made a new SAFE Browser Alpha release.
  • We made some significant simplifications and corrections to the code in both the Routing and quic-p2p libraries.
  • Vault Phase 2 testing continues, with a testnet in the wild edging closer.

Project Boards

This week we decided to remove the 3 shared project boards that we had on GitHub but which were a couple of months out of date (thanks to @Cryptoskeptic and @Antifragile for bringing this to our attention). We track our work each day through each individual repository’s project board(s) and although it is undoubtedly useful having shared boards where the progress from each individual board can be aggregated together, in practice it has become a maintenance overhead since we reduced our team size, and ultimately was left unmaintained and therefore misleading. We want to do everything we can to avoid distractions as we power forward. Where applicable, each section in each dev update should have a link to a project board, and we will work to keep those boards up to date to match our written progress updates.

Vaults Phase 2

Project plan

We’ve continued with Vaults testing and we’ve been making some interesting observations. One thing that we’ve observed was that the memory usage might be because of the holding of the PARSEC graph in memory. With this in mind, we are looking at different options to optimize the storage of this graph by potentially writing it to disk, or pruning it more often than we do right now. We are analysing various parameters before we take a step in this direction. Another observation that we’ve made was that there is a considerable spike in memory usage when a node goes offline. The Routing library attempts to retry sending the message a number of times before declaring it as ‘Failed to send’ and we’re looking at possible improvements that could be made here to avoid the memory spike. With these new observations we’re looking at a potential testnet (with certain limitations) so we could test the different components working together out in the wild. This has spurred a lot of internal communication and we are discussing ideas for re-ordering some work to allow us to iterate testnets quicker and with a clearer purpose in each iteration.

SAFE API

Project plan

We made some minor tweaks/fixes to our CI scripts as there were development builds required for mobile testing which weren’t being published on S3. These artifacts are not needed for users, only for mobile development and testing.

After that we invested most of our time analysing how to move forward with safe-api and CLI in terms of how they fit in the next milestones and plans, e.g. Fleming and beyond. We are still trying to finalise the plan but we seem to be moving towards having the new data types implemented end-to-end (E2E), i.e. from vaults to CLI, making sure the same use cases and scenarios still work as they do now with current safe-api and CLI features. Just to clarify, this is in parallel to the efforts and code base for the Fleming release, and we therefore will start to work on separate branches to bring in the new data types implementation in the corresponding repos, including safe-api.

We are also starting to run some E2E tests using the CLI test suite against a single-section network, lending a hand for the testing process already underway for Vaults Phase-2 releases. We are hoping that this can help in the process of troubleshooting and making sure that it’s stable enough for sharing with the community, as well as getting a better picture of how a single-section network is behaving with the CLI E2E tests and use cases we’ve been playing with during the last few months.

Labelled Data, Indexing and Token Authorisation

RFC, Project Plan

We’ve been firing deeper into tokens this last week, making good headway into storing tokens for reissue (which involves changes across the authenticator client), thus enabling already registered apps to update their token if there have been any permission changes. We’ve also been updating our mock test implementations to start requiring valid tokens from app connections. So right now we’re progressing through the tests, identifying where those tests are lacking token generation or need tweaking.

Data Types Refinement

RFC, Project plan

The work on this RFC keeps going on at a slower pace, in parallel to other tasks with higher priority for Fleming.

Some changes that came up during review of the Sequence PR introduced regressions, that were eventually solved. Now this branch is just waiting for merge. Next up will be converting safe_client_libs to handle the Sequence type, for running more tests. After that it’s safe-api and safe_vault, and when Sequence has been implemented and tested end-to-end, we’ll continue with Map in safe-nd.

Discussions in the Data Types RFC forum topic on resource usage for different use cases of Map in Private scope, led to some new ideas proposed by @tfa for solving this.
Additionally, in collaboration with @tfa, a new useful feature implementation for Private Map was modelled, which would allow resetting a key history back to an earlier version.

SAFE Network App

Feature Tracker Board

We’ve continued to develop the interactions around setting permissions for individual files, folders, and apps.

Now with the power of Labelled Data, we are able to extend this further, giving the option to users to set separate permissions for what an app can do with its own data — and the data you create with it — and permissions that apps are granted over the rest of your data.

This ring-fencing of data should allow a lighter touch to authorisation for new apps to get up and running, and provide a smoother experience, while still keeping a handle on data security. It has meant reworking some interfaces, and amending the approach for default permissions for new apps too, but worth a little extra investment for much more flexibility.

Along with labels comes the opportunity for ‘smart folders’ too. Not only a way of grouping data within a file management view without a rigid tree structure, but also a way to further enhance permissions as well.

For example, I might want to exclude all location data from sharing or publishing, or be asked to double-check before any photo can be shared, regardless of where these data reside or what app was used to create them.

There’s a lot of ways to cut and order this data, which means a nuanced and sometimes challenging flow to design, but good strides have been made and we’ll have more pixels for you to peep at soon!

SAFE Browser (desktop)

Project Plan

At the end of last week, we finally fixed a long standing issue in the browser which was blocking @anon57419684. You can now use the browser fetch API to request files directly from SAFE via the standard JavaScript (i.e., not safe) APIs.

You can check this out with the latest SAFE Browser Alpha release which we subsequently released early this week containing this latest fix, along with a safe-nodejs version bump, and various other dependency updates.

SAFE Browser (mobile)

Project Plan

This week we continued with the thorough testing of the mobile Browser app. We were able to fix multiple issues found and make some suggested improvements along the way. For example, the mobile Browser now shows a list of all the files from a files container if an index.html file doesn’t exist in the targeted files container - this replicates the behaviour on the desktop Browser and is particularly noticeable on sites where some versions have an index.html, while others don’t. The fixes include enabling the user to connect to a different vault after re-authenticating without having to completely close down the browser, and hiding the version change navigation buttons when any error page is displayed on the screen.

We are down to one or 2 final issues here, specifically around compatibility with older versions of Android and iOS which we would ideally like to support where possible. We will continue working on this into next week and see what we can come up with.

SAFE Authenticator (mobile)

Project Plan

This week our testing of the mobile Authenticator app found some inconsistent behavior where apps were retaining the new test coin permissions when revoked and re-authenticated. The issue was traced to the safe_authenticator lib in safe_client_libs and a fix was implemented by the SCL team. As a result, we updated the native libs that the Authenticator app was using to the latest version. We recently resolved another issue found during testing to prevent soft keyboard from overlapping input fields on small screen devices.

As with the SAFE Mobile Browser, we are down to our final couple of issues here. The main issue we want to resolve is also around compatibility with older versions of Android and iOS, so we will continue to work towards finding the best resolution for this.

SAFE App C#

Project plan

For the last couple of months we have been seeing frequent changes in the safe-api and to keep up with that we had released 3 NuGet Release Candidate packages. Since the APIs are more stable now, this week we released the first stable MaidSafe.SafeApp NuGet package built using the safe-api library :tada:. This release contains all the changes from all previous RC packages. To learn more about the changes, please check the complete changelog. It’ll be great to see you developing some cool apps using this new package and if you come across any problems, please open an issue in the GitHub repo.

Safecoin / Farming

RFC

This week we made a quick prototype of bounded counter (bcounter) CRDT in an interpreted language to get a deeper understanding of it, and wrote up a set of notes based on that experience.

There is a potential for data loss when a bcounter replica goes offline before syncing its latest changes to peers. In typical data center usage, this is not a big issue because they can make each data center a logical replica with multiple physical nodes for redundancy. For SAFE Network usage, an alternative approach must be taken. Additionally there are some complexities when replicas are added or removed from the group that must be handled. Work on these issues begins.

On the DBC front, a second draft design document was written based on discussions from the first one. A primary idea of this hybrid system is that DBCs could:

  1. be stored in a user’s wallet and sent around the network like a cashier’s check so only the recipient can deposit it. – or –
  2. be exported outside the system as a true bearer certificate that anyone can redeem/spend, more like cash or a gift card.

Routing and quic-p2p

Routing Project Plan

This week we have made some significant simplifications and corrections to the code in both libraries. This work is ongoing at some pace as these improvements are pre-release cleanups. The simplifications include:

  1. Remove requirement of getting certificates from quic-p2p and passing these back. This saves us handling mappings of certificates in Routing and allows code to be removed. This removes state that is not required and will aid restarts and upgrades. Note, that authentication of nodes (vaults) is already handled in routing via cryptographic keys. That means for nodes, SAFE already acts as a valid certificate authority from the perspective of authenticity and non-refutability. SAFE also handles authenticity revocations and does so extremely efficiently. Therefore all we need from TLS 1.3 and quic is secured and encrypted communications, but not authenticity.
  2. Handling connections in a more robust manner. This gets us away from seeing a connection drop as a failure and instead always try to connect to detect NotAvailable nodes. This will aid restarts and upgrades soon.
  3. Message unification has allowed us to clean up the message infrastructure significantly which helps security and code review. There is still a little to do here when we clean up the connection handling and responsiveness checks.
  4. Flattening the network structure to allow sections to communicate directly (via the section Elders). This will reduce hops considerably and will allow faster initial networks where we can introduce hops as the network grows. This also gives us a bit more information should any partition happen on a huge scale (such as a country cutting the Internet at the IP level and so on). The process here allows simpler code and the ability for sections to catch up as messages traverse the network (i.e. sections update in the background).

We are also looking at a robust non-responsive check for nodes. This allows agreement quickly on nodes that are not actively participating (closing an attack vector).

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

82 Likes

Almost first like. :slight_smile:
EDIT: But apparently first post!

22 Likes

nice update, feels like a very productive week really

23 Likes

That is awesome.

23 Likes

Music to my ears!!! Glad to see the continued progress :smile:

28 Likes

ya hopefully these testnets can help people on the outside gage progress. Right now every update reads to my non technical mind like “we are working on everything and making progress!” Maybe something a little more hands on would show me where we stand though a demonstration.

12 Likes

Yeah, I second that, to me as an outsider it doesn’t really mean much

3 Likes

Thanks so much to the entire team for all of your hard work!

18 Likes

This is the kind of lines that turn me on ! Great update again, thank you Maidsafe team for the continued efforts week after week. This must be what ‘commitment’ means.

20 Likes

Top 10! Now to read …

Beautiful work as always, guys—great to see the progress made on some of these projects!

11 Likes

This is very nice!

Good stuff, @Cryptoskeptic and @Antifragile!

Thank you for your persistence, @tfa!

And of course, a big Thank You to the whole official team!

20 Likes

You can sense David’s ever more and more experienced, applied ingenuity in his direction of separating Network functions, not doubling up on performance where none is needed—all while simultaneously knowing when to limit refactoring for these individual parts. I love Thursdays.

15 Likes

This is music to my ears

19 Likes

Music to my ears! Thanks for all your hard work, team!

Edit: just read comments and it looks like people like this tune! :grin:

19 Likes

Seems to me Jams could make a playlist from this update :musical_note: :drum: :rofl:

19 Likes

I don’t have a clue what it all means but the comments above are excited, so I am too​:grin::grin::grin:

10 Likes

Great update as usual. A small interrogation though:

Currently vault config file uses a network config from routing which is a reexport of quic-p2p config which contains a certificate. Then certificate of each of our peers is provided in its NodeInfo structure.

So, what does this mean?

  • peer certificates will not be available to safe_vault anymore

  • or they will be still available, but there will no code using it

I ask this question because a network name was managed in the past to avoid accidental insertion of a vault in the wrong network. Such a name is not managed anymore, but could be stored in the certificate. Another use case could be the opportunity for vaults to store complementary data in this certificate, like a vault name.

11 Likes

Weekly update already? Seems like I just read last weeks a couple of days ago!

As always great developments. I’m particularly interested in the DBC work. I know peeps said in previous comments of previous updates not to waste any effort making safecoin a DBC, but I think I’m in the other camp on this one. If we’re going to have DBC’s, then why not make safecoin one? Don’t mean to dredge up a debate here - ATM it seems DBC’s are still in the early research phase, but if it get’s past that point before Fleming, then I hope it will be considered seriously - it’s just too awesome a tool I reckon and would one up all other crypto’s by an additional order of magnitude … so why the **** not!?

Thanks for all the hard work team Maidsafe! :star_struck:

13 Likes

To extend a bit on this (originally I had a question but these links covered it)

From RFC0056 Secure Message Delivery: “In this RFC we make no distinction between close sections (neighbours) authority and remote sections. They all work in the same manner as far as section proofs go. However neighbours connect to each other and this requires neighbours know who makes up the Elder group in addition to recognising the aggregate public key of an elder group. The other difference for neighbours is that we push notifications of membership changes to them.”

This routing github issue: Have no neighbours, only local section and remote sections

10 Likes

There is also wild update on Routing Issue board https://github.com/maidsafe/routing/issues

Node Ageing Closed
Secure Message Delivery (SMD) Improvements Closed
fleming Open

With added all issues from these two projects and some new :grinning:

15 Likes