SAFE Network Dev Update - July 2, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • v0.4.0 of the MaidSafe.SafeApp NuGet package has been released, bringing the C# APIs in sync with safe-api.
  • We released a 1.0 version of the xor-name crate, which was recently extracted from Routing.
  • The Sequence CRDT is now considered ready for inclusion in the next iteration of the vaults-from-home testnet.
  • After another week of great progress, we are very much closer to an integrated AT2 implementation between SCL and Vault.
  • Some further SAFE Network App UX updates and details are shared via Figma.
  • Support for storing Private Sequence has been added and appending data to it using the API and the CLI. This will be available in the next release.

Vaults Phase 2

Project plan

We’ve been making progress with a number of improvements in Vaults. The introduction of signature accumulation in Vaults (previously done internally by Routing) has reduced the amount of memory used by the Vault binary since it can now directly validate requests approved by the section. We have already seen good results with a bare-bones signature accumulator that was implemented last week. We have now migrated this to use a generic signature accumulator exposed by Routing. There are a couple of failing tests that need to be fixed before this update is pushed to master.

We are also making a few changes to the test suite in Vaults for continuous integration and testing with the real version of Routing instead of using the mock version. This will give us a more realistic measure of performance when compared to the tests run in a mocked environment that is useful during development iterations. This shift away from testing mocks, towards testing production code is something we have been discussing as a team this week - this is a strategy we intend to apply, where possible and practical, across all our repos.

SAFE Network App UX

SAFE Network App: Screens, Flows and Feature Tracker (Figma)

A short update, but plenty to look at: we’ve added more screens and details to the Figma file. You’ll notice detail on the Settings Screens, we’ve laid out the Invite creation process, and we’ve also added to the Security options during onboarding.

SAFE API

Project plan

Experimental glob support, as discussed in last week’s update, has been implemented this week, though some final integration test cases are still to be written before submitting a PR.

We also noticed some strange behaviour with symlinks on Windows CI which merited further investigation - this turned out to be down to us having to jump through the hoop of configuring Git for symlinks on Windows CI boxes. You can read a bit more about @danda’s findings when digging into this here.

During the last few days we were able to also add support for storing Private Sequence and appending data to it using the API and the CLI. For example, a new Private Sequence can be stored on the network with $ safe seq store "my private note" --private, the command for appending to it is exactly the same as the one for appending to a Public Sequence. We’ve updated the CLI User Guide with an example of this.

SAFE App C#

Project plan

We released a new version of the MaidSafe.SafeApp NuGet package to bring the C# APIs in sync with safe-api. The updated package delivers a few new APIs (create/get/append) to work with Sequence Data APIs, and includes a symlink resolution option in the Files APIs. Check the changelog file to read about all the new features/enhancements.

CRDT

Last week we finished with the implementation of the Sequence CRDT, so over the last few days we have been putting it through its paces with E2E tests. Everything is looking good, so this is now considered ready for a first release so the community can start playing with it. This will be part of the very next iteration of the vaults-from-home testnet.

We’ve started looking at the options we have for migrating the existing MutableData type to CRDT, which is expected to support not only private data, but also a public version of it which was not available with current MutableData type. This will still be a key-value data type but implemented as a CRDT. We don’t have much more to share at this point but we’ll keep you up to date as we advance.

Transfers

SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan

This week we’ve been working on getting Vault code and SAFE Client Libs in sync. In doing so we’ve uncovered and fixed a few bugs (to do with section PublicKey handling, in Vaults, and some safe-transfers related bugs), and added more tests to the respective repos for the same. Replicas will now make use of SectionProofChain (a chain of old PublicKeys of a section) from Routing, to validate transfers signed with an earlier section key. We’ve also been rebasing code atop the latest and main repo code, and refactoring the SAFE Client Libs code into something much easier to digest.

There are still some small issues to be straightened out, but it feels like we’re very much closer to an integrated AT2 implementation between SCL and Vault.

After evaluating a few different alternatives for payment flow, we’ve picked one and started to align the Request structure to support it. This gives a better categorisation of the request types, which makes it clearer when, where and for what a payment is supposed to be handled. (Down the line there’s also a clean up of the request types, as we have types that are not necessary to implement on the network side.) After this is in place, we’ll be able to continue the work on the distributed AT2 Actor run by a Section, which then allows for the safe-farming repo to be integrated into safe-vault, and earning of rewards can be included in a testnet.

Routing

Project Plan

This week we merged the new SignatureAccumulator after addressing some additional requirements from the Vault team who originally requested it. We also ticked off a couple of tasks from the big remove parsec list which were mostly about removing some no longer needed components

The work on the DKG replacement is still ongoing, due to some unforeseen complications. But it seems like all test failures have been resolved now and most of the review points addressed, so it should not take much longer for it to be merged.

We also made some changes to the internal Message type to make it immutable and correct by construction which should hopefully make the code more resilient to accidental misuse.

We then fixed a rather embarrassing issue where the test suite would hang when run without any feature flags. This went unnoticed for a long time because we always run the tests with the mock flag. This came up in a discussion about the role of mocking in our tests, with the resolution agreed that we should try to use less of them, to increase our confidence in the tests.

Finally, we released a 1.0 version of the xor-name crate which was extracted from Routing two weeks ago, after polishing it up a bit. We even made the crate no-std compliant, which means it can now be used in the largest possible set of environments and platforms, including some very restricted embedded devices.

Useful Links


Feel free to send us translations of this dev update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

69 Likes

Exciting news! I think this will speed things up. We are getting closer all the time. Another great update team.

Also CRDT being part of the very next iteration of the vaults-from-home testnet sounds amazing.

24 Likes

Second will do and I’m grabbing it now because I want to tag onto the dev update. First though it is excruciating reading about progress in so many areas without being able to play - but encouraging too. Thank you team for keeping up the pace.

In the mean time, thank goodness for safe vault -t run-baby-fleming which I only discovered this week. I recommend giving it a go (see Run a local network) as it is a really easy way to explore and play with the SAFE CLI commands, which are very cool.

This enabled me to explore getting syncer to work with SAFE and to update safegit to work with the SAFE CLI.

Syncer

Syncer would provide a local disk on your PC which caches files locally, and so is fast, but saves everything to SAFE in the background as you work. This gives you effectively ‘infinite’ storage in a local directory, accessible at about the same speed as your local drive. More on the Syncer topic.

Syncer is written in Rust and uses ‘rsync’ to store and retrieve data. This is good because ‘rsync’ is similar to the SAFE CLI and I hope will be relatively easy to get working with SAFE.

I have syncer running and initialising its storage with SAFE using the CLI, but there are a couple of problems. These require some enhancements to the SAFE CLI, a bit beyond my Rust skills at the moment. I’ve raised an issue for one (#589) and given a +1 to another (#512), which should enable me to get syncer going so if you have Rust skills feel free to have a go. I am reading up on Rust and if I think I can do this I’ll have a go at one myself.

safegit

We have safegit live again. Updating this to use the SAFE CLI (instead of the old SAFE Drive which no longer works) was fairly easy. Most work was in updating the tutorial.

So we can again publish git repositories to SAFE, clone/fork them, make changes and offer pull requests back to the original all using the SAFE CLI and the safegit script. See:

39 Likes

:flushed: safe-farming repo you sayyy?

Great progress again team @maidsafe
You’re already grabbing attention with your choice to integrate at2. Once people can play with that it may well draw in much more attention.

Looking forward to the next testnet! :smiley:

24 Likes

Super exciting to hear about the state of CRDT, close integration of AT2, and shift to testing production-grade code. The next testnet is going to be amazing!!! :smile:

27 Likes

Exhausting just reading it!

Thanks for all the hard work everyone.

Sounds like you’re making great progress too @happybeing!

Everything seems to be coming together…

24 Likes

Thx for another great update Maidsafe devs.

jippy :clap: :clap: :clap:

@JimCollinson
Oooh keee some feedback
Imagine if invites are just like flyers, so instead of just having a + button that I press on to create invites everytime, I do this:
Create 100K accounts, provide each with 0.00001 SAFE

This actually leans more towards

I admit, it’s just me being lazy to press the + button for an invite every time, but an side effect would also being able to make mass payments. Hmmmmm maybe it would also be fun to claim the invite within a specified time period, if not then the token gets send back to the creator of the invites.

Enough out of me :stuck_out_tongue:

12 Likes

Thanks so much to all of the entire Maidsafe team for all of your hard work!

16 Likes

Very nice job @happybeing, no only in itself which is a great thing to have, but these type of developments will also help us (all of us here) in evolving the CLI in the correct direction. Many discussions are sometimes due to not realising the CLI is not just a user facing app, but also a tool (very powerful if we do it right) for other apps, scripts, backend systems, automation, etc. to interface with SAFE, and this aspect is key to make decisions around which are default values/behavior vs. optional/alternative behaviors behind flags, etc.

26 Likes

Funnily enough, we did some work on this many months ago, and we even had part of the account creation flow being “Request An Invite” where on could be dispensed to you by MaidSafe or the commity etc. I seem to recall we had a Persona worked through for it and everything. In the end though, it was—quite correctly in my opinion—bumped out of scope for the MVE. However much of the underlying workings to make this happen still are being made to make the narrower invite creation/management process. As you’d still need a way to:

  • Generate them
  • Keep track of redeemed/unused invites
  • Sort and filter them in some way
  • Withdraw them back to your account whenever you need to

So really what is missing, is basically a script or small additional “batch creator” app that you could load right from the Safe Network App to bulk generate the invites for you. That sound eminently doable, and perhaps even someone from the community might want to pitch in with that.

Just a note on this… an account invite isn’t creating an account per se, it is really just a wallet, with a small amount of safecoin in it, packaged in a way as to make it easy to redeem via the Safe Network App, right in the account creation flow. It will come with enough safecoin for the complete “Account Packed” to be created, plus we’ll overshoot it slightly, so the user has enough left over to do a bunch of post-account creation stuff, like set up their Safe ID and profile, change settings, personalise a few things, send a thank you message etc. Probably enough for several MB of mutations. That can be tuned of course.

But you’ll also notice on the feature tracker we’ve got a feature slated to be able to top up an invite with an additional amou t from your wallet to help get a friend going further.

Just be aware in all this, that as these are essentially small wallets, a single user could hoover them all up, if you made a faucet type arrangement on a website. They are better suited to 1-to-1 comms, and inviting friends.

We’re not preparing for a time limited arrangement, but as per the current designs, any invite that’s not been redeemed can be withdrawn back to your wallet at any time.

18 Likes

The suggestion of @19eddyjohn75 of provisioning mass invites (100K? Okay.) could really ignite exponential growth at an early point, maybe Beta-ish time period. Much more impactful than relying on one-to-one or one-to-few propagation. Of course, it would have to be done in such a way that the specter of spam would be avoided.

11 Likes

AT2 progress sounds promising!

13 Likes

Oh I totally get it, and that’s why we fleshed it out before pinning down the MVE—actually we even worked it in as part of the the onboarding flow.

But the MVE is about the what the starting point is: what is the minimum we need to do to get the Network usable for our given Personas. And all of the invite stuff as is would need to be provisioned anyway, to enable any kind of mass invite campaign.

Plus, you can’t have exponential growth unless you also provide for one-to-one transmission. I’m sure I read about that somewhere recently.

I’d be less worried about spam at this point, and more about mass theft of invites. Which is a tricky problem.

11 Likes

gogogo team!

11 Likes

Thank you for the heavy work team MaidSafe! Work hard, laugh harder! :safelove:

I add the translation into Bulgarian in the first post :dragon:

5 Likes

Here you can find a German translation of this weeks dev update

10 Likes