SAFE Network Dev Update - June 25, 2020

Summary

Here are some of the main things to highlight since the last dev update:

  • @jimcollinson provides another fascinating insight into the progress on the SAFE Network App UX.
  • The Symlink support and Sequence CRDT PRs have been merged into the safe-api codebase.
  • A new section has been added to the CLI User Guide describing how the new safe seq commands can be used to store data on a Public Sequence data type. These new commands will be part of the upcoming release of the SAFE CLI.
  • Work continues at pace to get SAFE Client Libs and SAFE vaults AT2 ready, with only a few more missing pieces to the jigsaw.
  • We extracted the XOR space module out of Routing into its own separate crate, allowing it to be used by other parts of the project.

Vaults Phase 2

Project plan

This week we’ve continued working on improvements based on the results from the last testnet. We have delegated the duplication process to the node responsible for holding the new copy of the chunk. This eases the stress from the Elders who were previously responsible for fetching the data and sending it to the new holder.

We have also removed the caching of ahead-of-time responses at Elders. Instead of holding on to these responses and processing them when the request eventually arrives, we use the accumulated BLS signature to validate that the section has approved the request and process it accordingly. These enhancements have already shown a lot of improvement in terms of memory usage and performance, and we are excited to share them with you. But first, there are a few more features that we’d like to add before the next iteration of the Vaults from home testnet.

SAFE Network App UX

SAFE Network App: Screens, Flows and Feature Tracker (Figma)

We’re pleased to share with you today a whole trove of screens, flows, and documentation.

Much of this you will have seen before, but there have been many hundreds of changes, tweaks, refinements, and iterations over the months.

We’ve dropped it all in one Figma file for you to dig in to—alongside the Feature Tracker — so you can keep abreast of the whole shebang. There are some screens still to drop in, and some still to be fully documented, but it’s a living file that we’ll be publishing week-to-week.

We hope this starts to paint a more detailed picture of some of the initial experience we’re building for Beta, and most likely before.

Please feel free to ask questions about any of it here, or directly in the comments on the Figma file if that’s easier when you are down in the guts of the designs.

SAFE API

Project plan

Symlink support was merged into the codebase this week after resolving an issue that turned up in testing at the end of last week.

Following on from that, work commenced on an implementation of glob() that operates in a FilesContainer. This is an experimental feature that enables pattern matching (e.g. *.txt) when resolving a SafeUrl. This could be handy for filtering results when using safe-cli commands such as files ls or files get.

A couple of API and CLI commands for the Sequence CRDT have been added this week and are ready for the upcoming release of CLI. These new commands allow users to store a Public Sequence on the network and append items to it using its XOR-URL. You can take a look at a new section added to the CLI User Guide describing how these new safe seq commands can be used to store data on a Public Sequence data type.

As part of the implementation of the Sequence API and commands, the use of AppendOnlyData has been removed and now the FilesContainers and NRS Map Containers are stored on the network as Public Sequence content. This doesn’t affect any of the CLI commands or APIs since it’s just an internal change in the implementation switching from one data type to this new one.

CRDT

This past week we finalised all the PRs in the full stack for a first iteration of the Sequence CRDT, while at the same time removing the AppendOnlyData type.

As anticipated last week, the last steps to achieve this were all about final clean up of the code in the safe-client-libs and safe-api crates, as well as creating tests for our CI test suite. We’ve been also testing all this E2E using our CLI tests suite with a Baby Fleming section, and we have all our functional tests passing.

We are also investigating an issue we still see on some of the vaults which sometimes don’t return a piece of content. We are still not sure the exact root cause of it.

Some of the next steps now are related to being able to attach an AT2 payment to a CRDT operation and that’s what we are starting to look at now. Another pending task is to transform our current MutableData data type to a new Map CRDT, which we believe can be done after we have our first iteration of the Sequence CRDT released and thoroughly tested by the community.

Transfers

SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan

This week we’ve been fully onto the integration work with AT2 SAFE Client Libs and SAFE Vaults. With Simulated-Payouts in action as mentioned in the previous update, both modules are now AT2 ready with only a few more missing pieces (choosing efficient routes for payments, refunds, etc.) that are yet to be written, alongside the integration testing. Almost all of the flows that deal with Money (PUTs/Mutations) have been updated to work with the underlying AT2 mechanism.

We are evaluating options in paying for data, which define, amongst other things, how sections use safe-transfer. Related to that we’ve progressed with farming. Tests have been written to verify the BFT properties of reward accumulation. Reward accumulation is designed to allow for rewarding a minimum unit of work, as well as leaving it up to upper layers to decide what it considers to be “work”.

Finally, we’ve been making corresponding updates in SAFE Client Libs for these latest changes to the PUT/Mutation process. Once we have these in we’ll be looking to verify everything against the updated vaults.

Routing

Project Plan

This week we finally merged the shared state signatures feature that we discussed in previous dev updates. We then addressed another requirement from the vault team - turns out a vault sometimes needs to manipulate the BLS signatures in a way that is not directly supported by the Routing API. We decided that instead of making the API more complex and potentially convoluted, we would let the vault perform these manipulations directly. But to avoid duplicating potentially non-trivial logic between vaults and Routing, we decided we would expose some low-level BLS helpers out of Routing. In particular, we implemented a Generic Signature Accumulator which handles the nitty-gritty details of collecting BLS signature shares, validating them and combining them into a full signature. The PR has been created and is currently being reviewed.

The work on the DKG replacement is progressing along nicely. A key issue when handling DKG around splits has been resolved in this PR - a very high level summary of our new approach to this is that we will now allow new Elders to vote for the new DKG. The PR is currently being reviewed and rebased against the latest fleming branch (because of the recent changes there, this will take a bit of time).

We also extracted the XOR space module out of Routing into its own separate crate, because it’s actually useful for other parts of the project. This crate defines the XorName structure which is a 256-bit long number serving as a unique identifier of nodes and data on the network. It also defines the XOR metric (distance between two names) which plays an important role in message routing. Finally, it provides the Prefix structure which forms the core of the disjoint sections concept.

Useful Links


Feel free to send us translations of this dev update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

76 Likes

Don’t read just comment to be first :wink:

Edit: Now read :dragon:

Edit2:

34 Likes

Number 2!! Now to read :smile:

Exciting!! Really appreciate seeing all the progress on the UXD front too!

Amazing! So much is happening in parallel process, that it feels like there will be a snowball effect with a ton of development coming together all at once. The velocity is certainly primed to accelerate :smile:

27 Likes

Way too excited about data types… feels like heat stroke :sunglasses: :sun_with_face:

Look forward to the new CLI to try the difference.

:+1:

Noting the obvious atm shows as

https://github.com/maidsafe/safe-api/blob/master/safe-api/src/api/app/xorurl.rs

pub enum SafeDataType

  • SafeKey = 0x00,
  • PublicImmutableData = 0x01,
  • PrivateImmutableData = 0x02,
  • PublicSequence = 0x03,
  • PrivateSequence = 0x04,
  • SeqMutableData = 0x05,
  • UnseqMutableData = 0x06

https://github.com/maidsafe/safe-api/blob/master/safe-cli/subcommands/dog.rs

  • NrsMapContainer
  • FilesContainer
  • PublicImmutableData
  • Wallet
  • SafeKey
  • PublicSequence

Not clear if wallet will be same as SafeDataType: SafeKey but can look forward to that.

Private is so much a better name than unpublished…

23 Likes

image

tenor

Data services in exchange for SAFECoin.

I can already taste the network.

30 Likes

Tasty. Steady progress on vaults, looking forward to diving into the UI stuffs. The seq and symlinks will be fun to play with in the CLI. THanks to all involved as always.

Just a wee point image is a lot like the Scotcoin symbol. Previously I was very relaxed about that but it is now certain that Scotcoin will relaunch within a couple of weeks with its V3 ERC-20 token. The logo still remains image and we will be using a simplied version of that as our currency symbol.
I say I was previously relaxed cos it seemed the coin was moribund but after much work it has been turned around. I just hope we don’t get too much confusion.

17 Likes

yeah, what happened to this guy?


can we make some use of it plz?

great update btw! :+1:

11 Likes

Depends whether its registered as a trademark and who owns the IP.

7 Likes

maidsafe owns the rights to it. At least im hoping so? a while ago we had a competition on 99designs.com and this was the outcome. same goes for the safenetwork logo. lets use it properly?

6 Likes

I’ve mentioned it before, but the Safecoin logo as shown above can’t just be used in a blanket fashion in all cases, and has not been designed to take into account use at the wide range of resolutions and sizes we need it to now, nor has it been adapted to use as a glyph and in a wide variety of typefaces. In this particular case we required it to work alongside a narrow numerical, and it ain’t cut out for that.

As a stop gap to designing a full glyph/unicode set for safecoin, I pinched a candidate community design from about 5 years ago, before we can get into the nitty-gritty of that particular design process. It might also serve as a starting point potentially too. However, it’s not something I’m sinking much time on at the moment!

Hope that makes sense.

19 Likes

Yeah, it’s amazing just how many screens and interactions even a ‘simple’ app can have, isn’t it?

Aye, don’t stress about that, that’ll likely change. The invites stuff is the one area that hasn’t been looped in to the latest iteration, hence why none of those sub-app screens are in the file yet. Hoping to get on to that next week.

Yeah, sorry, just to clarify, it will all be Safecoin (no dollars). This is due to lack of a Safecoin unicode character / font set. Mocking out a UI like this with just a dollar symbol at the moment saves a lot of pain!

18 Likes

Nice update, but I’m suffering from a bout of definition anxiety. (It may be the heat)

Can someone give me a succinct description of the word ‘Sequence’ in the SAFE context? So we have Sequence CRDTs, Public Sequence data type and Sequence API. Sequence implies order, so is this the same as versioned?

9 Likes

I don’t think it really aged well tbh.

Not to put anyone down, but with this logo i get the feeling 2005 wants Its logo back :grimacing:

6 Likes

Unfortunately I need to agree. :frowning_face:

5 Likes

Thx for the update Maidsafe devs,

Really wonder how you guys stay sane coding this up? Always good to see the merged PR, can’t wait to try out the CLI soon.

Sorry @JimCollinson can’t really give feedback (my computer is to slow to show Figma):computer::face_with_head_bandage:

Go outside sometimes super ants and enjoy nature, it’ll clear your mind from this alien task

Thx again and keep hacking

10 Likes

Yep, my understanding is that you (see the CLI docs example), create a Sequence by storing arbitrary data to it, and can then update that with new arbitrary data (which is appended to the sequence).

Each version in the Sequence is then accessible, and this will be the basis of the perpetual web and all published data.

17 Likes

Great job guys! Sounds like solid progress on many fronts and a real team effort!

10 Likes

Thanks. It makes sense. The word Sequence in that form doesn’t stick in my head for some reason. I prefer ‘Sequenced’ which implies the imposition of order rather than its simple existence. Verb/adjective versus noun, active versus passive. But I may not have the whole picture here.

13 Likes

The whole discussion is in this thread: [RFC] Data Types Refinement

10 Likes

Have you checked @JimCollinson’s avatar :slight_smile:

9 Likes