SAFE Network Dev Update - May 14, 2020


Here are some of the main things to highlight since the last dev update:

  • After some improvements, we ran another internal testnet with vaults from home and it’s looking good so far.
  • We released a new version of the MaidSafe.SafeApp NuGet package.
  • Work has continued on implementing and testing support for empty directories, with a PR imminent.
  • We migrated the FilesContainers API and CLI commands to be stored/synced-up/read using the Sequence data type and all basic manual tests with a local section worked just fine :tada:

Vaults Phase 2

Project plan

AT2 work is progressing with closer alignment to the paper, as well as adjustments for specific SAFE conditions. We’re making good progress with applying this to both the vault and the client. We are getting closer to a stage where the AT2 checks are in place and we can really start fleshing out our tests for this.

After the last internal testnet with vaults from home, we have made some improvements. The primary feature added was to remove the need for the --ip parameter which required users to pass their local IP address when starting the vault binary for port forwarding purposes. We have removed the need for this by connecting to the IGD gateway and querying our local socket address. This ultimately improves user experience by simplifying the process of starting a vault, minimising the need for technical details. With this in place, we started another testnet and it’s looking good so far. We’ll continue to monitor this in the coming days, and it is likely we will make further improvements and iterations before considering this ready for release.

In parallel, we have also been implementing some flows to maintain the minimum number of copies of data in case vaults go offline. In the initial implementation, a number of messages are passed around when a vault goes offline. All elders try to fetch the data and store additional copies. This gives us what we need, but, we’ve started to think of ways to optimize this already. Implement, iterate and improve is the way forward!


Project plan

Work has continued on implementing and testing support for empty directories. This required a small change to the way data is stored in a FileContainer.

Previously every FileItem entry in a FileContainer represented a file with a link property containing a safe-url identifying an immutable data object. In the proposed new code, a FileItem may represent a File, Directory, or Symlink identified by the type metadata attribute. Directory and Symlink entries do not possess the link property, so they exist only in the FileContainer.

Additional metadata is also created for each file, if locally available: original_created_time, original_modified_time, and unix_mode_bits. The aim of this metadata is to enable backup/restore that preserves local file metadata.

A small fix was made to files ls that enables ls to return details for a single file in a FileContainer, and also matches by path component rather than by individual character.

A first draft of the pull-request for these changes will be raised later today, for the team to review.

Support for hidden files (those starting with a .) and relative symlinks (linking within FileContainer) will be worked on next week.


Project plan

We released a new version of the MaidSafe.SafeApp NuGet package. In this latest release, we have renamed the XorUrlEncoder to SafeUrl and refactored the APIs accordingly. We also added some new APIs to generate the URLs for keys, mutable data, appendable data, and immutable data. We fixed a few issues in the FFI wrapper functions which were causing crashes and simplified the usage of the base encoding types for the URLs. You can check the changelog file for the complete list of changes.


This week we implemented, refactored and simplified the Sequence data type and supported requests in safe-nd, then adapted SAFE Client Libs and SAFE Vaults accordingly. We also attempted migrating FilesContainers API and CLI commands to be stored/synced-up/read using the Sequence as its underlying native data type (as opposed to AppendOnlyData) - all basic manual tests with a local section worked just fine :tada:

Most of the team have been on daily meetings with our CRDT contractor for that last 2 weeks. These have completed for now with a deep dive into AT2, ORSWOT and generalised byzantine fault tolerance for any CRDT in SAFE. This process has been very well received and we will likely invest a bit more there to have an audit of our specific use cases. This will help guarantee robustness and correctness of our data types and also section membership handling (which will be significantly improved).

Futures in Rust

Last week we got through the vast majority of changes required for safe_authenticator, though hit some issues when looking to update some event_loop code for both safe_authenticator and safe_core. This in turn has led us to take a closer look at the code here, and we’re trying to decide the best way forward, which is looking like removing this internal event_loop, to keep the libraries clean and simple, allowing higher-level libraries to use a Rust runtime as they require.


Project Plan

This week we implemented a couple of small API enhancements that were requested by the Vault team: First we added a method to retrieve the current section adults sorted by their XOR-distance to a given name and added events that are raised when a member joins or leaves the section. This functionality is needed by vaults to be able to distribute and redistribute the data replicas among the section members. Then we added a getter for the current threshold secret key share thus allowing vaults to participate in threshold signature and encryption schemes - something that up to now was available only to Routing itself for its internal needs.

We are still working on the pull model as described in the last dev update. Most of the new flow is already in place, we just need to debug some odd test failures and polish the code a bit before raising it as PR.

BLS - Distributed Key Generation

Project plan

Last week, we went deeper into testing the crate with real-network simulations which introduced malicious scenarios in the testing suite. This has revealed a few bugs for us to deal with, mostly regarding the way we handle DKG messaging among the participating nodes. These essentially show us different ways of implementing the messaging layer efficiently to suit the needs of our network, such as how we try to force 100% participation of the m of n participants, virtually removing the need for a Justification phase in the mechanism by making sure that all the nodes participating would have to be non-malicious, leaving no space for complaints.

Further improvements to the mechanism itself are also being worked on. For example, we are improving the transport layer of the crate to take decisions on its own using internal timers to successfully complete a DKG session by itself, rather than the users (such as vaults, clients, etc.) overseeing and handling these specifics, meaning integration and usability with other crates in future becomes much easier. We have also started discussions within the team on how BLS-DKG would be combined and used alongside the other modules that are being developed in parallel (such as with clients, Routing, CRDT, etc.), with the end goal of the crate to be a generic implementation.

Useful Links

Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:


Another great update - thanks team!


Thanks so much to the Maidsafe team for all of your hard work and speedy progress.


:exploding_head: I forgot it was thursday

Thx Maidsafe devs… now read…

This is why I love being here

The constant improvements and stubbornness to deliver the SAFE Network


More solid progress - thank you to all involved.


a light blinked here :wink:

Thank you again for the long work !! It is really neat to see things are still going on and in great shape. Keep that engine running guys


An enhancement for future I wonder might be to allow load balanced endpoints, not that I’ve done this myself yet but the option of having a fallback connection and then I expect necessarily an alternate IP, that is the same destination, might avoid loss of kudos or whatever the measure of a node state and status is.

images of old people panicing that they can’t find what it was they were looking for… I can relate. :laughing:

Will hardlinks be possible; if so, will those be limited to FileContainer?.. that is, literally the same file but pointers in different locations and potentially many of them. Aside from it being interesting, I wonder matching all kinds of file systems capability that exists, can only be viewed as a feature. Metadata and hidden files will be useful.



In the future we may well do. Of course we want others to also add features.

I am not sure it would make sense, but the guys are closer to this part.


Hey I’m 35th!

Looks like I win :rofl:

Glad this is being added to the end every week now. Really contextualizes the project. And you never know who will pop up and help out of nowhere

I’ll see if anyone I know who used to translate is still around. Lots of people seem to have fell off. Do you have any channels now where you like to post the update in other languages? Worst case scenario I could just walk outside and offer someone a few dollars to translate into Chinese, tamil or Malaysian here if that’s worthwhile for anyone


This is driving my curious mind nuts.
Did this person request that they not be named?


@Sotros25 When we get a marketing fund together, would this be a suitable case for recompensing @whiteoutmashups expenses for this - possibly in MAID?


Mandarin, Cantonese?
Malaysia has a big population, there must be a fair no of Malaysian speakers.
Tamil? Is this one of the languages spoken in South India where most of the Indian developers seem to be?
What about our own Maidsafe folk in India, we still have a few, don’t we? Can we incentivise them also to do some translating?


I don’t know but if it’s not explicitly mentioned then better not to pry I suppose. You can have a pretty good idea with some github stalking though :slightly_smiling_face:


That’s the maidsafe way :smiling_face_with_three_hearts:


hardlinks are built into the way that FileContainers already work. Each file item in a FileContainer has a link property with an xor-url that references an ImmutableData object, which actually contains the data. The URL is generated from the file data itself. So if you have two distinct paths in the FIleContainer that both reference a file with the exact same data, then both will have the same url in the FileContainer. This is even true if the content is referenced from different FileContainers, even with different owners. So, this is analogous to the way that hard-links work in file-systems, with multiple paths pointing to the same inode. Git works in a similar fashion.

For directories, the only planned linking is a symlink within the FileContainer. So if you try to PUT a directory with a symlink that goes outside the directory tree, it would likely be converted to a regular file (if a file) or skipped (if a dir). We may add some flags/options around this.


I was thinking symlinks might be able to point to a FileContainer. Is the any way to link from one FileContainer to another or must everything be self contained?

I’m not sure if it’s needed or useful, just exploring possibilities.

Another thing to consider is file metadata. I’m a bit out of touch now, but if you haven’t already it might be useful to look at the POSIX and FUSE support for this and try to support that, or ensure it can be added later. This includes support for extended properties (I forget the exact term).


d’oh… of course! xorurl … still later listing off what SAFE does against what is familiar as legacy normal, will be useful… that’s going to be a long FAQ :smiley:


At present, I’m just looking at it as a way of preserving local symlinks through a PUT/GET cycle.

What you are proposing goes beyond that. It’s interesting, but probably not a top priority right now, and there are some things to figure out about how it would work exactly. Do you have use-case(s in mind?

I expect there will be one or more standards around all this in the future. Remember that the FileContainer is being serialized and deserialized client-side. The network really doesn’t care what’s in it, so it is only the client-side API enforcing things. Thus apps could create their own method/standard. We run risks of incompatibilities when that happens, so I feel that some standards will naturally arise around 1 or more APIs over time.

Another thing to consider is file metadata. I’m a bit out of touch now, but if you haven’t already it might be useful to look at the POSIX and FUSE support for this and try to support that, or ensure it can be added later. This includes support for extended properties (I forget the exact term).

yeah, that could be interesting. I’ll keep it in mind. possibly as a flag on PUT. We probably shouldn’t include too much metadata by default though.


No use cases other than future proofing compatibility - forcing apps to build standards on top isn’t great at this level because it can lead to incompatibilities, but I realise you can’t build everything at this stage so some things will have to really. So I’m only making observations in case it can help later.

I experienced some difficulties when trying to build a FUSE drive on the old NFS API which was good but had some holes in it which were hard to fill because file APIs are far more complicated that I imagined.

I think what we have is already very good.


Hey Will,

currently, all of marketing is in the hands of the community. We have a separate group in the forum Social Media Organizers, which works for that. If you want to join, we can add you :dragon:

I am working on two international forums - Bulgarian and Espanol. As the Espanol forum is a few days old and @digipl mainly helps wiht it. My goal for this year is to launch more international forums and translations on the official website in different languages for the millions of people who do not speak English.