Summary
Here are some of the main things to highlight since the last dev update:
- After some improvements, we ran another internal testnet with vaults from home and it’s looking good so far.
- We released a new version of the MaidSafe.SafeApp NuGet package.
- Work has continued on implementing and testing support for empty directories, with a PR imminent.
- We migrated the FilesContainers API and CLI commands to be stored/synced-up/read using the Sequence data type and all basic manual tests with a local section worked just fine
Vaults Phase 2
AT2 work is progressing with closer alignment to the paper, as well as adjustments for specific SAFE conditions. We’re making good progress with applying this to both the vault and the client. We are getting closer to a stage where the AT2 checks are in place and we can really start fleshing out our tests for this.
After the last internal testnet with vaults from home, we have made some improvements. The primary feature added was to remove the need for the --ip parameter which required users to pass their local IP address when starting the vault binary for port forwarding purposes. We have removed the need for this by connecting to the IGD gateway and querying our local socket address. This ultimately improves user experience by simplifying the process of starting a vault, minimising the need for technical details. With this in place, we started another testnet and it’s looking good so far. We’ll continue to monitor this in the coming days, and it is likely we will make further improvements and iterations before considering this ready for release.
In parallel, we have also been implementing some flows to maintain the minimum number of copies of data in case vaults go offline. In the initial implementation, a number of messages are passed around when a vault goes offline. All elders try to fetch the data and store additional copies. This gives us what we need, but, we’ve started to think of ways to optimize this already. Implement, iterate and improve is the way forward!
SAFE API
Work has continued on implementing and testing support for empty directories. This required a small change to the way data is stored in a FileContainer.
Previously every FileItem entry in a FileContainer represented a file with a link property containing a safe-url identifying an immutable data object. In the proposed new code, a FileItem may represent a File, Directory, or Symlink identified by the type metadata attribute. Directory and Symlink entries do not possess the link property, so they exist only in the FileContainer.
Additional metadata is also created for each file, if locally available: original_created_time, original_modified_time, and unix_mode_bits. The aim of this metadata is to enable backup/restore that preserves local file metadata.
A small fix was made to files ls that enables ls to return details for a single file in a FileContainer, and also matches by path component rather than by individual character.
A first draft of the pull-request for these changes will be raised later today, for the team to review.
Support for hidden files (those starting with a .) and relative symlinks (linking within FileContainer) will be worked on next week.
SAFE App C#
We released a new version of the MaidSafe.SafeApp NuGet package. In this latest release, we have renamed the XorUrlEncoder to SafeUrl and refactored the APIs accordingly. We also added some new APIs to generate the URLs for keys, mutable data, appendable data, and immutable data. We fixed a few issues in the FFI wrapper functions which were causing crashes and simplified the usage of the base encoding types for the URLs. You can check the changelog file for the complete list of changes.
CRDT
This week we implemented, refactored and simplified the Sequence data type and supported requests in safe-nd, then adapted SAFE Client Libs and SAFE Vaults accordingly. We also attempted migrating FilesContainers API and CLI commands to be stored/synced-up/read using the Sequence as its underlying native data type (as opposed to AppendOnlyData) - all basic manual tests with a local section worked just fine
Most of the team have been on daily meetings with our CRDT contractor for that last 2 weeks. These have completed for now with a deep dive into AT2, ORSWOT and generalised byzantine fault tolerance for any CRDT in SAFE. This process has been very well received and we will likely invest a bit more there to have an audit of our specific use cases. This will help guarantee robustness and correctness of our data types and also section membership handling (which will be significantly improved).
Futures in Rust
Last week we got through the vast majority of changes required for safe_authenticator, though hit some issues when looking to update some event_loop code for both safe_authenticator and safe_core. This in turn has led us to take a closer look at the code here, and we’re trying to decide the best way forward, which is looking like removing this internal event_loop, to keep the libraries clean and simple, allowing higher-level libraries to use a Rust runtime as they require.
Routing
This week we implemented a couple of small API enhancements that were requested by the Vault team: First we added a method to retrieve the current section adults sorted by their XOR-distance to a given name and added events that are raised when a member joins or leaves the section. This functionality is needed by vaults to be able to distribute and redistribute the data replicas among the section members. Then we added a getter for the current threshold secret key share thus allowing vaults to participate in threshold signature and encryption schemes - something that up to now was available only to Routing itself for its internal needs.
We are still working on the pull model as described in the last dev update. Most of the new flow is already in place, we just need to debug some odd test failures and polish the code a bit before raising it as PR.
BLS - Distributed Key Generation
Last week, we went deeper into testing the crate with real-network simulations which introduced malicious scenarios in the testing suite. This has revealed a few bugs for us to deal with, mostly regarding the way we handle DKG messaging among the participating nodes. These essentially show us different ways of implementing the messaging layer efficiently to suit the needs of our network, such as how we try to force 100% participation of the m of n participants, virtually removing the need for a Justification phase in the mechanism by making sure that all the nodes participating would have to be non-malicious, leaving no space for complaints.
Further improvements to the mechanism itself are also being worked on. For example, we are improving the transport layer of the crate to take decisions on its own using internal timers to successfully complete a DKG session by itself, rather than the users (such as vaults, clients, etc.) overseeing and handling these specifics, meaning integration and usability with other crates in future becomes much easier. We have also started discussions within the team on how BLS-DKG would be combined and used alongside the other modules that are being developed in parallel (such as with clients, Routing, CRDT, etc.), with the end goal of the crate to be a generic implementation.
Useful Links
Feel free to send us translations of this Dev Update and we’ll list them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together
I forgot it was thursday
