After the flurry of releases in the past couple of weeks, this week has been one of taking stock within the Authenticator and Client libs with a focus on: code quality, general development processes and improving test coverage. As you will see, work within the networking layers has been more dramatic with the Routing design team focusing on messages flows and as ever looking to marry simplicity and efficiency. A major refactor is also underway in Crust in preparation for uTP integration. We have also fixed the issue where the Web Hosting Manager app was not allowing templates to be published. The Web Hosting Manager example application has now been updated with a patch version (v0.4.1) in the alpha-2 release on GitHub.
SAFE Authenticator & API
We’ve been focusing on code quality this week. This includes looking at improving test coverage across the applications, investigating some longer-standing testing issues, as well as diving in to refactor the example applications to the much easier to read async/await syntax (as some people have been asking for on here). The test cases are also being expanded in beaker-plugin-safe-authenticator and safe_app_nodejs. We were not able to focus on certain aspects such as automated packaging and release, which we are likely to pick up soon.
We’ve also been taking a look at our general development process, and checking out some libraries that we can use for faster UI development and to keep applications visually consistent. This would also have a knock-on effect in improving the quality/readability of any apps where we use these libraries.
We decided to first bring the code base to a better stable state and then add new features/enhancements. We hope that this exercise might extend until early next week.
We will also put out easy to read simpler examples of the Node.js API, to help the developers.
SAFE Client Libs
This week we simplified the public API for safe_app
by removing MDataKeysHandle, MDataValuesHandle, and MDataPermissionSetHandle. In their place we have new functions, mdata_list_keys
, mdata_list_values
, and mdata_list_permission_sets
, which return lists of their respective objects. This lets the caller of the API work on these lists directly instead of calling an FFI function for every operation. We also added a new FFI function in safe_core
for checking the mock-routing status of builds and a new environment variable for setting unlimited mutations in mock-routing. So with these changes, we continue our work on simplifying the public APIs and removing unnecessary indirection. We don’t intend to remove the Object Cache completely, but we certainly want to get rid of handles for value types, such as Permission
and MDataInfo
structs, which contain only simple values and no pointers to heap-allocated memory. Also, now that Alpha 2 is released, we branched off our frozen code base into the alpha-2
branch, and the current development goes on in master
now (now we have switched to it from dev
).
@adam has been working on making our concurrent tests deterministic. That means that before the changes we could get random results from the tests in certain circumstances, and soak testing (i.e. running the test suite continuously for a long period of time) has shown us that we have some rare bugs that can spring up at random times. @adam is reworking such tests to make their results consistent and stable, and it’s already helped to catch several concurrency-related bugs. We’re fixing them now and a pull request has been raised already.
Routing, Vault & Crust
Again, efforts have exclusively gone into the design of Routing message flows. The current front-runner is an algorithm which aims to strike a balance between simplicity (conceptually and in terms of code) and resulting traffic costs. The gist is that messages which need to traverse multiple sections across the network will primarily be handled by a subset of each such section; the subsets being chosen on a message-by-message basis. There are still several non-obvious issues which such an approach has to accommodate (for example dealing with sections splitting while passing a message between them, or dealing with malicious nodes in any or all of the sections) and identifying and addressing these has taken up most of the development effort. The Information Dispersal approach mentioned last week was developed a bit further in the early part of this week but has been mothballed in favour of the current approach due to the probable extra complexity it introduces. However, since it is a very general idea, it might still prove useful in some form in reducing the network traffic.
Work has begun on porting Crust over to using Rust’s Tokio and futures-rs libraries. This is a major refactor and will require significant testing; however, it will allow us to remove a lot of boilerplate code and make Crust simpler, more reliable, and more maintainable. After the refactoring, we will continue the work of integrating uTP using the new Tokio/futures paradigm.