Summary
Here are some of the main things to highlight since the last dev update:
- quic-p2p v0.6.0, which includes IGD, was released today.
- The new, work in progress, safe-transfers repo has been made public.
- Support for
PUT/GETof empty directories has been added to the CLI. - @jimcollinson provides an update on the progress on the SAFE Network App UX.
- CRDT progress continues - we are now ready to start adding the actual CRDT logic to the client.
- We merged the pull model PR into Routing, making section updates more robust and efficient.
Vaults Phase 2
Over the past week, weāve focussed mainly on implementing the way Vaults handle data when Adults join and leave the network. This is shaping up nicely and about 80% of the implementation is done. There are just a few more things to consider and of course a lot of testing before the changes are pushed upstream. This will fit in nicely with the latest changes in quic-p2p that introduce port forwarding using IGD. A new version of quic-p2p was released today with these new features. We have already completed integration testing with Routing and Vaults in the last couple of internal testnets and we look forward to the next one, which will include the data handling as well. We are looking at a community testnet in the future with these features included so stick around!
SAFE API
Support for empty directories was merged this week after fixing some test-case issues on the Windows platform. Also, a pull request has been raised to include hidden files (beginning with .) in PUTs. Previously they were excluded with no option to override.
Meanwhile, a fix for NRS paths in files ls (issue 510) is currently being tested locally.
Finally, a small refactor was made in the foreign function interface (FFI) to represent FilesMap as a native data type rather than a JSON string.
SAFE App C#
As mentioned in the SAFE API section above, this week we refactored the Files APIs to return a FilesMap instance instead of a JSON string. This simplifies the usage of the Files APIs as devs can directly query the FilesMap data instead of going through the JSON operations.
In parallel, we have made some progress on exposing the authenticator APIs using the bindings from the safe-ffi library. We still have to expose some more APIs and add a couple of unit tests to ensure the integrity of them. Once completed, the Authenticator app can be refactored to use the APIs from the package, meaning we wouldnāt need to maintain the bindings in two different repos. This will also simplify the development of the SAFE Network App in the future.
SAFE Network App UX
Slightly truncated sprint this week due to lockdown home-schooling shift patterns, but progress nonetheless. Not much razzle-dazzle, but working through accessibility tasks relating to the design system, as well as continuing through the UI layers on top of this all.
Weāre majoring on the functional here, so while weāre aiming to add just a smidgen of āSAFEā personality and branding into the UX, this should never be at the expense of usability, and function.
Weāve gone around the houses somewhat on that, and while youāll only see some subtle and incremental changes in the screens below, itās all tending towards something that will hopefully be highly useful, and highly used. And of course not forgetting all the componentisation behind the scenes that will make things easier to improve and build upon!
CRDT
This past week we continued with the implementation of the Sequence data type as a CRDT. As mentioned previously, the first steps have been about having all the core libraries support its type of requests. Weāve now added the same to our safe_client_libs mock-vault implementation which helps us validate all client-side logic for any type of request.
In addition to that, we migrated the NRSContainer and FilesContainers (in safe-api) to start using the Sequence data type, and we were able to verify this is all working seamlessly as before, with all our CLI E2E tests passing when using the mock-network.
We also tried to run these E2E tests with a Baby Fleming local network, and we are at the same stage as with older data types, which means we are now ready to start adding the actual CRDT logic to the client so these issues of content not being found in some unsynced vaults are resolved.
Transfers
Following up on the past few weeks of research into AT2, we put up the new safe-transfers repo, a work in progress that captures the essential properties of AT2, applied in the SAFE Network setting.
A short description
We have Actors and Replicas to carry out the different steps of the AT2 algorithm. An Actor instance is run by a client, and it initiates transfers from its key, to some other key. The Replicas are run by the network nodes - more specifically the Elders of a section - and will validate the request and use multi-sig to prove their agreement on its validity.
A section will run a single Actor to manage the section balance, and farming payouts, as well as receive payments for data uploads.
Next
Work will now continue with testing and ironing out details around section keys handling for their Actor instance.
Links
Futures in Rust
The last week has seen more intense Futures conversion work. Weāve chucked out the internal event loop, which was causing no end of trouble. This can easily be replaced in lib consumers by more specialised runtimes. And now we have safe_core compiling (after fixing some underlying issues in the self_encryption multi-thread conversion) and all tests are now passing too .
safe_authenticator and safe_app both also compile now, with conversion of the test suites underway, and weāre in the home stretch with the conversion of safe_authenticator_ffi now too.
Itās been a big job (a lot of code to go over), but as things are coming together itās looking tighter, cleaner and leaner than ever!
Routing
This week we finally merged the pull model PR we talked about in the previous dev updates. We then enhanced the API by adding a method to retrieve the current section BLS public key set and added an event which is raised whenever the section elders change. This was another feature request by the Vault team. We followed that with a small refactoring to further simplify the code and remove unneeded bits.
After that, we started working on improving the way nodes handle messages whose trust they canāt establish. Currently, this is handled rather crudely - the message is bounced back to the sender who then waits a short while before resending the message again, hoping that the node caught up and is now able to process the message. If not, the process repeats. The problem is that itās generally impossible to determine how long the āshort whileā should be - either it is too short and the node still canāt handle the message, or too long which slows down the message flow and hurts performance. This is a general problem whenever we introduce time into our logic and is why we usually try to steer away from doing so. The new way will get rid of the delay and instead we will resend the message immediately, appending any additional information the recipient needs to be able to process it.
BLS - Distributed Key Generation
Last week we wrapped up with our testing spree on the crate. With the feedback gathered, we are now working on implementing some tweaks to the mechanism that makes the user of the DKG no longer need to handle specifics. As it stands, the user has to call the APIs manually to reply to messages and force transition to other phases. Weāre updating them such that the user doesnāt have to do anything manually with them.
This led to us using internal timers for DKG to complete a session without any user intervention. By default, users/nodes who initiate a DKG session have to participate in six DKG phases, in between each of which messages/replies would be multicasted. In a happy case, the user would not have to intervene with the process as once all six rounds of messaging are done, nodes can conclude the session on their own without a problem. Whereas with unhappy scenarios (real-world problems), the six rounds of messaging might not get completed (due to, for example, dropping of nodes or messages) and would require the user to intervene with the process to complete the session.
Timers are being introduced to mitigate this intervention where we set a specific timeout to wait for all the inflight messages to be handled. On timeout, the DKG can automatically check the readiness of the Generator. If the six phases are not completed and the Generator is not ready, the mechanism would force itself to continue the session making it run the remaining phases, which can either be successful if the threshold had been reached, or fail if not. Note that this timeout value needs to be set based on the number of nodes in a group, as the amount of time required to complete the messaging rounds would increase if there are a higher number of nodes in the session.
Useful Links
Feel free to send us translations of this Dev Update and weāll list them here:
As an open source project, weāre always looking for feedback, comments and community contributions - so donāt be shy, join in and letās create the SAFE Network together
long time
anticipation is palpable!
Here they are. Perhaps a mod could drop them in to the main post again:
