MaidSafe Dev Update - September 28, 2017

After the flurry of releases in the past couple of weeks, this week has been one of taking stock within the Authenticator and Client libs with a focus on: code quality, general development processes and improving test coverage. As you will see, work within the networking layers has been more dramatic with the Routing design team focusing on messages flows and as ever looking to marry simplicity and efficiency. A major refactor is also underway in Crust in preparation for uTP integration. We have also fixed the issue where the Web Hosting Manager app was not allowing templates to be published. The Web Hosting Manager example application has now been updated with a patch version (v0.4.1) in the alpha-2 release on GitHub.

SAFE Authenticator & API

We’ve been focusing on code quality this week. This includes looking at improving test coverage across the applications, investigating some longer-standing testing issues, as well as diving in to refactor the example applications to the much easier to read async/await syntax (as some people have been asking for on here). The test cases are also being expanded in beaker-plugin-safe-authenticator and safe_app_nodejs. We were not able to focus on certain aspects such as automated packaging and release, which we are likely to pick up soon.

We’ve also been taking a look at our general development process, and checking out some libraries that we can use for faster UI development and to keep applications visually consistent. This would also have a knock-on effect in improving the quality/readability of any apps where we use these libraries.

We decided to first bring the code base to a better stable state and then add new features/enhancements. We hope that this exercise might extend until early next week.

We will also put out easy to read simpler examples of the Node.js API, to help the developers.

SAFE Client Libs

This week we simplified the public API for safe_app by removing MDataKeysHandle, MDataValuesHandle, and MDataPermissionSetHandle. In their place we have new functions, mdata_list_keys, mdata_list_values, and mdata_list_permission_sets, which return lists of their respective objects. This lets the caller of the API work on these lists directly instead of calling an FFI function for every operation. We also added a new FFI function in safe_core for checking the mock-routing status of builds and a new environment variable for setting unlimited mutations in mock-routing. So with these changes, we continue our work on simplifying the public APIs and removing unnecessary indirection. We don’t intend to remove the Object Cache completely, but we certainly want to get rid of handles for value types, such as Permission and MDataInfo structs, which contain only simple values and no pointers to heap-allocated memory. Also, now that Alpha 2 is released, we branched off our frozen code base into the alpha-2 branch, and the current development goes on in master now (now we have switched to it from dev).

@adam has been working on making our concurrent tests deterministic. That means that before the changes we could get random results from the tests in certain circumstances, and soak testing (i.e. running the test suite continuously for a long period of time) has shown us that we have some rare bugs that can spring up at random times. @adam is reworking such tests to make their results consistent and stable, and it’s already helped to catch several concurrency-related bugs. We’re fixing them now and a pull request has been raised already.

Routing, Vault & Crust

Again, efforts have exclusively gone into the design of Routing message flows. The current front-runner is an algorithm which aims to strike a balance between simplicity (conceptually and in terms of code) and resulting traffic costs. The gist is that messages which need to traverse multiple sections across the network will primarily be handled by a subset of each such section; the subsets being chosen on a message-by-message basis. There are still several non-obvious issues which such an approach has to accommodate (for example dealing with sections splitting while passing a message between them, or dealing with malicious nodes in any or all of the sections) and identifying and addressing these has taken up most of the development effort. The Information Dispersal approach mentioned last week was developed a bit further in the early part of this week but has been mothballed in favour of the current approach due to the probable extra complexity it introduces. However, since it is a very general idea, it might still prove useful in some form in reducing the network traffic.

Work has begun on porting Crust over to using Rust’s Tokio and futures-rs libraries. This is a major refactor and will require significant testing; however, it will allow us to remove a lot of boilerplate code and make Crust simpler, more reliable, and more maintainable. After the refactoring, we will continue the work of integrating uTP using the new Tokio/futures paradigm.

77 Likes

First?Âż?Âż???..

8 Likes

raaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaah second doh. Now read fast real fast

Thanks Maidsafe devs

(I was second because I had to preach to my niece that she had to be an independent woman raaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaah fam)

13 Likes

Good work all round nice one guys :slight_smile:

7 Likes

paracetamol price is dropping ! Neuro-surgeons unemployment raising

that is really nice , thank you for taking the time to translate the examples :smiley:

3 Likes

Great work guys exciting times ahead :+1:

3 Likes

oh nooooo - i’m way too late :frowning:

5 Likes

Thanks so much to all of you for working so hard. Make sure you manage your stress level and maintain a work/life balance.

4 Likes

This seems like a good move. Let’s hope it allows more people to get over the first step.

5 Likes

Will this also include a browser with mock routing?

2 Likes

Micro Transport Protocol or µTP (sometimes also uTP) is an open UDP-based variant of the BitTorrent peer-to-peer file sharing protocol intended to mitigate poor latency and other congestion control issues found in conventional BitTorrent over TCP, while providing reliable, ordered delivery.

µTP also adds support for NAT traversal using UDP hole punching between two port-restricted peers where a third unrestricted peer acts as a STUN server.[19][20]

Interesting.

Is congestion control the highlighted benefit of switching to uTP or are there other additional benefits?
Will it replace current UDP and Crust will still use TCP as a back-up protocol?

10 Likes

Since we have mutable data now, is it possible to create something like an app token ICO? My understanding was that SAFE coin was always just going to be a piece of mutable data whose ownership gets transferred.

3 Likes

I just realized today that as a developer, when I see the phrase “major refactor” in progress notes, I get excited.

Regular users (and probably many developers as well) probably feel a sense of dread when they see this phrase.

I hope all of MaidSafe is enjoying their refactoring period. The fact that I enjoy refactoring so much is probably the reason that I hardly get anything done.

13 Likes

I was riding so high off last weeks Alpha 2 release that I completely blanked that today was an update! Very excited to see message flows get finalized/optimized but even more excited for tokio futures micro transport protocol implementation!

9 Likes

What is a major refactor and why is it so exciting ?

1 Like

It’s like distilling alcohol. Refining the code to something more basic and efficient. Which is good in a lot of ways.

11 Likes

Essentially it’s just cleaning up the code. But it usually means that it’s easier to make progress in the future.

9 Likes

Good software goes through continuous refactoring phases. It is the process of turning experimental code into strong foundational code. Without strong foundations, you build bad software.

I am not sure it is exciting, but it is satisfying. It also demonstrates a commitment to the long term health if a project.

24 Likes

Oh yeah, it definitely is!

Pure pleasure to read that, keep up the good work folks!

5 Likes

Ha! I like to refactor as much as the next man, but thrill me it does not! :smile: With good tests, refactoring should be routine and part of any Definition of Done, imo! Ofc, sometimes you have to crunch for a release and then go back to clean up though.

4 Likes