There are a few staff movements worth noting this week.
Firstly, Andrew has been making himself very useful in providing expert suggestions, pull requests and RFCs, so much so that he will become the new primary maintainer of the critical Routing library once Ben leaves us at the end of this week.
Andrew’s also going to be working on an RFC for updating the Crust component. This will probably culminate in a significant overhaul of Crust to have it use more Rust-like paradigms than it currently does.
We’d like to thank Brian too for agreeing to reschedule leave he had booked for this week. He’s moved it in order to get the most out of Ben’s handover of the Routing library.
We’re also happy to see the arrival today of Spandan to our International HQ here in Troon. He’s brought miserable weather with him and his sideboards aren’t nearly as impressive as in his GitHub profile pic. Regardless, we’ll overlook these issues and I’m sure his stay will be beneficial for both the local team and him.
As for the recent sprint which has just finished, we unfortunately didn’t achieve all our objectives. There are more specific details about the individual targets below, but there was success for the Client libraries, the QA tasks and the Vault bugs. Crust and Routing which fell short did however make decent progress during the sprint, with unforeseen issues causing the bulk of the delays.
To expand a little on the Client libraries, we’re now in a position to say that the APIs for app devs are stable. The examples are currently running against a mock network interface, but even if that were to change, the actual API for apps is via a JSON interface in Launcher. This means that launcher can shield any network API changes from apps and once we’ve worked the final kinks out of Crust and Routing, we should be able to simply swap out the mock for the real Routing without impacting upon any created apps.
This week we have asked each of the teams maintaining the network components to provide a progress update on their individual library, so now you know who to blame for any incorrect information 
or who to ask if you have specific questions.
Crust (@Peter_Jankuliak and @vinipsmaker)
In Crust we were debugging the whole week, we managed to fix most of the bugs, but we still have two left. One of them we think has always been there (the one where occasionally Crust bootstraps to itself), and then there is the one which causes CI machines to pass but print out a panic, which is how it slipped into master branch. It also doesn’t seem to be related to UDP hole punching.Here are the weeks main tasks in more detail:
- Writing detailed tests with Vinicius in rust-utp to find out why our high level rendezvous test is failing, turned out the rendezvous connect function we recently implemented in rust-utp had a bug which made the new connection lose one packet at the beginning.
- Refactoring rendezvous connection in rust-uTP
- Resolving conflicts and merging old unmerged PRs which piled up during the sprint
- Fixing issue on OSX where we couldn’t bootstrap to more than 47 nodes.
- Change in API to associate tokens with connection attempts.
Routing (@benjaminbollen and @brian_s)
The sprint objective for routing concerned RFC-0011. This was a combined design with RFC-0008 from CRUST to achieve decentralised NAT traversal. From routings perspective this involved a redesign of the existing connection management. It has introduced for the first time connection management stateful objects necessary for a successful established connection (matching expected connections with unknown connections).
The RFC involved a significant change to the connection management of the routing library even without the explicit introduction of hole punching. Task specifications in Jira covered this thoroughly enough, although the points designated for the tasks were, with hindsight, optimistic. The increased complexity of the design introduced a number of bugs that ate into the time set aside for testing the library. Currently some bugs still remain and at the end of the sprint hole punching is still incomplete and we are still unable to test the latest version of connection management with the Vault library.
The issues discovered and addressed as the debugging continues is traced in a central Github issue #757. Some specific issues were; Having established an initial bootstrap connection between the first two nodes, the second node, after relocating, initiated a connect request as a managed node. The first node failed to respond to the request, since the only authority it was aware of was the client authority of the non-relocated second node.
Another so far unresolved issue is the intermittent occurrence Crust self-bootstrapping, which can cause a trivial blockage as Routing only keeps a single active bootstrap connection at a time now. A third issue arises from the multi-step nature of connection management where nodes can attempt to connect from both sides before the actual connection is consolidated, however, this should be easy enough to address.
Client Libs (@ustulation and @Krishna_Kumar)
SAFE Launcher, a gateway for the apps to communicate with the SAFE Network on behalf of the user, is now ready as a library. This is great news for app devs as it abstracts the working knowledge of the SAFE Network entirely from them. All the app devs are required to do is to go through and make use of the very friendly JSON IPC documentation in the RFC repository. Apps can be written in any language and as long as they adhere to a simple authentication mechanism, they can talk to Launcher using JSON requests and leave it to Launcher to carry out the actual tasks.
As the routing library is not ready, Launcher uses mock-network. This should be of no concern to the applications communicating with Launcher because as far as they are concerned there is absolutely no change in the interface - it all happens in the core libraries at the backend. So development and testing of such apps can start right away!
We are now on our way to designing a UI for the Launcher library and in the interim there will be a CLI version already coded and functional in the examples section. This Launcher CLI is a simple CLI executable that interfaces with the Launcher core and provides basic functionalities to carry out app development and testing. The pre-compiled release version can be found here The good news does not end there however.
We have already compiled and successfully run sample apps against the Launcher CLI. These can be tried straight from the releases here. These apps should be a great starting point, or a tutorial for the app devs. One of them is in NodeJS and is actually a functional UI. You run the Launcher CLI executable, follow a very simple but intuitive menu over there, add the app into Launcher (for it to manage) and simply ask Launcher to activate it. After that it will do a bunch of NFS and DNS operations to basically create a DNS record for the user. Then we have the FireFox Plugin example to actually see this being pulled into the browser when you type it in the address bar. Again if you don’t want to go through the process of building it locally (which also we have made it pretty easy - just follow the instructions in the README.md of the repository), you can try out the pre-built releases hosted here. All good stuff.
Another sample application is written in Rust itself. It has extensive inline documentation and shows how to create and fetch a directory from the SAFE Network. It is very simple to code that because of the way Launcher abstracts all the gruesome detail and the inline comments explain every bit of it.
These will hopefully get people excited and working on their apps. While the future scope for the launcher API Is huge, the immediate focus is to get a Launcher UI designed and implemented. In addition, Launcher does not support anonymous users at present - all users are required to be registered with the network by creating an account. This prevents things like browsers to interface with it because all that the browsers care about is the retrieval of data as opposed to the need for registration. The goal here is to ensure that everyone, provided they have an Internet connection, can fetch any data from the SAFE Network and if that data is public, i.e. not encrypted, then readable. So there is going to be some thoughts on how to make Launcher cater to requests from anonymous clients like the browsers. As usual we will go via the RFC route and people can see and comment on the suggestions.
Vault (@qi_ma and @Fraser)
- Network tests were refactored to be more efficient in terms of system resources and running time.
- Resolved issue 259 (the DataManager now fetches a copy of chunk which has been stored by a PmidNode which subsequently disconnects from the network)
- Resolved issues 261 & 263 (the DataManager persona now fetches a copy of chunk which has failed to have been stored by a PmidNode)
- Resolved issues 267, 268 & 269 (the ChunkStore now cleans up after itself properly)
- Resolved various outstanding FIXMEs and TODOs
- Began an RFC for the upcoming sprint regarding the naming of ImmutableData
QA (@Ross and @Fraser)
- Windows and OS X Vault installers are now signed
- Set up a Windows 7 test environment
- Linux Vault install process was updated to use company keys rather than test keys
- Vault installer workflows have been documented
- OS X Vault installer was fixed to handle upgrading the installed version
So, we didn’t achieve everything we set out to in the rust-5 sprint, however, the Launcher API is a significant step forward and a really nice deliverable. Despite the bugs in Crust and Routing we have made some really nice changes and once we have the remaining bugs fixed we will be able to move the whole network forward. We will obviously keep you updated with progress as we go. One of the great advantages of having a greatly reduced and less complex code base is that we should be able to identify and solve these problems much more quickly than before. Enjoy using the Launcher!
Cheers,
Fraser.
I’m broke as a joke so I can only give you a digital gift 