MaidSafe Dev Update :safe: 3rd November 2015

There are a few staff movements worth noting this week.

Firstly, Andrew has been making himself very useful in providing expert suggestions, pull requests and RFCs, so much so that he will become the new primary maintainer of the critical Routing library once Ben leaves us at the end of this week.

Andrew’s also going to be working on an RFC for updating the Crust component. This will probably culminate in a significant overhaul of Crust to have it use more Rust-like paradigms than it currently does.

We’d like to thank Brian too for agreeing to reschedule leave he had booked for this week. He’s moved it in order to get the most out of Ben’s handover of the Routing library.

We’re also happy to see the arrival today of Spandan to our International HQ here in Troon. He’s brought miserable weather with him and his sideboards aren’t nearly as impressive as in his GitHub profile pic. Regardless, we’ll overlook these issues and I’m sure his stay will be beneficial for both the local team and him.

As for the recent sprint which has just finished, we unfortunately didn’t achieve all our objectives. There are more specific details about the individual targets below, but there was success for the Client libraries, the QA tasks and the Vault bugs. Crust and Routing which fell short did however make decent progress during the sprint, with unforeseen issues causing the bulk of the delays.

To expand a little on the Client libraries, we’re now in a position to say that the APIs for app devs are stable. The examples are currently running against a mock network interface, but even if that were to change, the actual API for apps is via a JSON interface in Launcher. This means that launcher can shield any network API changes from apps and once we’ve worked the final kinks out of Crust and Routing, we should be able to simply swap out the mock for the real Routing without impacting upon any created apps.

This week we have asked each of the teams maintaining the network components to provide a progress update on their individual library, so now you know who to blame for any incorrect information :slight_smile: or who to ask if you have specific questions.

Crust (@Peter_Jankuliak and @vinipsmaker)

In Crust we were debugging the whole week, we managed to fix most of the bugs, but we still have two left. One of them we think has always been there (the one where occasionally Crust bootstraps to itself), and then there is the one which causes CI machines to pass but print out a panic, which is how it slipped into master branch. It also doesn’t seem to be related to UDP hole punching.Here are the weeks main tasks in more detail:

  • Writing detailed tests with Vinicius in rust-utp to find out why our high level rendezvous test is failing, turned out the rendezvous connect function we recently implemented in rust-utp had a bug which made the new connection lose one packet at the beginning.
  • Refactoring rendezvous connection in rust-uTP
  • Resolving conflicts and merging old unmerged PRs which piled up during the sprint
  • Fixing issue on OSX where we couldn’t bootstrap to more than 47 nodes.
  • Change in API to associate tokens with connection attempts.

Routing (@benjaminbollen and @brian_s)

The sprint objective for routing concerned RFC-0011. This was a combined design with RFC-0008 from CRUST to achieve decentralised NAT traversal. From routings perspective this involved a redesign of the existing connection management. It has introduced for the first time connection management stateful objects necessary for a successful established connection (matching expected connections with unknown connections).

The RFC involved a significant change to the connection management of the routing library even without the explicit introduction of hole punching. Task specifications in Jira covered this thoroughly enough, although the points designated for the tasks were, with hindsight, optimistic. The increased complexity of the design introduced a number of bugs that ate into the time set aside for testing the library. Currently some bugs still remain and at the end of the sprint hole punching is still incomplete and we are still unable to test the latest version of connection management with the Vault library.

The issues discovered and addressed as the debugging continues is traced in a central Github issue #757. Some specific issues were; Having established an initial bootstrap connection between the first two nodes, the second node, after relocating, initiated a connect request as a managed node. The first node failed to respond to the request, since the only authority it was aware of was the client authority of the non-relocated second node.

Another so far unresolved issue is the intermittent occurrence Crust self-bootstrapping, which can cause a trivial blockage as Routing only keeps a single active bootstrap connection at a time now. A third issue arises from the multi-step nature of connection management where nodes can attempt to connect from both sides before the actual connection is consolidated, however, this should be easy enough to address.

Client Libs (@ustulation and @Krishna_Kumar)

SAFE Launcher, a gateway for the apps to communicate with the SAFE Network on behalf of the user, is now ready as a library. This is great news for app devs as it abstracts the working knowledge of the SAFE Network entirely from them. All the app devs are required to do is to go through and make use of the very friendly JSON IPC documentation in the RFC repository. Apps can be written in any language and as long as they adhere to a simple authentication mechanism, they can talk to Launcher using JSON requests and leave it to Launcher to carry out the actual tasks.

As the routing library is not ready, Launcher uses mock-network. This should be of no concern to the applications communicating with Launcher because as far as they are concerned there is absolutely no change in the interface - it all happens in the core libraries at the backend. So development and testing of such apps can start right away!

We are now on our way to designing a UI for the Launcher library and in the interim there will be a CLI version already coded and functional in the examples section. This Launcher CLI is a simple CLI executable that interfaces with the Launcher core and provides basic functionalities to carry out app development and testing. The pre-compiled release version can be found here The good news does not end there however.

We have already compiled and successfully run sample apps against the Launcher CLI. These can be tried straight from the releases here. These apps should be a great starting point, or a tutorial for the app devs. One of them is in NodeJS and is actually a functional UI. You run the Launcher CLI executable, follow a very simple but intuitive menu over there, add the app into Launcher (for it to manage) and simply ask Launcher to activate it. After that it will do a bunch of NFS and DNS operations to basically create a DNS record for the user. Then we have the FireFox Plugin example to actually see this being pulled into the browser when you type it in the address bar. Again if you don’t want to go through the process of building it locally (which also we have made it pretty easy - just follow the instructions in the README.md of the repository), you can try out the pre-built releases hosted here. All good stuff.

Another sample application is written in Rust itself. It has extensive inline documentation and shows how to create and fetch a directory from the SAFE Network. It is very simple to code that because of the way Launcher abstracts all the gruesome detail and the inline comments explain every bit of it.

These will hopefully get people excited and working on their apps. While the future scope for the launcher API Is huge, the immediate focus is to get a Launcher UI designed and implemented. In addition, Launcher does not support anonymous users at present - all users are required to be registered with the network by creating an account. This prevents things like browsers to interface with it because all that the browsers care about is the retrieval of data as opposed to the need for registration. The goal here is to ensure that everyone, provided they have an Internet connection, can fetch any data from the SAFE Network and if that data is public, i.e. not encrypted, then readable. So there is going to be some thoughts on how to make Launcher cater to requests from anonymous clients like the browsers. As usual we will go via the RFC route and people can see and comment on the suggestions.

Vault (@qi_ma and @Fraser)

  • Network tests were refactored to be more efficient in terms of system resources and running time.
  • Resolved issue 259 (the DataManager now fetches a copy of chunk which has been stored by a PmidNode which subsequently disconnects from the network)
  • Resolved issues 261 & 263 (the DataManager persona now fetches a copy of chunk which has failed to have been stored by a PmidNode)
  • Resolved issues 267, 268 & 269 (the ChunkStore now cleans up after itself properly)
  • Resolved various outstanding FIXMEs and TODOs
  • Began an RFC for the upcoming sprint regarding the naming of ImmutableData

QA (@Ross and @Fraser)

  • Windows and OS X Vault installers are now signed
  • Set up a Windows 7 test environment
  • Linux Vault install process was updated to use company keys rather than test keys
  • Vault installer workflows have been documented
  • OS X Vault installer was fixed to handle upgrading the installed version

So, we didn’t achieve everything we set out to in the rust-5 sprint, however, the Launcher API is a significant step forward and a really nice deliverable. Despite the bugs in Crust and Routing we have made some really nice changes and once we have the remaining bugs fixed we will be able to move the whole network forward. We will obviously keep you updated with progress as we go. One of the great advantages of having a greatly reduced and less complex code base is that we should be able to identify and solve these problems much more quickly than before. Enjoy using the Launcher!

Cheers,
Fraser.

Transcript of Dev Update on Monday 2nd November 2015

44 Likes

It is great to see you guys moving forward. I am sure these bugs will be dealt with soon!

Edit: HAH! First one to comment.

5 Likes

hahaha … I could be endlessly tormented at the various airports on my way here - getting rid of them was a sane choice :smile:

10 Likes

I know this is probably in bad taste @Fraser but I have to ask, do you have any guess as to when we might be seeing the launcher CLI alpha?

1 Like

It’s there already - the update talks about it - here’s the link again anyway:

or if you want to build it yourself, just run cargo build --example launcher_cli --features use-mock-routing on the cloned repository.

5 Likes

Oh I know @ustulation but I was meaning when will be able to connect to the SAFE network proper and not running locally (mock build)

or am I missing something?

1 Like

It’s pretty difficult to answer accurately since it’s dependent on finding and fixing any critical bugs in the codebase. Suffice to say it’ll be as soon as we can manage it. I’m sorry I can’t be more specific.

8 Likes

cheers, may Jesus help you guys to solve the bugs asap. :grin:

4 Likes

thanks for the reply, The launcher and the example apps will keep me more than busy :grin:

He’s at the Troon office already:

15 Likes

Sorry but that’s just disappointing… Maybe because today’s my birthday idk :smiley:

2 Likes

Thanks @Fraser and team for this extra deep and thorough update. I like having the extra detail on the libraries from those handling them.

I thought this was an extra ambitious sprint and am not surprised it was too much for the time allocated. If anything I’m surprised you came so close - literally a handful of bugs short off a full NAT handling public test network! That’s an amazing feat. What’s apparent is that decoupling the libraries and particularly the JSON API makes it much easier for the different teams to work independently which I guess is a boost to productivity.

I’m hungry for more as well! So some questions… :smile: Thanks for all the Apps to play with - I haven’t looked at them yet, so if any can answer my questions please say so and forgive me. Here goes:

  • regarding building apps, the Lancher JSON API remains limited to authentication and Drive I believe, which means I’m still stuck personally because I need to get to grips with other areas (Structured Data, messaging/notifications for a start). I haven’t grasped the power of Structured Data yet and need to play with it in the absence of more info, and due to build difficulties with safe_firefox_addon / safe_ffi have not managed to find a way to create a test App to play with it yet. I posted some details of these problems (see link) but am not sure the best way to move forward. It sounds like waiting for Launcher to expose these APIs is not happening soon, so can someone help us (me and @smacz if he’s still interested) build safe_firefox_addon / safe_ffi? The first question that might help help should be easy I think: what build environment you used, but obviously that might not be enough.

  • I’m assuming that the differences between mock and real routing are purely behind the scenes, and for an App the difference is going to be negligible. Is that accurate, and are there any things App Devs should be aware of that might affect us?

There, and I didn’t even mention ARM and temp_utp … oops! :wink:

6 Likes

Love these updates guys. It’s now so easy to tell what’s happening and what to expect.

The API is my main concern, and I will be pouring over the code to figure out what’s going on. When new APIs are added (and I have no doubt that more will be added if only out of pure necessity) will this RFC be updated, or will there be further documentation that will be maintained that I can refer to?

4 Likes

As for delivering user value, I viewed this sprint’s goal as allowing users to put up simple web pages on a real network. Is that looking feasible for the next sprint? I know you haven’t committed to anything yet or set a goal, I’m asking more because there’s a lot of moving parts and I can’t totally interpret how they fit together from the dev update. (Thanks! It feels weird being a clueless user for once.)

1 Like

Welcome to the forum.

This should come around when the bugs are out. So at this sprint, although nobody knows if it’s the coming days or maybe after a week or more.

5 Likes

Ok 26 likes in the first 3 hours is GARANTEE a new record!! :smiley:

3 Likes

9 new users today on the forum, that’s another record in months. But sssstttt don’t wake the mods, it’s off-topic from the OP :wink:

10 Likes

Oops, I figured the team jumped from one sprint to the next. Not sure what to expect then. It seems like the ability to create public web pages (on a non-mock-network) is held up by underestimated issues in the routing library. I just hope it comes together this year, because there’s really no user value until we can connect.

1 Like

Is it true he turned the doughnuts into yum-yums?

2 Likes

Happy Birthday :gift: I’m broke as a joke so I can only give you a digital gift :stuck_out_tongue:

3 Likes