MaidSafe Dev Update - October 26, 2017

We have mentioned a few times recently that scaling the dev team is a significant priority in the near term. With this in mind, we’re delighted to announce this week that we have a new office focussed on front-end development in Chennai, India. Work has already started planning and designing the layout, look and feel. We have an optimistic timescale of 4th December for entry. We do hope to achieve that date all going well.

We have found that while remote working is very effective we also benefit from co-locating Engineers for certain tasks such as; Design, knowledge sharing and training. A well skilled local workforce makes Chennai a compelling choice for rapid team expansion. Krishna will head up the technical side of the office which will already have 4 team members (@Krishna_Kumar, @Shankar, @Srini and @Rachit) when it opens in December and we will recruit as aggressively as we can, without impacting quality.

In addition to building a developer team, the office in India also provides much better access to a large local market. With a population of 1.3 billion and an estimated 462 million Internet users, India has enormous potential, and as the office becomes more established (initial focus will be recruiting) attention will shift more toward marketing the SAFE Network to local app developers. We’ll keep you updated with our progress as we move forward.

This will not negatively impact our recent partnership in MaidSafe Asia, but will compliment such relations with closer technical ability and support being very important to partners. MaidSafe the company will extend its reach in many places over time, we hope. We will use this mix in Asia to test our assertions with a tight budgetary constraint. SAFE will feature very highly in both MaidSafe’s and our partners marketing, but there may be local bespoke solutions that are best handled directly.

Now on with the technical part of the update.

SAFE Authenticator & API

safe-node-app v0.4.1 has been published on npm, incorporating the following enhancements:

  1. Constants that were internally used by the package are now exposed in the API so client applications can also make use of them. The documentation was updated and you can see more details about these constants here.
  2. The notification/callback sent upon a successful network reconnection attempt is now generated and sent by the underlying safe_client_libs rather than by safe_app_nodejs.
  3. The safe_client_libs have been upgraded to incorporate the base64 library update (see the next section below for more details).

The work related to the refactoring of the example apps is almost done. @shankar will be integrating the tests for the web_hosting example and email example app. We will also be focusing on improving the CI scripts for building the example apps and browser. @rachit has been working to set up a scalable project structure for the Java API using Gradle. Once this phase is completed, the API implementation will begin.

The following fixes and enhancements were made to the safe_browser after the alpha-2 release:

  • Fix the issue with favicons which are now loaded and displayed for safe:// sites.
  • Warn the user upon a network disconnection event in the browser, not only from the Authenticator page but in any open tab, and attempt to automatically reconnect every 30 secs if the user doesn’t explicitly do it.
  • Assure that when reconnecting to the network, not only the Authenticator connection is re-established but also the ones for the safe_app plugin and the browser app so the browser state can be saved on the network afterwards.
  • Functionality to run tests on the DOM API functions within the browser, and creation of a checklist document for manual testing. The document can be viewed here.
  • Fix URI scheme registration for dev mode so the browser can receive authorisation requests even when launched with yarn start.
  • Minor enhancement to README for Windows build instructions.
  • Fix minor issues when fetching sites with a relative path.

We were aiming at releasing a new version of the browser today with all the above-mentioned fixes and enhancements, but during our internal testing, we found the browser was attempting to connect to the network as soon as it is launched which we don’t consider desirable. We therefore would like to fix this behaviour, which was recently introduced with the enhancements made to the disconnect/reconnect flows, before formally publishing a new version of the browser. We are now targetting a release of the browser for early next week.

@joshuef has been partially working on automating the execution of the browser tests using Spectron, and also looking into scoping Firefox and Brave as an alternative to the Beaker browser. He has summed up his findings in this post on the Dev Forum and we are looking forward to receiving inputs from community members to help us decide the best way forward. We therefore would like to encourage everyone to provide comments and thoughts about this on the Dev Forum.

We are planning to do some more research on the use of WebAssembly and how we can make use of it with our safe_app_nodejs package so that the same API can be used within the browser towards our intention to have unified APIs.

SAFE Client Libs

Starting this week, we are looking into ways to achieve binary compatibility. What this means is that we want apps linked to different versions of SAFE Client Libs to be able to read/write each other’s data on the network. That especially concerns the core apps, like Authenticator: as we improve them, we also modify the app data structures that are stored on the network, and currently there’s no straightforward and standard way to migrate existing data that are already stored on the network to newer versions. However, while we can migrate serialised data and evolve structure schemas, there’s a more complicated topic of evolving core data structures, like Mutable Data, without requiring to restart Vaults, leading to data loss. So for us, it is important to devise an approach that will allow upgrading the network versions without losing the already-stored data. We haven’t yet decided on the approach to achieve this, but we are in the process of intensively researching this topic and soon we will be starting with contemplating a new RFC that should answer these questions and involve the community in deciding which way is the best and what trade-offs we ought to make or not.

The base64 crate version that we were using (4.2) had some security bugs. We are upgrading to the latest base64 version (7.0) in our master branch and we have already upgraded in alpha-2. We also are fixing a bug where re-authorisation of apps was consuming 2 PUTs when this operation should not have been using any mutations. The issue was that we were updating the app container on every authorisation, even if the container already existed and there was nothing to update; now we update it only if it is missing some permissions. In addition, we had raised a PR a couple weeks ago which will be replacing all value structs in our FFI interface with struct pointers. We have updated the PR to change FfiResult to *const FfiResult everywhere in our API, removing the final inconsistency.

Routing, Vault & Crust

The backend team have largely been working on the design of Routing again this week. @bart has mostly completed a refactor of Ewok (the network simulator) to reflect the current notion of the proposed Routing ruleset, which is much closer to the original Data Chains. This refactor of Ewok should help us analyse the protocol, and see any implementation difficulties which might crop up when we’re building these into the real Routing library.

As well as the refactor, there have been several design Hangouts where the reworking of Routing and Vaults has been discussed at length.

The futures-based Crust implementation has been merged into the main Crust repository and is available on the dev branch. Our new Engineer Povilas (@povilasb) has been getting his hands dirty exploring the Crust codebase and asking really probing questions to the team (great to see). He has been extremely efficient understanding Crust and getting the test suite passing again. Work is ongoing porting our P2P NAT-traversal library over to futures+tokio so that it can be more easily integrated with the new Crust.

83 Likes

India wow :slight_smile::smiley::smiley:

Hats off guys :wink:

Tech update spot on as always.

I love how this project is like one giant jigsaw puzzle with the pieces slowly and methodically being put together on a number of different levels.

We all, I think, underestimate the enormity of what is going on behind the scenes preparing for the approaching tomorrow today. Can’t help but admire @maidsafe in many many ways.

27 Likes

Very detailed update, as always :+1:. Great to see the expansion.

15 Likes

Excellent news !! Thank you very much for keeping us up to date, it is really inspiring to see that the project and the team is vibrant , well alive and kicking !
I am a very pleased to learn that new team of highly skilled people are taking this stuff seriously and are going to work on it all around the globe.
Neat stuff on browser reconnect and ability to snake around versions !! These make sound basis for the future :slight_smile:

safe-node-app v0.4.1 has been published on npm

Hey isn’t this a great milestone !!

14 Likes

An office in India!? That’s awesome! I’ll sleep incrementally better as we tick off continents that have a Maidsafe office :stuck_out_tongue_closed_eyes: go @Krishna_Kumar!

17 Likes

What are the hangouts reworking routing and vault addressing?

so am I to hear this! smart move :wink:

5 Likes

Thanks Maidsafe devs for another great update.

Show your teeth aggressive ants and get other super ants on board. Recruiting is always good news…:stuck_out_tongue:

4 Likes

Data chains, although we have tons of hangouts all the time. That and slack is how we communicate really as we have everyone all over the world. So we try to limit local conversations and use these tools to keep everyone involved as much as possible without distracting them.

9 Likes

Scotland and India in the house!!! Scotch and curry niiiiiice

11 Likes

Not to get anyone off their work, but can someone give like a little insight in what the problem is here? And how the solution should look like? It has to do with one group of Vaults getting a message from another group of Vaults. And the questions (problem) is: Is this indeed a valid group in the SAFE network. Is that correct? I remember talking with @AndreasF about this and I thought the idea was to let every group (HOP) sign the message that comes by:

Disjoint routes
With this design, each section member will know all members of the connected sections. Therefore route n can always pass the message to the n-th closest (to the recipient) member of the next section. That way, none of the routes can intersect.

I’m currently failing to get a grasp of the problem here. Is it this part, or is there more to it?

6 Likes

Welcome to Chennai! Let me know if you need Non technical support :slight_smile:

4 Likes

This is the case for secure message transfer where each section confirms message validity.

The routes point is where we used to (still do but wont soon) send messages via various routes, delegating the send to a single node. Then we try each of the next 7 routes. Now the network will use a series of votes between sections to speed up and basically send a smaller amount via every route in parallel, if that makes sense. The routes was delegated sender, new design is all nodes send to all nodes, but only send a vote, not the whole message.

10 Likes

I understand that a lot of spring cleaning is happening in preparation for the next big milestone but what I’m really hanging out for is a picture of the new headquarters with Maid signage. Please put me out of my anxiety

7 Likes

There has been pictures posted already as I’m sure you’ve seen, and signage is to come which I think David said.

https://sjc3.discourse-cdn.com/business5/uploads/safenetwork/optimized/2X/9/96ec61171de9eff69cdb4c900863ea2ace88f0d5_1_666x500.jpg

6 Likes

Thanks for the recap neo. Impatient me is hanging out to see the interior and Maid signage to show to my friends who still think Maid operates out of a rented flat, due to the Google pointer error. Hopefully it will be a good Christmas present for forum members.

1 Like

Haha yeah, imagine the story to the uninitiated when they in the years to come look back over the projects history and read about…moving from the ‘flat’ over into ‘Ayr’.

The project that shaped the digital world which was looked at by most outsiders for many a year as vapourware actually existed the whole bleepin time right in that there Ayr.

You just can’t make that stuff up :rofl:

6 Likes

Arrr this topic


https://sjc3.discourse-cdn.com/business5/uploads/safenetwork/optimized/2X/d/d16eba50682b8c26ef5c6201b9fc5a21d703428c_1_690x262.png

https://sjc3.discourse-cdn.com/business5/uploads/safenetwork/optimized/2X/8/80fa09c1646d4d319fe8ea18a95c00fa2b282d36_1_690x328.png

We’ve still to start work on the interior @Cryptoskeptic, it’s really a very blank canvas at the moment. We’re currently working on making sure that as we expand globally we maintain consistency across all our sites and we’re working to establish that look and feel at the moment. We will hopefully be able to reward your (im)patience soon enough.

11 Likes

Nice! Across from a Funeral home, where blockchain is buried…

6 Likes