MaidSafe Dev Update - May 18, 2017


#1

Recent updates have been relatively brief and focussed as the team worked through the ongoing development issues. In this update we intend to provide a little more detail so this one might be a little longer than you’ve been accustomed to.

Firstly, as you all know we have been (trying) to build the team. The task of finding network engineers continues to be a challenge, but in this last week we are delighted to welcome @hunterlester to the team. Hunter has proven himself to be a skilled, highly motivated and passionate about the network and we think he will make a great addition. Hunter is yet another developer we have been fortunate enough to pick from our excellent community and he’ll be joining @Krishna_Kumar’s team. He is also currently plotting reviving the San Francisco meet up so likely that you guys will be hearing a little more from him in the coming weeks and months.

For quite a few weeks now majority of the Routing team have been looking into Data Chains integration and other parts that need to get added to address some of the requirements to reach our next milestone of enabling Vaults from home with Data Republish and network restarts. While the dev updates themselves have been quite brief, it’s been quite the opposite within the team with lots of proposals and discussions related to tackling these objectives. Two concepts stood out amongst the rest and we’re looking to summarise and publish both to the community to showcase the work that has been ongoing in this part of the team, and also give some details of the thought process and library components coming together to achieve these objectives. We’re hoping to have these out next week, but disclaimers first to say these have by no means concluded. However, they’ve matured a bit over the past weeks and we feel discussions on the dev forum would be of benefit and help move the process along. As you’d see from the routing section, the guys are working on simulations that can be used to test these concepts, confirm their reliability, while considering various other factors that can be expected in a live network.

From the side of MutableData we’re looking to start some internal network testing with clients early next week. The front end team with the guys from Safe Core have got the Vault implementation up to speed and with a few pending PRs the test suite in the Vault library is looking good :slight_smile:. Once we can confirm the same with some mini network tests via the safe core modules, we can start testing the network exhaustively for the various new features brought in via MutableData before moving them forward to the next test network with the Authenticator paradigm.

It has also been great to see the community helping each other out working on building mock routing. The MaidSafe team would love to have the time to help out but as you know we must prioritise the network and APIs. While we certainly wouldn’t discourage the community from trying to work build mock from scratch, it may make sense to use the [binaries] (https://github.com/maidsafe/safe_client_libs/blob/dev/safe_app/README.md) provided. Not only will this save frustration and enable you to focus work on your app, it would also help us receive some initial feedback on the APIs. For the die hards we will publish instructions in a future dev update. We also wonder if it would make sense to keep this type of work to the dev forum, as it would then help keep relevant discussions in the same place as any new potential dev to the system might be looking for help in the corresponding forums.

SAFE Authenticator & API

@gabriel is refactoring few API function names to keep the API consistent and friendly. The changes are also implemented in the DOM API side by side. We are reviewing the documentation and examples in safe_app_nodejs. We will be updating the example applications based on the safe_app_nodejs API changes.

We will be working on DOM API documentation and few simple snippets to explain its usage. Though the DOM API and safe_app_nodejs API are similar, DOM APIs return handle instead of Objects just like previous REST APIs. Hence, we are planning to create simple javascript snippets to demonstrate the DOM API usage. While reviewing the APIs, we noticed that the APIs are not yet exposed for freeing the handles returned from the DOM API. @shankar is updating the launcher UI based on @shona’s designs and it’s getting closer to completion. He will be working on exposing the DOM APIs to free the handles shortly.

@krishna-kumar and @kumar have made good progress with JAVA api. You can follow the progress with the Java API from the repository which should now be live.

SAFE Client Libs & Crust

@nbaksalyar is closely working with the frontend team in sorting out the API related concerns and bug fixing either in Javascript side or rust side as may be the case. Improvements such as bringing consistency to the APIs are always trickling in as can be seen in this PR.

Vault tests have now been updated to work with fake clock just like routing tests with mock-crust, so that there is precise and deterministic control of time manipulations to make sure containers and cache artifacts are indeed what we expect. This has lead to discovering a few further bugs which are being worked on and squashed.

Test cases have been expanded in order to cover more scenarios that we find as we think about additional permutations. After putting in these new test cases and fixing the new bugs discovered with the help of fake clock we will get down to reviewing this mammoth PR again and if nothing else stands we’ll merge it.

The Crust library has been updated and now the upper libraries can templatise it for a given Uid of choice, which with the help of traits can be expanded in the future to integrate secure serialisation in Crust. Given the templated nature of current Crust module this should be relatively easy to integrate.

Routing & Vault

The main part of unifying node/peer names/IDs is done. This concludes a significant portion of a cleanup in Routing Peer Manager and node modules. Quite a few containers in PeerManager have been made redundant as a result of this change while functionality remains the same. Based on this we will be able to do a few further simplifications, which will come later as smaller, individual changes but overall it helps structure up and coming feature work in a more concise manner.

The data chain design discussions have also been progressing along well with multiple hangouts discussing the various portions of multiple concepts currently being considered to integrate these features. We are writing more simulations to evaluate different proposed variants in terms of security and stability.

To give you some perspective, the most contentious and central questions are about how to get from what each node observes about the network to a cryptographically signed and independently verifiable chain recording the network’s history. For example, one node A might see a node C disconnect, and then a node D, while another node, B, might see node D disconnect first, and then C.

  • Is it relevant whether C or D went offline first, or can the chain leave that open and just state that both left?
  • Do the nodes arrive at a strict, linear history of the network via a consensus algorithm like PBFT involving several rounds of message exchanges before committing that history to their chain, or do they just inform each other about what they observe, and record those observations as the history?
  • The latter case is easier at first, but produces a more complex kind of chain which requires an elaborate set of rules to define how to read this chain as the network’s history.
  • Are the blocks in the chain “change events” like “remove node D” or “add node E”, or are they “states”, like “nodes A, B, C, E are in our section”?
  • Can we use a simple majority as a quorum, or do we need to use something like ⅔, as required by most consensus algorithms? keeping in mind not all consensus algorithms deal with a fully decentralised locally sourced groups with members fluctuating often and reliability that cannot be taken for granted.

Some of these questions don’t necessarily have a right or wrong answer, and through thorough discussions and extensive simulations we are trying to figure out which approach will best fit our use case. We’re hoping to have these and lot more such design discussions summarised and published to the dev forum hopefully soon :slight_smile:


#2

Surely not??..


#3

I am!..:grinning:


#4

Congratulations @hunterlester!, I find this to be an excellent news for the project.


#5

Great work can’t wait to start playing with the JavaScript api


#6

Congrats @hunterlester hard work lies ahead!


#7

Congrats hunter!

:slight_smile: :clap:


#8

Nice work @hunterlester ! And as always great thanks to @maidsafe for all the hard work.


#9

Awesome to see the team growing and congratulations @hunterlester . Hey @nicklambert quick question, why do you think finding network engineers has been so challenging? Is it Rust? Maidsafe? Are they all developing their own ICO? Working for Vitalik? What do you think?

Did the Asia trip help at all? Irons in the fire?

Thanks for the update, nice.


#10

Good to see thinks ticking along nicely .:slight_smile:

I’m sure we’d all love to help out. As a non-developer (or at least a very rubbish one) my contribution would be pretty limited, but from reading these forums I think others would appreciate some direction as to what needs testing, why and how.


#11

I think it is a combination of the newness of Rust and the fact that we’re quite demanding and specific about what we need. We certainly get a lot of CVs, but not many make it through.

It is still early days, but we’re optimistic about the future of the partnership. We have probably held them back a and they are straining at the lead a little, but getting through the next couple of dev iterations should see us in good shape and the opportunities in that neck of the woods are almost endless.


#12

Makes sense. Probably a dumb question but would infiltrating the wide list of Rust Meetups with some presence be of consequence? Sponsor a single evening with BEER & PIZZA and a remote presentation?

EDIT:

Meetup organizers are always looking for: Content and members. - Sponsors bring content, Beer and Pizza bring members

EDIT:

1 beer, each.


#13

Thanks so much for all of your hard work! And welcome to the new members!


#14

Congratulations @hunterlester it was only a matter of time. Saw this one coming a mile away :tada:


#15

Thanks Maidsafe team for all your hard work, always good to read that you’ve added another super ant to your team.

Congrats @hunterlester

Jippy :kissing_heart:


#16

Not dumb at all. I think it would. @ustulation recently presented remotely at the Rust London meet up and we should look to do more in future.[quote=“BIGbtc, post:12, topic:13726”]
1 beer, each.
[/quote]

1 beer each, we’re not millionaires you know :slight_smile:


#17

And rightfully so, you guys are shinning example of how things should be done!
Even though we wish you could do those things faster :joy:
Quality over quantity!


#18

A gentle reminder @BIGBtc : MaidSafe is a Scottish company :wink:


#19

If building the developer pool moves this project along, thats what we need to do.

So if we were to connect with all the Rust Meetup organizers and offer a slice and a beer to each of their participants in exchange for:

  1. Including the Maidsafe Logo on their meetup.com sponsor page for the event.
  2. Using social media to promote the event.
  3. Including a full text intro to Maidsafe, Safenet on the meetup page including some commentary about developer opportunities at Maidsafe. (prepared by Maidsafe)
  4. Including either a live feed of a (insert duration here) presentation or a prerecorded presentation as the focus of that specific meetup. This presentation would likely be along the same lines of what @ustulation has already done in London.
  5. Sharing images from the sponsored meetups - in real time- on this forum and other pertinent social media to further promote the cause. The cause being developer recruitment.

…Maidsafe would foot the bill - subject to cost?

Ideally this would be all done on the same date as in simulcast event, but that is a bit unrealistic, so if the first live event/s could be recorded, they could be played back to other event holders within a short period of time.

I would gladly work this with @whiteoutmashups and look for Maidsafe supporters to attend any event in any city. Im in Toronto and Gulf side Florida and can easily get to Miami.

First steps would be to connect with all Rust Meetup organizers to gauge interest. I’m up for doing that.

Thoughts?


#20

Thank you for the great update!

I’ve a question concerning [quote=“nicklambert, post:1, topic:13726”]
it may make sense to use the binaries provided.
[/quote]

Maybe I’m misunderstanding or missing something obvious, but that link leads to Build Instructions, not binaries…?

Thank you in advance for your answer!