MaidSafe Dev Update - November 30, 2017


#1

Today, we were planning to release new versions of the SAFE Browser and the example applications which have been updated to work with the latest API changes. The changes are mostly related to the safe_app/DOM APIs and also to improving the developer experience.

However, there is a limitation in the browser which we are working on fixing at the moment. If we open a web page, say www.example, it loads fine, however, www.example/index.html is not being loaded. We will be releasing new versions of the browser and the example applications as soon as this issue is resolved (tomorrow hopefully).

As we mentioned in a previous update, we would keep you up to speed with developments in our new Indian office. Circumstances have enabled us to take a bigger office for the same monthly costs within an IT-specific business park. The additional square footage will facilitate up to 30 staff as opposed to the previous 15. The one caveat is that the bigger office space requires more internal work and will therefore push out the anticipated opening of the office to late January, however, we felt that the bigger space was worth the slight delay. With the front-end team starting to make regular new releases, there is much to be excited about bringing this team closer together and growing its capability even further.

SAFE Authenticator & API

SAFE App Node.js and SAFE Web API have been updated to adapt to recent API changes in SAFE Client Libs, summarised as follows:

SAFE App

Addition of following functions to app object in safe_app_nodejs and window.safeApp in web API: isNetStateInit, isNetStateConnected, isNetStateDisconnected, clearObjectCache, isMockBuild.

Crypto

Addition of a signing key API on app.crypto object in safe_app_nodejs and window.safeCrypto in web API: generateSignKeyPair, pubSignKeyFromRaw, secSignKeyFromRaw, generateSignKeyPairFromRaw.

  • signKeyPair.secSignKey.sign or window.safeCryptoSecSignKey.sign to sign data on the network
  • signKeyPair.pubSignKey.verify or window.safeCryptoPubSignKey.verify to verify signed data on the network
  • Note that the safeCryptoKeyPair object is replaced by safeCryptoEncKeyPair and safeCryptoSignKeyPair in safe_app_nodejs andwindow.safeCryptoKeyPair is replaced by window.safeCryptoEncKeyPair and window.safeCryptoSignKeyPair in web API.

Mutable Data Permissions-set

Removes MutableDataPermissionsSet and window.safeMutableDataPermissionsSet.

Mutable Data Permissions

forEach changed to listPermissionSets, which now returns an array of permissions represented as strings, instead of the no longer existent permissions-set object.

Mutable Data

  • removed free function
  • removed newPermissionsSet function
  • getKeys returns array of mutable data keys
  • getValues returns array of mutable data values

Mutable Data Keys

Removes MutableDataKeys and window.safeMutableDataKeys.

Mutable Data Values

Removes MutableDataValues and window.safeMutableDataValues.


It’s suggested to browse documentation here to learn the APIs:

And to read the tests for each API to view examples of how each function works:

We have an issue with the loading of certain URLs in SAFE Browser. We are working on addressing this issue. We will be releasing the browser and the example applications as soon as this issue is resolved (tomorrow hopefully).

@joshuef has been working on the custom browser over the last couple of weeks and has made considerable progress. We are testing the initial version of the custom browser which makes use of the latest APIs and allows safe:// only browsing. The UI is still naive as the focus was on functional aspects.

A pull request for desktop support for the Csharp APIs has been raised and is currently under review. We do hope to get it merged once it is tested. @rachit has been expanding the APIs and test cases. @joy is beginning the integration with the Java JNI bindings provided by the core team. We are hoping to get the Java API covered as soon as possible for Android and desktop platforms.

SAFE Client Libs

We’re nearing the completion of the binding generator for Java and C#. We’re starting to work with the front-end and mobile teams to collect their input and their requirements for the bindings, and so far we’ve gathered some useful feedback on the C# bindings. With that feedback, we intend to cross off most of the TODOs during this week and start to integrate it with the actual .NET and Android/Java projects.

@marcin has completed the automated tests for the Alpha 2 network, which we mentioned in one of the past updates:

These tests already helped us to uncover some obscure bug which we previously didn’t encounter, it concerns the apps revocation on the Alpha 2 network using the current master branch build of SAFE Client Libs. We’re actively working on pinpointing the exact cause of this bug and fixing it.

Routing & Crust

As data chains discussions and designs are nearing finalising phase, we decided to start coding features to better express the fundamentals and continue testing the design. Work has started at last. We have branched off Routing master into dev. There one can now clearly see the fundamental types we will be using in the implementation of data chains. We’re currently working on a document defining and detailing the different parts of the changes that data chains integration brings and hope to have this ready and published on the Dev Forum next week (barring any nasty surprises early next week). In the interim, here are some of the key definitions that have been fleshed out in that proposal:

Complete group: GROUP_SIZE peers with age >4.

Elders: the GROUP_SIZE oldest peers in the section. If there is a tie, we use the tie-breaker rule for node aging.

Tie-breaker rule for Node Aging: If there are multiple peers of the same age then XOR their public keys together and find the one XOR closest to it.

Adult: a peer with age >4 in a section with a complete group.

Infant: non-elder with age ≤ 4.

Vote: a node’s detection of a network event plus its signature of it.

Proof: a node’s id and its signature against a network event.

Block: network event with a vector of proofs collected.

Group consensus: a block containing a quorum of proofs of the elders.

Section Membership: All the members of a single section, comprising the elders, adults and infants.

Neighbours: sections we are supposed to be connected to differing in exactly one bit from us.

Sibling: A section differing from us in the last bit.

Corresponding code is also being integrated into the dev branch, for instance here are the events that will be voted on. Once quorum or more have voted we form a block for the event with their proofs. Some of these blocks will go into the data chain. For context, here is a Vote which is exchanged between Peers directly. A Proof is extracted from a Vote to create a Block which gathers the Proofs and when it reaches a quorum of Valid voters the Block becomes valid. These valid voters are deduced from the chain order.

As we had stated earlier, in a section we will usually have GROUP_SIZE elders, some adults and some infants. Basically, elders are the oldest nodes in the section and it’s their votes that matter. If an elder is lost, an adult is promoted to take its place. Infants are the ones whose age is <=4 so they must stay long enough to see enough churn and get relocated for them to become an adult. Until then they do not contribute towards churn. This is because such young nodes have a much higher probability of coming and going offline thus the network does not rely on these nodes for general operation.

With that brief categorisation in mind (much more detail will be available in the document we are currently formulating), the (local) events witnessed by a section will be voted by the elders and when a block is formed as described above, we will decide whether that should make it into the data chain or a state-container. For e.g. if we were accepting some peer who is going to be in the elder group we put that block in the chain. If it was an adult or an infant we wouldn’t. This will reduce the size of the chain considerably because it’s elders who actually matter in the network in our current design. They are connected to the neighbouring sections’ elders etc. Infants can just come and go (because they have a small age, so they don’t have great incentive to remain online or they could just be spawned by casual/malicious users for appearing temporarily on the Network) and if we recorded all that, we would have ended up with an enormous chain. So one needs to “earn” one’s position in the chain - by becoming an elder and by that very virtue meaning that one has spent enough time in the network being a good node and also disincentivising them to act badly which would compromise their age which they’ve earned over a long time, few will wish to harm their nodes farming rate in that manner :wink:

This is also a very important defense against any botnet type attacks, or large adversaries with boundless cash who may wish to attack the network in huge numbers. The network simply won’t allow such mass joining in short periods, but will instead ensure joining is a process the network controls through need. Therefore the network will require nodes when it needs them at the rate it needs them and upon accepting nodes, the network will monitor and validate the capability of the nodes (infants) over time as they mature and “grow up” to become valuable members of the network. A very natural design as inspired by the natural world, as always, if you cannot find your design in nature, it’s probably wrong. We are delighted we can and these steps are very powerful in their simplicity.

Now that we have started with the code, we also thought it would make sense to quickly run some simulations to see things like what is the section size before we split (because we’re currently hoping to only split once we know both post-split siblings will have enough elders and adults) so that if we end up with sections with a great many infants then we quickly devise ways to not have so. E.g. for a start we could say in a section with proper categorisation of peers into elders, adults and infants, don’t accept any more infant with age of 1 if we already have such an infant. @bart’s created a small simulation to test this and it requires some finishing touches but it has already given us some very useful results.

@fraser and @qi_ma are working on finalising the detailed data-chains document which will have the definitions and explanations and also rule-sets for various scenarios such as merges and splits. This will include the message flow, which makes looking at the algorithm much easier so that it can be reasoned about or improved and finally put to code and tested.

In Crust, work has nearly finished with the integration of the p2p library. There’s still a couple of bugs that we’re aware of in p2p (e.g. it doesn’t close router ports once it’s finished with them and so ends up hogging all the ports), but we expect these to be simple to fix. Once those PRs are merged and tested we will be able to start integrating our uTP library as well and Crust will no longer be limited to the slow and peer-to-peer-unfriendly world of TCP. A significant step to a secured multi-protocol, randomised port encrypted network library that will benefit a great many projects.


No progress being made?
SAFE Browser v0.8.0 - Release
MaidSafe Dev Update - December 7, 2017
SAFE and Mesh Networking
#3

first :smiley: hooooooooooooo


#4

that was fast!
second …


#5

Awesome! Great improvements on the API.

Keep rocking!


#6

Holy moly - this seems to become a bigger deal in the near future than I expected :hugs:


#7

It’s very exciting to read this detail about sections and data chains :slight_smile:

Can you say more about the corresponding designs in nature? I get the general point about infants, adults and elders, but am wondering if there are particular examples from nature being alluded to, and any other elements (sections for example) which have natural counterparts. I find this aspect fascinating and think it would be a great subject for MaidSafe or somebody else, to write some publicity material about - the ways MaidSafe’s solution to building a never before seen autonomous network has taken its cues from natural systems. I think a lot of people will relate to that and find it fascinating too.


#8

Magnificent design: simplicity is the ultimate form of sophistication. Indeed, nature is simple yet complex, like the Mandelbrot Set. Great post!


#9

thanks for the update, exciting about the office in india!
@nicklambert: how did it go at google? anything you can share?


#10

@BambooGarden @nicklambert discussed that here - MaidSafe Dev Update - November 23, 2017


#11

This whole piece is amazing. Very interested to dive a bit deeper when the document is out :+1:.

:+1::+1::+1:

Thanks for the update, as always.


#12

You have to be careful you don’t build an oscillator. I can imagine borderline systems that can’t handle the full-node responsibility (limited or non-constant bandwidth, limited CPU, etc), doing just fine as infants, and then failing when they are finally promoted to adults. Then they get busted back down to infant and they’re fine again, because you’re not working infants very hard, so they get promoted back to adults, and the cycle repeats. Probably be taken care of by the initial capability testing that you do when a new node comes online. As long as that remains, probably no worries.

EDIT:

Never mind, it looks like you’re already thinking about this.


#13

Nice update. Great to see everything continuing to gel. Can someone define GROUP_SIZE for me?


#14

I think the point is that infants are worked, and have to perform. It’s just that they aren’t trusted until they’ve done that reliably for long enough.


#15

Just a suitable number of nodes(the reliable bunch - elders) we’ll have to pick(currently 8) to achieve a quorum with a majority(5) and the remaining portion(3) being “churnable” / out of sync nodes at a given time whose lack of participation doesnt impact consensus but would have consequences for them to deal with for not participating ofc. its also the size that when the section drops to (ignoring infants for now), it’d trigger merging with its sibling and so on.


#16

OK, so it’s different from section size in that a section can contain non-voting (child and adult) nodes whereas a group contains only elders?


#17

Not so much Mark, but the network uses the growing/maturing of nodes through a process of nurturing and testing as it grows in capability (more on this will relate to the AI advances well after launch, where improving neural networks will be linked to this). This ageing process is well defined in many species. The superpods/sections are intersting as well, they have a form of collective behaviour ( such as https://en.wikipedia.org/wiki/Collective_animal_behavior ) but a bit more, as a roaming individual tries to join a new herd (section) then it’s evaluated much quicker than an Infant, but still evaluated (like our resource proof).

The network does some more though, where the marauding Node that is being relocated brings with it a signed certificate of authenticity (like early humans in some ways). Even though they are still evaluated by the section they wish to join.

If a node is killed, it then cannot join any part of the network again with that id, so it can be reborn as a different id and start over. This is interesting as even a very distributed network like this any section can kill a peer for misbehaving and all it can do is die, it cannot rejoin the network as a valid rejoining node. So it is much more like the real world, where death means death :wink:

There are a few other wee bits and pieces. we should do a few blog posts on this biomimicry that we get inspiration from.


#18

yep, certainly different from section size. A section still would contain elders, adults and infants. non voting peers are just not part of the GROUP_SIZE nodes.


#19

:bulb: Got it! Thanks @viv


#20

Always nice reading. How many people is currently working on this project? Office in India for 30 new people? I thought there is like 20-30 people working on this now. Is there any plan on how many new people will join the project in next year/years?


#21

We are working on several expansion plans to allow us to expand quickly after Alpha3 :wink: