MaidSafe Dev Update - March 15, 2018

Here are some of the main things to highlight this week:

  • We have three new team members joining us this week. Welcome to Pierre, Lionel and Kayley!
  • We’re considering a different topic for the second CEP video. After discussing a few ideas, we believe that Proof of Resource is now a prime candidate for the video treatment.
  • Today, the front-end team is releasing a new version of the Beaker SAFE Browser (v0.10.1). Please use this link to download SAFE Browser v0.10.1 and to see more details about this release.
  • The front-end team is also sharing a new set of Peruse packages, available here. There are now development builds of Peruse, which should simplify the app development process and testing with mock networks.
  • The SAFE Client Libs team is continuing to straighten up the language bindings. They’ve finished the Java/JNI part and are currently working on merging the Java bindings together with the C and C# bindings.
  • The Routing team completed the vanilla implementation of the push-pull gossip protocol and have got some results from running the test with different network sizes.
  • The Crust team slightly refactored all of Crust’s examples and spent quite some time on the peer-to-peer chat example.

Marketing

New team Members

First up today, this week saw us add three new members to the team. As is tradition around here, each has penned a quick blog post so you can learn a little more about them - welcome to Pierre, Lionel and Kayley!

CEP Videos

As mentioned in past updates, our Safecoin CEP video is progressing nicely. Our plan had been to use the extra CEP money that was raised to produce a second video explaining the consensus mechanism for the Network (see this post for more context). However, in developing the script, it soon became clear that we could do little more in this format than introduce the concept of consensus. Whilst we intend to tackle this topic by video in the future, it feels that at this stage, we could have a greater impact by focusing on an alternative topic.

We’re now considering a different topic for the second video. After discussing a few ideas, we believe that Proof of Resource is now a prime candidate for the video treatment. It’s a defined area that we can tackle clearly in order to set ourselves apart from much of the existing blockchain scene. We now have a script pulled together but we’re interested to hear your thoughts below - please do comment below if you have any input here.

Meetups

On the community front, @harvindar and @wong worked hard to organise a meetup last weekend in Kuala Lumpur (for a report see this post) and @PeterRobertson ran a meetup in Melbourne with the SAFEDev Melbourne Group. We’re also in the process of organising a call next week with all of the individuals who’ve shown an interest in running SAFE Meetups where they live.

DevCon Update

There have been a few conversations during the past week about SAFE DevCon 2018 on April 23rd. Just to recap for everyone’s benefit: if you have a space at the Conference, we’ve already booked you a hotel room for two nights in Ayr (on Sunday 22nd April and Monday 23rd April). The full agenda will be released closer to the date but SAFE DevCon 2018 itself will run from around 9am to 6pm. At that stage, we’ll transport folk to another venue where we’ll put on some food to keep the conversations going. So if you’ve still to book your travel, plan to be in Ayr for Sunday evening and leaving Ayr Tuesday morning.

SAFE Authenticator & API

Today, we are releasing a new version of the Beaker SAFE Browser (v0.10.1) after a few days of internal testing and fixing some minor issues encountered in this process. This new version upgrades safe_app_nodejs to v0.8.1, the native SAFE client libs to v0.6.0, and it incorporates the corresponding DOM API binding functions for the two new functions recently exposed in safe_app_nodejs v0.8.x: readGrantedPermissions and getOwnContainerName. A minor issue related to error codes not being returned from the DOM API functions is also fixed in this version (thanks @larsmanson for reporting it and for helping us troubleshoot it!). Please use this link to download SAFE Browser v0.10.1 and to see more details about this release.

During our internal testing of the new version of the browser we discovered a bug in our packaging script for safe_app_nodejs v0.8.0, therefore a new patch version (safe-node-app v0.8.1) was recently published to fix this problem. The safe_app native lib was being included in the package, which is unnecessary because it’s actually downloaded at the target platform when the package is (npm) installed. Due to this, we changed the way the files are chosen to be included in the package from using a blacklist, with .npmignore, to a whitelist with the use of the files field in the package.json (more information about this npm feature can be found here). We believe this is better to avoid delivering/including unnecessary (or even unwanted) files within our package as it now needs to be explicitly done by changing the whitelist.

@nbaksalyar helped in fixing the issues we had with the autogenerated bindings for Java. We will be starting the integration and wiring up the tests alongside to help identify issues and fix them early. With Lionel joining as a Java developer, the work should be progressing faster.

While we’ve made some great progress with UX wireframes and mockups, these have also led to further questions and debates. We’ve come up with a few possible directions we could be taking authenticator UX enhancements in Peruse, so we’ve set about outlining these various options, their pros/cons as we see them and will be presenting them to the community for feedback soon.

We are also sharing a new set of Peruse packages, available here. We have not only upgraded the SAFE libraries dependencies in these new packages, but we have also automated the creation of a development build of Peruse SAFE Browser. As you will notice the dev builds are larger, this is because they contain the required libs for both accessing the Alpha 2 network and a mock network. These are intended to simplify the app development process and testing with mock networks.

To use this feature, ensure you’ve downloaded a dev- prefixed version of Peruse package, then for using a mock vault, you should start the Peruse browser via the command line with NODE_ENV=dev set, and use the --mock flag, e.g. (on macOS): running NODE_ENV=dev open Peruse.app --args --mock from inside the folder containing Peruse will open Peruse against a mock network. If you mistakenly try running --mock on the non-dev package version of Peruse, you will get an error that we are planning to replace in the future with a more explanatory warning message.

As a reminder, these are early stage builds of the Peruse browser and we encourage everyone from the community to report issues and/or provide feedback :slight_smile:

SAFE Client Libs

We’re continuing to straighten up the language bindings. We’ve finished the Java/JNI part: now we have updated to the latest stable version of the JNI library and added proper error reporting for Java exceptions (this should considerably simplify the integration process for the front-end team). We’re currently working on merging the Java bindings together with the C and C# bindings. There are also several pending tasks for the C# part that need to be done before pushing these changes upstream.

Routing & Crust

We’ve completed the vanilla implementation of the push-pull gossip protocol and have got some results from running the test with different network sizes. In the table, the leftmost column indicates the network size, also referred to as n. Since we see gossip being used to replace swarming messages within a section, probably the top two rows are most relevant (network sizes of 20 and 200).

  • “missed percentage” indicates how effective the protocol was at delivering all messages to all nodes.
  • “rounds” indicates how long it took for the delivery of a message to complete.
  • “empty push & pull” refers to the tiny messages nodes exchange to indicate they have no outstanding info to gossip about; the figures there indicate the total number of these exchanged in order to gossip a single user-message. These numbers are possibly somewhat high since if there were many user-messages being given to the network rather than just one, there would be far fewer empty push/pulls on average.
  • “full copy sent” shows how many full copies of the user-message were exchanged. This column and “rounds” are probably the most important since they roughly equate to volume of traffic and speed of delivery.
  • “n * n” is for comparison with the naïve swarm approach where every node gives a copy of the user-message to every other node. This in theory should happen in a single round and hence be quicker than gossip. However, in practice that may not be the case since the volume of traffic exchanged in that one swarm round could well take far longer than several gossip rounds with very light traffic.

The figures are in line with the theoretical values from the paper, and highlight the benefits of using a gossip mechanism, while the code gives us a good starting point should we decide to integrate this gossip protocol into Routing.

We are now investigating ways of linking messages to create a Directed Acyclic Graph (DAG). This is being done with a view to integrating it with a gossip protocol and considering how such an approach could benefit the SAFE Network. This feature would mainly help the SAFE Network with the absolute ordering of group consensus. We strongly feel this would remove a substantial portion of eventual consistency issues in that part of the code and thereby reduce the amount of testnets we will require to move on to a feature complete network.

In Crust, we implemented some bootstrap cache tests to check if the cache is actually updated when possible. We also fixed some tests randomly failing on Windows - service discovery tests would fail because of too short timeouts (200 milliseconds might not be enough for peer discovery on LAN :smiley: ). In addition, we slightly refactored all of Crust’s examples and spent quite some time on the peer-to-peer chat example. The PR is not merged yet, but, in short, the chat example will have a nicer output and will even be able to exchange files! Also, our net simulator library is at a very infant stage and requires a lot of attention. We’ve witnessed occasional segfaults which are hard to debug and unfortunately we haven’t fixed them yet. Although, the library is usable and it helped us to implement automated tests to check how Crust bootstrap and rendezvous connections react to laggy, lossy network connections.

76 Likes

Define: CEP …20

2 Likes

Community engage program

7 Likes

I just added a link to this post to give more context about the second CEP video :slightly_smiling_face:

15 Likes

Oo safe utilizing decentralized close group consensus, Torrent-like DHT-tech, data chains for restarts and security, node aging + possibly DAG-technology where it looks promising :open_mouth:

now this is beginning to become a picture with more facets than i ever expected but could become the perfect composition of decentralization tech :thinking:

20 Likes

Great update!!

Am using the Memes to draw peoples attention on social media to the thursday night update.

image

13 Likes

3 new hires? welcome welcome :slight_smile:
i don’t know any other project in this space that is hiring so aggressively at the moment.
what you need all these ppl for? re-invent the internet? :wink:

16 Likes

Welcome to all the new folks! Great update as always. For those interested… That gossip protocol is explained here in 5:30 min.

There’s also a full topic on this. Will look at this video tipped by @Viv as well. Lot of interesting stuff. And you never know when you’re writing an update to the SAFE Primer with @JPL about this stuff isn’t it :smile:.

17 Likes

Welcome Pierre, Lionel and Kayley!! :smiley:

18 Likes

Excellent update!
Welcome @pierrechevalier83, @lionel.faber and @kayley!

So segfaults are still possible with the Rust Language, it seems:

9 Likes

Just found this link that I think it can be interesting to read.

14 Likes

Pierre and Lionel welcome. Your backgrounds are very impressive (to a non technical like me). Kayley, you and me will get on fine. I will even understand what you say. An ex hairdresser and Media make up person. How cool is that

10 Likes

Could the Proof of Resource script or perhaps just an outline of touching points be posted? I think Proof of Resource would make a good video and it would fit best after a Safecoin explainer video as well. Describing incentives to join the network is a great start. Then moving on to consensus and then data chains would seem like a natural progression.

To me I feel like a really big picture view of the network, showing nodes joining and leaving, sections splitting and merging, nodes gaining rank (indicated by color), popular data propegating the network and opportunistic caching. Basically a visualization of all facets of the network in action would be the ultimate video. I’m not sure how you could visualize data chains but that would be cool too.

9 Likes

is this on facebook?

Thanks for the update Maidsafe devs

Welcome @kayley, @lionel.faber and @pierrechevalier83

POR CEP :exploding_head:

9 Likes

New Rust Dev, Android Dev and Accounts Admin is excellent news.

All those meetings are paying off :sweat_smile:

6 Likes

Who is this “off” that the meetings are paying. blame the coffee for the bad joke

10 Likes

Winning bigly, yuge, Vitalik

7 Likes

It is and much of it seems valid, but not all. In our case we don’t wish to scale a dag as its contained in a section and sections don’t scale for us, the numbers of sections does but not individual sections. This is where we get a win, almost like we started with solid sharding and now use something dag like for ordering (shared state agreement). Seems like we are in a good place with that part.

23 Likes

Is this the goal now, or is the goal still to deliver a Minimum Viable Product (MVP)? I never hear MVP mentioned anymore and, while a significant quantity of work is being accomplished each week as documented in these weekly updates, it is not clear how much of it is contributing directly to an MVP version of the network. It sometimes seems to me that work is proceeding mightily along a fractal line segment of infinite length and that we will never be able to close the distance between point A and point B.

1 Like