MaidSafe Dev Update - March 16, 2017

Today, we had hoped to release Test 15. We had fixed and patched the crust bug we had found in 12c, but decided on much more rigorous testing. We have found an issue that we could not reproduce, it showed up after several thousand iterations of testing and is related to section updates whilst two sections concurrently require to merge. There is a very small error window in that message flow. So we have decided that rather than put out Test 15, we will delay another few days to also fix this very rare, but damaging bug. The code is written and we are preparing the patch for review tomorrow and will run many testnets ourselves over the weekend and early next week. All things going well, we will release Test 15 at that time.

While this is not us saying we wish to be 100% bug free, we do feel that now, as we put out testnets, they should all be more feature complete, but importantly more stable. Such a bug is just not acceptable anymore. We understand there will be disappointment, but please bear with us as we continue the steady tested progress to Beta and beyond. We still feel Alpha 2 is very close. So we are all very encouraged by the results we are seeing in our internal testing.

On a bigger picture view, we have almost 70% of the routing team in design meetings with 2 current threads, one of these is comparisons of our consensus algorithms with respect to disjoint sections and comparisons with others such as PBFT and Tangaroa, to name a few. These meetings are producing results that are all very good and we’re hoping to get the discussions, once matured, moved over to the Dev forum while initial implementation can occur in parallel. We have not one way to proceed here, but several, and now will focus on the design that allows secure data republish and all that brings. It’s a busy time there, but a good time.

Much of the current code paths in routing are complex and we see these designs will simplify this a lot, leading to more efficient and more secure networks. In parallel, there are a couple of obvious simplifications we can make in production at the moment. This is also a process which again allows us to look at simplifying the system and hopefully be able to explain the details to interested parties in a much simpler way than we currently can.

@dirvine and @viv were at Glasgow University to promote SAFE to the 2nd year Computer Engineering students. The lecture went well and a further event will take place there in the next two weeks with some industry partners of the University.

Also, here’s Krishna’s presentation from the Jakarta conference. The sound is shaky, but it’s as good as we can get for now. We’ll make sure we coordinate with our partners in the future to ensure we get better quality. Enjoy!

SAFE Authenticator & API

The DOM API for SAFE Browser is now implemented. The safe_app beaker plugin will be injecting the DOM API into the browser, just like safe-js previously did. @bochaco and @shankar will be creating simple web applications to test the APIs and eventually publish the samples. We are also planning to port the existing examples to the new APIs. We picked the markdown editor sample as the first example to port to showcase the usage of the DOM API.

@lightyear is porting the email example to the latest APIs. Also, we pushed minor improvements to the build process of the safe_app_nodejs library.

We’ve got a few new team members starting soon here. One of them is experienced with Android and should help with taking the Authenticator and clients libs over to mobile (don’t worry if you don’t use Android, we’re also going to be covering other mobile OS soon). We’ve also got @Shona joining the team again very soon. Most of you might already know Shona as the front-end designer who was away for a while, so we should be seeing some UI & UX updates in Authenticator and the client apps coming soon :slight_smile:

SAFE Client Libs & Crust

The MutableData implementation in vaults is progressing and we have a WIP PR. There were many potential test cases missing so those have been added for ImmutableData, covering churn scenarios too. MutableData handling under churn is ongoing and there is a simple test for that added too although it requires more functions in data_manager.rs to compile. There are also broken integration tests because of the changes which will be fixed once module-level code is ready and tested.

safe_client_libs has had a couple of updates, one of which is adding some public APIs to list the content of access_container as demanded by the front-end team.

A new version of crust (v0.22.1) including the patch for the bug from Test-12c has now been published to crates.io.

Routing & Vault

While a lot of routing-related work is currently going into planning and design, we also fixed several issues in preparation of Test 15:

  • For historical reasons, routing was using several different hash functions in different places. We replaced them all with the secure SHA3: #1376
  • Routing now tries re-establishing lost direct connections before looking for a tunnel: #1389
  • More fixes related to tunnel nodes: #1391, #1392

Two patches, one which is already in a PR and another to fix two stages of the issue described in the intro section, are also being worked on which should hopefully address the merge issue rarely putting merged sections out of sync between the members of the new prefix.

56 Likes

Am I really first? I can cross this off my bucket list!

Thanks so much to the team for all of your hard work!

13 Likes

More time for me to think on how to call that nodejs function webFetch… still haven’t quite figured that out.

Glad to see effort being put into ensuring a stable Alpha 2. :thumbsup:

12 Likes

Thanks for being thorough and careful! I’m sure alpha 2 is close enough.

11 Likes

Looking forward to Alpha 2 when it’s ready and great to hear new team members incoming :slight_smile:

12 Likes

Doh, Oh well, at least it’s probably only ‘a few days’ rather than next week.

Better to get it right the first time than go off half-cocked I agree.

/Jabba puts his bottle of good whisky back in the cabinet for another day and sticks the kettle on :stuck_out_tongue_winking_eye:

15 Likes

What can I say, nice update as always. The Vaults are getting more stable after each bugfix so not disappointed to wait a few more days. Welcome back @Shona.

This is very great to hear. SAFE on it’s way to an even more userfriendly experience with the SAFE Browser as the main entrance. Keep up the great work!

21 Likes

badaaaaam :smiley: excellent news =) [damn… i might have to get an android phone next … i love my ubuntu phone … but i want to test mobile safe apps Oo]

16 Likes

Great to see MaidSafe went to Glasgow University to keep a lecture.

I just got my Scottish (haha) CS bachelor degree last year from there and I wish I had that chance to see you guys :wink:

15 Likes

Love the level of testing we’re doing internally prior to rolling out additional tests. Also Love that we’re having to go through thousands of iterations prior to a bug showing up! Hopefully we can take care of this bug and release Test 15 C next week. Been extremely busy this week but should be more free next week and looking forward to testing 15 next week as its released!

@everyone keep up the great work! Welcome to the new members to the Maidsafe team!

14 Likes

Thanks Maidsafe team for another week of hard work.

Huh how can people be disappointed? it’s you guys who’s doing the real hard work. If people get ideas of being disappointed they should start helping you guys with coding or hire someone to help you guys with the code. :stuck_out_tongue:

:kissing_heart:

:kissing_heart:

I’m really missing the SAFE Network, maybe we should always have a parallel community powered network running, so that we can show newbies the network at these intervals… just a thought

9 Likes

3 posts were split to a new topic: Everyone group (access denied)

First of all great work and good decision to do further testings to make Alphas as stable as possible.

What confuses me though is
“We have found an issue that we could not reproduce” … “The code is written and we are preparing the patch”

Were you able to pinpoint the bug without reproducing it? Or are you just hoping to fix it with the patch?

Also, if the bug occoured only after thousands of iterations, wouldn’t it be better to get it out in front of as many people as possible in form of a test net? Or is a release too much overhead / effort?

Would be interesting to hear your thoughts about this.

5 Likes

From my perspective, the guys found an issue that only occurred after thousands of attempts which highlighted a very small error in that message flow. So although not reproducible in our tests, it did exist and we thought better to get it fixed before we release the next test net as repercussions from bugs cannot always be fully understood straight away.

Setting up the test nets does take some time, but this is more a case of putting the best foot forward that we can during each iteration. We’ll have test 15 with you before you know it.

18 Likes

The bug was that we managed to get a failing section. On verifying the message flow the only way we could see that happen was in a small window of error. This error was a group merge happening concurrently with a neighbouring group not seen by all members of the first merged group (half of them knew of the remote group) and during this time the remote group sent the new group prefixes to that half of the current merging group. This would not accumulate securely so was not accepted, however then we had an issue that a remote section was not as it should have been.

The fix is to cache remote section changes whilst we are merging, on merge complete then ask again the remote sections smallest prefix for a section update.

This resolves any in flight section changes whilst a section is merging.

All of this actually may go away with some of the design teams work as it’s job is to find fundamental rules/algorithms and ensure they are applied consistently. Prior to that we do need to find edge cases and fix these, as this gets complex then the design team step in an look for the fundamentals. It’s all hard dedicated work, but that’s our job :slight_smile:

TL;DR We found a code path that could fail and seen a test iteration that may have experienced this failure, so we patched it.

23 Likes

Great job! And welcome (back, I guess, but I wasn’t around at that time) @Shona!

4 Likes

Glad to hear @shona has returned!

we all missed her voice on the SAFE videos!

7 Likes