MaidSafe Dev Update - March 2, 2017 - Test 12c

Not able to access 12c via launcher or vault today. My vault has been running constantly since last Thursday.

I am unable to log in also. I can browse to my webpage, and some of it comes up, but it’s missing the picture. We may have had some data loss when we dropped from 174 vaults to 66

100% we did. Some fixes today and possibly again tomorrow (well today :wink: )

6 Likes

Is the fix out yet? Is the entire network disrupted or is this only a hiccup?

12c as a testnet is dead really. The fix in crust is there (in github), we are also looking at a couple of things in routing where it should have recovered from even that weird connection issue. It’s good actually as the issue created what would be like a massive ddos attack (which is really really hard to do on the network) . SO this has been a really good test and has given us an awful lot of info for this bug but importantly much more faith that the huge changes we made in the last few months also work. So good news.

We know tests are a pita for us all with many folks working hard on each test, but the rewards are really superb, we could never have found this in local testing, well not for months.

31 Likes

Got it. Sounds successful, keep up the tireless work.

3 Likes

Just a small update for anyone checking this thread later. Test-12c is now offline. We’ve identified(well the patches are already in PR now too and have cleared CI) a bug in crust as @dirvine mentioned earlier. We’ve also got routing updated slightly to handle similar issues from crust if it arises better than it was handling it in this test. More info about the bugs and changes should be in the Dev Update coming out today. Thanks again for taking the time to try and run vaults repeatedly through these test networks. They help so much and aren’t easy to debug and diagnose without the help from the community these tests are having. If you haven’t already hit the reply button to ask the next question, next test we can expect to go live sooner than normal :wink: Just need to let it run the internal soak test suites to prevent any accidental regressions.

25 Likes

any interest in running a public vault network on this version?

give me a shout if so:

rupert@btl.io

normal ip: 185.16.37.149

rup

1 Like

Also, is there a problem with merge operations?

I ask this question because there were sections with few nodes in them, below 8 nodes which is the min section section size. I am not sure of this value, but one section was empty, which is clearly not expected.

2 Likes

There would have been problems with every operation type because of this crust issue I think as it caused the section to loose consensus in the validation stage(even if they agreed in the content portion), which then meant any action that section wanted to do did not succeed. So Merge/Split/Data-Handling/New-Node/… all fall under that criteria

It is the right value for min_section_size right now :slight_smile: It’s defined by vaults and provided to routing.

It’s a good question and without the crust bug it would have indicated an issue “in most cases”. Reason I say in most cases is when a merge is ongoing, the nodes would end up with sections that are empty. Wouldn’t be their own sections ofc or the partner section they’re merging with but neighbours of their partner section which they previously weren’t connected to but would be after the merge under their new section prefix to satisfy the network invariant. Once the Merge completes, those sections would start getting filled in. So seeing a persistent empty section would always indicate an issue(If the merge itself was the source of the issue becomes the next question).

9 Likes

This is doing my F5 key in, guys…

2 Likes

$0.74 well spent :smiley:

1 Like

Visualisation of test 12c network on youtube (partial part from 2017-03-03 00-52-01 to 2017-03-06 21-35-38 UTC). It includes 4 section splits:

  • at 2017-03-03 03-54-37, position in video: 0 mn 33 s

  • at 2017-03-03 13-43-45, position in video: 3 mn 39 s

  • at 2017-03-04 22-02-27, position in video: 10 mn 31 s

  • at 2017-03-05 13-47-58, position in video: 12 mn 15 s

Nodes can be represented in a 2D coordinate system by taking the high bits of the node ID and splitting them in two parts: the bits at odd positions and the bits at even positions. For example if the 8 higher bits are named b7 b6 b5 b4 b3 b2 b1 b0 then the node coordinates are (b7 b5 b3 b1 b0, b6 b4 b2 b0).

This way node proximity is preserved in the generated points in the euclidean plane.

In the safe network neighbouring nodes are grouped by sections. As an application of the proximity preservation, these groupings remain visible in the projected plane and can be emphasized by drawing a line from the point corresponding to a node in a section to the center of the section.

17 Likes