MaidSafe Dev Update - February 23, 2017

So this week had a few surprises in store for us, some good and some not so much in terms of Test-12c. Firstly, we completed the routing patches (that we mentioned in the last dev update) on Friday and early this week, we also had the routing soak test run about 1000 iterations successfully. These are tests that use the mock crust feature to simulate different types of network setups and specific cases we’d like to test. This passed all the way through and was great news.

We then moved on to testing the production vaults in droplets with internal tests. This was where the not so great news occurred. We hit a few issues very early on that we patched, however, we continued to see a few more issues through the week. It’s worth noting that each issue that needs to get debugged and patched takes a bit of time since this involves collecting about 1800 log files and spotting the source of the issue, (fun times :slight_smile:). The Routing team went at it hard (so hard that we’ve made Andreas and Diggory sick) to try and get these resolved ASAP. This was further complicated with the new Crust version we were using in the live network, and involved us running two internal tests to confirm Crust wasn’t the variable here (it certainly wasn’t, so looking good from that side, more good news from Crust coming up).

Now for some more good news, we’ve currently had an internal test network running since yesterday afternoon that has gone through about 2100 churn events. That translates to about ~3500 nodes joining and leaving the network. The internal tests also test some particular corner cases such as lots of merges/splits happening in the network at the same time as nodes joining/leaving. As of writing this dev update, that network is retaining data while the invariant is holding, this is certainly great news. All is not completely done yet though. We have one other issue we’ve spotted that can occur when a candidate (new node going through resource proof check) is dropped by a section after it splits which in an edge case prevents the split section from ever re-connecting to that candidate. The bug meant that the section did not get rid of the candidate properly after the split which prevented them from even re-establishing the connection to that candidate (who by that time might have been promoted to a full node). This is certainly an issue and @qi_ma is patching this today. We’ve not run into this edge case in the network within the last 24 hours.

Once we have the patch for this in place, we intend to keep repeating the internal tests until we can confirm a bug-free network before progressing to the most elaborate test of user run vaults (Test-12c) :slight_smile: From the results this week we’re hoping that we’re not far away, but please do take that with a grain of salt as this situation is very fluid as you can imagine.

With all this happening at one side this week, we’ve got some really good news to share from some of the other teams, specifically the progress being made in protocol support from Crust for uTP, and various hole punching tests that we’ve tried in combination (all this will increase the number of people able to join the network and not just rely on TCP). With frontend APIs getting more reliable, @ustulation and team are also starting their work in Vault to support — yep, you guessed it! — mutable data :slight_smile:. This will enable the test networks to move from the SD/AD data types, introduce the new data type MD which will then facilitate the Authenticator paradigm in place of Launcher, which would get deprecated at that stage. Just to note this isn’t a patch job that we expect to be done with in a week, but we’re certainly in motion there to get vaults to support this data type. We’ve also got the frontend guys going hard at their applications and docs which we intend to share more with you this week. Please check the individual module sections for more information in all these aspects.

As you already know, we have some of the team in Asia which is great, but as a small team it does hurt dev progress a little back home. So this balance is a fine one that we choose to make, even though it does add a lot of time pressure. We feel sure though that getting the SAFE name out and solidifying relationships with our Asian partners will benefit the SAFE Network and all that is stands for in the long run.

So in summary, it’s been a pretty hectic week all around, got a few people ill (looks like they’ll survive just fine), got some of us in Asia spreading the SAFE Network concepts, patching some network bugs from internal tests and progressing along with the MD implementation and Crust updates for protocol support and doc updates of frontend APIs.

SAFE Authenticator & API

Last week, @happybeing (again we must thank him for his amazing contributions to this project) had hosted the API documentation for safe_app_nodejs for easier access. We have now published the documentation for safe_app_nodejs’ master branch, which we will keep up-to-date with the latest changes.

We’re currently improving the build process for the Node.js library and also the safe_browser. The library will be downloading the native dependency at run time and won’t have to install Rust and build locally to get the native libraries. @lightyear has created a dependency downloader which is a postinstall tool for safe_app_nodejs to download the native library for the current architecture and the platform being installed on.

@lightyear and @shankar have been working on packaging the example app – which depends on safe_app_nodejs – using Electron Forge. This is tested on macOS and we are currently setting it up for use on Windows. We hope to have it integrated soon and test the same across all three platforms. Minor feature updates to the example app have been implemented by @shankar.

Once we have these tested across platforms, it will make the installation of the safe_app_nodejs library seamless. The browser build process will also be simplified with this WIP PR getting merged.

@bochaco, has been working on fixing minor issues in the API along with continued testing. At the same time, @bochaco is working on identifying the helper functions which can be moved to the Rust FFI layer so that the integration with FFI will be easier across languages.

SAFE Client Libs & Crust

Quite a few good things are planned for Crust and Rust networking in general. Carl having solved the issues with the internals of Mio, is planning to work on an async uTP support on top of Mio. We had given him a few pointers to where the protocol was itself, which is implemented in C++ as libutp, and had asked if he could come up with a list of TODOs. He has given us a rough idea of what API he plans to expose in this new crate. We were recently discussing on having a coded multiplexer in the uTP listener logic that functions on just one socket vs spawning multiple sockets per accept. The general consensus is to spawn a new socket per accept instead of using multiplexers. Although multiplexer would have helped us keep number of used descriptors under control, they would have a significant overhead in terms of CPU cycles and could potentially affect performance. Once we have uTP support, we will certainly be much better off in terms of reaching peers. UDP hole punching trumps TCP by a good factor and uTP wraps a UDP socket with reliability and congestion control layers added on top of it.

@nbaksalyar in the meantime has been writing more test cases for Crust and looking into the challenges of setting up a virtual network within a machine to test external reachability and other parts of code, which necessitates peers on non-loopback addresses thus posing challenge. While there are ways on *NIX platforms, Windows as usual needs more looking into and out of the box support seems bleak there. If he does figure it out, it might prove to be highly useful for tests in Crust because they would be closer to the real world and will help us iron out deficiencies if at all observed.

@adam started working on the implementation of MutableData in vaults. We already have this supported in safe_clients_libs via mock vault. Based on the churn handling in the vault library which isn’t catered for in the mock, we might have to optimise or revisit certain components of this implementation. MaidManagers should be easier to deal with than DataManagers just now, and that’s where we have started. Every successful mutation request now would deduct account balance (previously POST was free and went straight to the DataManagers). Also the final cap is based on total number of mutations, not just PUTs as previously was the case. See this post for more info.

@ustulation had intermittently done some NAT-traversal testing with different kinds of routers including Symmetric NATs and observed some expectedly unfriendly router behavior. NAT traversal is a cat and mouse game in which we need to find out and try and trick through the weird behavior of routers. All the observations until now are documented in detail with workarounds for successfully reaching peers in this repository. This is meant to be an independent pluggable crate for robust p2p communication. This is a work-in-progress and current design (and other) details (which are aimed to explain all observations and techniques thoroughly) can be found in the crate docs. Though still in early stages, a somewhat working chat-engine is already written and tested here.

Routing & Vault

As mentioned in the introduction, this week has mainly involved us patching and updating the routing library based on internal tests. Some of the changes that have been applied this week include:

  • #1357 adds a test with an increased level of churn.
  • #1363 makes the log messages a bit more concise.
  • #1370 fixes problems that occur if a candidate (a joining node doing the resource proof) joins a section that is currently merging with another one.
  • #1371 addresses another section merge bug.

Developer outreach

This week’s Tech Conference in Jakarta was a great success! See this blog post by @nicklambert for more info.

Also, Ben (@lightyear) is speaking at Bob Conf Berlin tomorrow. His talk is titled “Privacy-first data structures on SAFE”. We hope it’s as good as the excellent talk he gave at FOSDEM 2017 :smiley:


Lots happening and mostly good news. 2017 is starting out great for SAFE :+1:


All the best to the sick folks. Indeed mostly good news. Once all the logic in Routing is solved it will be there for 1, 2 or even 10 years ;-). So better safe than sorry. I know you are all doing an amazing job. Keep up the good work and we’ll hear the bells ring when 12c is among us.


:heart_eyes::sunglasses: :hugging:

all sounds very promising! :slight_smile: very very cool! (i especially like the “dependency-downloader” :smiley: … just getting a working lib without any hassle sounds perfect! )


The new crate p2p with some-what working chat-engine.

Could it be possible to use this for multiplayer gaming?


I do see mentioning about the server, and peer connections. One of the reason why I brought up the question. Does this focus on low lag, high response times?

For Webrtc to work, it needs a server, or using safe SD to set up a room to communicate. Is p2p server example is similar to signaling_server? Does this mean this would enchant the MD and safe communication even better in long term?

this crate uses a technique to trick the routers. The problem is we don’t want to reach the other end fast (thus getting ourselves blacklisted) while updating the filter at our routers. TTL (time-to-live) to the rescue. While punching hole, we start with the lowest reasonable TTL (of say 2). Note that some routers were found to drop the packet when TTL was 1 after decrementing while some still send it but drop if the the TTL reached the value of 0. With 2 it will definitely go past the first router. We put a delay, increase the TTL by 1 and send again. Both sides (peers) do this. In pracitice it’s usually the 1st couple (or 3) routers that do NAT while others are non-NAT. This gives ample amount of time for the NAT-routers to update their filters to allow the peer’s incoming packets in the future while not reaching the peer quickly themselves and getting blacklisted. By the time we hit TTL of around 12, we would likely have reached the other end.

Nice! That should curb man in the middle attacks. Sets up a honest communication.

I see a lot of potential with this crate.


So much happening in parallel! This is great, can’t wait to see all the wrinkles ironed out and once we do it’ll be a huge leap forward. Likely most outside of this community won’t see it coming. Im pulling for you @maidsafe team and get better soon to our sick comrades


Thanks for the update (and mention :blush:) and all the best to the team, far flung, healthy and poorly. So much is going on in each update these days :slight_smile:

Is this a change to the pricing model - I thought @whiteoutmashups asked recently if this was changing and was answered no. So I thought plan was same as for SD, e.g. pay a premium for creation but then nothing for updates.

No objection here, just seeking clarification on the plan / options.


I love reading these updates, even though I struggle to understand all the different developments. What is abundantly clear though is the immense amount of work that is going and has gone into MaidSafe. It’s really humbling for someone who isn’t involved and fills me with a sense of gratitude. Thanks everyone!


Thanks so much to the team for all of your hard work! Also, I hope the sick members get better soon!


Thanks for the update! Keep up the good work!!!


Now this is really yummy ! I think I’ll have a piece of that one cake :cake:


Yeah this is very strange; kind of thrown in to the update which skipped right along.

If this is true, and SafeCoin transactions will no longer be free, then that is a…


…change to the SAFE Network and needs to be discussed!

I remember many many many posts, charts, (BTtF biz plan) etc where MaidSafe promised free SafeCoin transactions, and this was supposed to make SafeCoin accessible to the masses (billions) out there with barely any money.

If MD means the end of that, then I am not for it at all, and am very very worried!


It’s also a possibility there be a dedicated data type for safecoin. But I am curious about this too, so the more clear this is the better.


Really looking forward to updates to the safe_app_nodejs library and more explicit documentation of how to get it up and running (maybe i’m just thick?) Tried to follow the instructions earlier but when I tried to node create_public_name.js from the examples directory I got an error about an unhandled promise rejection. Not sure if I launched the browser correctly or if I built it wrong? copied the files into the wrong place?

Either way, keep up the good work and hope this can be made easier for numpties like me to start building nodejs apps with.


uuuh - i missed that part :open_mouth: veeeeery cool!


Brilliant update! So much being done and I can almost feel the delight in the progress being made. Great to see things coming together well!


I suspect the cost would be tiny, even if it exists. We are talking about changing a small data item after all.

Personally, it worries me more when something is free. Every data change has a resource cost, which presents a DoS attack vector if not appropriately handled.

Ofc, it would be good to hear more about these changes though, so we are not discussing what ifs.


I agree as long as it doesn’t apply to safecoin w/ a tx fee because then it would really lose its granularity and ability to be the first truly frictionless micro transaction capable digital cash


This is a concern. It wont apply to safecoin but there may be some limiting factor on mass changes to data exactly for dos protection. It may be a small charge or another limiting factor, but it’s not final yet. Safecoin does need transactions to be free.

I think this is an ongoing discussion really, but we are not focused to much on the limiting factor, but there likely will be some kind of limit.


Argh! A Berlin talk and I’m out of town. Hope all goes well @lightyear!

Thanks for another good update @maidsafe!