MaidSafe Dev Update - June 15, 2017

The soak tests that we ran last week had very positive results. We’ve now started running an internal testnet with Mutable Data and we’re trying out the authenticator and the example apps against this internal testnet.

There are still some things we need to implement before releasing a public testnet. While this will be a client-only network, tests still involve churn to deal with special cases where a few droplets may go offline and these are cases where churn is expected to handle the data requirements as we’d expect. This also allows us to add more nodes to the network if there is a requirement to add more usable space to the network. Initial tests have identified a few bugs that did not occur with mock network tests which the front-end and client team are working on right now while the back-end team are also addressing some of the spam prevention requirements in parallel.

After 18 months of working with MaidSafe, Andreas (@AndreasF) has decided to move on. We discussed him leaving, at least for a while, he is more than welcome back in the future and the door is open. We would like to thank him for working under such pressure and achieving so much in doing so. Heading up the routing design team, Andreas has been a committed and well respected member of the company. As you know he was very much involved in the recent changes to the Routing library and we all wish him very well. One of the benefits of the guys working so closely together is that a number of our engineers are very familiar with that part of the system and its future design direction.

Andreas spent a lot of time with Spandan assessing potential new recruits and scaling the team continues to be a focus. We are working with a number of recruitment agencies and are starting to see a larger than ever before volume of CVs. While the front-end team has taken on a couple of new developers in the last few months, and we are close to confirming more, the network team has not grown much and we are making changes to improve that situation with a view to speeding up development and spreading the huge workload much more evenly. We recently dropped the necessity of requiring each candidate to have Rust experience, and are also looking for system level devs (C, C++ and Rust) who also have experience working with other P2P systems, or network security, or other distributed technologies, such as Hadoop, Redis or Cassandra. This means we are looking for system level devs experienced in Rust, or interested in learning it :slightly_smiling_face:

SAFE Authenticator & API

We tested the front-end APIs and applications against an internal test network. We came across a few issues which were not present with the mock network. The entire team has been focusing on fixing these issues and we are now almost done fixing them. However, applications don’t work as expected when revoked and reauthorised. We also found a gap in creating the unregistered client and it was wired up in safe-core. The unregistered client related changes were integrated with authenticator and the Node.js API. In general, it is expected that an unregistered client should be created without even needing an account in the network. At present, it is required to have an account to create an unregistered client. This issue will be addressed in the next few days.

Other than the functional issues, we have few a UI updates that we will be working on as we continue to test and fix the issues.

The changes that were made in the client libs must be integrated in the Java API. The test cases are being wired at present for the existing integrated APIs.

SAFE Client Libs & Vault

We keep catching and fixing bugs in SAFE Client Libs and we’re adding missing functions found by the front-end team. As described in the section above, we’ve been working on adding missing functions to propagate Crust bootstrap configurations to unregistered clients. We’ve also fixed account registration and login functions that weren’t following the calling convention that we use for all other functions exported from Rust.

Currently we’re working together with the front-end team to correct the app revocation and reauthentication flow and we’re adding more automated tests to make sure that these features work as intended.

There are some more bugs in mock-routing that were discovered by @shankar, concerning handling of MutableData addressing and storage. The current mock-routing implementation assumes that only one MutableData instance can be stored with a given XorName, while in the real network that is not the case: apps can store multiple MutableDatas with various type tags under the same XorName. Another issue that we’ve found is that network status is not propagated to the front-end properly and it is not properly tested on the Rust side. So this is what we’ll be addressing for the next couple of days.

For spam prevention, we are trying to see what numbers are good enough a compromise. We have also identified more areas that can potentially be patched and some of them are already underway. That for e.g. addresses the problem that a client can send RPCs which it shouldn’t (but does it because it’s malicious). Soon we will be able to come up with a list of approaches and then we’ll present that information in this Dev Forum topic.

On the vaults side, we’ve added tests and checks to forbid duplicate message IDs for client requests. This is required because if clients send two (or more) request using the same message ID, the account balance checks could be bypassed as the balance would have been charged only once. We’ve also added some new tests and increased the test coverage.

Routing

The mutable data branch has finally been merged into master! That’s a huge change, although not quite as big as the one for safe_vault is going to be. We are now internally running test networks and adding the last few pieces to the implementation.

Apart from minor tweaks to Option B, we are now working on extending the simulation in two directions:

  • It used to assume for simplicity, and to first only test the convergence aspect of the proposal, that every piece of information — every vote — is sent to every node in the network. We need to work out, test, and simulate the details of who actually needs to send the votes to whom, and how to recover from cases where messages failed to arrive.
  • The simulation is too memory-hungry to create a large number of nodes, but there are ways to deduplicate some of the information shared by the nodes (votes and blocks that are held by the simulated nodes) without changing the semantics. And of course the first point will also help: if less information is exchanged, the code will run faster and with less RAM.

And finally, a new option to mock crypto primitives in tests speeds up the test suite more than 4-fold!

49 Likes

I am so fist it hurts.

14 Likes

Dammit - the bankers win again!

14 Likes

#Video Q+A call!! Here!

Here to help the community get answers and info about SAFE & today’s update! Ask anything

5 Likes

For anyone wanting a plain but thorough summary of where we are and what is next, I believe this post from David is still up to date:

10 Likes

@AndreasF NOOOOOOOOO! Don’t go! The work you’ve done here will never be forgotten by us, you along with the others on the team have been such a valuable resource and what you’ve accomplished with them in routing is admirable. When I got to watch sections split and merge with one of our community members network visualizor, it was a moment where I felt like the network was alive, truly amazing. Take care Andreas I wish you the best

24 Likes

Thanks Andreas, wish you all the best!

@maidsafe thanks for the update, keep up the good work :thumbsup:.

20 Likes

Every week it’s coming closer and closer :tada:

Success @AndreasF on your next career and great work!

13 Likes

Thank you all! :blush:
It was a fun and interesting time and a pleasure to work as a part of this team! I’m looking forward to seeing the SAFE Network grow to completion!

43 Likes

So first congrats @AndreasF

Now, sorry to be this guy, but the departure of a key member from a team can always be considered a red flag. It may have nothing to do with it, but the crypto space is currently booming and SAFE is supposed to be a few quarters from completion. Why leave now ?

16 Likes

What’s more, as I understand it, Maidsafe should be also flush with cash due to the recent/yearlong cryptosurge since the last round of funding reached that million dollar goalmark.

2 Likes

@AndreasF Thank you for your contributions to such a great project. Hope all is well and wish you the best!

10 Likes

I will try and answer your understandable question

Andreas spoke about leaving a while back to get new challenges, but stayed a bit longer at our request to get to a point where it would be in our (SAFE and Andreas) best interest. As we said though we are not even sure the story is over in terms of Andreas working on this project, but right now he is very keen to work on something different for a while. I understand as he has probably lead the team through what I think is the most pressured and tough part of the network internals to date, That has allowed us to clear a road through to where we are much more comfortable, but we did push Andreas hard and he allowed us to like a champ.

So really it’s not about us launching, cash or anything like that. Like most of the Engineers there is very limited view of price trading and the monetary side of things. Although Andreas probably is more aware than many of the devs, but still we are pretty much removed from the markets and focused instead on the code etc. I know he in uncomfortable with such questions and like most of the team does not want to get involved into detailed introspection of personal decisions. No harm though, but no underlying issues, just a capable Engineer working on hard challenges and looking for the next one. All cool with us in house anyway for sure :wink:

tl:dr Andreas likes to work on different projects and switch tasks once in a while, and recently got an offer that he doesn’t want to miss out on.

35 Likes

Oh I see, a developer burnout. I think we’ve all (devs) experienced that at one point or another. Unless your Linus Torvalds or something, after a couple years the same project/code can grate on the vision.
To be clear, I was just noting that salary probably wasn’t the issue since Maidsafe should be fine cashwise.

3 Likes

Yea, I don’t think m/any of the team are salary driven, we are looking all the time to improve that though as we move on. I think we all find it tough, but that’s not the case really, some Engineers just like new challenges every year or two, Andreas certainly does, he has a very wide range of skills and I think he needs to use them in different projects. Not sure burn out is correct, but certainly it’s not easy, there’s a bunch of us debating in slack right now as usual, more detail and algorithms, it never stops. It’s just life, in tech a couple of years is the time on a single project for many Engineers, we see that a bit in this project as well. Anyway it’s not all about Andreas, he would not like that. The update is about SAFE and moving on :smiley: We are getting there, but have many more long days and nights to go, so we get on with it.

11 Likes

So rather than burnout it could also be that @AndreasF felt he’s completed the hardest parts of the project, and it’s all boring details now. Meaning we’re close to Beta? :wink:

5 Likes

Aaannnddd none of us will answer that one :smiley: :smiley:

20 Likes

Nor do I expect you to. In fact, even though it was in jest, I feel slightly bad even asking. I just wanted to balance the fear with some positive speculation.

The devil really is in the details, last 10% takes 90% of the work, etc, etc.

7 Likes

That and another classic: success is 10 percent inspiration and 90 percent perspiration.

4 Likes

just in case you missed the hidden message :grin:

25 Likes