MaidSafe Dev Update - March 30, 2017

Last week, we released Test 15. Our initial plan was that if Test 15 proved to be stable, it would become Alpha 2. Test 15 was stable for the first few days and we have been very pleased with the results. However, after a few days Test 15 experienced an issue related to client account spam. It appears that someone ran a script that created lots of new accounts and stored lots of data, to the point where most vaults are running at 50% of their capacity, and it no longer became possible for the network to store any new data (hence the NetworkFull error). Currently, vaults will only go up to 50% of their capacity before rejecting new client data. The remaining is left over to accommodate for churned data being relocated to existing vaults. As things turned out, this situation has been quite useful for us in understanding how the network reacts when full, as this has not been something we have publicly tested before.

As most of you know, it’s quite easy to create new client accounts using SAFE Launcher: you just need to provide an “account secret” and an “account password”. And since test safecoin is not implemented yet, vaults are coded to give 500 PUTs to each new client account (a PUT is a request to store a single chunk on the network). This makes it easy for community members to participate in test networks. On the other hand, it also makes it easy for an attacker to spam the network. We would prefer to wait for test safecoin to be implemented, but at least this spam attack showed us that the NetworkFull error works as expected :smile:

As @fraser said in this post, the main takeaway is that we need to accommodate for such scripts being run in the upcoming testnets. Earlier this week, we discussed various potential options that could help solve this issue. We want to implement a temporary solution that’s not too complicated and that we can easily remove later on when we start implementing test safecoin (see this dev update for more info on our short-term roadmap).

We are currently implementing a solution called invitation-based accounts. Here’s how it will work:

  1. Open SAFE Launcher, go to “Create Account”.

  2. Before asking for an “account secret” and an “account password”, Launcher will ask for an “invitation”. Here’s an example of how an invitation might look like: OfM2QD5aWqZpOjof3ysA6MB3Gtp4Y38aW5W8vEuPJ7KPFl6iT4wEexCOOKBPHgvkBPl4sbi7zjBLulWmglqk1Yn7OD.

  3. You can either enter an invitation token (if you have one) or click on the “Claim an invitation” button.

  4. After clicking on the claim button, you will see an OAuth dialog with a few options (“Sign in with Google”, “Sign in with GitHub”, “Sign in with Twitter”…etc…). The options haven’t been fully decided yet and might change over time.

  5. After choosing one of the options, you’ll need to log in and grant the required permissions (the OAuth dialog will only ask for basic info). There will be a server (controlled by MaidSafe) with a database that keeps track of which OAuth users have received an invitation. The server will only give you an invitation token the first time you log in using a specific option (e.g. your Google account). If you try again with the same account, it won’t work.

  6. If the “Claim invitation” process was successful, Launcher will automatically fill the invitation token you obtained from the server into the invitation box. You can then continue the account creation process as normal.

As a fallback option, we will also distribute invitation tokens to forum mods. Users who prefer to not sign in using OAuth could go to the SAFE Network Forum and ask mods for an invitation token and then copy-paste it into the invitation box. This could also be useful for people (e.g. app developers) who want to have multiple client accounts. They could simply contact the mods and ask for a couple of invitations.

The invitation tokens will be generated by MaidSafe (in batches). Each invitation will be represented by a structured data with type tag 8. The name of the structured data will be the hash of the invitation token.

The way it will work is that when you try to create a client account, the MaidManagers will expect an invitation token along with the normal information. They will hash the invitation token and try to update a structured data from that location in the network to version 1. If the update operation fails, it means that the invitation has already been claimed. The vault config will be accepting a new entry for storing a public signing key. MaidManagers will have a rule to only allow PUT requests for structured data of type tag 8 if it’s signed by the public key contained in their vault config.

Please keep in mind that this solution is intentionally small so that we can get back to the normal day-to-day work :slight_smile: It’s going to take us a few days to finish implementing and testing this. Once it’s ready, we will release Test 16 (which should become Alpha 2 should testing go as anticipated).

SAFE Authenticator & API

With continued testing, we identified a couple more bugs with the Authenticator. We are holding back on a simple tutorial that shows how to build the SAFE Browser and the SAFE Hosting Manager example app, and also a simple tutorial that shows how to use the new DOM APIs until the root issue has been solved in safe_client_libs and then tested again from the front-end side thoroughly. The issue is with the MutableData delete functionality. Once you delete an entry from a MutableData, fetching the same MutableData again was throwing an error. Similarly, after you authorise an app and then revoke it, if the same app tries to authorise again, an error was returned. This is being addressed by @nbaksalyar.

We have created an initial wireframe for the developer website. This is only a first cut wireframe which is yet to be discussed within the team, and after couple of iterations, we will be happy to share the plans with community.

@krishna_kumar has been working on a POC for a Java API using the jnr-ffi library. An Android developer is joining our team next week. He will be able to make use of this POC to come up with a full blown safe_app_java library for building desktop applications using Java.

SAFE Client Libs & Crust

Carl has progressed with his uTP implementation and now has a partially working solution here. It does not yet have a full-fledged congestion control (it’s one of the things he will be working on next). However, there is an example which shows how to use it in simple cases and the API looks pretty similar to TCP which is what we would have expected. It also allows multiplexing of uTP streams over the same local socket, which might come in handy in future.

Many crates have been updated to use Serde serialisation instead of rustc-serialize (which is flagged to be deprecated since procedural macros are now stabilised). Crust benefits from this because in the past we needed our own custom SocketAddress and IpAddress wrappers since the standard ones were not serialisable. With serde, this is no longer the case and thus crust code will see some good clean-up in getting rid of manually coded wrappers (leading to better and less confusing code).

safe_core now supports the new invitation-based scheme in combination with the vaults. It also supports the creation of a specialised seeded client that the vaults will recognise as special. It is for populating the invitations on the network so that future clients can be enforced to be creatable only if invited. A binary in the examples folder is almost complete which will allow various invitation-related tasks. Once the chosen seed is given, it will output a public key which will be made known to the vaults via the vault config. Then account creation with the same seed can happen, which the vaults will allow without checking for an invitation. This seeded client will also be allowed unrestricted PUTs to allow it generating invitation tokens as required.

Routing & Vault

We’re working on implementing an invitation-based account creation in safe_vault, as mentioned in the introduction.

In routing, we resolved a few issues with section merges and tunnels, and at the same time started investigating the introduction of section version numbers, which would be a first step towards data chains.

From Test 15, we’ve also identified a bug thanks to @rupert which seems to occur at times when a tunnel connection is promoted to a direction connection causing a state inconsistency between the peer manager and the routing table. We’ve got a patch in place for this and are working to confirm it via tests before bringing it into the master branch.

68 Likes

first! … 20 :smiley: …

10 Likes

Damn!!! So close lol I was like what do I say after first? Time to read

8 Likes

Third… is more special.

Is it useful to keep Test15 running or should we let it fall?

:thumbsup: Would be interesting to see an example of hello world for Android. Another simple hurdle that could be flattened and encourage noobs like me to try putting together an app, would be fun.

7 Likes

This looks like an elegant and lightweight solution ! Thank you for this very fair and positive attitude :slight_smile:
And now, to testing seriously !

10 Likes

Thanks so much to all of the team for all of the hard work! :slight_smile: :slight_smile:

7 Likes

Nice solution :+1:.

Cool!

Well done @rupert and of course @maidsafe

Lot of great news, really like the simple solution to the account spammer attack. As always keep up the good work :+1:.

13 Likes

Great stuff, team!

Only remark I’d make is the need for Captcha (intelligence check), even I being a moron could buy a batch of oauth enabled accounts and launch an attack on the network, my efforts would be thwarted by Captcha - unless I had an army of leftover gold farmers at my disposal.

4 Likes

What I forgot to ask @maidsafe. Will the Vaults have some extra storage as well? In TEST 15 the default is 5 GB. I guess most people with a Vault wouldn’t mind to provide 15 GB or even 25 GB. If Alpha 2 goes live (and TEST 16 becomes Alpha 2) we might see quite some extra users than in a testnet.

2 Likes

It’s fine to take your vault down :slight_smile: We’ll let the network run for a few more days while getting ready for Test 16 and continue to monitor for any other issues that may crop up, but it should be fine for users to take their vault down and save some cash, especially if they’re paying cloud providers to run vaults right now.

10 Likes

We can look into adding a captcha, which would of course make the OAuth process not as seamless. It’s worth noting that spam of tokens isn’t going happen from the server as it’s going to have a restricted amount of tokens to hand out at a given time.

This again is something being discussed. The main reason to bring the default size down was to allow users running nodes in Vultr or other providers with limited space to not need to edit their default config options. (Users who want to increase the default size can edit safe_vault.vault.config and change the max_capacity value.)

For Alpha 2, we might have to provide a larger number of nodes to counter overall capacity or increase the space per vault accordingly, but we haven’t picked an option there just yet.

11 Likes

That invitation system seems like a proper fit for the issue at hand.
Keep up the good work!

5 Likes

Ugh…

Such an ugly and slow way to address this IMO… :stuck_out_tongue:

But we’ll see

I wont be doing any of that.

Just make a page that generates these keys and put a bot protection on it from google. Simple.

2 Likes

Who cares if it is ugly? - its a fix to get us “over the hump” until SafeCoin is ready for use.

Got a better idea to solve the problem? Quickly?

Submit your patches, then.

Yes several people were asking for very basic and simple coin to regulate uploads.

But not up to me. Can wait

And can you make that work in the next 2-3 days?

can make a key generator that has a bot check against it.

1 Like

That type of page can be made in hours