Everything that we had as objectives for Alpha 2 has been met with Test 16. But based on what we’ve seen from Test 16’s limitations, these limitations were the main things that were abused in various ways (e.g. nodes bypassing the relocation process, clients spamming the network with GET requests, nodes having a lot of tunnels). As @dirvine explained in this post, these are all things that are currently just spot checks and that were put in and highlighted as limitations that would be resolved down the road with features like node ageing and data chains. So features that were meant to be for Alpha 3 are limitations right now, because we haven’t implemented them yet, but those are exactly the things that would solve the kind of issues we experienced in Test 16.
We have discussed the idea of helping set up a community testnet, it’s a double-edged sword though. Test 16 did not show any bugs and, barring the attacker’s participation, was fine and would be Alpha 2. We are not sure what can be achieved even with a community testnet right now. We know vaults from home work, we know the limitations and we know devs really want to use the new SAFE API with Mutable Data.
With this in mind, and instead of delaying Alpha 2 and taking features from Alpha 3 and rushing them into implementation now, we’re now going to swap the objectives of Alpha 2 and Alpha 3. We have done the donkey work here, mostly, but need to spend a few short weeks at least soak testing and finalising the Mutable Data integration to a point where we are happy to release the next testnet. We will try to speed through this with half the team while the other half are already on the routing & vault updates we need for Alpha 3. Alpha 2 will also affect the Alpha 3 code but it’s pretty clear where and we can easily cope with any conflicts there. However, to ensure we do not knee-jerk this, we will measure the results carefully and with luck we should do this internal testing over the next few weeks and of course update you all as we progress.
Alpha 2
- Mutable Data and authenticator pattern (with the new SAFE API)
- SAFE Browser DOM APIs and Node.js SDK along with example apps, tutorials and documentation
Alpha 3
- Vaults from home (more people should be able to run more nodes and on more platforms)
- Node Ageing
- Data Chains, Part 1
This means that the upcoming testnets will focus on Mutable Data. Like Alpha 1, these will be client-only networks (MaidSafe will run all the vaults). We hope these testnets will trigger a lot of discussion and feedback from the community in terms of how to use the new APIs, how to migrate from SAFE Launcher to SAFE Authenticator, what are things that might need to be addressed or done better, documentation, supporting different platforms, etc. It should be an exciting period!
Alpha 3 is going to have everything that the vaults had with Test 16 (what we currently have now), plus the missing features without the hacks. We can add temporary hacks here and there, but this is not something we feel is good in the long term of the project and there is not much we would gain out of doing this right now. Instead, we’re going to go ahead and implement the actual features that are planned for Alpha 3 in terms of data chains, node ageing, nodes key as name, etc. But we’re not going to wait for that to be complete before allowing app devs to start using the new SAFE API with Mutable Data, which is why the next testnets will focus on Mutable Data. Mutable Data support will be added to the master branch of routing & safe_vault and, in parallel, we’ll implement the features for Alpha 3 in development branches.
Once the Mutable Data objectives are complete and at a stage where we consider the API stable enough for a “version 1”, and if we are not done with node ageing and the first part of data chains (which might be likely), then we will call that the new Alpha 2. Once we are done with the original Alpha 3 objectives (as far as vaults are concerned), we will release Alpha 3 and community members will be able to run vaults from home again. Alpha 2 and 3 may appear very close together in this case.
Beyond Alpha 3
- Data Chains, Part 2
- Data republish and network restarts
- A security audit of the network
- Test safecoin
- Real-time network upgrades
- Network validated upgrades
In between alpha networks, there will be many testnets. Each alpha will focus on a few major components. Major features will be added one by one in testnets. Once all those things are in place, there may be other alpha networks, or we may move to beta.
Upcoming Dev Forum discussions
We are going to have quite a lot of technical discussions on the SAFE Dev Forum over the next few weeks. This will involve discussions from the Mutable Data side and also from the back-end network side (e.g. data chain integration into vaults).
Here are some of the topics you can expect to see on the Dev Forum in the next few days/weeks:
- Data Chains (Part 1) design proposal
- Vault chunk store optimisations (as pointed out by @mav in this topic, which has lead to @qi_ma looking into improving the chunk store performance)
- Client RPC spam prevention approaches
- Solutions for how to address the 100 entries limit of Mutable Data
- Operation recovery for Mutable Data from the network. So a transaction-based process (for some operations initially) to allow rollback in case of failed operations midway (this one is likely going to take a while longer as the guys are still in the early stages of working out the details)
These will all be open discusions, but at the same time, we will need to consider what is going to work well for the project in the long term and decisions will then need to get made accordingly. To be clear though we will still have ongoing and possibly more blunt debates in house, these are critical to achieve the best outcomes but not always good for public consumption as there would be too many ways that people who would wish to cause harm could take this kind of frank discussion out of context. Any conclusions though would immediately be made public as you would expect and deserve.
SAFE Authenticator & API
We finished porting the Markdown editor web application and email application to use the new APIs, but we haven’t merged the code to the safe_examples repository yet because of an API issue related to Mutable Data. The test cases for Mutable Data are passing, however the API fails when invoked from Electron. We created a CLI sample app to test the Mutable Data APIs and can confirm that the APIs are working fine from the Node.js environment. We are looking into this issue and will have it resolved soon. Once the issue is fixed, we will be able to get the updated examples merged to the safe_examples repository.
@Kumar tested a few approaches for Java JAR packaging and distribution. He is also trying to get a simple FFI sample tested on Android. JNR seems to be a problem on Android. If JNR is not going to work we will switch to JNI. We would like to have a working sample before starting to implement the actual code for safe_app_java. Ideally, it would be easier if we could use the same source code for both Android and Java desktop apps.
SAFE Client Libs & Crust
We are looking into viable solutions to prevent GET and MUTATION spam on the Network. GET spam can be caused by either registered or unregistered accounts, where a (malicious) client makes repeated GET requests of large chunks of Mutable Data (Immutable Data is cached so the benefit to the malicious client is not that huge) and makes the Network do work for them with relatively no effort on their part. It can also be GETs to non-existent data, which will result in failure response but still cause the Network to process such requests. MUTATION spam, while not as obvious as GET spam, can also cause the Network to do a lot of work. In this case, frequent bound-to-fail mutations can be done. As an example, consider a PUT request for a chunk of data (largest allowable size) that already exists on the Network. It will travel to the MaidManagers who will deduct account balance, forward it to the DataManagers who will complain that data exists, MaidManagers will undo the balance deduction on getting failure from DataManagers and will finally relay the failure back to the client. Thus, currently, without spending any balance it is possible to make the Network do a lot of work (thus a spam attack vector). Also, invalid mutations can be done which get rejected at the MaidManagers themselves (e.g. mutations by someone who has run out of account balance or by an unregistered client). Though not as severe as the previous example, they will still cause some level of spam.
The problem and some of the solutions are being discussed here. We have some more ideas to eliminate these issues and we’ll continue to discuss the merits of each before putting out a proposal, but it seems a pretty solvable issue.
Then there is the cap of 100 entries for MutableData entries. In the current paradigm we do not actually delete entries but merely blank out the value of the key-value part, and this still occupies space and counts towards the capacity limit (as it needs to be refreshed in churn etc. and treated just like other data). We are also thinking on how to solve this and once we have a proposal we will share it with you.
Apart from that, there were some bug fixes (e.g. this one) in safe_client_libs in the dev branch as the front-end team continues to test against it.
Routing & Vault
We added section versions to routing that increment whenever a section splits or merges. This simplifies some message flows and already allowed us to get rid of some types of messages, just because it makes it trivial to determine whether the information in a message is more current than the one we already have, or whether it is outdated and can be ignored. It is also a first step towards data chains, which will produce an independently verifiable record of the network’s and the data’s history.
In parallel, we have been working on the next step in that direction (Data Chains, Part 1), which will provide a log of all section changes, but not yet data changes. I.e. it will not only version sections, but also produce cryptographic proofs for them, and it will not just include splits and merges, but also additions and removals of nodes.
We are finally taking the last few steps towards deterministic tests.
@qi_ma has been looking at possible alternatives to ChunkStore implementation to improve its performance. With this topic, we can hopefully analyse the findings and pick a suitable solution to move this forward.
Finally, in some other news, Diggory (@dhardy) is off from next week. We’d like to thank him for the time he spent at MaidSafe, he’s been of great help in the Routing Design team and in the production cycles as well. We are close to closing some additional positions in the Engineering teams so again there will be more new faces to get to know 
Enjoy!
