MaidSafe Dev Update :safe: 21st April 2016

Firstly, sorry for the delay in this week’s update, we had a few things we wanted to complete before sharing this update and I would also like to thank @dirvine for picking up and producing a bumper Dev Update last week. As David highlighted in that update, Crust and Routing are now signed off as MVP ready with Vaults in the final phase of testing and it is all looking good to be signed off as well. So no MVP release as part of today’s update, we need to tidy up a few loose ends before wrapping and tying the bow around about it. However… yes there is a however, if you are super keen to get involved in community Vault testing a little earlier - we are in the position to share Vault (TCP only) binaries that can be run from home (subject to some limitations please read before jumping in, details below). I really want to be 100% clear, this is NOT the MVP… I repeat …not the MVP :wink: but it is a Vault running from your own computer and joining the SAFE Network (we are running 100 Vault nodes) - which is in itself extremely cool and another milestone as part of the rolling community testing release cycle.

So how do you do this…

##safe_vault v0.6.0

  • Download safe_vault binary and unzip files.
  • To run a Vault, it is currently necessary to be either directly connected or forward a TCP port in your router and then specify the port you have forwarded within the Crust config file (safe_vault.crust.config).
    At tcp_acceptor_port replace null with the port you forwarded e.g. 1234
  • Execute the binary (from the command line, although not necessary on Windows) - you have now joined a Vault to the SAFE Network :slight_smile:

##SAFE Launcher v0.4.2

##Limitations

This Vault test comes with some limitations:

  • To run a vault, it is currently necessary to be either directly connected or forward a TCP port and specify that within the Crust config file.
  • File size will currently be limited to 5Mb
  • Vault size will currently be limited to 30 MB (30 chunks)
  • Storage per client account is limited to 50 MB (50 chunks).
  • Churn handling - replacing connections and relocating data after nodes joined or left the network - is still very rudimentary. It will almost certainly fail if there are hundreds of megabytes of data to relocate, which is why we set a low Vault size limit for now and ignore the one set in the Vault config file.
    But too high a churn frequency might well do even more damage to the network; we’re not sure how much - that’s what real-world tests are for! :slight_smile:
    However, for most of these limitations we already have concrete solutions that we will implement in the coming weeks.
  • Zombie data: If there is heavy churn it can happen that a client requests deletion of data while that same data is being relocated. In that case, sometimes the data comes back from the dead after its deletion.
  • Known issue - occasionally the client may not receive enough “agreeing” responses from a network operation in order to decide the result. This causes the client to hang. It should just be restarted if this happens.
  • There is still a lot of work to be done in the Routing library for it to make the security guarantees needed not only for safecoin but also for preventing anyone from taking over the network entirely. There are known attacks that the network is vulnerable to.

This really is just for folk that are super comfortable in the command line and messing with their routers to port forward, however it delivers the expectations that David outlined in last week’s update … I don’t know you go on holiday for a week and folk start to “convey our feelings and desires” and as none of us want to see David “slaughtered” by the community, so we thought we better share working Vault code for folks to test :slight_smile:

This is the next phase of community testing - aimed squarely at techy community members and moves us yet another step forward. For the rest of us that prefer things simple and easy to use, the download and click to install completed MVP will not be far behind.

As shared in the postponed dev update post we published the Roadmap here. We have received a lot of good feedback and as @nicklambert has already stated the front-end guys have taken all the feedback onboard and are implementing improvements.

##Crust - uTP / API : Vinicius / Andrew

MVP (Minimum Viable Product) ready and feature frozen.
Looking beyond the MVP implementation, work has already started on the transport library which will eventually replace parts of Crust’s internals. @vinipsmaker has been working on adding heartbeat / keepalive messages to Crust.

##Routing

MVP (Minimum Viable Product) ready and feature frozen, resources reallocated to other libraries.

##Vaults : Andreas / Brian / Fraser / Qi / David

After more advanced testing this week of the Vault code (in particular using a mock Crust interface to simplify testing and fault-finding) we’ve made some simplifications to the code.

These may persist in light of how well the testing is now going, but further development of the simplified code will be needed to allow the Vaults to handle all that will be asked of them. An alternative would be to look at working through the complexity that was revealed in the previous code.

Either way, it appears that the simplified code is likely to be robust enough for supporting the upcoming MVP. Current tests using the mock Crust are indicating that a small network can handle things like three Vaults within a close group all going offline simultaneously. The tests aren’t exactly full stress tests, but we are approaching the limits of what we would expect to be realistic scenarios and the Vaults are handling this well.

On the droplet Vault testnet (a 50 node network), testing has been going very well. In a test run earlier this week, Vaults successfully retained all data on 2433 churn events (1211 started and 1232 stopping) with an event taking place every 20 seconds. We are keen to continue gathering and begin sharing more of this type of data as the testing of Vaults expands into the community.

##Client - Launcher : Krishna / Spandan / Shankar

maidsafe_utilities logging was formatted to follow a standard JSON structure which will be sent to the log visualiser server. There is a little bit of work still left in Vaults logging to determine how an ID can be associated to Vaults which can then be used for identifying the Vaults. This ID can be used in the log visualiser server to analyse the logs of a Vault.

A Work In Progress Pull Request is raised for handling the ImmutableData operations. Once in place the library should ensure no Immutable Data chunks of size > 1 MB (currently) are stored on the network. Moreover this PR will help us to take the Low level API RFC discussions forward with snippets to showcase.

We tried a few samples with SSL integration for Launcher using a self-signed certificate, but things don’t seem to work as we would like. The first problem is that we need to accept the untrusted certificate and also the clients (JAVA, nodejs) should add the CA or allow all untrusted certificates which seems to be more hassle. The proxy doesn’t seem to redirect for a self-signed certificate. Thus we have decided to park this for now and will try incorporating other standards that can be added to the Launcher APIs. The changes will be detailed in a separate RFC later this week. Once the RFC is discussed and agreed, the changes detailed in the RFC must be implemented across all the APIs that are exposed so far.


To repeat the message from David last week, MaidSafe resources are now being much more focused into areas that will have a direct impact for App Devs and early SAFE adopters or as he put it “the people that matter”. So now we have working Vault code and an imminent MVP release, even the more reserved amongst us can “convey our feelings and desires” and begin to allow ourselves to get (a little bit) excited.

Thanks as always for your support; here is a link to the weekly transcript.

50 Likes

Yeah, first! This is a great next step, congratulations on this one :thumbsup:.

11 Likes

oh boy!! oh boy!! oh boy!! oh boy!! Time to join in.

4 Likes

Great work guys! (and girls)

7 Likes

HOT !!! I guess I know what I do tonight

5 Likes

WOOT! Much appreciation to all!!!

4 Likes

Crazyyyy!!! I love it!

8 Likes

It really should not be a problem handling untrusted certificates on a localhost-only API. There are other ways to validate an X509 cert besides trusting the issuer. Also, it is not that difficult to add a CA in node or in the JVM keystore (even at runtime, I have done it many times). Regardless I’ll wait for the RFC to discuss these things. The best answer is to stop encrypting localhost only traffic unnecessarily. I mean if I have an app that statically links the rust libs and doesn’t have to use your encryption, what have you really gained by encrypting in the launcher only on local loopback? Think of localhost-HTTP as no different than named pipes or any other IPC mechanism but with a friendly HTTP interface. I’ll be looking forward to the RFC explaining why the encryption is needed when I can show a snippet of direct-lib-linked code that circumvents it.

3 Likes

This is AWESOME !

Good job folks!

6 Likes

Heck yes!..exciting characters to make it to 20!!!

5 Likes

Nice, quick question. Does running this help you test or is it for us to test?

3 Likes

vault config: are we atm limited to ipv4 and portforwarding if on nat? or is the port valid also for ipv6 global addresses?

1 Like

very nice. in for the long haul. can’t wait to test this when i get home

2 Likes

So this is a preview of upcoming releases. You have a vault to play with but you will need to set up port forward on your router to try it. You can continue with client tests as usual, but these are severely limited. Expect this to last a very short time then be replaced over some vary quick iterations as we progress.

12 Likes

also did we respawn the whole network since the last launcher and demo app from march 2016, or are our credentials and accounts still valid and existing on the current interation?

This is EXACTLY what was needed. Being able to create a vault from home and connect to the SAFE Network for the 1st time is history in the making. The baby is out of the womb!

8 Likes

Creating a better future
Requires creativity in the present


#step by step the era of the creative is rising

6 Likes

I know most people on this forum are very computer literate but for those that need help with port forwarding this is a great site - http://portforward.com/ I used it a lot when doing tech. support over the phone.

8 Likes

self-reply: i manually added a port on the local fw here, but then the vault binary upon launch complained about igd/upnp fail. when we have upnp activated then we dont need to open the port manually? enhancement for the future to not fail when port is already opened.

my port is open, but all i get is this:

WARN 21:33:57.972685600 [w_result lib.rs:288] Failed to find an IGD gateway on network interface {CD227C91-7DED-4057-8631-4FB9F7D669BA} 192.168.10.139. igd::search_gateway_from_timeout returned an error: IO error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (os error 10060)
WARN 21:33:57.972685600 [w_result lib.rs:288] Failed to find an IGD gateway on network interface {C8B796B8-8406-458D-991A-251370718CBE} 192.168.56.1. igd::search_gateway_from_timeout returned an error: IO error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (os error 10060)
ERROR 21:33:59.786789400 [crust::connection connection.rs:489] Invalid crust msg on tcp accept: Deserialisation failure
ERROR 21:33:59.787789400 [crust::service service.rs:298] Failed to start listening for peers.
WARN 21:34:10.492401700 [w_result lib.rs:288] Connecting to endpoint 139.59.169.137:41964 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
WARN 21:34:10.492401700 [w_result lib.rs:288] Connecting to endpoint 139.59.169.137:41964 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
WARN 21:34:10.839421500 [w_result lib.rs:288] Connecting to endpoint 139.59.169.144:37591 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
WARN 21:34:10.839421500 [w_result lib.rs:288] Connecting to endpoint 139.59.169.144:37591 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
WARN 21:34:10.921426200 [w_result lib.rs:288] Connecting to endpoint 139.59.169.145:37771 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
WARN 21:34:10.921426200 [w_result lib.rs:288] Connecting to endpoint 139.59.169.145:37771 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
WARN 21:34:10.971429100 [w_result lib.rs:288] Connecting to endpoint 139.59.169.122:46590 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
WARN 21:34:10.971429100 [w_result lib.rs:288] Connecting to endpoint 139.59.169.122:46590 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
WARN 21:34:10.972429100 [w_result lib.rs:288] Connecting to endpoint 139.59.161.28:42020 failed: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)

4 Likes