MaidSafe Dev Update - March 2, 2017 - Test 12c

Development Updates
1

Today, we are releasing Test 12c. :tada:

We’ve been testing these binaries in a few internal networks since last week. The test we ran over the weekend and through this week went through about 8500 churn events, so roughly 4250 nodes added/removed respectively. The network state has been stable through the duration of the test and data relocation functioned as expected too.

Here are the changes compared to Test 12b:

  • Port to the new paradigm of mio-v6.
  • Crust can now provide echo/stun service on Windows.
  • Extended tests for tunnelling and churn.
  • We have fixed several peer manager issues related to tunnel nodes and candidates.
  • Send SectionSplit messages as PrefixSection authority to allow resending.
  • We have fixed several issues related to merging sections.
  • Log file management has also been updated. Previously when a vault process quits and restarts, it’d overwrite the old log file. Now, log files also include a timestamp as part of their name which should help prevent this. This should help debugging issues if the need arises, especially if people wish to share their log files. There is a log file issue here as currently the process also creates an empty Node.log file (please ignore this file).
  • From what we saw in the user run network topic after Test 12b and limited space availability, we’ve also reduced the default vault storage capacity from 20 GiB to 5 GiB.

SAFE Vault v0.13.1

3022×1670 846 KB

Only people who either have a TCP port forwarded and specify it in their Crust config file OR people who have UPnP enabled will be able to participate as a vault.

If you forwarded a TCP port on your router (see this website for more information on how to do this), you need to specify the port in the safe_vault.crust.config file (which is located in the same directory as the safe_vault binary) in the tcp_acceptor_port field.

Just a reminder: you can run only 1 Vault node per LAN – running more than 1 will result in this message:

More than 1 routing node found on LAN. Currently this is not supported

Download

SAFE Launcher v0.10.2

1724×1344 101 KB

To connect to Test 12c, you need to use SAFE Launcher v0.10.2.

Each client account is limited to 500 PUTs. A PUT is a request to store a single chunk on the network (a file may contain many chunks). The maximum chunk size is 1 MB.

Please be aware that we might need to restart this test network, wiping all data that is stored on it. We will announce this in a Dev Update (when possible) prior to taking it down.

Download

Documentation

SAFE Demo App v0.6.2

2244×1624 198 KB

The latest version of SAFE Demo App (v0.6.2) should continue to work fine.

When uploading files using SAFE Demo App, the maximum file size is 25 MB per file.

Download

Documentation

SAFE Browser v0.4.3

3824×2300 2.09 MB

The latest version of SAFE Browser (v0.4.3) should continue to work fine.

Download

Other apps

Apps that were working with TEST 11 should continue to work fine with TEST 12c.

Support

If you need help with anything related to SAFE Vault, SAFE Launcher or SAFE Demo App, please use the Support category of this forum.

Where should I report issues?

If you know which component a bug is associated with, feel free to report it on GitHub in the corresponding repo.

SAFE Authenticator & API

The dependency downloader is now integrated with safe_app_nodejs. It downloads system_uri on installation (npm install) of the safe_app_nodejs library. We’re also planning to do the same thing for safe_app (it will be downloaded using the dependency downloader). safe_app_nodejs can be installed for now by adding the repository in the package.json of the application as shown in the example application.

On a few Linux versions, the URI gets registered but the app/authenticator is not being invoked. @lightyear has identified the issue and has raised a ticket to fix the issue.

We also had internal team discussions on improving the APIs and also to move certain helper functions to the Rust FFI layer. The discussion mainly focussed on providing more sophisticated functions on top of the existing available APIs. For example, the API can abstract the exposure of versions. The changes won’t be breaking to the current available APIs. This will help us stabilise the API in one language (Node.js) for now and the same can be easily ported to other languages. @bochaco has been detailing the changes and improvements for the API.

The example application is being tested with Electron Forge to make it easier to package the application. On Windows, the package script hangs, but we are still able to run the application locally.

A new Java implementation of the Email example was added to the safe_examples repository. This was added to help demonstrate the SAFE Network to the Glasgow University students in the upcoming talks that we’re about to give them. New examples are very much going to be tailored for the Authenticator paradigm and this was just a one-off example created to help explain some backend parts of the system to the univeristy students. See this blog post for more info about our partnership with Glasgow University.

SAFE Client Libs & Crust

We are continuing with what we started last week. The mutable data implementation in safe_vault has progressed and the parts that have been done include passing tests. A work-in-progress PR was raised. We also intend to have our first uTP spec discussion on Friday. We expect to break work into week long chunks, starting with a working uTP, moving to an integration with Crust, in addition to accepting UDP hole punched sockets. We do expect several weeks of intense testing there. On completion of that, we will enable key passing from Routing (to ensure man in the middle attack resistance (no key exchange at network layer)) as well as re-enable bootstrap cache (automatic seed nodes if you prefer). Crust will see a lot of activity in the coming weeks below its API to routing. While these are going to be great features to increase the user base that can run vaults and work with the network, it’s also going to have the advantage of being seamless updates to most libraries above Crust.

Apart from that there were various changes carried out in maidsafe_utilities and crust. For instance, it’s now possible to create log files with timestamps if the file_timestamp key is specified in log.toml. If this key is supplied, maidsafe_utilities will provide timestamped filenames so that we don’t overwrite the previous files in case there is a vault restart.

Routing & Vault

We continued improving our tests and fixing issues to get routing ready for Test 12c. Much of the work has been dealing much more effectivly with network growth and decay, merge and splitting of sections. This was an intense period as some of the bugs we chased down were extremely difficult to find. A parallel task of message flow simplification has also meant that almost half of the team have been focussed there. This relates to security and eventually data chains type integration and the consequent data republish.

  • #1374 fixes section split-related issues.
  • #1355 improves the tests for tunnelling.
  • #1380 addresses some bugs in the peer manager, which handles connection states.

Developer outreach

As many of you will be aware, @dirvine and @nicklambert spent the last 2 weeks touring Asia. This included Penang, Jakarta (where @Krishna_Kumar also attended) and Shanghai. The trip is discussed here. Nick also described the trip in our blog here. Needless to say, this was a busy trip and very enlightening. It represents the tip of the iceberg for us in terms of outreach in that part of the world though.

@nbaksalyar is also continuing outreach in Russia and beyond from the Rust perspective. @lightyear is also continually on alert to spread the word. Gabriel is arranging a meetup as well. Much of this is individuals just doing their bit and it’s working. We will again though regroup and try and push outreach further as we go. The balance between talking the talk and walking the walk is a continuous challenge for us. All in all the word is spreading, but Alpha 2 and Beta will spread it much faster :slight_smile:

64 Likes
2

first :grinning:

Ok now Im reading it …

9 Likes
3

Yay ! I am first. Now , reading :slight_smile:

Naaaah

7 Likes
4

oh man, that first line - it’s just pure poetry. so simple yet so beautiful :wink:

13 Likes
6

Guess I gotta go learn node!

@dallyshalla was always telling me to :stuck_out_tongue:

4 Likes
7

so I am in with a vault running from Netherlands, known as : bc7720…(10)

letting the log fill and heading to bed !
Thank you for the very nice update, foundations are getting sound !!

11 Likes
8

Well its working…

safe://hello
safe://wall.knot

Now just got to wait for @whiteoutmashups to learn node and write a REST API with it. ;p

11 Likes
9

1432×101 29.3 KB

got connection without any problems - works like a charm! :slight_smile: terrific!

hmhmm - yeah i was looking into how to connect to the api via cffi in python and have to admit i got a little bit stuck where i had to define the data structure the API exposes :open_mouth:
(i’m not any experienced in writing C API connections for python :wink: … so not really surprising)
I’ll have a look into it again within the next days … but no promises if i’ll find much time in the near future :-.… i wasn’t sure after all if i was looking at the right places in the rust code (then i decided to learning rust basics last weekend to understand the data structure i wanted to connect to)

Is there already a description of how the structs exposed by the api are looking inside that i didn’t see …?

8 Likes
10

Keep up the good work!

2 vaults running (1 on AWS): estimated network size 96 and 114.
The code 11… “Resource temporarily unavailable” error messages in Node logfile are normal?

Let me promote some Flemish pop: safe://zietemduun.draw.

8 Likes
11

up and running, great work team!

I 17-03-02 21:00:03.835139 ------------------------------------------------------------------ 
I 17-03-02 21:00:03.835261 | Node(6884f0..(011)) PeerId(761e21b4..) - Routing Table size:  57 |
I 17-03-02 21:00:03.835308 | Estimated network size: 116                                      |
I 17-03-02 21:00:03.835350  ------------------------------------------------------------------ 
I 17-03-02 21:01:00.422984 Stats - Sent 15000 messages in total, comprising 246933994 bytes, 1 uncategorised, routes/failed: [1824]/0
I 17-03-02 21:01:00.423002 Stats - Direct - NodeIdentify: 44, CandidateIdentify: 12, MessageSignature: 472, ResourceProof: 0/11820/0, SectionListSignature: 540
I 17-03-02 21:01:00.423007 Stats - Hops (Request/Response) - GetNodeName: 1/0, ExpectCandidate: 0, AcceptAsCandidate: 0, SectionUpdate: 0, SectionSplit: 0, OwnSectionMerge: 0, OtherSectionMerge: 0, RoutingTable: 12/0, ConnectionInfo: 16/44, CandidateApproval: 0, NodeApproval: 0, Ack: 1677
I 17-03-02 21:01:00.423012 Stats - User (Request/Success/Failure) - Get: 90/90/0, Put: 0/0/0, Post: 0/0/0, Delete: 0/0/0, Append: 0/0/0, GetAccountInfo: 0/0/0, Refresh: 181
8 Likes
12

/Jealous

Looking forward to the vault requirement coming down a little bit so I can have a proper play with it.

Great stuff from Troon though. Fingers crossed for some really positive test results on this one. :grinning:

10 Likes
13

b31a8761 @ static IP 104.154.104.21:5483

:seedling::evergreen_tree::four_leaf_clover::herb::deciduous_tree:

7 Likes
14

Bah! I’m getting bootstrap error from both AWS and home machines (Linux)
Running safe_vault v0.13.1
==========================
D 17-03-02 21:10:00.534808 [crust::main::service service.rs:454] Network name: Some(“test_network_12_c”)
T 17-03-02 21:10:02.836173 [routing::states::bootstrapping bootstrapping.rs:131] Bootstrapping(7c4bdc…) Listener started on port 5483.
E 17-03-02 21:10:07.041994 [crust::main::bootstrap mod.rs:185] Failed to Bootstrap: (FailedExternalReachability) Bootstrapee node could not establish connection to us.
I 17-03-02 21:10:07.042381 [routing::states::bootstrapping bootstrapping.rs:216] Bootstrapping(7c4bdc…) Failed to bootstrap. Terminating.
D 17-03-02 21:10:07.042388 [routing::state_machine state_machine.rs:268] State::Bootstrapping(7c4bdc…) Terminating state machine

Anyone else?

15

Is there a planned life span for this test net? If it is relatively bug free, will it run for a while, or will it only be a few days?

16

This may mean that you need to explicitly allow external connections at port 5483, otherwise by default on AWS, no connections are allowed. I think…

4 Likes
17

All the testnets in reality are short lived (maybe a week or so) as we move to Alpha II which will be longer lived. The new client API’s right now are looming so I would expect that we move rather quickly, but I always think that :wink:

I would imagine a testnet will go away in a few days, but recently the community keep them going with varying results. We have a couple of parts disabled like bootstrap cache that would allow folks to share a file simply that uses existing nodes on the network to seed of off.

So long winded and I am sorry, but bottom line testnets are probably considered short lived. Alpha/Beta will be much longer lived though.

We really need to get to secured data republish though to save everyone continually uploading again, we know it’s a pita, but we have to prioritise other things right now. More interviews this week though so hopefully more resources will help us, the recent hires have all proven very valuable, although a lot of the work is not yet visible to the community.

29 Likes
18

OK thanks. I’ll check.

19

Vultr instance I’ve just setup is $2.50/month+VAT(uk) and less in reality if I stop it sooner a fraction of that.

For those wanting to play then here is a rough list of some of the buttons I pushed…

================
Goto
SSD VPS Servers, Cloud Servers and Cloud Hosting - Vultr.com

Deploy New Server/Instance
Vultr Cloud Compute
[Any location]
[Ubuntu 16.10]
20GB SSD $2.50/month
[scroll down and enter a server hostname]
[deploy now]

manage
get IP address and a copy of the password

[start a local terminal…]
ssh root@999.888.777.66
[password]
[you are now logged in to your server!]

mkdir ./SAFE
cd ./SAFE
wget https://github.com/maidsafe/safe_vault/releases/download/0.13.1/safe_vault-v0.13.1-linux-x64.zip
sudo apt-get -y install unzip
unzip ./safe_vault-v0.13.1-linux-x64.zip
cd ./safe_vault-v0.13.1-linux-x64
screen
[Enter x2]
./safe_vault

[wait a while…(410s)…for resource proof… ta da!]
[detach from screen and leave it running with “Ctrl-a” “d”]

then “exit” will drop your ssh connection and the vault will be there when you ssh again and “screen -r” to return to the screen with whatever is the latest output from the vault.
There are other options to screen but I can’t recall atm the detail of nohup etc to advise those.
also that setup is for a short burn and not as secure as I’d make a long standing server with new user and ssh only etc.

:thumbsup:

14 Likes
20

That is cheap. Thanks for the info. I’ll see if I can work it out tomorrow. I’m a bit of a windows-using tech-tard though, so there’s a good chance I’ll get confused, annoyed and give up :laughing:

I see myself as the perfect tester for how idiot-proof and user friendly SAFE is when it’s ready to launch :wink:

8 Likes