[status: offline] SAFE Network Testnet - Vaults from home with IGD

After the release of the Baby Fleming shared section we have been designing, planning and implementing some new features that take us forward on our road to beta. After a number of internal testnets we are happy to announce the first testnet (in a long time) with vaults from home. :tada:

What it does:

Successful results from the Baby Fleming shared section allowed us to add some additional features which have taken us forward towards our end goal. Today’s release includes a single section network of ten nodes that are deployed on DigitalOcean. Seven of which are the initial nodes to join the network and hence they are the Elders. The remaining nodes are a couple of Adults and an infant. See the Further Information section at the bottom of this post for an explanation of Elder, Adult and Infant nodes.

As a user you can run a Vault that will join the section, will eventually be promoted to an Adult and given the responsibility of holding Immutable Data chunks. The Vault uses the IGD protocol to forward the internal port number on your computer to an external port on the router. This will allow the other nodes to seamlessly communicate with your running vault. As vaults running from home may go offline, we need to ensure that a minimum number of copies of each Immutable Data chunk is always available on the network. The Vault binary released today includes a data recovery mechanism to maintain this copy count.

In summary, here are the features included in this testnet:

  • Single section testnet with node ageing.
  • Vaults use the routing’s messaging API with signature accumulation for intra-section communication.
  • Users can run vaults from home to eventually become Adults in the section, with the responsibility of holding Immutable Data chunks.
  • Vaults will use the IGD protocol to forward the local port that the vault uses to an external port on the router.
  • When a node holding some chunks goes offline, the network will create an additional copy of this data at a different node to maintain the minimum number of copies.

What it doesn’t do:

We are taking one step at a time towards our end goal and hence there are a few limitations to this iteration of the testnet.

  • Section splits have not been implemented in this testnet iteration and so if the single section network we are running breaches its size limit then we may need to restart it. There isn’t an exact number of nodes which will trigger a split, this depends on how the nodes on the shared network are prefixed and so divided into subsections, with node churn making prefix members vary. We can say for certain that the minimum number of nodes that could trigger a split is 1000. We have increased this value to delay the section split in this testnet iteration.

  • If your router does not (correctly) support the IGD protocol, you will not be able to take part in today’s testnet iteration. Don’t worry, we have solutions for you but would like to save them for another iteration of the testnet so we can concentrate today on ensuring that our IGD solution works as expected, and to get a general overview of how many participants out there actually have routers with working IGD. If your router does not correctly support IGD then you will see the message below in your vault logs

    Automatic Port forwarding Failed. Check if UPnP is enabled in your router's settings and try again.    Note that not all routers are supported in this testnet. Visit https://safenetforum.org for more information.
    

    If you do not see this exact message in your vault logs, leave your vault running. We have seen instances of log messages such as Could not send the IGD response: sending on a disconnected channel - "SendError(..)" which could be interpreted by some as IGD failing, but this is not the case and your vault should eventually start contributing to the network.

Instructions to join:

You can bootstrap to the network using the nodes running on DigitalOcean as hardcoded contacts, then watch the logs as your vault joins the network, progresses to Adult status, and plays its part in hosting Immutable Data Chunks. Of course, you will also be able to create an account on this network, login, upload data, create keys and wallets, and all the other commands described in the CLI User Guide. This guide will take you through connecting to the shared section started by MaidSafe, but of course it can be applied to connecting to any shared section, hosted by anyone.

To connect you should first make sure that your CLI, Authenticator daemon and Vault are all updated to today’s latest releases:

$ safe update
$ safe auth update
$ safe vault update

You now need the network configuration containing the details of a hardcoded contact that will bootstrap you to the shared section. If you have connected to this or previous iterations of our shared section then you may already have a shared-section network profile saved on your machine. You can confirm this and update it to the latest configuration using safe networks check:

$ safe networks check
Checking current setup network connection information...
Fetching 'my-network' network connection information from '~/.config/safe-cli/networks/my-network_vault_connection_info.config' ...
Fetching 'shared-section' network connection information from 'https://safe-vault-config.s3.eu-west-2.amazonaws.com/shared-section/vault_connection_info.config' ...

'shared-section' network matched. Current set network connection information at '~/.config/safe_vault/vault_connection_info.config' matches 'shared-section' network as per current config

If you don’t have a configuration in your results which points to the exact S3 location listed in the results above, you can add using safe networks add:

$ safe networks add shared-section https://safe-vault-config.s3.eu-west-2.amazonaws.com/shared-section/vault_connection_info.config
Network 'shared-section' was added to the list

Now you need to ensure you are set to use this shared-section configuration that we have updated/added, we can use safe networks switch shared-section for this:

$ safe networks switch shared-section
Switching to 'shared-section' network...
Fetching 'shared-section' network connection information from 'https://safe-vault-config.s3.eu-west-2.amazonaws.com/shared-section/vault_connection_info.config' ...
Successfully switched to 'shared-section' network in your system!
If you need write access to the 'shared-section' network, you'll need to restart authd, login and re-authorise the CLI again

We now have our CLI, Authenticator daemon and Vault components up-to-date, and we have the latest hardcoded contact details to bootstrap to the shared section so everything is in place and we’re ready to launch our vault and add it as a node. This is achieved using safe vault join as follows:

$ safe vault join
Creating '/Users/maidsafe/.safe/vault/local-vault' folder
Storing vaults' generated data at /Users/maidsafe/.safe/vault/local-vault
Starting a vault to join a SAFE network...
Launching with vault executable from: /Users/maidsafe/.safe/vault/safe_vault
Node started with hardcoded contact: 161.35.36.185:12000
Launching vault...
Vault logs are being stored at: /Users/maidsafe/.safe/vault/local-vault/safe_vault.log

Your vault will now launch and attempt to connect to the shared network. You can keep an eye on its progress via its logs, which can be found at ~/.safe/vault/local-vault/safe_vault.log.

Do I need to run a vault to participate?

The highlight of this release is being able to run a vault from your own home (IGD dependent) and watch as it plays a part in a wider network, but you do not need to run a vault in order to participate in the shared section testing. Whether you are running a vault or not, you can still connect to the network as a client and do everything you could do on the previous single section networks that you ran from home or that MaidSafe hosted and you connected to. See the CLI User Guide for a comprehensive list of available commands and instructions on how to use them.

Further Information

What is IGD?

IGD is short for Internet Gateway Device Standardized Device Control Protocol. As most people running vaults from home will be behind a router, IGD allows us to automatically map the port that your vault runs on, and so seamlessly allows data to flow in and out of your vault. Without correctly functioning IGD, a user would need to manually set up this port forwarding on their router.

Wikipedia entry

Incorrectly Functioning IGD

Many routers that claim to support IGD through Universal Plug and Play (UPnP) actually do not implement it properly and so will not connect to this version of the testnet - your log file will display this message:
Automatic Port forwarding Failed. Check if UPnP is enabled in your router's settings and try again. Note that not all routers are supported in this testnet. Visit https://safenetforum.org for more information.

If you do not see this exact message in your vault logs, leave your vault running. We have seen instances of log messages such as Could not send the IGD response: sending on a disconnected channel - "SendError(..)" which could be interpreted by some as IGD failing, but this is not the case and your vault should eventually start contributing to the network.

If your router has a UPnP option in its settings, please ensure it is enabled. If it still doesn’t connect to the shared section, with the logs reporting the same Automatic Port forwarding Failed, then sit tight and skip this particular iteration of the testnet. We would like to isolate those with IGD correctly implemented at first, fix any issues resulting from them, then open up to allow others to enable manual port forwarding and join in. We don’t anticipate there will be a lengthy delay before an iteration which includes manual port forwarding is made available.

Where are my vault logs?

When you launch your vault you should see the location of your log file printed on screen - this will be ~/.safe/vault/local-vault/safe_vault.log. You can tail your logs with a command such as tail -f ~/.safe/vault/local-vault/safe_vault.log

What are Elder, Adult and Infant nodes in the SAFE Network?

These terms are related to Node Ageing. Node age means that as vaults prove themselves over exponentially long periods of time (age) then they grow older by 1. Each time they age they are moved to a different section (for security) and take on a new name.

Many people will run a vault briefly and that is great, but we don’t want the network to do much work with unknown new vaults. We call these Infants and as infants we don’t ask much of them, but we do monitor that they are there. We don’t remember Infants so they cause no lasting network footprint when they restart. We intentionally don’t record infant information to save network resources.

As an Infant does some work and hangs around long enough it is promoted to Adult by increasing its age by 1 and relocating it to a new section.

The 7 oldest vaults in a section are the most trusted, we call these Elders and they make the decisions, provide network (section) authority, and decide what Adults will store what data. The Elders are the controllers of the section and the only vaults in a section that communicate with other sections. We can consider Adults are the disk space of the Elder group.

For more information on Node Ageing in the SAFE Network, have a read of the SAFE Network Primer section on it.

Where can I report any issues found?

The ant colony is only as strong as its worker ants, and you are the backbone of this colony. Please let us know if you come across any issues in your testing. You can do so by reporting these issues in the comments on this topic’s thread. We will monitor and investigate these as soon as we can.

66 Likes

Where do you see your node age?

8 Likes

Well, the age of your node isn’t something that’s being logged anywhere at the moment. Its not useful in the context of the Vault. Routing uses it to determine if the node is an Adult or an Elder and the Vault behaves accordingly. But I see how that is an interesting property to keep track of. Will include in the test nets to come :slight_smile:

21 Likes

Don’t you know you should never ask a woman’s or vault’s age?? :stuck_out_tongue:

17 Likes

Can I limit amount of storage used by Vault with this version?
Or put it at disk, where I have enough free space.
If vault eat all of my available space on disk C: (7GB), that will be bad.

7 Likes

Not in this version, this is early stage

11 Likes

We haven’t limited the use of disk space in this iteration. You can, however, choose the storage location of the vault. It’s not possible via the safe CLI. But you can do this.

cd ~/.safe/vault/ # or the directory where the vault binary is located
./safe_vault -h "[\"167.172.55.70:12000\"]" --root-dir <location for files>

Let me know if you need additional help.

EDIT: Added the option needed.

13 Likes

I don’t see any activity in log files.
Last lines are:

INFO 2020-06-11T19:23:59.939225700+03:00 [C:\Users\runneradmin\.cargo\git\checkouts\routing-616e72cf40f12bc0\4a092f0\src\node\mod.rs:807] 698ecc..(?) This node has been approved to join the network at Prefix()!
INFO 2020-06-11T19:23:59.939225700+03:00 [C:\Users\runneradmin\.cargo\git\checkouts\routing-616e72cf40f12bc0\4a092f0\src\consensus\parsec.rs:348] 698ecc..(?) Init new Parsec v6
INFO 2020-06-11T19:24:02.449369300+03:00 [C:\Users\runneradmin\.cargo\git\checkouts\quic-p2p-9a2715614ed5c99d\c6ae051\src\lib.rs:292] Could not send the IGD response: sending on a disconnected channel - "SendError(..)"

Is that ok?
I have public IP, so vault should work without port forwarding.
But how to be sure?

8 Likes

That looks ok to me, eventually you should be receiving requests to store ImmutableData files, and these type of logs should show up there:

INFO 2020-06-11T12:12:19.940162410-04:00 [src/vault.rs:517] Responded to our data handlers with: Response { response: Response::Mutation(Success), requester: App(PublicKey::Bls(b18224..), owner: PublicKey::Bls(b72224..)), message_id: MessageId(6b575f..), refund: None }
INFO 2020-06-11T12:12:25.500362529-04:00 [src/vault.rs:347] Received message: 000..c4f
 Sent from Section(Prefix()) to Node(23c8d9..)
INFO 2020-06-11T12:12:25.508410591-04:00 [src/vault.rs:517] Responded to our data handlers with: Response { response: Response::Mutation(Success), requester: App(PublicKey::Bls(b18224..), owner: PublicKey::Bls(b72224..)), message_id: MessageId(7359c4..), refund: None }

Once that starts to happen, you should also see the ImmutableData chunks being stored at ~/.safe/vault/local-vault/chunks/immutable/

11 Likes

Exactly what @bochaco said.
For now, will this do? :smiley:
image

14 Likes

Awesome work, I have a few questions.

Can I run the safe vault join command attached to the terminal with a special flag like “-a” so I can ctrl-d to disconnect and view logs live?

When I installed the safe cli using

curl -so- https://safe-api.s3.amazonaws.com/install.sh | bash

it failed to detect and install into my .zshrc profile.
I had a look at the GitHub and it seems you check for the .bashrc file and skip looking for .zshrc . This didn’t work for me because I have both profile files and I use both terminals.

Is there a verbose logging mode? I did look for --help, but its very basic, please share secret nerd commands.

4 Likes

I guess I’m in…

INFO 2020-06-11T18:38:00.746208800+02:00 [src\vault.rs:128] Initializing new Vault as Adult

Last log lines…

INFO 2020-06-11T18:38:06.338585+02:00 [C:\Users\runneradmin.cargo\git\checkouts\routing-616e72cf40f12bc0\4a092f0\src\node\mod.rs:807] c9bfb3…(?) This node has been approved to join the network at Prefix()!
INFO 2020-06-11T18:38:06.338585+02:00 [C:\Users\runneradmin.cargo\git\checkouts\routing-616e72cf40f12bc0\4a092f0\src\consensus\parsec.rs:348] c9bfb3…(?) Init new Parsec v6

10 Likes

Same here :+1:

5 Likes

These are the magic words right? from safe_vault.log :sweat_smile:

INFO 2020-06-11T18:38:31.000628740+02:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/4a092f0/src/node/mod.rs:807] eb4629…(?) This node has been approved to join the network at Prefix()!
INFO 2020-06-11T18:38:31.000766830+02:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/4a092f0/src/consensus/parsec.rs:348] eb4629…(?) Init new Parsec v6

10 Likes

bad news…

INFO 2020-06-11T18:48:31.333222900+02:00 [C:\Users\runneradmin.cargo\git\checkouts\quic-p2p-9a2715614ed5c99d\c6ae051\src\lib.rs:292] Could not send the IGD response: sending on a disconnected channel - “SendError(…)”

3 Likes

Once you launched it with $ safe vault join it will run detached, so you can exit and/or see the logs live, unless I’m misunderstanding you.

Not sure what you are looking for, assuming you are already aware of the commands explained in CLI User Guide?

6 Likes

As explained in the OP, you can ignore that error, let it run, eventually you should start getting some requests to store and serve ImmutableData files as other users start storing files on the section.

10 Likes

Yep. Too fast reading…

7 Likes

Yes, I can tail the log file myself and watch it, but its not logging a great deal. Can I run some sort of debug mode? (do I need to build my own version)? I just wanted more insight in what it was actually doing.

4 Likes

Ah right, yes, you can kill it with $ safe vault killall and start it again with $ safe vault join -y for DEBUG level, or $ safe vault join -yy for TRACE level.

8 Likes