[Status: offline] SAFE Network Testnet - Vaults from home with IGD - Iteration 2

Noting that the browser already seems to be auto-updating to .5

I’ve yet to understand github for submitting the simplest fixes… so, the text on the abc suggests --test flag
should be --testing

https://github.com/maidsafe/safe-api/tree/master/safe-cli#run-a-local-network

Run a local network for testing: --test

The run-baby-fleming command accepts a --test or -t

Happy now, I have a local network running. :smiley:

The vault seems just to complain about authentication timeouts at the start, which is unfortunate because truly it’s working.
I wonder it could do with declare success initially, to help the user ignore these kinds of output errors:

Setting up authenticator against local SAFE network...
[2020-07-21T19:43:14Z ERROR safe] safe-cli error: [Error] AuthdClientError - Failed to execute authd from '/home/safe/.safe/authd/safe-authd': No such file or directory (os error 2)
Creating a SafeKey with test-coins...
Sending account creation request to authd...
Sending login action request to authd...
[2020-07-21T19:45:19Z ERROR safe] safe-cli error: [Error] AuthdClientError - Failed to establish connection with authd: [Error] ClientError - Failed to establish connection with remote QUIC endpoint: timed out
[2020-07-21T19:45:24Z ERROR safe] safe-cli error: [Error] AuthdClientError - Failed to establish connection with authd: [Error] ClientError - Failed to establish connection with remote QUIC endpoint: timed out
3 Likes

Can you confirm the version of your authd with $ safe auth update? it should be v0.0.11 otherwise you will have issues connecting to it. Also make sure it’s up and running with $ safe auth status

Edit: actually I just saw the error you are getting is '/home/safe/.safe/authd/safe-authd': No such file or directory (os error 2), perhaps you need to install/re-install the authd with $ safe auth install

2 Likes

That was a clean install following the abc that has the install vault ahead of the install auth… but yes auth is v0.0.11

However, on a restart is doesn’t upload
and one more frequent glitch looks to be that it’s not setting balance.

eg: below “Account was created successfully!” but then “No SafeKey found at specified location”… so, I don’t know if that’s important for test network… kind of surprised it was doing that and not waiting for the user just to create an account… but even if I create account, login etc it’s not actioning an upload. I’ll keep at it in case it becomes clearer.

$ safe vault run-baby-fleming -t
Creating '/home/safe/.safe/vault/baby-fleming-vaults' folder
Storing vaults' generated data at /home/safe/.safe/vault/baby-fleming-vaults
Launching local SAFE network...
Launching with vault executable from: /home/safe/.safe/vault/safe_vault
Network size: 8 vaults
Launching genesis vault (#1)...
Genesis vault contact info: ["127.0.0.1:12000"]
Launching vault #2...
Launching vault #3...
Launching vault #4...
Launching vault #5...
Launching vault #6...
Launching vault #7...
Launching vault #8...
Done!
Setting up authenticator against local SAFE network...
Stopping SAFE Authenticator daemon (safe-authd)...
No running safe-authd process (with PID 6662) was found
Starting SAFE Authenticator daemon (safe-authd)...
safe-authd started (PID: 6925)
Creating a SafeKey with test-coins...
Sending account creation request to authd...
Account was created successfully!
SafeKey created and preloaded with test-coins. Owner key pair generated:
Public Key = 832e56f78aee8e0afe1c67a888d6a348b4bed14287df94ef843837b1780240bd7368f24696681ae0c0b759d86ed297f2
Secret Key = e1429cdb40ea65f8ff290cb51ff5a3f850c090810a8d91e2d89f814a66cf3c47
Sending login action request to authd...
Logged in successfully
Authorising CLI application...
Waiting for authorising response from authd...
[2020-07-21T20:23:09Z ERROR safe] safe-cli error: Application authorisation failed: [Error] AuthdError - [Error] AuthenticatorError - Failed to authorise application on the network: Core error: Data error -> Balance does not exist

$ safe keys balance
Enter secret key corresponding to the SafeKey to query the balance from: e1429cdb40ea65f8ff290cb51ff5a3f850c090810a8d91e2d89f814a66cf3c47
[2020-07-21T20:30:15Z ERROR safe] safe-cli error: [Error] ContentNotFound - No SafeKey found at specified location
$ safe vault killall
Success, all processes instances of safe_vault were stopped!
$ safe auth stop
Stopping SAFE Authenticator daemon (safe-authd)...
Success, safe-authd (PID: 6925) stopped!
2 Likes

Right, cool, you are now getting authd working properly there and creating the account correctly. Those other errors are semething different now, which could be what we experience in our CI sometimes and which we still didn’t get to the bottom of. I assume you are running latest vault binary right? Perhaps you can give it a try by first removing the /home/safe/.safe/vault/baby-fleming-vaults folder? I think there was an issue with accounts not being found after the local network is restarted.

2 Likes

yes, I deleted those vault folders, looking for a fresh restart (for no good reason than OCD) and then tried a few times because it wasn’t just working to upload… tail -F on the vault logs shows activity and it acknowledged requests for balance on keys but the upload doesn’t seem to action… and wondering atm if it is on the occasions that there is not balance…but then I’ve also created an account and auth subscribe logged in etc… so, not obvious cause for not doing the safe files put --recursive. works now on the back of initial balance error; so, my experience is 1/5 times it works and I can’t see the difference atm but keeping this instance as it works.

3 Likes

So, I just put a load of directories as upload and watched as it ate memory ~0.1% of 15GB each 5secs or so, which I guessed is 20MB/s and cut it off at 90%.

TIL

renice +10 on all the vaults, seems to have no effect on its appetite for memory.

The vault logs become huge and seem to be by far the largest files being created.

I guess not but worth asking, is there a way to minimize the repetitive volume of the same messaging?.. I guess it’s deliberate from there for now but a switch off might help it function in a more balanced way??

So, too much of this drive messaging:

INFO 2020-07-21T22:38:04.485002915+01:00 [/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/quinn-0.6.1/src/connection.rs:228] drive; id=5
INFO 2020-07-21T22:38:04.485026439+01:00 [/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/quinn-0.6.1/src/connection.rs:228] -> drive
INFO 2020-07-21T22:38:04.485071577+01:00 [/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/quinn-0.6.1/src/connection.rs:228] <- drive
INFO 2020-07-21T22:38:04.485088095+01:00 [/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/quinn-0.6.1/src/connection.rs:228] -- drive

then
/mnt/vault/.safe/vault/baby-fleming-vaults$ ls -Rl

I was surprised that it’s not immutable files being created but seemingly all centred on /chunks/sequence and /chunks/mutable

Edit: noted the logs summed to 500MB so, it likely not just logs held in memory causing this creep over 10GB

4 Likes

For now the work around as detailed in OP is to set the following before launching the local net:

$ export RUST_LOG=safe=trace
1 Like

Without doing puts and now without logs as large, it’s doing the same memory creep.

The only oddity I can see is that one vault instance is not the same resource usage… memory for the bottom one on the list is much smaller - that instance is mostly asleep and rarely flickers back as running and sleeps again. So, perhaps one vault at a disadvantage…

Otherwise simple observation that x4 CPUs are maxed out at 100% and not obvious to me what work it’s doing with a default near zero content to balance. I guess loops are unlikely unless it’s just made to check as much as practical that content is as should be.

2 Likes

Probably the infant and they have little to do. We are looking at resource usage now and @lionel.faber got some pretty good pointers. We will see if we can find this, but I am dead keen to get parsec and that gossiping it does out of there. Too many messages with no real reason is a killer for these networks. Looks great in mock testing and all that jazz, but real world, it’s a big no no.

12 Likes

Just attempted to join and it did all work until the last step which as explained above means I need to wait for next iteration, cool to get it running nonetheless! Waiting for the next version to launch and build some app on it if possible.

ERROR 2020-07-29T15:03:42.464906166+08:00 [src/bin/safe_vault.rs:169] Cannot start vault due to error: Routing(Network(IgdNotSupported))

5 Likes