Support Issues with Baby Fleming Version 1 (Vaults Phase 2a - single-section network)

This release has been described in the recent dev update
https://safenetforum.org/t/safe-network-dev-update-march-5-2020/31227/62

Please discuss issues/problems here

9 Likes

After 6min 24, it fails this time with a 2MB .mp3 file
willie@gagarin:~/.safe/vault$ time safe files put ~/Downloads/B17_KBDL_Edited_for_Time.mp3
[2020-03-06T01:43:16Z ERROR safe] safe-cli error: [Error] NetDataError - Failed to PUT Sequenced Append Only Data: Error(CoreError(Request has timed out - CoreError::RequestTimeout))

real	6m24.710s
user	0m0.535s
sys	0m0.097s
willie@gagarin:~/.safe/vault$ ls -l ~/Downloads/B17_KBDL_Edited_for_Time.mp3 
-rw-rw-r-- 1 willie willie 2016170 Oct  3 05:04 /home/willie/Downloads/B17_KBDL_Edited_for_Time.mp3

But a 1Mb file works OK - not quickly but its just a wee baby still

    willie@gagarin:~/.safe/vault$ time safe files put ~/Downloads/WP_20170601_14_41_14_Pro.jpg 
FilesContainer created at: "safe://hnyynyzc4udp1jeu65kjnw5m8rx7xte798dosq69dozpahm8kog7n7ha1nbnc"
+  /home/willie/Downloads/WP_20170601_14_41_14_Pro.jpg  safe://hbwybondwan37j4b4uy8gjhtz5yitiawd6rwaxbw3pkrx7jaobxhfhph9g 

real	0m44.005s
user	0m0.236s
sys	0m0.072s
willie@gagarin:~/.safe/vault$ ls -l ~/Downloads/WP_20170601_14_41_14_Pro.jpg 
-rw-rw-r-- 1 willie willie 1016218 Jun  2  2017 /home/willie/Downloads/WP_20170601_14_41_14_Pro.jpg

so lets try a 1.5MB file …

willie@gagarin:~/.safe/vault$ time safe files put ~/Downloads/CP-168U.pdf 
FilesContainer created at: "safe://hnyynysde4s9dq9ktfpi1ebp1cmgyf8j5a4u7d89zjuxffota1k9wrjii6bnc"
+  /home/willie/Downloads/CP-168U.pdf  safe://hbkyjon1bmbzimuug5kgc7zgh68skqzbeb4m6jjf5fpg6padb3oh4jr66b 

real	0m37.465s
user	0m0.259s
sys	0m0.045s

Slightly faster for a larger file. Obv the file type matters so lets get sensible and instead of picking something roughly the correct size from ~/Downloads lets Do It Right and copy @maz 's method.

It seems the maximum size it will handle is between 1.5 - 2 Mb.

Or is it?
At least this time it fails faster

willie@gagarin:~/.safe/vault$ dd if=/dev/urandom of=/tmp/rand.dat bs=1k count=1500
1500+0 records in
1500+0 records out
1536000 bytes (1.5 MB, 1.5 MiB) copied, 0.0417652 s, 36.8 MB/s
willie@gagarin:~/.safe/vault$ time safe files put /tmp/rand.dat
[2020-03-06T02:22:25Z ERROR safe] safe-cli error: [Error] NetDataError - Failed to PUT Sequenced Append Only Data: Error(CoreError(Operation forbidden - CoreError::OperationForbidden))

real	0m34.721s
user	0m0.237s
sys	0m0.045s

I tried rand.dat sizes of 1.75MB and 1.625MB, they both timed out around the 6min mark.
Next step will be to set logging to DEBUG and increase rand.dat size until it fails again. I’ll zip up the logs and sent to the devs if it helps.

Mods - should this now move to support?

9 Likes

See the section of this post under ‘Quirks’.

It has a timeout
pub const REQUEST_TIMEOUT_SECS: u64 = 180;
which is 3 minutes. My guess is two of these are happening for some reason.

I also explored a similar thing in the topic upload limit of 500 MiB which is due to quic-p2p but I think the particular timeout seen here in baby-fleming is unrelated to that. Just some ideas from previous research, might be related might not.

11 Likes

Now I get a failure at only 800kB…
Saved the logs, I’ll send them in the morning if it might help @bochaco @StephenC
well past kip time … TTFN

12 Likes

I ran this same 3MB upload on a local 8 vault baby-fleming network, but with a much more powerful cpu and 64 GB ram.

Client timed out as expected after 6m0.148s

Vaults ran at 100% cpu for over an hour, consumed all the memory, no crashed vaults but something is obviously very wrong there… I decided to kill the vaults in the end.

Just ran it as per the basic instructions, will do some deeper digging when time permits.

12 Likes

I’m on Windows 10. I started cmd in admin mode, did the install, started the daemon without problems.

> safe auth status

Sending request to authd to obtain a status report…
±-----------------------------------------±------+
| SAFE Authenticator status | |
±-----------------------------------------±------+
| Authenticator daemon version | 0.0.5 |
±-----------------------------------------±------+
| Logged in to a SAFE account? | No |
±-----------------------------------------±------+
| Number of pending authorisation requests | 0 |
±-----------------------------------------±------+
| Number of notifications subscribers | 0 |
±-----------------------------------------±------+

But when I create an accout with test coins, it times out:

> safe auth create-acc --test-coins
Passphrase:
Password:
Creating a SafeKey with test-coins...
[2020-03-06T10:29:32Z ERROR safe] safe-cli error: [Error] NetDataError - Failed to allocate test coins: Request has timed out - CoreError::RequestTimeout

Also when trying to restart, using:

> safe auth restart
Stopping SAFE Authenticator service (safe-authd)...
[2020-03-06T10:31:33Z ERROR safe] safe-cli error: [Error] AuthdError - Failed when invoking safe-authd executable from 'C:\Users\Joost\.safe\authd\safe-authd.exe'

Probably missing something obvious here, but what?

8 Likes

Hope you got some sleep @Southside!
Do send me over the logs for your 800kb failure if you can. While we expect it will timeout with larger files, this is quite small. Wouldn’t mind having a peek to see if anything unusual happened in this instance.

There are other factors at play with timeouts as well, for example, how busy the vaults were at the time of the put, how quickly the put was initiated after baby fleming was created (i.e. were the vaults still bootstrapping to each other), and several other factors.

We are aware of the timeouts you guys are all seeing, and there are other issues as well, hence why we are calling it a baby. You can see what direction it’s going in :rocket:

We have so many improvements in the pipeline, but we know you all want a chance to have a play at the earliest opportunity so get stuck in! We will iterate at every opportunity and always keep the community in the loop.

21 Likes

Hi @Joop77!
Hmm I’m not a Windows user myself so I’ll grab a VM & give this a try from scratch as you have and report back

4 Likes

Managed to get this running on a Windows 10 VM. Should have gave the VM more resources, it was close to grinding to a halt when all 8 vaults were running.

I followed the steps as laid out and didn’t have any issues.

Some things to look out for just in case you missed them first time around:

  • If you have ever run the CLI & authd on your machine in the past then you will need to explicitly kill the authd service before the new authd service can take over. This can be done with $ ./safe auth stop. Once you have then launched the Baby Fleming vaults you can then run $ ./safe auth start.
    (I’m not convinced this is your issue as I see your auth status showing v0.0.5, but worth trying anyway).
  • Ensure the console window that you run all commands in has administrator privileges.

Assuming neither of the above are the issue then the error you get when restarting auth is our best lead:

  • Does the safe_auth.exe file exist where it should be, i.e. C:\Users\Joost\.safe\authd\safe-authd.exe
  • There should also be a logs folder located at C:\Users\Joost\.safe\authd\logs so it would be useful to have a look in the .err and .out files to see what is reported there. If you want me to have a look then feel free to obfuscate any IPs, etc in them and post here, or you can send them to me in a PM. (Note I’ll be leaving for the day shortly).
6 Likes

Same here. I was able to upload folders with files of a few hundred KB. But as soon as I try with a few MB. the system starts using huge amounts of resources.

7 Likes

Logs are here

4 Likes

My auth version is also 0.0.5 I thought this was the latest?

image

I will tidy up the wee bash script I was using to ensure everything got shutdown and re-initialised between failures and post it later. Be nice if this could get checked by others.
Conscious I dont want to put too much work into this as it will be a swiftly moving target.

Can someone point me at the format for horribly insecure passphrase/password config files or remind me of the env variables. I cant see them in the latest version of the CLI Users Guide

4 Likes

0.0.5 is indeed the latest version - what I meant was that Joop77’s version was showing as the latest so I don’t think they would have been running the old authd (but still worth them killing all authd processes/services)

Out on the move at the moment so I’ll have a look at those logs over the weekend :+1:

6 Likes

I checked all your points above. I do not have any previous versions.
But there is also no folder called \logs, perhaps there still is some kind of privilege issue? I am sure I run the CLI in admin mode.

5 Likes

@Joop77
The logs are in each of the vault subdirs
In linux its like this
/baby-fleming/safe-vault-genesis/safe-vault.log
/baby-fleming/safe-vault-2safe-vault.log


/baby-fleming/safe-vault-7/safe-vault.log

Pretty sure it will be similar on Windows

3 Likes

I’m noticing a random failure to create an account on about 50% of my attempts. I think someone else noticed this as well.

I am running this script to save previous logs, clean up old versions and start a new set of vaults.
Note this script is likely to change frequently over the next few days, check for the latest version.

Please help me improve this script .

7 Likes

I’m also hitting the timeout issue on Windows 10. First, a sub-1MB text file uploads OK. Next a 1 MB png image uploads OK, but CPU usage ( i7-4720) hits 100%. A subsequent 1 MB png file times out but the CPU keeps running at 100% til I kill the vaults. I notice that Vaults 2 to 7 contain immutable chunks but vault 8’s immutable chunk store is empty. Wild guess, but could the vaults be trying to share the chunks with vault 8 but failing in some way?

More (possible clues). It looks like vault 8 only ever makes it to Adult stage while vault 2 makes it to Elder. The routing table size climbs to 7 but then drops off to 6, 3, 2 back up to 7 again … (see logs).

Logs from vaults 2 and 8.

12 Likes

Guys, i am no tech-savy person, but want to tell you that i find it so amazing to see so many incredibly driven community members helping filter out bugs in the network. The drive is just amazing!! Thank you so much :blush:

26 Likes

Hi @Joop77, I checked my Windows 10 VM and there’s no \logs folder in my .safe\authd\ directory either, in fact I ran a search on my VM and got no results back for safe-authd.out - this appears to be a bug as the authenticator logs should be stored there, and they are successfully stored in that location on macOS.
I’ve logged an issue for the lack of authd logs on Windows here.

Note that the logs that Southside refers to are vault logs, but your issue sounds related to the authenticator, hence why I was looking for those specific logs.

So this leaves me a little bit in the dark over what the issue you are seeing is. I assume the safe-authd.exe file exists in the location that it should and it’s only the log files that are missing from that directory. I’ll have a think and run this by the team next week and see if they have any suggestions.

12 Likes

@StephenC
Couple of quick questions - yes I know its the weekend…

  1. How can I increase logging verbosity in ~/.safe/authd/logs?
  2. What is the format for the password config file used with the -c flag for login?
  3. Alternatively, ISTR we could also set env variables for password and passphrase, can you detail these. I was sure this was in the CLI users guice but Im not seeing it in the latest version

I want to try and improve my wee testing batch file . I start a 12 on 12 off contract for a couple of weeks on Monday and I wont be able to get too much done with SAFE if I am to get a sensible amount of sleep, so Id hope to get tore in to the max over the weekend.

6 Likes