[MemDebugNet] [4/10/23 Testnet] [Offline]

Something is off with safeup. It does not give you the latest version as a default. I removed everything, run safeup without specifying the version, and the version I got was:

sn_cli 0.83.21

But a while ago @neik already had:

I managed to download without problems:

Saved AnarchyInTheSouthside.mp3 at /home/topi/.local/share/safe/client/AnarchyInTheSouthside.mp3

real	0m18,873s
user	0m30,124s
sys	0m7,538s

2 Likes

That’s not working for me:

Downloading JoshsUbuntu from 6cfa28d385d5af711893744362aaa32e9116aacce06287614163c20e1b5064df
Client (read all) download progress 1/3666
Client (read all) download progress 2/3666
Client (read all) download progress 3/3666
Client (read all) download progress 4/3666
Client (read all) download progress 5/3666
Client (read all) download progress 6/3666
Client (read all) download progress 7/3666
Error downloading "JoshsUbuntu": Chunks error Chunk could not be retrieved from the network: ab2214(10101011).

On a second trial a different chunk failed:

Downloading JoshsUbuntu from 6cfa28d385d5af711893744362aaa32e9116aacce06287614163c20e1b5064df
Client (read all) download progress 1/3666
Error downloading "JoshsUbuntu": Chunks error Chunk could not be retrieved from the network: d8b678(11011000)...

real	6m1,599s
user	9m30,009s
sys	1m21,952s

I was checking my system monitor while downloading, and it seemed to download a whole lot of stuff. At the same time the information in the Terminal seemed to be lagging behind. I was receiving a long time at about 24MB/s, before nothing happened there.

Is it downloading in parallel, in whatever order, but updating the progress in order of chunks from smallest number to largest?

3 Likes

I think you’re hitting a hang there. It’s worth checking logs for any activity,

Yeh kind of. I think we’re only progressing the counter when we have the next chunk necessary at the moment.


edit:

sn_node 0.92.4 has removed the code that was aiming to prevent “lost nodes”. It’s suspected to be involved in the mem leak. If folk fancy restarting a couple of nodes to use that, it could be interesting to see what/if there’s any difference.

(it’s also worth noting, I’m wondering if the new network-contact setup, with many many more contacts than in the older testnets might well help with the ‘lost-node’ issues…)

6 Likes

How are Maidsafes nodes running? Down a lot?

4 Likes

Yes it’s the size of the Transaction. Since each cashnote contains the transaction it was created in:

  • batch 1: means 6 outputs (1*5 payments + 1 change) in the Transaction, meaning small transaction, resulting in smaller cashnotes
  • batch 21: means 105 outputs (21*5 payments + 1 change) in the Transaction, meaning bigger transaction, resulting in big cashnotes

We currently keep a local copy of the cashnotes we created for the nodes as payment. In both cases (any batch size) there will be the same amount of those as we ultimately pay the same nodes. The difference is that instead of having 105 small ones (with batch 1) we will have 105 BIG ones (batch 21)!

One transaction means 1 change output:

  • so batch 21 would mean we add 1 change cashnote to our wallet because all is settled with only one transaction.
  • and batch 1 in means we need 21 transaction resulting in 21 change outputs.

All in one transaction:

  • 105 cashnotes to pay notes each roughly 28K
  • 1 change cashnote for us of roughly 28K
  • 1 origin cashnote that you had before roughly 2k

105 * 28k + 1* 28k + 2k make roughly 3.0M

Divided in 21 transactions:

  • 105 cashnotes to pay notes each roughly 4K
  • 21 change cashnote for us of roughly 4K
  • 1 origin cashnote that you had before roughly 2k

105 * 4k + 21* 4k + 2k make roughly 528K


Now intuitively I would want to end up with a smaller cashnote dir when batching payments! The reason it is not is because we currently keep the cashnotes we created for the node payments for debug purposes. Ultimately those will not be kept anymore as we don’t need them (we don’t own them), resulting in smaller cashnote dir when batching! Which is indeed the intended behaviour!

Hopefully this has helped clear some confusion!

14 Likes

It worked pretty consistently, never hang so to speak I could watch it slowly work through verification. Completed verification after 8 hours.
It was only on the batch-size 1 upload that verification took this long, are they connected somehow?

With batch-size 1 the upload took, 21 hours but was only missing 1 chunk after the check, way less than other attempts, after batch-size 2838 (all chunks) the first check found 1700 missing chunks.

⠓ [08:06:14] [################################>-------] 2323/2838                                                                                                                                                                                           ⠈ Verified 2838 chunks in 35191.643637228s
1 chunks were not stored. Repaying them in batches.
Failed to fetch 1 chunks. Attempting to repay them.
⠤ [00:00:30] [----------------------------------------] 0/2838                                                                                                                                                                                                Repaid and re-uploaded 1 chunks in 30.687921004s
**************************************
*            Verification            *
**************************************
1 chunks to be checked and repaid if required
Verified 1 chunks in 44.79699842s
Verification complete: all chunks paid and stored
**************************************
*          Uploaded Files            *
**************************************
Uploaded ubuntu.iso to 3fb74b0c141f57f2028efee188c26e34d9b29e1b752782129fcfe4e73e5a19d1

I am failing to download it though.

How do I limit concurrency?

Thank you! @Anselme I much better understand now.

Started 8 under 5:x

5 Likes

still struggling to download this file on my home connection even after increasing time out to 120s

ubuntu@safe:~$ safe files download  --timeout 120 AnarchyInTheSouthside.mp3 eaa0b39813183323b491e6715a0b4ea9f3cfdeede6f04a73c5b849b5210ad20d
Logging to directory: "/home/ubuntu/.local/share/safe/client/logs/log_2023-10-10_08-58-14"
Built with git version: 985e1d0 / main / 985e1d0
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network w/peers: ["/ip4/138.68.175.236/tcp/38365/p2p/12D3KooWBk7HqragMtCdmfTYVAy8T9jJuPRbade5BjE3q7sDXuYC", "/ip4/134.122.110.132/tcp/42537/p2p/12D3KooW9x133NoUsif3aXLw33MinDLXMKvdQwL8Fy6FPnKk1U1p", "/ip4/138.68.178.41/tcp/35753/p2p/12D3KooW9t3EoKMrxY98Hd8m92rkKzDguyP7qmNP1Yb5nzKHsUsX", "/ip4/209.97.183.90/tcp/35353/p2p/12D3KooWBJQSD2qrpqdHau6dti5tihkK8NBFAyjeRuHR5aZgZ7EZ", "/ip4/209.97.134.128/tcp/36563/p2p/12D3KooW9uycSAMUxnVMXBFDtvFkmSmZJ2p5vJy6mvyv6986WkVw", "/ip4/165.22.114.243/tcp/43507/p2p/12D3KooWA88RP5cNNxQZrnLHGxU21EYFvfDjVYZkn3aestkqDyze", "/ip4/165.227.226.209/tcp/37861/p2p/12D3KooW9rENjfWvuoHahcQiziWk4B9Yfzw7kBSi6BrBFp4rUhp3", "/ip4/167.99.89.78/tcp/39385/p2p/12D3KooWAUUTizhiGYqFcJksDk7YbWXMra7DhkJormQY5LwstXbV", "/ip4/178.62.5.22/tcp/41709/p2p/12D3KooWA4WvSGtGkFVJT14q7kSss8UZZHm1PMSntuJNE5tSoSut", "/ip4/134.122.110.125/tcp/43885/p2p/12D3KooWCUxxiNK4TU98yiSbZf1pAVTai5qcH9sTvkN2QjHRiTno", "/ip4/138.68.182.225/tcp/34247/p2p/12D3KooWA4YKEECUHGBmAAGsLsuCihdb5mMgkJSfbvt9ysaLASFQ", "/ip4/138.68.190.1/tcp/37523/p2p/12D3KooWADMxubnokBfg1xNwMUwY2Z4yqYjrRjTB6CXkrh48L7Ne", "/ip4/142.93.35.1/tcp/44573/p2p/12D3KooWBLJGVswdvJvvBLU33WCwJdLSxzKmWdPyghLvKaxHt9xN", "/ip4/167.99.89.55/tcp/36549/p2p/12D3KooWAbPRA4MjFnpcdsTEHdiifLctywvanMQgRj91cSgm2eq4", "/ip4/165.227.238.161/tcp/39591/p2p/12D3KooWB5PKufJLJ2Lzm16uBzFHPXK46Qz2RPbmm6K4mTVxeHBF", "/ip4/165.232.98.177/tcp/42309/p2p/12D3KooW9rHbExmGFDu3rEx5Cei57vH5ahftncZXRBCuQjAxhjrv", "/ip4/209.97.128.181/tcp/45271/p2p/12D3KooWAkJybhpGKrpurFN3XkLxKnKc8CLSSJDNMaEPgZrk6nQs", "/ip4/68.183.33.142/tcp/34385/p2p/12D3KooW9qo2TEV6jTrFe8JCTs4CdGwjoZHKVFwpiWmjyQGRxN9a", "/ip4/209.97.143.12/tcp/45085/p2p/12D3KooWCKZyxndJoL7snv1w6q4WkgsDzQB6n3rPFVCCSpcePbVd", "/ip4/161.35.161.196/tcp/44469/p2p/12D3KooW9uZJEnTm2eg6CrMvbepn5RSgkAMU4RQsNzk9nmBbwX55"]...
🔗 Connected to the Network                                                                                                                                                                                                                             Downloading AnarchyInTheSouthside.mp3 from eaa0b39813183323b491e6715a0b4ea9f3cfdeede6f04a73c5b849b5210ad20d
Client (read all) download progress 1/132
Error downloading "AnarchyInTheSouthside.mp3": Chunks error Chunk could not be retrieved from the network: f0a250(11110000)...

log

safe.log to 8a9009e1b5c4ebc2e81fe899ff09bb8648d06b37b199a3c944b8f2e0074bc884
2 Likes

And again arrived here in no time:

Client downloaded file in 59.442593798s
Saved AnarchyInTheSouthside.mp3 at /home/topi/.local/share/safe/client/AnarchyInTheSouthside.mp3

real	1m2,430s
user	0m41,540s
sys	0m12,135s

I’d like to try to upload something, but the faucet is not working. Could someone send me a penny? @neik, remember the guy, who once helped you out, when you were in trouble :disappointed_relieved:

Or are the prices so high that no money can buy any storage?

aa06a51ccae374ee595e2e881834d2056e1d0d025083401a1fcf9085d1a63ef72860911b15dbb74adc6eae2daf89292c

4 Likes

this is prety much all iv got just for you @Toivo

safe wallet receive 5ec1a8603e163021e9fd31c1fcc62834148482994c779df62c77e801824f5fd78fd45a2849408942200e79e1a2ab060832b4e41b424f7f06dd8764fdd5496d793d1c4206a8550269d23c80b866d0e409008107d297179cbfccb6443918363b8e6b37cb97b55e274d458b2f9ae46093deeaf983c830831eaff963b25f0c96212bd37101336589f0477212cfc8975fbae2770c96a1611fdc0a369f86dff09ccc6b967d256fab99df71a30d000edf3af943d9670c247cfd3f324efa58deda82c88d7b00000000000000614e9fd0c316495837b3140a43df6b77291141398701f40f4e3491cbaf1449879bd60b03f5995f44a187d68512eea23eb90000000000000001
3 Likes

Input was split into 185 chunks
Will now attempt to upload them...
Error: 
   0: Transfer Error Transfer error: Not enough balance, 99.991793956 available, 1890599.515990176 required.
5 Likes

Thanks a lot @neik, I knew I could trust you :heartpulse:

So for you it is 1890599 / 185 = 10219 per chunk.
But I uploaded mine with:

Chunking 1 files...
Input was split into 4 chunks
Will now attempt to upload them...
Uploaded 4 chunks in 20 seconds
**************************************
*          Payment Details           *
**************************************
Made payment of 0.000352771 for 4 chunks

EDIT:

OK, larger file is way more expensive:

Chunking 1 files...
Input was split into 115 chunks
Will now attempt to upload them...
Error: 
   0: Transfer Error Transfer error: Not enough balance, 124.999647229 available, 76072388.474115903 required.
5 Likes

im getting

Made payment of 0.000009418 for 1 chunks

restarted my router and increased time out to 500 seconds and still failing on same chunk

3 Likes

Interesting, I tried another file…

Uploaded 4 chunks in 14 seconds
**************************************
*          Payment Details           *
**************************************
Made payment of 0.000026177 for 4 chunks

EDIT: I see, price goes with file size

Input was split into 58 chunks
Will now attempt to upload them...
Uploaded 58 chunks in 1 minutes 14 seconds
**************************************
*          Payment Details           *
**************************************
Made payment of 0.009446603 for 58 chunks

EDIT2: It is not about file size I think. Price varies greatly between nodes and with more chunks, there is bigger probability of hitting really expensive nodes:

Input was split into 4 chunks
Will now attempt to upload them...
Uploaded 4 chunks in 16 seconds
**************************************
*          Payment Details           *
**************************************
Made payment of 0.253467645 for 4 chunks

Btw I thought chunk size is 512 kB, how come 112 kB file was chunked in 4 chunks?

5 Likes

upgraded to the latest client and am now failing on a different chunk

Error downloading "AnarchyInTheSouthside.mp3": Chunks error Chunk could not be retrieved from the network: f0a250(11110000)

edit ------------------

it changed chunk on the last try before I upgraded the client to sn_cli 0.83.36

3 Likes

And my file was only 36,7kB, uploaded earlier, and still pricier than yours.

Me: Made payment of 0.000352771 for 4 chunks 36,7kB
You: Made payment of 0.000026177 for 4 chunks 112kB

EDIT:

Three times the same upload back to back:

Made payment of 0.001811038 for 11 chunks
Made payment of 0.000034194 for 11 chunks
Made payment of 0.001809202 for 11 chunks

File of 16 chunks is super pricey. A steep rise from 11 chunks:

Input was split into 16 chunks
Will now attempt to upload them...
Error: 
   0: Transfer Error Transfer error: Not enough balance, 124.992304823 available, 1890599.508113001 required.

The command safe files download

downloaded my files successfully. But a file that was a 4,8MB jpg before uploading, is now a 28,7MB jpg file. But it opens normally. I uploaded it six times, and so it downloaded six times, so that’s where the number comes. But weird thing anyway.

4 Likes

At the mo you cannot. Also, this statement mayy have been a bit premature. I’ve had good/bad runs with and without this change. (Which was me manually coding in another semaphore).

After that I’ve been poking at more libp2p ways to perhaps tame concurrent up/down, and there’s a couple of PRs that mayyy help here.

A lot of the slow upload situation appears to have been due to the kad timeout though, and us loosing some responses (or libp2p… I think standard packet loss), but basically not allowing us to progress if we had one failed StoreCost check.

sn_cli 083.36 features a reduction here that may help smooth things out.

This is likely just more transfers taking more time (and requiring verification before we progress).

@toivo (i think) asked how our nodes were doing… at ~200mb ram usage, ~50% capacity. And we’ve started to lose some (between 2-5 on most machines are down).

7 Likes

Faucet isn’t working for me either.

Any spare tokens so I can test the upload with the new client? Contributions appreciated to:

8a9c82c95d9e93c29ae7ac6b7f9464f7b39fa8d9f414ff2e8358a6c96e58d5368c117af9d6a4ca4c6bb7010f05d360fc

2 Likes

Here you go, 65:

36aa24138ecc4118233ae12b75a527d489f3b6fe66383c0af522fa1703b4a4e7a86cb869656987b36ff301fd30be570e813bd50d366c0c548e6399e0e47f5836aac003660800f184aee2a12489f0da495b6f3f4c55f8c24b00449abdb94bc9b9cb9f9f98e9680fdf928fc1844fe884b815267019c44a41348ef2eb9ab1e72e3169a51d79d84e0a6ad67ec952be7050fb411ff80cbe1cfc487652175b4bca8b1967b8da3173d7adbf9addd9a2a232c35511326b802798180dd874546ee0c0369eb5fc588c10920000000000000066dd5db138cfc42a8c089e625cc273fdf2a24cfd07239b7999e55c9cce5b53ef6aaebc987f776f36ccc650952eff1287b90000000000000001

But be aware the price gets astronomical somewhere between 11 and 16 chunks - at least for me. It seems the network doesn’t treat everyone as equal.

2 Likes

Not sure if it is a proven fix but i removed safe and reinstalled. Faucet worked after.

4 Likes