[MemDebugNet] [4/10/23 Testnet] [Offline]

Didnt know you got bamboo in .fl
I thought the alligators and burmese pythons would have eat it all.
I now imagine you as some machete wielding superstar hacking his way through the sup-tropical wastelands of Florida suburbia.
I know - I have been reading too much Carl Hiaasen

3 Likes

Faucet works from time to time and most uploads fail. Only some relatively small files manage to survive.

The only thing that hasn’t given me errors are transfers.

4 Likes

I’m getting this error now:

Uploaded 80 chunks in 107.896549102s. Current progress is 80/4058.
Getting upload costs from network for 80 chunks...
Cannot get store cost for NetworkAddress::ChunkAddress(5777c0(01010111).. -  - 5777c03309e321ad1e1198e02894120180b2b187fe0eab9b4101c4ec30a39515) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Cannot get store cost for NetworkAddress::ChunkAddress(ec2462(11101100).. -  - ec246299845176311e66dec8f9326e60dbf9a3ca61045d8f9626ef137ec6413c) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Cannot get store cost for NetworkAddress::ChunkAddress(04b482(00000100).. -  - 04b482824f76be6acf6a9c7ec03229c7b5be4fc78553098a15b9b04897054745) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")

4 Likes

Just watch those fingers and toes :see_no_evil:

1 Like

I’m not sure if it is the cause but don’t think you’ll get useful output building for release as the debugger symbols won’t be there to point you at the code. I’m not sure what heaptrack will do in that case though.

3 Likes

@joshuef this looks like the hang error I was getting before:

Connected to the Network                                                                        Total number of chunks to be stored: 53458
Getting upload costs from network for 20 chunks...
Transfers applied locally
All transfers completed in 5.795880271s
Total payment: NanoTokens(2829) nano tokens for 20 chunks
Made payment of 0.000002829 for 20 chunks
Stored wallet with cached payment proofs. New balance: 321.999928923
Uploaded chunk #cdbf8a.. in 0 seconds
Uploaded chunk #c2e240.. in 0 seconds
Uploaded chunk #3e19c0.. in 0 seconds
Uploaded chunk #57cc05.. in 0 seconds
Uploaded chunk #b2b702.. in 0 seconds
Uploaded chunk #6b8aaf.. in 0 seconds
Uploaded chunk #f5e24e.. in 1 seconds
Uploaded chunk #376ae2.. in 0 seconds
Uploaded chunk #d2cb7d.. in 1 seconds
Uploaded chunk #279d49.. in 0 seconds
Uploaded chunk #1fbb39.. in 0 seconds
Uploaded chunk #021204.. in 1 seconds
Uploaded chunk #f19082.. in 0 seconds
Uploaded chunk #59863b.. in 1 seconds
Uploaded chunk #4c735e.. in 0 seconds
Uploaded chunk #f9898d.. in 0 seconds
Uploaded chunk #9e280b.. in 0 seconds
Uploaded chunk #012f55.. in 0 seconds
Uploaded chunk #27f6aa.. in 0 seconds
Uploaded chunk #6bec82.. in 0 seconds
Uploaded 20 chunks in 24.904734518s. Current progress is 20/53451.
Getting upload costs from network for 20 chunks...
Cannot get store cost for NetworkAddress::ChunkAddress(eac886(11101010).. -  - eac88667da749d7a6197349861b7063673c276ca635f6ec6af7a6ca6c9420d8a) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")

safe.zip (44.1 KB)

2 Likes

Downloads are working fine. Just downloaded a 60MB and a 550MB file.

3 Likes

Well, I wish I could of be of more help here in figuring out why your nodes get killed.

My issue was, user error. I did indeed mess the port forwarding up. :roll_eyes:

4 Likes

So CLI v 0.83.21 should work better and mayyy unblock some upload efforts. It will reuse previously generated cashnotes, and or top up. So worst case if you reupload it might get past the chunk you were blocked at (and had generated a cashnote, but note found it to upload).

The errors @digipl and @neik and probably some others have seen is this client side concurrency error. (The ones that were not faucet related).

I’d recommend trying the latest CLI and using a lower concurrency value if you’re hitting issues (try -c 1 as a last ditch effort).

I’m not sure what causes it exactly. There’s a race condition afoot there we need to handle better. But we’ve a read on it so should hopefully get a fix along for that in the nearish future.


@happybeing I’ll have a scan of your logs. Worst case can you try with new client and be sure to prefix SN_LOG=all to your hanging cmds (that version should print the commands run in the logs too, cc @Toivo !)

edit: @happybeing okay, with the extra logs actually it’s clear, one of the storecost requests is not returning at all.

What’s weird is that for most folk , this is the NotEnoughStoreCostsRetrieved error… and we carry on/ retry. But for you it’s hanging awaiting the completion of a request :thinking: .

A step forward at least. But an SN_LOG=all will likely be quite helpful here if you’re able to repro with that set. :bowing_man:

5 Likes

For me too it seems to hang at times and carry on at other times. :man_shrugging:

3 Likes

Oooh, i think we’ve a lock with the semaphore there. Your upload batches are overlapping, claiming semaphore permits but that’s blocking another from finishing (I suspect :male_detective: ). Bit of a guess atm, but I’d wager as that’s the only concurrency limiter we have, (no locks etc), and this looks very deadlocky.

If you’re seeing a hang, have a go with -c 9999999 eg, I think our setup is fine for smaller file,s but much more likely to trip up with more data…

It may be that now the client doesnt chunk everything in mem that we can drop the semaphore entirely. That may well be the cleanest solution.

edit: aha! and with the semaphore removed i’m hitting the Failed to read cash_note file errors :muscle:

10 Likes

Ha, I don’t understand any of that, but I started an upload with the new client with:

time safe files upload -c 9999999 --batch-size 80 ubuntu-18.04.4-desktop-amd64.iso

There’s some Cannot get store cost for NetworkAddress but it doesn’t hang there.

Something else I haven’t at least noticed earlier, so might be normal even though seem weird:

Transfers applied locally
All transfers completed in 5.56931569s
Total payment: NanoTokens(907) nano tokens for 32 chunks
Made payment of 0.000000907 for 80 chunks
Stored wallet with cached payment proofs. New balance: 99.999963287

Total payment for 32… made payment for 80… what?

EDIT:

Ended up failing:

Uploaded 80 chunks in 76.211837582s. Current progress is 1040/4058.
Getting upload costs from network for 80 chunks...
Transfers applied locally
Error: 
   0: Transfer Error Failed to send tokens due to Network Error Could not retrieve the record after storing it: 01b4ceb235f612e6f9b1ba6c9ad197abd6da00886b5ee04092fd8b9cf8460c5b..
   1: Failed to send tokens due to Network Error Could not retrieve the record after storing it: 01b4ceb235f612e6f9b1ba6c9ad197abd6da00886b5ee04092fd8b9cf8460c5b.

4 Likes

Is there any chance the default nodes are being hammered and not responding? I couldn’t download BegBlagandSteal (still never listened to it!) until I used one of my own nodes as a peer. It was just spinnning around at Connecting.

3 Likes

Was the first file I ever managed to upload to a test net so it’s become my test net mascot.

Plus I know the guy did that remix so hopefully he won’t be to pissed over his copyright :joy:

4 Likes
Uploaded 50 chunks in 150.690881264s. Current progress is
3250/5861.
Getting upload costs from network for 50 chunks...
Transfers applied locally
Error:
   0: Transfer Error Failed to send tokens due to Network Error Could not retrieve the record after storing it: 3eb5f33b7f73eb3a1c25f2cee8a16340f30c6c6d8c09d70abceb3aa8214d684c..
   1: Failed to send tokens due to Network Error Could not retrieve the record after storing it: 3eb5f33b7f73eb3a1c25f2cee8a16340f30c6c6d8c09d70abceb3aa8214d684c.

This with cli 0.83.17.
Logs:

Uploaded safe.log to 2a6b1b8374d5d87ddddb939e6e0be3a1156098d7f2133abc6c3e6b5e97c47d3b
Uploaded safe.log to 2e25e6fd42fd2ce960699e4c78fab5c90ace95309f629a0a0de190ee46b8c62d
Uploaded safe.log to 4d883c295daafe88aa3f4f246d0dd775045a9d7ddce3bbe17c6dc548f2c8391c
Uploaded safe.log to 5647130d2409f656ec698f72160755de7bfbe01f123b9a8e0d461e5d22fd948c
Uploaded safe.log to 7b13d92419e593988b6af2537f0af5654503fba2700ca7c0f5051065336127e4
Uploaded safe.log to 832b0e23cba3ebc189aca41271498e987f28800e4c7067475749c62866ae8fe8
Uploaded safe.log.20231004T133816 to 8428832ab20b00d98b7037c3d200932caa5ebf306db54a05b1e4c2d4dcb014bf
Uploaded safe.log to 98fdff90d33128771260b19e1a19649953d4df75f6793471a87c4888a7168913
Uploaded safe.log to a7069609a57a01df6c35e6a1a9b2abe3192e770d7288fd6c7867925e2e837f6f
Uploaded safe.log to af7a87fafd0ee886cf636f55bf86dd886d374d1047d3d4795bbd76955b7688bd
Uploaded safe.log to ee25aad366e2bdab29fd3c5823cdf7c42bfb57365a1b0a7721eaae6a6c3fc83f

I’ll try with the new cli

1 Like

Mm, with client 0.83.22:

Uploaded 20 chunks in 138.348392486s. Current progress is 1620/5861.
Getting upload costs from network for 20 chunks...
Transfers applied locally
Error:
   0: Transfer Error Failed to send tokens due to Network Error Could not
retrieve the record after storing it: b718bc279b4b0012d27fb6d032e88545c071
4a81c83f7ff1cb11157a1253d724..
   1: Failed to send tokens due to Network Error Could not retrieve the re
cord after storing it: b718bc279b4b0012d27fb6d032e88545c0714a81c83f7ff1cb1
1157a1253d724.

Logs:

Uploaded safe.log to 2a974ee892b58b0a359882ebaa307f2d7f50218d8a0e
8158d9564768ed170126
Uploaded safe.log to 492f07f50aabc93a0ea05806c67a1b152099d6fc22a1
387f9351e592e26c3001
2 Likes

Here’s a successful one from laptop:

======= Verification complete: all chunks paid and stored =============
Uploaded all chunks in 8 minutes 40 seconds
Uploaded full_greater_sydney_gtfs_static.zip to 5de9fec83dece6fd6a1219f816793f22b512c0bbbd904c62742011fbb65b278f

Log:

safe.log to 520ba7e87907f26c959cb83374f8a9fd69f9b84e442f055ea2a7a90027728db9
2 Likes

the good news


safe@MemDebugNet-southside01:~$ ls -lh full_greater_sydney_gtfs_static.zip 
-rw-rw-r-- 1 safe safe 238M Oct  5 01:32 full_greater_sydney_gtfs_static.zip
4 Likes

uploads from laptop are going well so far; a few for you guys to check:

PowerToysSetup-0.67.1-x64.exe to de944eba12b471c434dd33aa010f8601e3df2b73e69dd031ed0061a7604ba368
Insta360 Studio 2023_Winx64_4.8.4_build_202307281619.exe to 36ecb47f0ce0d81839b8b58ac0d8bd2d9c17b9cd7ebce70feb26901e58580349
Docker Desktop Installer.exe to 6162a3cecde1f26231a89bbc6aa3947890e0fe945e804d0d9586bed1b15f9075
3 Likes

Faucet not working for me:

Requesting token for wallet address: 82a1e9f43adf4a85392d94f786ebdd4c0e5020745a47eec385a11b73b2e4af308bfa4ea99fb757c33bcaf3e23f925588...
Failed to get tokens from faucet, server responded with: Failed to send tokens: Transfer Error Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Network Error Could not retrieve the record after storing it: 049a552856588f2143ee28b5584cd1b4245419b7b33674a47dd0d5096bb81d10.").

Also my node had stopped last night, NAT → private:

[2023-10-04T19:18:34.577551Z WARN sn_node::api] NAT status is determined to be private!
[2023-10-04T19:18:34.601738Z INFO safenode] Node is stopping in 1s..

@joshuef do you want logs from this?

3 Likes