[MemDebugNet] [4/10/23 Testnet] [Offline]

It does. It seems the download is where we’re having problems.

I’m still not sure what’s up. Seems like a successful download is fine, and can be repeated. BUt if it fails, something is going on, and we end up pulling in more data that there should be (presumably the prior failed runs).

We’ll be digging in there today. We have a failure case at least!

10 Likes

Okay, the issue is somewhat simpler it seems. I have successfully dlded the video.

A key part: After a failed attempt (or even successful one), erase the existing download. It seems like we’re appending to the content there. (I’m not sure why only after failure. If a file exists we can re dl fine w/o corruption :thinking: )

Anyway, getting closer to the main issues there!

edit:

It’s because we’re now Streaming decryption. And so appending new unencrypted parts to a file (improves client mem massively for large files; makes them feasible). But clearly we are a) not cleaning up after failed dlds and b) not removing a file before starting!

14 Likes

I removed the dl from “client” folder and tried again got.

Client (read all) download progress 33/433
Error downloading "Ourwealth.mp4": Chunks error Chunk could not be retrieved from the network: 6be040(01101011)...

Removed dl with error above again and tried again.

It works, dl was successfull and I could play it back on VLC

Thx for the feedback

9 Likes

Trying a new approach to my 1.5GB ubuntu.iso batch-size 1 taking a little while but I wonder if that may be a good thing later.

⠲ [15:07:26] [#############################>----------] 2098/2838
Made it to 2500 with previous batched uploads so nothing to indicate the result will be any different this time, yet.

Here is a odd thing, with this attempt my cash_notes dir at this time is only 39.4MB each cash_note is only 3.1kB.
This is hugely different to other attempts where my cash_notes dir would be multi GB by now. ping @joshuef

I’ll rerun after this without batch-size to confirm but I am 99% sure something sketchy is going on with cash_notes.

8 Likes

Just wanted to say that it’s delightful to see how people find different angles to test things out. :vulcan_salute:

And I still can’t get over:

:joy:

5 Likes

Confirmed, significant difference in the size of the cash_note dir when using batch-size 1 compared to all other attempts.

  • safe files upload 10MB.txt

Uploaded 21 chunks in 1 minutes 24 seconds


du -sh cash_notes 2.5M cash_notes

find cash_notes -type f | wc -l 108


  • safe files upload --batch-size 5 10MB.txt

Uploaded 21 chunks in 2 minutes 3 seconds


du -sh cash_notes 1.2M cash_notes

find cash_notes -type f | wc -l 111


  • safe files upload --batch-size 21 10MB.txt

Uploaded 21 chunks in 50 seconds


du -sh cash_notes 3.0M cash_notes

find cash_notes -type f | wc -l 107


  • safe files upload --batch-size 1 10MB.txt

Uploaded 21 chunks in 7 minutes 47 seconds


du -sh cash_notes 528K cash_notes

find cash_notes -type f | wc -l 127

6 Likes

The batch size is effectively how often we pay for the current chunk batch. ie, pay for 5 chunks and then upload them, or pay for 10000 chunks and then upload them.

So a size > the number of chunks will do one payment eg.

You’ll effectively have less change dbcs, really.

7 Likes

If I understand correctly a 21 chunk file uploaded as batch-size 21 should be a single payment whereas a 21 chunk file of batch-size 1 is 21 payments right?

But 21 payments results in a smaller dir.

Is that what you are saying? Sorry if I am slow on the uptake here.

Paying for more chunks upfront means more change?

5 Likes

exactly.

This may be due to the weight of the transaction… im not sure. :thinking:

There will be 21* the recipients in that one tx. I’m really not sure how that would all stack up w/ encryption etc around there. (cc @Anselme maybe you can shed some light here?)


In other news, a fix should be in for the @19eddyjohn75 issue w/ downloads. The next client build should clear the file before redownloading.

Aaaaalso @happybeing I have reproed your hang! No idea what’s going on yet, but at least now i can dig in!

13 Likes

@happybeing I suspect this pr will fix things for you (i know I’ve said that before :stuck_out_tongue: ; one of these times I’ll be right though! )

Basically i think you have some failing queries. And the timeout is set at a mad 5mins just now. If you’re uploading a lot, there’s a lot of GetStoreCost etc going on, and if anyone of those hung, well, you’d be waiting 5 mins. A lot of chunks, and batches, a lot more of that could go on…

Let’s see how you go when that’s in :crossed_fingers:

10 Likes

saw the name and thought of you @Southside

safe files download AnarchyInTheSouthside.mp3 eaa0b39813183323b491e6715a0b4ea9f3cfdeede6f04a73c5b849b5210ad20d
1 Like

For now it must remain a mystery :frowning:

🔗 Connected to the Network                                                                                                                              Downloading AnarchyInTheSouthside.mp3 from eaa0b39813183323b491e6715a0b4ea9f3cfdeede6f04a73c5b849b5210ad20d
Error downloading "AnarchyInTheSouthside.mp3": Chunks error Chunk could not be retrieved from the network: 85f32a(10000101)...

Chunks error Chunk could not be retrieved from the network: 85f32a(10000101)...

is a consistent error - 4 times now

1 Like

you using the latest client ?

Client (read all) download progress 128/132
Client (read all) download progress 129/132
Client (read all) download progress 130/132
Client (read all) download progress 131/132
Client (read all) download progress 132/132
Client downloaded file in 20.062829581s
Saved AnarchyInTheSouthseide.mp3 at /home/mccnex/.local/share/safe/client/AnarchyInTheSouthseide.mp3

2 Likes

So difficult to keep up these days…

willie@gagarin:~/projects/maidsafe/safe_network$ safe -V
sn_cli 0.83.34

Thats the latest, I think?

3 Likes

yea thats what im on im just trying to download it here

1 Like

ok im getting that as well but i shared it with a non forum member who is having a play right now and he managed to download it first time

🔗 Connected to the Network                                                                                                                                                                        Downloading AnarchyInTheSouthside.mp3 from eaa0b39813183323b491e6715a0b4ea9f3cfdeede6f04a73c5b849b5210ad20d
Client (read all) download progress 1/132
Client (read all) download progress 2/132
Error downloading "AnarchyInTheSouthside.mp3": Chunks error Chunk could not be retrieved from the network: f026db(11110000)...

1 Like

Piece of pish when I try from my cloud instance

Client (read all) download progress 131/132
Client (read all) download progress 132/132
Client downloaded file in 5.229893955s
Saved AnarchyInTheSouthside.mp3 at /home/safe/.local/share/safe/client/AnarchyInTheSouthside.mp3

Yes its been safeup’d to 0.83.34 as well.

It was nice of @Josh to send me that miniPC/Wyse terminal. Think I will start from scratch with that for all SAFE stuffs from home and just leave this big box for footering and flight sims…

3 Likes

Crapped out on a different chunk whereas mine b0rked at the same chunk consistently 85f32a(10000101)...

Dunno if there could be any significance in that… @joshuef - do you want client logs?

2 Likes

just tried again and same chunk f026db(11110000) which is diferent from your missing chunk

Downloading AnarchyInTheSouthside.mp3 from eaa0b39813183323b491e6715a0b4ea9f3cfdeede6f04a73c5b849b5210ad20d
Client (read all) download progress 1/132
Client (read all) download progress 2/132
Error downloading "AnarchyInTheSouthside.mp3": Chunks error Chunk could not be retrieved from the network: f026db(11110000)...

1 Like

I downloaded it fine on first attempt in 45 seconds. Surprising as this machine is still on 0.83.31 and I hear much talk of the benefits of new clients.

2 Likes