[Offline] Update 08 December, 2022: A wild testnet appears!

We’re working on how to deal with full nodes

Important, but also there should be a disincentive for nodes getting too full. Some way to make sure nodes don’t go over something like 50% of their capacity, otherwise, if global disaster strikes and a huge amount of data needs re-spreading, then there might not be enough space. I’d suggest having junk ‘bonus’ data designed specifically to occupy node storage space.

Maybe for every chunk of user data stored, an equally-sized random bonus chunk could be stored on the same node, and presentation of both could be necessary for the full GET reward (if those are still a thing, if not then some equivalent).

If a node is offered a user chunk but is already full (of user chunks and bonus chunks), then the node should tell the others in its section that it is full rather than risk not getting a bonus for another chunk. If the section needs the node to take the chunk (due to catastrophe maybe), then it could exchange two of the node’s bonus chunks for a single bonus chunk applicable to two different user chunks.

Just some improvised ideas.

4 Likes

Just regarding the niggles with running on Windows, the install script actually modifies the PATH variable, so that would have helped there.

Interesting bug you’ve found with the encoding. Highlights the importance of cross-platform testing coverage.

15 Likes

Uploading a 1.5 MB file took 7.673s

Downloading the same file using safe cat took 2.695s

Downloading the same file using safe files get took 3.224s

These numbers look pretty good!

Uploading 8 MB file 10 times gave these results

Min: 9.596 seconds
Avg: 11.277 seconds
Max: 15.203 seconds
Stdev: 1.850

Avg Mbps: 5.930 Mbps

All tests:
10.482
10.473
10.56
10.712
11.031
15.203
9.596
14.166
10.409
10.136

Uploading various sized files indicates the lag on startup (larger files are faster because they can use the existing connection from the early chunks)

size (KB) upload time (seconds) speed (Mbps)
1 2.78 0.00294
10 4.319 0.01896
100 5.059 0.16192
200 5.158 0.31764
300 5.977 0.41117
400 5.57 0.58829
700 6.434 0.89126
1000 6.502 1.25992
2000 7.19 2.27872
3000 8.03 3.06052
5000 9.014 4.54404
8000 10.482 6.25224

22 Likes

That’s my upload been running for nearly 9 hours and so far 1319 files uploaded with no errors.

Haven’t had a chance verify any files yet as I’m on my mobile.

Again anyone wanting to check in can do so with

safe files ls safe://hyryyry1e3i46spukkrh1md7grw5z6ye3yjpz11ksh3669q6zfnhfzp6ydanra
11 Likes

So atm I’m thining nodes want to report storage because they don’t want to get full, as they will then be kicked off for being unable to return data.

ie, they want the network to grow so they can keep earning. Seems simple + clean.


Again anyone wanting to check in can do so with

a subset:

| 8529920 | 1670562028 | 1670562028 | Beyond - Tunes - Tunes - 12 - Nirvana - Smells like teen spirit.mp3                                                                               |
|---------+------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------|
| 4698112 | 1670562083 | 1670562083 | Beyond - Tunes - Tunes - 13 - P.C.D - Dont cha.mp3                                                                                                |
|---------+------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------|
| 3991552 | 1670562118 | 1670562118 | Beyond - Tunes - Tunes - 14 - Robbie - Trippin.mp3                                                                                                |
|---------+------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------|
| 5578752 | 1670562152 | 1670562152 | Big Country - Crossing - 1,000 Stars.mp3                                                                                                          |
|---------+------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------|
| 6004736 | 1670562194 | 1670562194 | Big Country - Crossing - All Fall Together [Jimmy Iovine Remix][Version].mp3                                                                      |
|---------+------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------|
| 5941248 | 1670562236 | 1670562236 | Big Country - Crossing - Angle Park [US Version].mp3                                                                                              |
|---------+------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------|
| 6369280 | 1670562278 | 1670562278 | Big Country - Crossing - Chance.mp3                                                                                                               |
|---------+------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------|
| 18      | 1670528808 | 1670528808 | uploads.txt                                                                                                                                       |
+---------+------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------+


FYIs nodes are currently reporting stored data sizes thus:

node-20: 24K   
node-12: 24K   
node-19: 12K   
node-24: 1.5G  
node-10: 1.7G  
node-6: 1.5G   
node-18: 1.2G  
node-25: 1.5G  
node-23: 1.8G  
node-13: 1.5G  
node-11: 2.9G  
node-22: 1.7G  
node-16: 1.7G  
node-8: 1.7G   
node-17: 2.9G  
node-21: 2.8G  
node-26: 3.4G  
node-5: 1.8G   
15 Likes

Ok, update on this, re-tried to cat the file and it now works fine…not sure what was happening before but please ignore and don’t waste time investigating that…weird stuff

6 Likes

Console is designed to work with text.
Using it for binary data transfer is a hack.
No surprise that it fails in unpredictable ways.

I hope that binary cat will be eventually replaced with more reliable commands.

1 Like

to use code in the forum posts you need to use backticks before and after

`

e.g. code

or triple backticks in the before and after line of code ```

e.g.

code
code

the backtick is under the ESC key on the keyboard
backticks

3 Likes

also here you would need to update your instructions,

to get a file you need to prepend safe cat

so in your example:
safe cat safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr > 1.jpg

there is also another (better IMO) way to download files

by using the container xorurl

safe files get <container-xorurl>

this way you download all data from that container and filenames automaticly too

3 Likes

seems like your upload stoped at the letter b? I imagine you have more mp3s with other first letters too?

so far 1909 mp3 files!

edit: NO! I was wrong! now there are 1961 mp3s in the container! so the upload continues normally but I guess its just a huge collection and it takes time to upload!

I found out how to restart the download with preserving the existing data and downloading only the new!

safe files get -e preserve <container-xorul>

awesome!!!

3 Likes

OK I wont waste any more time on that.
If we get similar reports we can raise an issue if need be.
Or as @Vort says, develop a more suitable tool.

I always have thought of cat as a hacky “internal” tool that would not be directly exposed in the long awaited API.
Just something to use in the early stages for proving to ourselves that it worked. Or didnt…

5 Likes

I think cat would be useful only with downloads as in you click download and it uses cat and you set the filename, but it would need some other tool to get what kind of file it is (extention) like .mp3

Allright, 1.6GB of photos (jpg), 194 photos alltogether put succesfully, not a single error message:

real	46m23,250s
user	5m52,402s
sys	1m51,864s

Thanks once more @Southside for the instructions!

11 Likes

Awesome!
Then you want to keep checking that they are all still there :slight_smile:

10 Likes

Why are some nodes only storing a few K (12K to 24K)?

7 Likes

elders? 20char

4 Likes

Good thought, but it’s only three out of twenty. The rest are storing GB :man_shrugging:

5 Likes

GET ting the whole container:

real	4m23,231s
user	1m0,919s
sys	0m24,324s

That was quick in my opinion. No error messages. I opened a couple images randomly, no problems.

Ok, so there’s some benefit in trying to get them a few times in a day for example?

How much space there is left in the nodes? Should we try to fill them up?

7 Likes

Yes the logs will be interesting. Perhaps some nodes were adults then promoted, but most likely, they started storing while the network was loading, and with no adults they stored the data. Anyway we need to analyse these logs and update everyone with the results.

Then we move on to the next test. We want to push into DBCs and then, after that have to pay for uploads (interesting problem) and so on.

Nodes form home is likely where there are silly comms bugs and that testnet will be interesting, but we are doing a lot of work in coms in parallel so that it may pull together at the right time. At least then all we are testing is comms and nodes from home.

16 Likes

More interesting than making decentralization actually work?

I propose intermediate step:
Nodes are still centralized, but randomly goes offline and online.

6 Likes