[Offline] 50GB Static Network - Round 3

Development Updates
1

A new testnet: 50gbRound3 :bell: :bell: :bell:

Here we have round three of the 50GB/25-node testnet. (You cannot join this network with a node just yet.)

We’re looking to verify the changes applied since the prior 50GB testnet:

  • some infrequent comms situations appeared to be causing mem spikes
  • node storage level messages could potentially have caused deadlocks
  • elders now store data
  • nodes have min/max storage (50/100GB), so we should likely not see storage errors even as we exceed 50GB stored on some nodes :crossed_fingers:

Getting involved

As noted above, we’re not looking to check out node joins just now.

As such, it’s only safe client and PUT/GET data that we’ll be testing out. The CLI is limited to files of less than 10MB.

To get involved you can follow these instructions to set up the safe cli. The testnet-name is 50gbRound3, and the recommended safe version is 0.68.4.

What’s helpful and reporting issues

Right now, keep uploads small. < 10MB / file.
There is a temporary limit added, and you will get an error if surpassing 10MB.

If you are consistently seeing issues PUTting data or retrieving data you have PUT, please run your command with RUST_LOG=sn_client prefixed to it (on Linux/Mac at least). The output there, and MsgIds that have been sent/failed will be key to debugging.

We’ll try to report back stored data sizes at nodes as we go, so we can see if there’s a correlation between capacity and reliability.

Please note, nodes are not evenly distributed in XorName space, so with a limited crop, we will not see data uniformly spread across them.

This is what we’re using to verify data-storage just now.

safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr 
safe://hygoygym19bdbzg6jakfzkmacxurpkbuphgsxi6af3aecugfqjfaq66shww
safe://hygoygyq1xiofs8kx11jwkcguzuocd1xa5qp767bjjz39yrh44eo1mjkjch
safe://hygoygypnrqirzo7r5w464mnda8s6sdpatyrco5pda1w8nh5appa3ycb98h
safe://hygoygykajkeropyam6yihj9exdhn5o4b1pbjbeet4bazqo8m47ga3pw8mo 
safe://hygoygyp8c1wy7ijrpnesz9cj7enf4ms9oxhmp5m4r4yw4pjcemgywekf9r
safe://hygoygye13iznm1yxzpxp9seh6j8weh84c5ib3fq9mxozz4h6hb95o3uduo
safe://hygoygycc11b4pmrcb97ro6ro3wxh3cfrgi7ctbiwyya8mkktq1j35ysmzh
safe://hygoygykzkguckqygm985f5jrhcw87wiskbc97bottab47tq6fm9zm85f7r
safe://hygoygyewnaj8o7za3qycckfy95o5j9gex6s3b8jygr9j48aiyzthc36ffo 
safe://hygoygykime3s7rup1ezeiqeqo1zy1o4bqmckewhriun889c64n4umxcyih
safe://hygoygyc98c1ozh1cn9edxeworctuagt465zyybr61m8judfry4a4zycf8c
safe://hygoygyqcq8bift7urnftiqxzn7d877ckwtk85haawg7oigm3bhn3r1dx7e 
safe://hygoygyk7dzmcqxm3q4np87unu6tb9d8qt44fowxgqrhjy1xs6xgt69hade 
safe://hygoygymq98z86j1oagpctmg7pnixz94ckekpgytu1jkb45du8xw49qrr4w 
safe://hygoygypoq7y31uedr7c6q6e3jaxhd94id3r8bwn7od3f79hgdj7xbqibjr 
safe://hygoygykw8iw7wt9o3c7w36hi5xc1coyjdmdfm6dima3y9ehefxipxkwg3w 
safe://hygoygyc67ozi6m9i9o8xq5wti7ysa4g6nodxetmdgw64g5pw1qmucheeac 
safe://hygoygyqsxzn4p6mjx7ggj98pqc4he53xk1c54kw3hq1967n4yga1nbo3iw 
safe://hygoygyxojetcr5553xfyjnfoh3ywge7grs8d7z66e8nf678stecagdkomw 
safe://hygoygyqf4au3yf7ouyi7imu1hty5qrsjppy1mj5uba5tahnxkmocys5g7h 
safe://hygoygyx41a9samc35ep6h7y6zwku5ess3377f8awtuwn44r1pc5w3ragky 
safe://hygoygyej97u6irwmgq1od4s8m6h8epbkgmmwqtghhhy1hbcsntizpjmh3c 
safe://hygoygypn9urw4mhoqfabnke71e8usekjeg8m54t1fkoeigdyr5srgsqoia
22 Likes
2

FilesContainer created at: "safe://hyryyry1eu6seqjrboen3per3bestenb1mhwpnuzbcjpj1a7k3fu1tpk94ynra?v=hahocsaw9zffwykgc7qe1exux54hj4qqfjtk8zpft1pz4ohi7nuoo"

first test jpg

image
image560×794 74.2 KB

last test jpg

image
image560×794 78.1 KB

10 Likes
3

Cock and bull story


FilesContainer created at: "safe://hyryyryixbdtza34nch7mxhs91yq335q9ntaeeq1e9dyneo3p5wxb5tc9fanra?v=h4nk3kna4bjgbkyyticu4cbd4zndhr8taowbj3wyjwqb7orsc6qto"
+---+----------------------+--------------------------------------------------------------------+
| + | /home/willie/coo.jpg | safe://hygoygyxgwmpqj65ku8upyisggtkhbkzbq31au1q5aqwd1q415f4nbq4tqe |
+---+----------------------+--------------------------------------------------------------------+

FilesContainer created at: "safe://hyryyry1byr37tzhfrpxdka19uch3jscw9ekg75i7ybmmd163hkcbk8mn7hnra?v=h84dxmj1dg9mw7ndxugc1m5nfs4ofj6eb45cg5c35qxuzabadmcdo"
+---+-------------------------+--------------------------------------------------------------------+
| + | /home/willie/cock01.jpg | safe://hygoygyqzr1t99g38e5orf6o6ts4cja8ck7wo1hnn1rqcte4b19shnpbqeh |
+---+-------------------------+--------------------------------------------------------------------+
6 Likes
4

uploading now. :smiley:

got one of these:

<ClientError: Timeout after 90s when awaiting command ACK from Elders for data address fdd5e9..>

That’s the only err message so far – all the rest of attempts to upload have gone though without any error.

7 Likes
5

Everything working smoothly so far.
Might have to learn rust now…

Screenshot_20230106_213131
Screenshot_20230106_2131311080×1760 166 KB

Rocker’s up. Container at:

safe://hyryyryz83hnpdmyg6xmwj8xa67e98wwudietz8fmfu31t6d8fnthzmnz9onra?v=hzkuhkttp9cehnctm8bbhkuuaqssgwsg9k5u84e8s7kw8jjar95zy

:tada:

12 Likes
6

Currently:

node-20: 3.3G        total     
node-24: 3.2G   total
node-18: 3.2G   total
node-16: 3.3G   total
node-11: 3.2G   total
node-13: 3.2G   total
node-7: 4.0G    total
node-1: 3.1G    total
node-17: 4.0G   total
node-23: 4.9G   total
node-12: 3.1G   total
node-9: 4.7G    total
node-15: 3.2G   total
node-14: 4.0G   total
node-22: 4.8G   total
node-3: 3.2G    total
node-21: 4.8G   total
node-5: 4.8G    total
node-25: 6.0G   total
node-2: 3.2G    total
node-6: 4.0G    total
node-4: 4.9G    total
node-19: 6.0G   total
node-10: 5.6G   total
node-8: 4.8G    total
12 Likes
7

Sensational Alex Harvey Band
FilesContainer created at: "safe://hyryyryuz738x4iwoa71k7ihhrd5bwo1i47f93egw7ew86oh1feqfg4r55onra?v=h4dozhoyjadtske95y1ih4ckychbw6fzs7dcbb3fkgegmrirme53o"

lots of this missing cos tracks are >10MB

Got few timeout errors as well


|---+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------|
| E | SAHB/05 - Alex Harvey - Swampsnake.mp3                                      | <ClientError: Too large file upload attempted (11642880 bytes), at most 10485760 bytes allowed currently. Try storing a smaller file.> |
|---+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------|
| E | SAHB/06 - Alex Harvey - Next.mp3                                            | <ClientError: Timeout after 90s when awaiting command ACK from Elders for data address 57dec7..>                                       |
|---+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------|
| E | SAHB/06 - Alex Harvey - Penicillin blues.mp3                                | <ClientError: Timeout after 90s when awaiting command ACK from Elders for data address 03ae76..>                                       |
|---+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------|
4 Likes
8

Are these timeout errors normal/expected?
They didn’t seem to happen, at least not to this extent, in the previous 50gb testnet

2 Likes
9

A little Lama Drama
safe cat safe://hy8ayqyersgnyyouyc6ap84589z56rketetjx5tn66h5qqtnzamph8sywyw > lamadrama.mp4

3 Likes
10

I’m uploading ~1000 unique files of 10Mb at the moment (10Gb).

Is there a way to see the XOR file link where my files are uploaded? I ran with RUST_LOG=sn_client

5 Likes
12

1.2GB of audio uploaded in 1h 24 mins, no errors

safe://hyryyrywxif6mnohj5157mx4axt4a5eyp7nh8h4rpqxy79n1ki73girrrtenra?v=h5wia71474nj3xcapn1s489iu9hjd1ddq6bgwdxqj1178bzq7feky

11 Likes
13

And 2GB of images here, no errors as well !

FilesContainer created at: "safe://hyryyrysp9d7ss17u9t3aqsnggh8pntbypyrfxqecg3nqktp6f65i41zqpenra?v=hq8gfk7b7za4piemth6xzp1fm5ww631trka3px97n4o5gqqgk57so"
+---+--------------------------------------------------------+--------------------------------------------------------------------+
| + | ./sn_photos_test                                       |                                                                    |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/4a7dc4b6.jpg                          | safe://hygoygyjzhim9h151k47qph1jz883o8oczyru6zrfmh9rqzxxhe7tqrso7h |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/SAM_0551.JPG                          | safe://hygoygyk7ikqkd3szk84fq3dzgyb1n8weghesskhgshmuuws98pmihjeyyr |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/ce92d553.jpg                          | safe://hygoygymgdmxhcu7xq5e84occggeutqdrpuhwdqu8ieipkcrs8apj1bqx6h |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/f1f2ebf3.jpg                          | safe://hygoygyczsbgkeizyj4x8nhnig96gykor8yjmb4qufno3wgdnfofamypdxh |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/photo-1415650328328-1909c4ceabdb.jpg  | safe://hygoygykocf8kye3jme6zcazktyse33bzpt7umfnpk3z4eew8se3od5k1uw |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/photo-1416312189147-9a686b7fb250.jpg  | safe://hygoygyqm1tqkmqtwkem7fdbjm69fau181pfgpd4u8u54t6ghtsrumntpne |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/photo-1417722009592-65fa261f5632.jpg  | safe://hygoygycdtz5uunetrry5n57w5rfn55hdjyrkcdai47yg7wwj7oep3amsxo |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/photo-1420553850395-dccec107a66d.jpg  | safe://hygoygymmbh7eph97j7a9irdr6ki997tc939cw8kw3e7je6bbhrjqk9ft1a |
|---+--------------------------------------------------------+--------------------------------------------------------------------|
| + | ./sn_photos_test/photo-1422004707501-e8dad229e17a.jpg  | safe://hygoygyq3yt5j1ndfnq4fi5nrhh7aire9k6bm9tuyjim9hbdejmsgq945oy |
...

Now let’s try another 16go of images !! I prepared that for last 50GB testnet but it was ready after the party :smiling_face_with_tear:

I’ll make quick instructions later today for anyone wanting to have another big batch of images as well.

11 Likes
14

Uploading!

When does the test end? What is the hoped for “does not work anymore” scenario?

4 Likes
15

1.7Gb of the Stones at: "safe://hyryyryosrzt8i4oztgxw9tm96mitdc6dggb1aogrhodw43p9u55hdyjddwnra?v=hpbojh3pedukyn615ja9fbjysbnwk79nddcww7k8dmhw5sescg4fo"

8 Likes
16

I have 0.68.3 where to download latest? thx

2 Likes
17

Have you followed the instructions here? Setting up Safe for PUT/GET

5 Likes
18

Some music:

FilesContainer created at: "safe://hyryyryzj5m7wksyboppcfu7ugyn3mmzyy8ewr99kf1nd81fw7e14bc71ocnra?v=hmj3mshhjz49rfawk6pjtkf4ux4c6ndpbjsam9dh4dmicsfjmumpy"
2 Likes
19

They could be indicative of elders under load perhaps. What exactly are you seeing?

Loads of errors are not expected.

I’m hoping that we reach 50gb on a few nodes and we’ll call that the end. We wont reach it on them all, due to xorname space fun. But we should also not see “could not store” errors, as we have a storage margin on the nodes too, now.

FYIs: Metric beat is looking fine so far, and I can dl all test-data :+1:

node-16: 13G total                                                                  ─╯
node-13: 13G    total
node-11: 13G    total
node-20: 13G    total
node-12: 12G    total
node-15: 13G    total
node-23: 19G    total
node-18: 13G    total
node-17: 16G    total
node-24: 13G    total
node-3: 13G     total
node-14: 16G    total
node-9: 19G     total
node-10: 22G    total
node-22: 19G    total
node-7: 16G     total
node-2: 13G     total
node-6: 16G     total
node-25: 23G    total
node-5: 19G     total
node-21: 19G    total
node-19: 23G    total
node-4: 19G     total
node-1: 12G     total
node-8: 19G     total
13 Likes
20

I’m getting these timeout errs about 40% of the time. Didn’t at first, but happening more now. I suspect because more people are uploading now as compared to when I started so nodes are getting overloaded?

<ClientError: Timeout after 90s when awaiting command ACK from Elders for data address 18f208..>
6 Likes
21

Are you seeing those on upload? I haven’t seen them at all.

7 Likes