[Offline] A pre-christmas playground present

anyone still managing anything ?

im trying a put and a cat and its looking like its hanging :frowning:

5 Likes

Damn I take 2 mins out from charades to catch up and you broke it already…

6 Likes

45 nodes, vcpu4-8gb droplets.


I’m seeing some node full errors! That went faster than I’d anticipated

10 Likes

This seems to be stuck for me too now.

5 Likes

yeap it seems down

safe cat safe://hygoygyybjxqmgxq83kq8me8fei6ohfg8q4ph8tg15h999rsais1as88871eo > racer.jpg
^[[A^[[B^[[D^[[CError:
   0: NetDataError: Failed to GET Blob: NoResponse

Location:
   /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1914

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
6 Likes

Dead?

Was about to give it a go :smiley:

2 Likes

it was a nice present anyhow!!! :love_you_gesture:

7 Likes

:thinking: someone didn’t read the OP?

Nice to see, hope it was useful.

3 Likes

Aye we seem to have flooded it. Definitely I did not expect that to go so fast. But seeing:

"Avg storage usage among Adults is between 70-80 %" logged (and they’re considered full at 90%.

I’ll trawl the logs over the weekend to see if that’s a valid number or the storage calc is off.

It also doesn’t make sense that GET should fail either.(edit: as the network fills I mean) So something to look at there.

Hopefully we’ll be back with another one for you soon enough!

24 Likes

Fair play.

From sounds of it, it was super fast until it got full :ok_hand:

6 Likes

I observed some “lag” in this testnet, on older testnets it would be instant, not in this testnet!

4 Likes

image

@Southside trying to safe cat when he gets home later

15 Likes

Oof, blink and you miss it. Or, in my case, slumber in a chair with a book on your lap, because of a hard week and a large glass of mulled wine, and you miss it. Check the logs, dot the i s and cross the t s. Awesome effort Safe and community.

4 Likes

This is what a call a flash net, it came and went so fast I couldn’t even turn on my PC to give it a try. MerryX

5 Likes

I suspect we need to consider folk will flood it as soon as they can. So we probably do need farming and DBC transfers in place. Otherwise folk will upload the world as soon as we make it available, which is probably a good test.

28 Likes

In the end of course, but meanwhile bigger nodes and expanding network should give longer uptime for sure.

4 Likes

Absolutely, also though flooding should only fill it and definitely not have it lose data. So we need to kill that bug. I suspect the fill flag is stopping data relocating. Paging @yogesh on this one

14 Likes

Good work all! Too late to the party, but I see that there are some interesting results, and that’s what counts.

Waiting eagerly for a next testnet. Hope I will not be too late for the next one…

7 Likes

An observation from combing the testnet logs: There seems to be an issue with nodes trying to join the network. The network, once it notices that it reaches an X%(configurable) of avg storage capacity, it is designed to start accepting new nodes as demand for resources increases. This consequently triggers data relocation within the section and its Adults rearrange the data they hold. Moreover, we do have the always-joinable flag set to true in this testnet, meaning the network should take new nodes no matter what capacity it is at.

Since nodes are not able to join, which mainly seems to be due to connection timeouts at first glance, we start snowballing downhill and the network does not take any new data nor does it relocate the already present data. And if Adults that are present, leave at this point, they leave with the data resulting in a data loss and coupling that with connection timeouts during GETs makes data ungettable. So seems like we have some communication shenanigans when dealing with high throughput amongst nodes here(which we haven’t seen to this extent when testing internally, so thanks to the community for showing its strength :slight_smile: )

26 Likes

We should though relocate the data to the remaining Adults at least. Something to look into.

12 Likes