[Offline] Fleming Testnet v6.2 Release - Node Support

I will plot 2s, 60s, 180s on one chart once you upload 180s data.

2 Likes

https://pastebin.com/wkRY8zr5

1 Like
Wolfram code
d2 = {15.358, 18.258, 19.246, 22.73, 24.068, 25.281, 27.485, 29.269, 31.432, 33.002};
d60 = {20.765, 23.03, 23.538, 28.403, 28.119, 28.558, 31.97, 33.713, 36.626, 35.514};
d180 = {23.368, 26.523, 27.466, 28.939, 32.91, 34.488, 34.25, 36.231, 37.996, 39.392};
d = {d2, d60, d180};
ListLinePlot[d, PlotRange -> {All, {0, Max[d]}}, PlotLegends -> {"2s", "60s", "180s"}]

uplchart345

Conclusion: slowdown is perfectly reproducing with delays in range 2s…180s.
To understand what causing it may be useful to collect profiling data for first and last run, then compare it.

It is interesting why lines have different offsets.
But I think it will become obvious during problem chasing.

5 Likes

These are very small samples though - Any point in letting it run over night on 50 iterations of 2, 5, 30, 60 and 180 secs delay?

Should I run 10 iterations on 2 sec delay of 10, 50, 100 200 and 500kB? Or something else? If we can get others to run this script we may well see very different results on different hardware.
This is on an elderly 4 core Intel(R) Core™ i5-2500K CPU @ 3.30GHz 16Gb RAM SATA 2Tb disk

1 Like

Tests can be extended in many ways.
But these results are enough for me:
Every upload increases execution time, which not depends on network load (in 2s…180s delays range).

100% reproduction rate and such consistency in results are gift for debugging.
If Rust is supported in advanced profilers, then finding guilty process should not be hard.
I may be wrong, bugs sometimes makes surprises, but it looks like easy thing to track (which tells nothing about complexity of needed fix however).

5 Likes

@Southside , @Vort
Appreciate your efforts. These are interesting and necessary procedures to go through.

I have myself been doing some similar tests in the branch without transfers, to check the performance sans payment step (since that step is being reworked).

I have played around a bit with the way data is stored to disk, and such things.

On local network, I could notice a very consistent 1.7 chunks per second being handled by Adults regardless of parallel/sequential upload, which was up from 1.2 chunks/s with the storage solution used in main now, so a 40% improvement.

When running the same loop directly on the db, a thousand chunks are handled in less than half a second.

So there are severe bottle necks in the message handling. NB this is local network so the transport layer should not be a major contribution.
Serialisation and signing are known to be heavy, but I do suspect there is more at play here. I have some ideas for what can be done, but need to look closer at those parts.


I noticed this as well. I observed 20 byte chunks taking longer than 1kb chunks.

9 Likes

Would testing 1, 2, 5 and 10kb files with a constant delay be helpful? or even smaller files?

Any poking around is good. I can’t say for sure just how helpful that particular one would be (not saying it wouldn’t be, just unsure). I think the diff in size is less of a contribution compared to the per msg overhead (except for when going bigger as it chunks up to many chunks).

I’m at a msg routing isssue currently which will need to absorb me entirely until I’ve solved it, because I’m off on vacation next week, but I will circle back to these questions and dig deeper there.

6 Likes

Increasing the no of iterations shows little change in the shape of the graph. THis is with 20 runs of a 2kb testfile with 2 sec delay

2 Likes

Just for fun, you should try 1MB and 10MB.

1 Like

Just for fun you should grab the reworked script at GitHub - safenetwork-community/upload-performance: PUT a series of standard size files to exercise the SAFENetwork testnets and try it yourself :slight_smile:

EDIT: Silly missing " error fixed now.

The script now accepts parameters so it should be easy enough to wrap it in a loop and run it overnight with various values for filesize, iterations and delay.
We really need to get a wide range of results from a variety of machines to get more meaningful trends.
I still need to do more work to get nicer output so its less hassle to create graphs - all PRs welcome…

and they’re off and running…

From the first few results it looks like the shape of the graph will be pretty similar to the others , a sharp rise from the first to second run then a gentle slope up. I also spotted a wee formatting blooper in the script. Funny how you only notice these things after you post them :slight_smile:

At run 14, I was watching a video of the underwater pipeline fire in the Gulf of Mexico. I think this highlights the need for a wide range of results from different machines…

EDIT: I meant to add the link to the raw data for this graph last night

6 Likes

~/random-files-upload-test.sh 500 10 2 ? :pensive:

that will fire off a test with a 500kb testfile for 10 iterations and a 2 sec delay between each iteration

In your output folder you should get a file named 500data.txt . The time in seconds is on every third line, first two are the file container and actual xorurl…
I tried to make this filename 500k-data.txt but that confused the tee command so I just left it at that somewhat misleading filename for now.
I need to tidy this up of course and am just about to write another post suggesting a common format for capturing data from these tests and possibly further data profiling tests in general.

For now I think I want an array for each data point of
[time elapsed(real), testfile size (kb), iteration no, delay(secs)]
but it may make more sense to add data re processor type and speed, RAM fitted, connection type (local, ADSL, cloud) and most likely more parameters.

1 Like

it creates testnet-file-uploads-20210703_0947 but it remains empty I don’t think it runs. What is the command you use to run it?

1 Like

Give me a couple of secs to grab the file again from github and check it myself. I am working on a local copy right now.

2 Likes

Ah When you grab that from github you will need to chmod +x random-files-upload-test.sh

If you just run it without any arguments, it will fire an error message explaining the arguments required test file size, iterations, delay in that order.

When it is all finished /home/$USER/tmp/testnet-file-uploads-$(date +%Y%m%d_%H%M) should contain a file named 500-data.txt or wahatever filesize you chose in kb

Like I say, all this needs much tidying up.

2 Likes

I think I found the problem.
sn_node 0.7.1 is not allowing me to create test coins.

2 Likes

I have just replaced the screen on my laptop ( long story dont ask) and was bringing it up to date with the latest.
I too cannot create coins with sn_node 0.7.1. I am building both node and cli from source right now to see if that solves it.

I have 0.6.1 working fine on my desktop, I think I’ll just put that on the laptop as well for now and let the devs sort it out on Monday

3 Likes

:slightly_smiling_face: back on 6.1 an it works just fine. Thanks

3 Likes

0.7 specifically had transfers removed. It’s normal behavior.

ps: loving your independent exploratory work!

3 Likes