tl;dr just wanna make nice graphs and help provide some useful data.
Looking at upload speeds and performance and I have written a wee script to get some performance data.
The aim being to build a body of test results to inform the devs of immediate problems now and to monitor performance enhancements in the future.
When I think harder about this, I wonder if I should take a step back and ensure that the data captured is in a agreed JSON schema for sharing and analysis by the devs and the brainier members of the community.
Another much bigger point is that a time in seconds on its own is no use without the context of filesize, iteration number, delay between iterations, software versions, PUT or GET, CPU type and frequency, RAM fitted, disk type (SSD, sata), local or remote, connection type(ADSL, cloud etc) and a host of other parameters that need to be captured to make each individual point meaningful. Should I be capturing real, sys and user times instead of just the real time? - probably…
So which of these parameters should I capture and although most of the data points will be numeric, would it make more sense to store them all as JSON strings?
Not being a data scientist, I’m going to be trying to learn quickly probably using python about data visualisation. However if there is any point to this, I’d like to use an agreed schema - which may well already exist internally in the project but I dont know about it yet.
Also right now I collect the real time formatted in seconds only using GNU time 1.7 NOT the bash builtin time command. See
/usr/bin/time -f "\t%e" safe files put
willie@gagarin:~/tmp/testnet-file-uploads-20210703_1345$ /usr/bin/time -V GNU time 1.7 willie@gagarin:~/tmp/testnet-file-uploads-20210703_1345$ time -V -V: command not found real 0m0.458s user 0m0.039s sys 0m0.383s
see the difference? The problem here is GNU time 1.7 is not POSIX-compliant which will have implications perhaps for Windows users but it does let me format the time in seconds only instead of as 0m0.458s. I suppose this can be worked around with regexps/sed but I’m lazy…
In post 55 [Offline] Fleming Testnet v6.2 Release - Node Support - #55 by Vort @Vort is using Wolfram and C# and that may be the best route for visualisation and Im going to have to bite the bullet and try learn some C#.