Having further investigated the bootstrapping and now able to reliably start the full 27 nodes (the issue was around the disconnection of peers based on whitelisting), the results are still slower than pre-disjoint-sections, but much more consistent.
Results
96.2 MiB file (hippocampus.mp4) was uploaded several times.
Upload Time (minutes)
33.6
33.1
33.2
32.8
29.4
33.2
33.7
The upload time is still slower compared to pre-disjoint-sections (around 21m), but not as bad as first tested.
There was a second factor beside network bootstrap which contributed to the previous very slow upload time (this really took me by surprise). These most recent tests were done on my normal test setup, but the prior test was done using a computer for launcher and demo_app running qubes-os, which I noticed to be quite slow in general use. I didn’t expect that slowness to affect the newtork layer, but these results indicate it may have. It amazes me how many variables there are in this testing, and how there can be no assumptions about which ones may be ignored. This also feeds my unhealthy need to be a control freak in these tests, since it can be very hard to isolate all variables. I was considering switching to cloud servers, but now I think that environment would lead to too many uncontrolled variables.
Deviation
It’s extremely pleasing to see such an improvement in consistency of upload time.
This is the previous variability test.
A normalized standard deviation is used to compare the variability (ie normalizing upload times with the average then calculating the standard deviation).
Standard Deviation (higher is more variable is worse)
-----
Pre Disjoint Sections 0.0742
Disjoint Sections 0.0456
More reliable upload times means
- better user experience
- greater predictability in the software product which means
- easier to identify areas for improvement
- easier to measure the impact of changes
Very keen for Test15 to come out. Still feeling a bit disappointed with the slower upload times, but I have some ideas I want to test once Test15 is out. Now that consistency in upload time is better, I’m confident that improvements made to upload time will also be consistent.