It’s testnet time! Yes folks - it’s time to dust off those terminals and fire up those clients once again, testnets are back in town. We don’t expect this one to be perfect and there are a few issues we’re working through with client-elder communications, but we’d really appreciate your help with testing a few things including data retention, and timeouts. To avoid complications, we have removed section split functionality for now, so we’ll be testing one big section. @joshuef explains more below.
General progress
The whole team has been working on getting the testnet up and running in the past week, but of course other work continues in the background.
@bochaco has been digging into a client connection/reconnection issue where messages seem to be getting dropped. It may be a quinn/qp2p issue, but further digging is required. Hopefully this will be fixed for the next testnet.
Meanwhile, @roland has been working on the implementation of the SectionTree, the record of all section keys back to genesis that proves users are on the right network.
The implementation of the ABBA consensus protocol is also going ahead, with Mostafa and @davidrusu making some final tweaks. Once implemented, the next step will be to integrate VCBC with ABBA to arrive at the full MVBA (Multi-Value Byzantine Agreement) consensus protocol.
@oetyng has been tidying up a few operations, including how we report storage at adults (a PR which also paves the way for avoiding nodes ever becoming full), and @chriso is looking into forwarding node logs for ELK, the main observability stack we use to monitor what’s happening at node level.
A new testnet
As we mentioned in the last update, we’ve been down in the trenches for a while, but are now starting to look at things holistically once more. And we want to do that involving the community as much as possible.
So we’ve got a 25 node testnet up. Each node has 50GB of space. We’re aiming to:
- Ensure that we’re not losing data until nodes are full. (We’re working on how to deal with full nodes). Until then, our current tests have shown data sticking around quite happily as long as we’re not filling up nodes. (So full nodes will herald the end of this run).
- Have no nodes join. We’ve limited this in the nodes themselves to remove one testnet variable. (We’re also aware - and have a pending fix for - some AE issues post-split, so don’t want to dive in there just now).
-
Get a better handle on certain timeouts. Elder->Adult bi-directional comms timeout after
7sjust now. Client->Elder in 45s. We want to see how relevant those values are under load.
Getting involved
As noted above, we’re not looking to check out node joins just now, as we have a blocker post-split that we need to test more before we can get the fix in.
As such, it’s only safe client and PUT/GET data that we’ll be checking out. We’re asking folk not to upload more than 10mb at a time to see how things go there.
Sadly we’ve not full-release as yet. (We’re changing up our test flow to check all PRs going into main on their own testnet now; once that’s in, releases from main should be much smoother). So we’ve manually created one and uploaded safe binaries here:
Note: macOS will not run the binary because it can’t verify the developer. To circumvent this, run xattr -d com.apple.quarantine ./safe. Then make sure it can be executed with chmod +x ./safe . Then the CLI can be run with ./safe.
To get involved:
- remove
~/.safe - download your relevant
safebin from the release and store in/usr/local/binon mac/linux or%USERPROFILE%.\safe\cliin windows.- Note for Windows users - you may wish to add the bin directory above to PATH, alternatively you can use
./safewhen run from the directory for all proceedingsafecmds.
- Note for Windows users - you may wish to add the bin directory above to PATH, alternatively you can use
- run
safe --versionto make sure you’re on the correct version (0.67.0of the CLI). - run
safe networks add wild-testnet https://sn-node.s3.eu-west-2.amazonaws.com/testnet_tool/main/network-contacts safe networks switch wild-testnet
And you should be good to go 
Example: Upload a file
The simplest thing we can do is upload a file using the files put command:
$ safe files put ./to-upload/file1.txt
FilesContainer created at: "safe://hyryyryynamznbfsgn7ccfquqmx6y8yzhq6tn7uzz775hrkyj4g8ipcy3ke6yeuy?v=h9jxxwwpy1cwf3pnb86ahkfk5ju8eb3miegnuehc99f5r5x83d9go"
+ ./to-upload/file1.txt safe://hy8oycyyb7jfqswhktzn9ahhk1hnz53dhfnrfp6h34emgrmjzggro75eikpoy
This will create a container with a single file. The safe:// URLs assigned here refer to the container and the file, respectively. This means the file is addressable using either safe://hyryyryynamznbfsgn7ccfquqmx6y8yzhq6tn7uzz775hrkyj4g8ipcy3ke6yeuy/file1.txt or its direct URL.
Read more about how to upload files, in the CLI docs.
What’s helpful and reporting issues
Right now, keep uploads small. < 10MiB / file.
There is a temporary limit added, and you will get an error if surpassing 10MiB. The error is not informative currently, but you will know by the size that it’s due to surpassing this temporary file size limit.
You may well see issues around Cmd Ack Validation timeout, (our timeouts need to be dialled in better, see above), this means the network is probably under load, so have another go in a few minutes. Your data may well have been stored, just not as fast as we expect right now.
If you are consistently seeing issues PUTting data, or retrieving data you have PUT. Please run your command with RUST_LOG=sn_client prefixed to it (on Ubuntu/Mac at least). The output there, and MsgIds that have been sent/failed will be key to debugging.
We’ll try to report back stored data sizes at nodes as we go, so we can see if there’s a correlation between capacity and reliability.
The network is up on the back of a slightly less reliable (client-connectivity-wise) testnet that we’ve had up all week. Data retention has been fine there, so 
hopefully that will continue here (or a bug will be highlighted 
)
Most of that’s junk test data, but there is a small set of jpgs uploaded at the following xorurls for anyone wanting to pull some data (e.g. safe cat safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr > 1.jpg):
This is what we’re using to verify data-storage just now.
safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr
safe://hygoygym19bdbzg6jakfzkmacxurpkbuphgsxi6af3aecugfqjfaq66shww
safe://hygoygyq1xiofs8kx11jwkcguzuocd1xa5qp767bjjz39yrh44eo1mjkjch
safe://hygoygypnrqirzo7r5w464mnda8s6sdpatyrco5pda1w8nh5appa3ycb98h
safe://hygoygykajkeropyam6yihj9exdhn5o4b1pbjbeet4bazqo8m47ga3pw8mo
safe://hy8ayqyjy9obx5kr5g9f1r95hshhfswt7yqh6cy9t34sap7zqid4ox7ucuo
safe://hygoygyp8c1wy7ijrpnesz9cj7enf4ms9oxhmp5m4r4yw4pjcemgywekf9r
safe://hygoygye13iznm1yxzpxp9seh6j8weh84c5ib3fq9mxozz4h6hb95o3uduo
safe://hygoygycc11b4pmrcb97ro6ro3wxh3cfrgi7ctbiwyya8mkktq1j35ysmzh
safe://hygoygykzkguckqygm985f5jrhcw87wiskbc97bottab47tq6fm9zm85f7r
safe://hygoygyewnaj8o7za3qycckfy95o5j9gex6s3b8jygr9j48aiyzthc36ffo
safe://hygoygykime3s7rup1ezeiqeqo1zy1o4bqmckewhriun889c64n4umxcyih
safe://hygoygyc98c1ozh1cn9edxeworctuagt465zyybr61m8judfry4a4zycf8c
safe://hygoygyqcq8bift7urnftiqxzn7d877ckwtk85haawg7oigm3bhn3r1dx7e
safe://hygoygyk7dzmcqxm3q4np87unu6tb9d8qt44fowxgqrhjy1xs6xgt69hade
safe://hygoygymq98z86j1oagpctmg7pnixz94ckekpgytu1jkb45du8xw49qrr4w
safe://hygoygypoq7y31uedr7c6q6e3jaxhd94id3r8bwn7od3f79hgdj7xbqibjr
safe://hygoygykw8iw7wt9o3c7w36hi5xc1coyjdmdfm6dima3y9ehefxipxkwg3w
safe://hygoygyc67ozi6m9i9o8xq5wti7ysa4g6nodxetmdgw64g5pw1qmucheeac
safe://hygoygyqsxzn4p6mjx7ggj98pqc4he53xk1c54kw3hq1967n4yga1nbo3iw
safe://hygoygyxojetcr5553xfyjnfoh3ywge7grs8d7z66e8nf678stecagdkomw
safe://hygoygyqf4au3yf7ouyi7imu1hty5qrsjppy1mj5uba5tahnxkmocys5g7h
safe://hygoygyx41a9samc35ep6h7y6zwku5ess3377f8awtuwn44r1pc5w3ragky
safe://hygoygyej97u6irwmgq1od4s8m6h8epbkgmmwqtghhhy1hbcsntizpjmh3c
safe://hygoygypn9urw4mhoqfabnke71e8usekjeg8m54t1fkoeigdyr5srgsqoia
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
