Teeny tiny testnet time! Following a successful comnet last week (thanks @josh 
) we’re rolling out a new testnet with 1GB nodes. And this time we’re allowing tiny nodes from home too. With additional stabilisation fixes over the last couple of weeks, we’re hopeful of seeing a successful split - so we encourage as many of you as possible to jump on board.
General progress
@anselme reports encouraging news on DKG (elder voting process), forcing DKG termination and key generation whenever a SAP (shared information about current membership) is updated. Stuck votes have been a problem for a while and, while still being tested, this looks like a definite step forward.
The Davids @dirvine and @davidrusu have been exploring the stable set idea introduced last week to see what a non-DKG/BLS elder set could look like and how that could handle forks.
@bochaco is looking at in-memory storage of data and seeing how it performs compared to data written to disk.
@oetyng has been working on simplification of code, especially in comms. In addition to that we’ve been looking into ways to have messages between clients and nodes forwarded without expensive deserialisation at the elders.
And @chriso and @roland have set up OpenSearch on AWS and are just tweaking it to enable highly detailed tracing.
Testnet Ahoy! 
OK mateys, this week we’ve launched a new 42 node 1GB testnet into the briney. With a favourable wind, you may be able to join with nodes from home, or from your favourite cloud provider. One advantage of small nodes is that free cloud VMs should now be more viable.
The last comnet was encouragingly stable and apparently failed after filling up with no new nodes to join. Obviously, with small nodes this can happen quite quickly. So this time we’ll be looking to see if we can get a second split to happen without the network falling over. We’ll also be checking out
- how easy it is to join from home
- has the joining node memory issue been sorted?
- whether performance is affected as the network fills up
- whether DBC transfers are working properly
- how smaller nodes with min 1GB - max 2GB and elder storage are working out
For the first time, we have an OpenSearch server set up to help us monitor this testnet. OTLP functionality is built into the safe binary, so that’s another thing we’ll be looking at.
Hopefully we will see a second split, in which case we will be able to test improvements to the relocation process we’ve been working on. You can see if your node has relocated by looking for RelocateStart and RelocateEnd log messages.
Getting involved
Once again, so we can test what we need to test, the CLI is limited to files of less than 10MB.
To get involved you can follow these instructions to set up the safe CLI. The testnet-name is feb2, and the recommended safe version is 0.69.0.
Joining as a node
To join as a node, once safe is installed and you’ve switched to feb2 as per the instructions, run:
safe node install
Then
safe node join --network-name feb2 [optional flags]
Success is most likely with a cloud VM, but during the previous comnet folks managed to get in using nodes from home, both with a VPN and without one, so it’s definitely worth a try. Start with:
safe node join --network-name feb2
Then with combinations of --public-addr <your public addresss>:12000 and --skip-auto-port-forwarding to see if you can find one that works. You may also want to setup port forwarding on your router, but please note, NAT traversal is not implemented and there’s no guarantee of success. This thread could be helpful SBC Network? NAT nightmares.
Submitting Traces
We now have Open Telemetry enabled for the node binary, so you have the option to submit traces/logs from your node to us. The traces are stored in an index in an OpenSearch cluster we’ve deployed on AWS. To submit your traces, before running the node join command, set the following environment variable like so:
export OTEL_EXPORTER_OTLP_ENDPOINT="http://dev-testnet-infra-543e2a753f964a15.elb.eu-west-2.amazonaws.com:4317"
We’re still getting to grips with how to make use of the data in OpenSearch, so at the moment we don’t have something to show, but soon we will offer read only access to the dashboards. We’d also be interested in any contributions here from people who know or who have worked with things like Elastic/Kibana, who might be able to show us how to do interesting things with the data. If you have any experience here and would like to help, please let us know 
!
What’s helpful and reporting issues
Right now, keep uploads small. < 10MB / file.
There is a temporary limit added, and you will get an error if surpassing 10MB.
If you are consistently seeing issues PUTting data or retrieving data you have PUT, please run your command with RUST_LOG=sn_client prefixed to it (on Linux/Mac at least). The output there, and MsgIds that have been sent/failed will be key to debugging.
We’ll try to report back stored data sizes at nodes as we go, so we can see if there’s a correlation between capacity and reliability.
Please note, nodes are not evenly distributed in XorName space, so with a limited crop, we will not see data uniformly spread across them.
May you have fair winds and following seas.
This is what we’re using to verify data-storage just now.
safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr
safe://hygoygym19bdbzg6jakfzkmacxurpkbuphgsxi6af3aecugfqjfaq66shww
safe://hygoygyq1xiofs8kx11jwkcguzuocd1xa5qp767bjjz39yrh44eo1mjkjch
safe://hygoygypnrqirzo7r5w464mnda8s6sdpatyrco5pda1w8nh5appa3ycb98h
safe://hygoygykajkeropyam6yihj9exdhn5o4b1pbjbeet4bazqo8m47ga3pw8mo
safe://hygoygyp8c1wy7ijrpnesz9cj7enf4ms9oxhmp5m4r4yw4pjcemgywekf9r
safe://hygoygye13iznm1yxzpxp9seh6j8weh84c5ib3fq9mxozz4h6hb95o3uduo
safe://hygoygycc11b4pmrcb97ro6ro3wxh3cfrgi7ctbiwyya8mkktq1j35ysmzh
safe://hygoygykzkguckqygm985f5jrhcw87wiskbc97bottab47tq6fm9zm85f7r
safe://hygoygyewnaj8o7za3qycckfy95o5j9gex6s3b8jygr9j48aiyzthc36ffo
safe://hygoygykime3s7rup1ezeiqeqo1zy1o4bqmckewhriun889c64n4umxcyih
safe://hygoygyc98c1ozh1cn9edxeworctuagt465zyybr61m8judfry4a4zycf8c
safe://hygoygyqcq8bift7urnftiqxzn7d877ckwtk85haawg7oigm3bhn3r1dx7e
safe://hygoygyk7dzmcqxm3q4np87unu6tb9d8qt44fowxgqrhjy1xs6xgt69hade
safe://hygoygymq98z86j1oagpctmg7pnixz94ckekpgytu1jkb45du8xw49qrr4w
safe://hygoygypoq7y31uedr7c6q6e3jaxhd94id3r8bwn7od3f79hgdj7xbqibjr
safe://hygoygykw8iw7wt9o3c7w36hi5xc1coyjdmdfm6dima3y9ehefxipxkwg3w
safe://hygoygyc67ozi6m9i9o8xq5wti7ysa4g6nodxetmdgw64g5pw1qmucheeac
safe://hygoygyqsxzn4p6mjx7ggj98pqc4he53xk1c54kw3hq1967n4yga1nbo3iw
safe://hygoygyxojetcr5553xfyjnfoh3ywge7grs8d7z66e8nf678stecagdkomw
safe://hygoygyqf4au3yf7ouyi7imu1hty5qrsjppy1mj5uba5tahnxkmocys5g7h
safe://hygoygyx41a9samc35ep6h7y6zwku5ess3377f8awtuwn44r1pc5w3ragky
safe://hygoygyej97u6irwmgq1od4s8m6h8epbkgmmwqtghhhy1hbcsntizpjmh3c
safe://hygoygypn9urw4mhoqfabnke71e8usekjeg8m54t1fkoeigdyr5srgsqoia
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
Thanks
or 
?