Update 15 December, 2022

After last week’s successful testnet we’re launching a new one, this time allowing nodes to join. :tada:


Note: This testnet is now offline. Latest one is here


Last week’s testnet allowed us to track down a bug where we’re not handling registers (directories) that cross a size limit (thanks @stout77 and @neik). We had always imagined that you’d need to pay every X entries for registers, so this issue has just brought that question to the fore. To solve this, we’ve added a fix which simply limits a register to 1024 entries for now, which will give an error when it fails. Down the line we imagine there will be app layer logic to enable linking together registers so that this size limit can be overcome.

We now want to test some other changes we’ve made around messaging, stripping as much as possible out of qp2p, which we believe is causing some issues with dropped communications, and leaving it all to quinn. This is in a testable state now so we’re keen to see what happens.

Once again we have removed split functionality, so we’re operating with a single section, but this time you’ll be able to join with your own nodes if you fancy it. The more testers the merrier, so dive on in, the water’s lovely!

Well we think it’s lovely but we’re well aware that for many of you the water may seem shark-infested or otherwise unwelcoming. We want to understand what the sharks are and how to remove them, so please take our anonymous poll at the end of the update.

General progress

It’s been all hands on testnet, but @oetyng has found some time to scope out systematic tests for two different aspects of our network; data retention in presence of churn and network health in presence of message drops.

Mostafa and @davidrusu continue to make good progress in the world of consensus.

With the first major qp2p update in, @bzee is looking at removing our custom wire format, which may simplify things if we go all in on the bidi flows.

@jimcollinson is working through some onboarding flows for data, such as where files might be transferred from a centralised service to a user-sovereign one, likely a very common use case.

And the entire team has been experimenting with ChatGTP - it’s not perfect by any means but it actually comes up with some pretty good coding suggestions. Minds have been duly blown.

Another week another testnet

We’ve got a 27 node testnet up. Each node has 50GB of space, and our single big section can go up to 1,000 nodes or more.

Goals

Once again we’re aiming to ensure that we’re not losing data until nodes are full - apart from the known case of registers hitting a limit. Our internal tests have shown data sticking around quite happily as long as we’re not filling up nodes. Full nodes will herald the end of this run, should we get there. We’re continuing to study client->elder->adult timeouts too. We want to see how relevant those values are under load.

Like last time, we’re asking you to avoid larger uploads. We’re not enforcing it this time but please keep them under 10MB, so we can rule out large files being the cause of any errors.

Churn

All info is good info, as far as we’re concerned, but if you want to join as a storage node please leave it running for as long as possible .

The stability of data will be related to the amount of churn that takes place. Right now we have four copies of each data, so we’ll also be seeing how that fares here too.

We’re testing the waters here, though ideally we don’t want this one to be brought down by churn. Data replication bugs (moving data to a new home when one goes down) will be the topic of a future testnet.

Getting involved

We really want to make it as easy as possible for everyone to get involved, but understandably it’s a daunting prospect for some. Below are some detailed instructions for Linux, macOS and Windows. Follow these and it should work, or at least fail in an understandable way (probably router related). Whatever happens it won’t blow up your machine! :boom:

Two modes - upload/download and storing

You can help us with this testnet in two ways: (1) by uploading and downloading data; and (2) by offering your device as a storage node. If you’ve not done this before we’d recommend you start with (1) and move on to (2) if you feel inspired. We’ve provided instructions for both.

Uploading/downloading

To upload and download (PUT and GET) data to the test network, you just need to install the safe binary on your operating system. See the instructions below.

The bad news…

First the bad news. This is bleeding edge stuff and we don’t yet have a nice GUI to help you upload and download data.

And the good news

But using the command line isn’t too hard. Mostly it’s just copy and paste. And for the folder creation/deletion tasks you can always use a GUI file manager, of course.


Linux and macOS

Installation

If you have run safe before you need to delete the old version.

# Delete the existing `~/.safe` directory.
rm -r ~/.safe

# Check that safe has been uninstalled
safe --version # --> should give a 'not found' message

If not it may be installed in /usr/local/bin:

sudo rm /usr/local/bin/safe
[password]

To keep things simple, we’re now going to recommend running the installer as the root user. It puts the binary at /usr/local/bin, which is always on PATH on any Linux/macOS distribution. For future updates the installer will just overwrite it, so you won’t need to clear anything as an additional step.

On Linux:

curl -so- https://raw.githubusercontent.com/maidsafe/safe_network/master/resources/scripts/install.sh | sudo bash

safe networks add main2 https://sn-node.s3.eu-west-2.amazonaws.com/testnet_tool/main2/network-contacts
safe networks switch main2

safe --version # should be 0.68.0

On macOS:

# switch to sudo shell
sudo su

# run install script
curl -so- https://raw.githubusercontent.com/maidsafe/safe_network/master/resources/scripts/install.sh | bash

# return to your own user profile
exit

safe networks add main2 https://sn-node.s3.eu-west-2.amazonaws.com/testnet_tool/main2/network-contacts 
safe networks switch main2
safe --version # should be 0.68.0

All subsequent instructions are the same for both operating systems.

Uploading

Linux upload both

Uploading a file
# Upload a file in your current directory
safe files put [filename]
e.g. safe files put examplePic.jpg

Uploading a container (a directory plus contents)
Choose a directory (e.g. /home/[yourUserName]/Pictures) or make a new one and add some files. Then cd to the parent directory (e.g. /home/[yourUserName]/) and run:
safe files put ./[directory] --recursive
e.g. safe files put ./Pictures --recursive

Downloading

Linux download

Downloading a file
Use safe cat safe://[address] > filename.txt to download a file:

safe cat safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr  > coast.jpg

Downloading container (directory plus contents)
safe files get safe://[address?v=version]

# For example this will download the container to your current directory. This is just an example, it's not live data)

safe files get safe://hyryyryikpk16oxay7wa1midtbkibxoec6eg4fsgd853gx5xuhs19ujmgzwnra?v=hc7xjae7f8o96xk9446gzyy13j9z7es47rpqdcu81iap61jdcu6no

Running a storage node

To run a storage node (an adult) you need at least 50GB free disk space plus a router that allows you to connect.

safe node install

safe node bin-version # should be 0.72.2

Now download the testnet config file and switch to that network by running:
safe networks add main https://sn-node.s3.eu-west-2.amazonaws.com/testnet_tool/main/network-contacts && safe networks switch main

Now try to connect to the network with:
RUST_LOG=sn_node safe node join --network-name main

If that gives an error try:
RUST_LOG=sn_node safe node join --network-name main --skip-auto-port-forwarding

If that doesn’t work it may be you can’t join from that machine / router.

If successful the folder ~/.safe/node/local_node will start to fill up with new folders and chunks.


Windows

Installation

Press the Windows key or click on the Start button and type “Powershell”. Then right click on the Windows Powershell entry on the menu and select Run as administrator.

Paste the following text into the terminal by right clicking on the bar at the top of the window, then select edit → paste:

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/maidsafe/safe_network/main/resources/scripts/install.ps1'))

Now hit enter to run the install. If you have a previously installed binary, you’ll be asked if you want to overwrite it. Say yes. Now exit the session.

Start a new session without administrator privileges and now run safe --version. It should be at 0.68.0. Then:

safe networks add main2 https://sn-node.s3.eu-west-2.amazonaws.com/testnet_tool/main2/network-contacts
safe networks switch main2
safe --version # should be 0.68.0

Downloading


Downloading files
safe cat safe://address > filename
e.g.
safe cat safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr > coast.jpg

Downloading container (directory plus contents)
# For example this will download the container to your current directory. This is just an example, it's not live data)

safe files get safe://hyryyryikpk16oxay7wa1midtbkibxoec6eg4fsgd853gx5xuhs19ujmgzwnra?v=hc7xjae7f8o96xk9446gzyy13j9z7es47rpqdcu81iap61jdcu6no

Uploading


Uploading file
# Upload a file in your current directory
safe files put [filename]

e.g.
safe files put examplePic.jpg

Uploading a container (directory plus contents)
Choose a directory (e.g. /home/[yourUserName]/Pictures) or make a new one and add some files.

Then cd to the parent directory (e.g. C:\Users\[your username]\) and run:
safe files put .\[directory] --recursive
e.g.
safe files put .\Pictures --recursive

Running a storage node

To run a storage node (an adult) you need at least 50GB free disk space plus a router that allows you to connect.

From your Powershell session, run safe node install to get the latest version of the node. Then use safe node-bin version to check you are on 0.72.2.

Download the testnet config file and switch to that network by running:

safe networks add main https://sn-node.s3.eu-west-2.amazonaws.com/testnet_tool/main/network-contacts
safe networks switch main

Now try to connect to the network with:
$env:RUST_LOG = 'sn_node'; safe node join --network-name main

If there is an error, try:
$env:RUST_LOG = 'sn_node'; safe node join --network-name main --skip-auto-port-forwarding

If that doesn’t work it may be you can’t join from that machine / router.

If successful the folder C:\Users\[your username]\.safe\node\local_node will start to fill up with new folders and chunks.


Test data

To follow


Poll time

We really hope you’ll give this a try. If you get stuck there’s plenty of folks who’ll be more than willing to help out.

In the meantime we’ve a couple of poll questions to help us gauge what we can do to increase involvement.

What puts you off trying a testnet? (Pick one)
  • I’m not comfortable with the command line
  • Too busy to get my head around it
  • I’m not at all techie but like seeing other people have a go
  • I think I’d need a lot of help and don’t want to be a burden
  • Nothing - I’m happy to pitch in when I can

0 voters

What would most help you become practically engaged with testnets? (Pick up to 3)
  • Video walkthroughs
  • More regular testnets so I know when they are
  • More structured feedback from the team
  • GUI so I don’t have to use the command line
  • Podcasts / audio
  • Clearer goals for each testnet
  • Clearer instructions for each testnet
  • Something else - please expand below
  • Nothing - testing software’s really not my thing!
  • Nothing - I’m already involved

0 voters


Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

43 Likes

That was a cheep first now to read can’t wait to see what’s been happening this week :slight_smile:

17 Likes

Second ! Wow 2 testnets in a row, ok we’re at full steam !! :cowboy_hat_face: Let’s try that

18 Likes

These two things :metal:

15 Likes

Instructions probably missing switch testnet -type of command, or…?

I mean, with current instructions, upload gave:

Error: 
   0: Failed to connect: ConnectionError: Failed to connect to the SAFE Network: NetworkContacts("File read/write error: Error opening SectionTree file from /home/topi/.safe/network_contacts/default: Os { code: 2, kind: NotFound, message: \"No such file or directory\" }")

This solved the problem, but it was presented only in the “join node from home” part of the update.

10 Likes

Node ate 700 MB of RAM within first minute after start.
I doubt that it will be possible for me to keep it running if RAM consumption will go further up.

7 Likes

Thank you ! That solved it for me as well.
I could get the coast.jpg file :slight_smile: The download felt slower than last week testnet though

8 Likes

Correct. Yeah, this should probably be immediately after the install instructions. Apologies.

@JPL could you edit the post to that effect please?

12 Likes

Got another kind of error:

Error: 
   0: Failed to connect: ConnectionError: Failed to connect to the SAFE Network: NetworkContacts("failed to make initial contact with network to bootstrap to after 3 attempts, result in last attempt: Err(NoResponse { msg_id: MsgId(7de4..cfd5), peers: [Peer { name: bd0f18(10111101).., addr: 178.62.18.48:12000 }, Peer { name: d4af52(11010100).., addr: 138.68.135.35:39862 }, Peer { name: c27658(11000010).., addr: 178.62.72.166:42939 }] })")

Location:
   sn_cli/src/operations/auth_and_connect.rs:83

6 Likes

Thx 4 the update Maidsafe devs

Super exiting that we have another TestNet up and running so quick

My clueless share :heart_eyes:

:clap: :clap: :clap: @stout77 and @neik, but also for the rest of you super ants who stubbornly keep hacking…

Will dig in deeper later

13 Likes

Well, those IP addresses are definitely valid nodes. Not sure why there is only three of them. Maybe the rest get truncated as part of the error output.

Hey @bochaco do you know anything about that?

8 Likes

You don’t have any errors there. You could check and see if your node directory has logs in it.

6 Likes

Error trying to put a jpeg (4.8 mo)

➜ ~ safe files put 14424.jpeg
FilesContainer created at: "safe://hyryyrytgfjb3tb5ur615maahtchyg168pa8qsjwrg8n8bhig6kr3ibb8fcnra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+------------+------------------------------------------------------------------------------------------------------------------------------------------+
| E | 14424.jpeg | <ClientError: Error received from the network: InvalidOperation("Failed to perform operation: CmdSendError") for cmd: MsgId(2888..3a54)> |
+---+------------+------------------------------------------------------------------------------------------------------------------------------------------+```
6 Likes

Those should be the ones it tried to send the msg/query to, i.e. the closest three Elders to the query dst addr.

7 Likes

Yeah … similar

willie@gagarin:~$ safe files put cock01.jpg 
FilesContainer created at: "safe://hyryyryznq6xzrjnjrkyqinw7xey6dmhg1f45hb1519rqzsb9967eqzo68hnra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+------------+------------------------------------------------------------------------------------------------------------------------------------------+
| E | cock01.jpg | <ClientError: Error received from the network: InvalidOperation("Failed to perform operation: CmdSendError") for cmd: MsgId(62b7..c96b)> |
+---+------------+------------------------------------------------------------------------------------------------------------------------------------------+
6 Likes

I double checked, the error message pasted here had everything the terminal output for me.

5 Likes

Does everyone get similar results for safe networks sections?

willie@gagarin:~$ safe networks sections
Network sections information for default network:
Read from: /home/willie/.safe/network_contacts/default

Genesis Key: PublicKey(01d8..fc68)

Sections:

Prefix ''
----------------------------------
Section key: PublicKey(1261..52ba)
Section keys chain: [(PublicKey(01d8..fc68), 18446744073709551615), (PublicKey(1261..52ba), 4), (PublicKey(029d..b999), 5), (PublicKey(107e..0398), 2), (PublicKey(0ae3..51c1), 3), (PublicKey(11a1..aeac), 0)]

Elders:
| XorName  | Age | Address              |
| 1f1944.. |   5 | 178.128.36.45:54295  |
| c27658.. |   5 | 178.62.72.166:42939  |
| cc2b5f.. |   5 | 161.35.36.148:50238  |
| d4af52.. |   5 | 138.68.135.35:39862  |
| d818b7.. |   5 | 138.68.189.108:57817 |
| eae2f6.. |   5 | 161.35.34.29:43661   |
| bd0f18.. | 255 | 178.62.18.48:12000   |
5 Likes

Is it possible to prevent bringing register to the state where such linkage becomes impossible?

4 Likes

Yes exact same results

4 Likes