Fleming Testnet v6 Release - *OFFLINE*

Fleming Testnet v6 Release

:trumpet: Fleming Testnet v6 is LIVE! :trumpet:

Friday testnet anyone? :sweat_smile:

We are excited to announce that Fleming testnet v6 is now LIVE.
Released versions:

  • sn_node v0.49.8
  • sn_client v0.61.1
  • sn_api v0.29.1
  • sn_cli v0.29.2

We’d like to dedicate Fleming Testnet v6 to @sharontannahill :woman_superhero:

Sharon has been an absolute rock for MaidSafe through thick and thin. We depend on her for so many things, and the work she does behind the scenes just doesn’t always get the credit an increasing number of us realise she deserves. From administration, salaries, coin management, general organisation, and communications with exchanges and other 3rd parties, to name but a few of the tasks we rely on her for.

Sharon, this one’s for you from all your Safe pals :heart:

IMPORTANT NOTE - IGD connection issues.

In the build up to today’s release we have observed some IGD issues which we believe is in our IGD implementation. These issues can mean that trying to join the testnet with your node will more often that not fail with an IGD error, unless you have a public facing IP address, or have set up port forwarding on your router and launched your node specifying your internal & external IP and port numbers to reflect this.

We have a very small sample size within the company so we are looking to get some feedback from the community here on whether you can queue a node up to join the network, or whether you are receiving IGD errors.

If you do receive IGD errors, you don’t have a public facing IP address, which the majority of us would not, but you feel that you are advanced enough to set up port forwarding on your router, you can also give this a try. We can’t provide instructions for all the different variations of routers out there, but we will put instructions below on how to launch node when you are port forwarding.

If you are getting IGD errors and you don’t feel that you are advanced enough to set up port forwarding then please continue to use the testnet as a client. No matter whether you run a node or not, you can still follow the instructions in the Joining the Testnet section below and use the CLI as per the User Guide.

We are working to fully identify and resolve any IGD issue and hope to release a fix ASAP.

v6 Changelog

With the community’s help, we identified several issues with previous testnet iterations, and have been working hard to resolve as many as possible.

Note that in the build up to today’s release, we discovered a bug with rewards not being paid out and therefore decided to disable rewards so as not to block the testnet release. We will not be fixing rewards or transfer based issues as it seems the DBC work will transform those processes to be simpler and more privacy-aware.

The fixes and improvements for v6 are:

1. Only Accept “Reachable” Adults

There has been a major change to which nodes the network accepts to join as Adults - now only nodes which are reachable are accepted.

However, as detailed in the IMPORTANT NOTE above, we’ve discovered a bug in our IGD implementation. Once this has been resolved you will see that not all nodes will be allowed to join the testnet, only those with correctly functioning IGD or a public IP.

How will you know if you have a correctly functioning IGD or a public IP on your node?
Once the IGD bug is fixed, when you follow the joining instructions below and attempt to run your node with safe node join, if your logs, accessed with the command tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log, report:

Unfortunately we are unable to establish a connection to your machine either through a public IP address, or via IGD on your router. Please ensure that IGD is enabled on your router - if it is and you are still unable to add your node to the testnet, then skip adding a node for this testnet iteration. You can still use the testnet as a client, uploading and downloading content, etc. https://safenetforum.org/

This means that IGD does not appear to be functioning correctly, and you don’t have a public IP.
You do not need to take any further action as your node will not be running.

We’re not sure at this point how many people this will affect, wider testing with this testnet iteration will give us an idea, once the IGD bug is fixed.

Can I still do anything with the testnet?
Yes! Everyone can use the testnet as a client. This change only affects those wishing to add a node to the testnet. No matter whether you run a node or not, you can still follow the instructions in the Joining the Testnet section below and use the CLI as per the User Guide.

Will I ever be able to join another testnet if my IGD doesn’t work and I don’t have a public IP on my machine?
The short answer is yes. The long answer is, at the moment, it will take some manual configuration with different instructions for different people depending on your router. You would need to forward a port on your router to your machine, plus you would need to specify your machine’s IP:port when launching your node, which is not currently supported as an option when launching a node via the CLI.

We do have a sn_cli PR ready to test to support this change, but we feel that our resources should be 100% directed at the core testnet functionality right now rather than router configuration and other associated issues. This can be added, enabled and improved once we have stability with the testnets. In the meantime, if advanced users want to try port forwarding they will find some instructions below on how to manually launch their node.

2. Anti Entropy

As a recap, a major hurdle any decentralised network which will experience churn, such as Safe, must overcome is keeping the nodes up to date with the network’s current state. Who has joined and who has left? Who are the current Elders in any given section?

Elders are by definition relatively dependable and churn amongst them is minimal, but Safe is asynchronous and communications between section Elders and external actors may be few and far between. The Elders in a section may have churned many times between contacts.

It’s vital actors wishing to make changes on the network have an up-to-date ‘understanding’ of the section before they are allowed to do so, otherwise data could be written to Elders that no longer exist.

Entropy, or increasing disorder, is a characteristic of a constantly changing network. We cannot reduce it but we can wait for periods of zero change among Elders before allowing a change in their section. Anti-Entropy (AE) is the mechanism for doing this.

Even during periods of massive change such as a large global event, AE enables the network to inform actors attempting to mutate data that they must wait until the change is over before they can try again.

On Safe, the AE mechanism is applied to all messages that can mutate the network, such as a PUT request. It requires the requesting actor - be that a Client, an Adult or another section - to prove that the information they hold on the affected section is current. Until then they cannot make a change.

AE reduces the synchronisation issue to a series of questions to which the answer is yes or no. If the answer is no (your information is not current), you need to update your records and try again. There is no halfway house.

In this way AE forces all nodes that want to perform data operations on the network to prove they are up to date.

We’ve had AE implemented in sn_routing for quite a while now, and more recently have been working hard to integrate it into many more of our crates, the biggest challenge being sn_node. Kudos in particular to @yogesh for the many, many hours invested in making this a reality.

3. Out Of Memory issue

As you will know, Fleming testnet v5.1 seemed to be working spectacularly well until memory usage on multiple hosted nodes spiked and eventually led to those nodes dying, and so sections becoming unresponsive.

The reason for the memory spike was that the network was experiencing a huge explosion in the number of messages being sent across it, so we’ve spent a lot of time since reviewing what generates all these messages and whether we could cut them down - the answer was yes, significantly!

Some examples of unnecessary network messages include the reserialising the whole message for every dst update, or sending some messages to all Elders in a section when we actually only need to send to a subset, and also sending response messages, again to multiple Elders, when actually in many instances there is no need to send any response.

The sheer volume of messages was choking the system, we witnessed memory usage north of 1.5Gb per node. Our internal tests indicate we’ve managed to cut that down to between 100-200mb.

4. Performance Improvements and Logging Changes

A nice side effect of the network message optimisations made with a view to resolving the OOM issue is that we expect that less network message traffic will equate to performance improvements. We’ve not gathered any stats on this, it wasn’t our aim when we set out to tackle the OOM issue, but it should make a difference.

We’ve also reduced the amount of logging on our Digital Ocean nodes, which seems to have improved performance - we have noticed the tests running a fair bit quicker after this change. This is configurable so we can increase logging again to reproduce issues and gather more data, when required.

Note that as a result of these log changes, the log file that you will be used to tailing in previous testnet iterations has now changed name to sn_node_rCURRENT.log. You will also notice that the log files now rotate when they hit a capped size limit, i.e. the contents of sn_node_rCURRENT.log are copied over to sn_node_r00000.log when it fills and the current logs are clean again, next it will copy to sn_node_r00001.log, and so on. Searching for historic logs may therefore mean they are not in the sn_node_rCURRENT.log file.

We still intend to look at overall performance and implement some ideas we have to improve the testnet’s efficiency and performance, but our initial goal is to get all the features working and the testnet’s stable, otherwise we will have a constantly moving target. It seems that as we go along improving stability and reliability, performance improvements are increasingly becoming a bonus side effect of this.

5. Account for Fluctuating StoreCost

We’ve observed during testing that sometimes the quoted StoreCost can change between querying a section and sending a PUT. To account for this we’ve added a StoreCost buffer to avoid unnecessary errors here and allow the PUTs to go through. (Note: This should be resolved a little further down the line with DBCs and PUT contracts.)

What happens to previous testnet iterations?

All previous Fleming testnet iterations have been taken offline, with all data removed.

This is to allow us to concentrate on any issues that arise from the new testnet iteration, and avoid any community confusion over which testnet they are connected to, or any errors as a result of contamination from previous configurations and data. We ask that everyone who attempts to use this latest testnet iteration removes their $HOME/.safe folder before trying to interact with this testnet, i.e. $ rm -rf $HOME/.safe/ to ensure there is no contamination.

All of the bootstrap nodes for previous iterations that we had on Digital Ocean have been destroyed, with new, clean D.O. nodes created for this testnet iteration - please follow the full list of instructions below to remove all previous testnet settings and data from your machine and to get you started with the latest testnet iteration.

IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.

Are We Working on Other Fixes?

Yes. Multiple other fixes and improvements are already in progress, some even from before we released testnet v1.

We also anticipate that the feedback from this testnet iteration will direct us where else to aim resources as a priority.

Where can I report any issues found?

If you come across any issues in your testing, start by checking the Known Issues section of this post (see below) to see if it has already been acknowledged . If it has not yet been added, you can post your issue in the comments of this post. You can also report issues in the Online Spreadsheet for Testnet Results and Issues - see the section below.

We will monitor and investigate reported issues as soon as we can.

Online Spreadsheet for Testnet Results and Issues

If you will be uploading data to the testnet and/or providing safes (nodes), please consider posting your data and results at:

SN Testnet Review
(Massive thank you to @VaCrunch for creating this spreadsheet :clap: )

This is a new version of this spreadsheet, with the versions used for previous iterations now locked.

For those posting:
• No need to Sign In. The white cells are for your data entry.
• One row for each of your devices.
• Scroll down until you get to the first empty row.
• You do not need to use your Forum name for your ID, but please use the same ID for all your devices.
• Supporting/analysis tabs are at bottom of screen: Error_Msgs, Summary, Thumbnails, Charts, Matrix, Top 10, Map, Resources, Lists, and Comments.
• Use the View/Zoom feature at top to reduce the visible size of the spreadsheet to match your screen. 75% will be the best fit for most people.
• If you choose not to record the name of your country please select “Unspecified” at the bottom of the list.
• If you change the number of nodes (safes) you are running, just edit the existing figure, do not add another line for the same device.

Known Issues

Any reproducible issues that we, or the community, come across will be added to this section:

  1. Although we’ve started making some optimisations, we still expect to see performance issues, particularly around $ safe files put .... operations with larger files/folders. There has been very little optimisation of write times, nor any general analysis of speed so far, with the majority of our efforts concentrated on getting features in place and fully functional before we spend more time looking at it from a performance perspective.

  2. We have removed authd from the stack for normal users right now, instead focussing on Safe CLI. To that end, we have updated the CLI to be able to do all things that previously required authd. This means no GUI right now, mostly. The focus is on network stability and then speed. As we optimise for speed, the team will also focus on authd again, perhaps removing it (our goal) in favour of a simpler approach. So test as much as possible with the CLI for now. It reduces user error, reduces the potential bug surface, and allows total focus on the prize - a fully autonomous decentralised network.

  3. We have discovered a bug with rewards not being paid out and therefore decided to disable rewards so as not to block the testnet release. We will not be fixing rewards or transfer based issues as it seems the DBC work will transform those processes to be simpler and more privacy-aware.

  4. In the build up to today’s release we have discovered a bug we believe is in our IGD implementation. This bug means that trying to join the testnet with your node seems to more often than not fail with an IGD error, unless you have a public facing IP address, or have set up port forwarding on your router and launched your node to reflect this.

    If you don’t have a public facing IP address, which the majority of us would not, but you feel that you are advanced enough to set up port forwarding on your router, you can give this a try. We can’t provide instructions for all the different variations of routers out there, but we will put instructions below on how to launch a node when you are port forwarding.

  5. It’s been noticed after v6 launch that large file uploads can hang as transfers get out of sync. A potential fix for this is known and already planned, as per here.

Let’s try it out!

Please read and follow the instructions below carefully.

IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.

Installing/Updating to Latest

To avoid any pollution from previous testnet iteration settings and data, we ask that everyone removes their $HOME/.safe/ directory, then installs the CLI, authd and node again.

$ rm -rf $HOME/.safe

Note that Windows users may be required to install specific software before being able to install the CLI using the command below - see full instructions for Windows here.

Now download and install the verified latest CLI binary via our install script with the following command:

$ curl -so- https://sn-api.s3.amazonaws.com/install.sh | bash

This script downloads the correct CLI binary for your OS (Windows, macOS or Linux), installs it in the correct directory, creating that directory if it doesn’t exist, and adds it to your system PATH.

You may need to restart your terminal window at this point for any changes to your system PATH to take effect. You can now confirm whether the CLI is installed and set up correctly:

$  safe -V
sn_cli 0.29.2

Next, you should install the latest node with the command below.

$ safe node install

Before proceeding we’ll just make sure we kill any old sn_node processes which have been left running:

$ safe node killall

You should now be equipped with the latest CLI and node.

Note that you can find full installation instructions in our user guide:

Joining the Testnet

As with previous public testnets, we are hosting some Elders and Adults on Digital Ocean to kick off the Network. These act as hardcoded contacts which bootstrap you to the network, therefore you will need a network configuration file to inform your CLI which network to bootstrap to. We store and update these connection details on S3 for you to easily point your configuration to.

If you have followed the instruction above to remove your $HOME/.safe folder before installing everything again, you should have no existing network configurations - you can confirm this with the $ safe networks command.

You can add a profile for the latest testnet which points to our S3 location, this is done using $ safe networks add like so:

$ safe networks add fleming-testnet https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config
Network 'fleming-testnet' was added to the list. Connection information is located at 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config'

Now you need to ensure that you are set to use this newly added fleming-testnet configuration, we can use $ safe networks switch fleming-testnet for this:

$ safe networks switch fleming-testnet
Switching to 'fleming-testnet' network...
Fetching 'fleming-testnet' network connection information from 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config' ...
Successfully switched to 'fleming-testnet' network in your system!
If you need write access to the 'fleming-testnet' network, you'll need to restart authd (safe auth restart), unlock a Safe and re-authorise the CLI again

(Ignore the “…you’ll need to restart authd…” advice in there)

Now to get write access to the network, you can use the following command via the CLI before proceeding to uploading files or transfer testcoins with CLI:

$ safe keys create --test-coins --for-cli
New SafeKey created: "safe://hyryyyy8ogt4yxmtgqd5yj9kuim911yuchbz77mhcrsdqh6dc5noypi9nqa"
Preloaded with 1000.111 testcoins
Key pair generated:
Public Key = f0347407ae2670f604fd53aaff29026ce06fdeaf8c2586ee786cd8a006d7e276
Secret Key = 7c2839cfda20f3c0f7872372d20a28080c7713688f16d9199c1ec02a7b0d185b
Setting new SafeKey to be used by CLI...

We now have our CLI and node components up-to-date, the latest hardcoded contact details to bootstrap to the public testnet, and we’ve generated some test Safe Network Tokens that we can use later to upload data.

To Add Your Node To The Testnet

Some differences in this section today due to the IGD bug described at the beginning of this post. We encourage all users wanting to try connecting a node to try the usual method of using $ safe node join as follows:

$ safe node join
Creating '/Users/maidsafe/.safe/node/local-node' folder
Storing nodes' generated data at /Users/maidsafe/.safe/node/local-node
Starting a node to join a Safe network...
Launching with node executable from: /Users/maidsafe/.safe/node/sn_node
Node started with hardcoded contact: 161.35.36.185:12000
Launching node...
Node logs are being stored at: /Users/maidsafe/.safe/node/local-node/sn_node_rCURRENT.log
(Note that log files are rotated, and subsequent files will be named sn_node_r[NNNNN].log, with values starting at 00000 and up to 99999.)

BUT we’ve found during in-house testing that in the vast majority of cases this will fail with an IGD error, when it shouldn’t. We’re working to resolve this now.

The alternative/workaround is to set up port forwarding on your router, then launch your node with:

$ $HOME/.safe/node/sn_node --public-addr <public ip:port> --local-addr <localnet ip:port> --hard-coded-contacts=[\"46.101.59.28:12000\"]

We’ve had success with this in-house, but it’s not for everyone.

We’re working hard in the background to resolve the IGD issue which should allow more people to join.

If the above steps are successful for you, your node will now launch and attempt to connect to the public network. Keep in mind that it can only join the testnet if the Network is accepting new nodes at that time (see NOTE below). You can keep an eye on its progress via its logs, which can be found at $HOME/.safe/node/local-node/sn_node_rCURRENT.log.

If there is no space on the testnet for your node to join, your node will automatically try to rejoin until it is successful, or until you kill the sn_node process ($ safe node killall). This is an anti Sybil attack feature which we have put in place to only accept new nodes onto the testnet when resources are required, therefore you may be attempting to join the Network but your logs tell you that the Network is not accepting new nodes at this time:

The network is not accepting nodes right now. Retrying after 3 minutes

This is expected behaviour which will happen each time you try to connect until the testnet detects that resources are running low. Your node will automatically try to rejoin in a loop, you do not need to attempt to join again manually, or via a script. We recommend keeping a watch on your node log file to check if you have been accepted - $ tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log

You can help to speed up the process of the network needing new resources by adding some data yourself - you don’t need to run a node to upload data! See the Do I need to run a node to participate? section below.

To Connect to the Testnet as a Client

Before working your way through the CLI commands to perform various actions on the network, following the steps above gives your Safe some test Safe Network Tokens to use. This means that there is no need to farm first to earn rewards before being able to try operations such as uploading to the testnet. You can then proceed with the various CLI commands to perform operations on the network.

You can even connect to the testnet in read only mode using CLI, i.e. once you’ve installed CLI you can try fetching content uploaded by other users. For example, try download the following image and open it locally afterwards:

$ safe cat safe://hygoyeyx768nkst7qqjjk8khjm67tr1n4otbctcts5j7dten43zae3ga83y > ~/safe-the-planet.png

IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.

Do I need to run a node to participate?

No, you can join with just the CLI to experiment with data, tokens, etc. You can follow the instructions in the Joining the Testnet section above, but you don’t need to run $ safe node join - at this point, just go straight to using the CLI as per the User Guide.

Further Information

Where are my node logs?

When you launch your node you should see the location of your log file printed on screen - this will be $HOME/.safe/node/local-node/sn_node_rCURRENT.log. You can tail your logs with a command such as $ tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log

Note that as a result of these log changes, the log file that you will be used to tailing in previous testnet iterations has now changed name to sn_node_rCURRENT.log. You will also notice that the log files now rotate when they hit a capped size limit, i.e. the contents of sn_node_rCURRENT.log are copied over to sn_node_r00000.log when it fills and the current logs are clean again, next it will copy to sn_node_r00001.log, and so on. Searching for historic logs may therefore mean they are not in the sn_node_rCURRENT.log file.

Where are my rewards?

As per the Known Issues section above, in the build up to today’s release, we discovered a bug with rewards not being paid out and therefore decided to disable rewards so as not to block the testnet release. We will not be fixing rewards or transfer based issues as it seems the DBC work will transform those processes to be simpler and more privacy-aware.

       ____
      /  \ \
     / /\ \ \
    / / /\ \ \
   / / /__\_\ \
  / / /________\
  \/___________/
82 Likes

yass. woohoooooo!

21 Likes

Congrats! Let’s test it. @sharontannahill you rock!:muscle::partying_face:

21 Likes

@sharontannahill is off right now, but I hope she sees this. She is our :muscle: power :superhero: for all things business and more.

35 Likes

No way, wohooo hoooo!!! Great! Great! Great! :beers:

15 Likes

Amazing, can’t wait to see the results, we are getting closer people :stuck_out_tongue_closed_eyes::heart:

15 Likes

Legends. Here we go!

13 Likes

Perhaps needs update above as appears to be now
sn_cli 0.29.2

and noting the change of log location that notes the error
look to

tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log

which shows as

[sn_node] INFO 2021-06-18T13:31:29.819607549+01:00 [src/bin/sn_node.rs:112]

Running sn_node v0.49.8

[sn_node] ERROR 2021-06-18T13:32:11.093105153+01:00 [src/bin/sn_node.rs:138] Unfortunately we are unable to establish a connection to your machine (82.41.99.238:57660) either through a public IP address, or via IGD on your router. Please ensure that IGD is enabled on your router - if it is and you are still unable to add your node to the testnet, then skip adding a node for this testnet iteration. You can still use the testnet as a client, uploading and downloading content, etc. https://safenetforum.org/

12 Likes

1st attempt → fail. Maybe my mistake. Will recheck.

[sn_node] INFO 2021-06-18T15:28:55.762772+03:00 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\sn_routing-0.77.8\src\routing\bootstrap\join.rs:292] Sending JoinRequest { section_key: PublicKey(00eb..2c66), resource_proof_response: None } to [(0906ba(00001001).., 206.189.22.44:39409)] 
[tokio-runtime-worker] ERROR 2021-06-18T15:28:55.997772+03:00 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\qp2p-0.12.3\src\connections.rs:300] Failed reading from a bi-stream for peer 206.189.22.44:39848 with error: StreamRead(ReadError(ConnectionClosed(ApplicationClosed(ApplicationClose { error_code: 0, reason: b"" }))))
[sn_node] ERROR 2021-06-18T15:28:56.000772+03:00 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\sn_routing-0.77.8\src\routing\bootstrap\join.rs:353] Invalid JoinResponse::Retry bad prefix: Retry(SectionAuthorityProvider { prefix: (1), section_key: PublicKey(0e88..758e), elders: {(9e2968(10011110).., 46.101.59.84:56270), (b5bdfc(10110101).., 138.68.153.41:46626), (c7511e(11000111).., 167.99.199.121:48299), (c796dd(11000111).., 206.189.22.117:59235), (f3838f(11110011).., 206.189.125.149:12000), (f3851f(11110011).., 206.189.123.68:44729), (f7686b(11110111).., 206.189.123.59:43633)} }) 
9 Likes

I love you maidsafe team!

15 Likes

This looking like it’s hung but perhaps is unfair on a new network without enough nodes?

safe wallet create --test-coins --preload 8000000

7 Likes

Unfortunately we are unable to establish a connection to your machine…

Not for me! But its expected, I don’t think I’ll go down the advanced route though :smiley:

4 Likes

try safe keys create --test-coins --preload 1000000 --for-cli

10 Likes

updated, thanks!

10 Likes

I also see this error, thought I was the only one! If others do I will add to the OP, thanks.

EDIT: I believe @yogesh is already on the case looking at fixing this :muscle:

10 Likes

For now I have 3 “bad prefix” and 2 “Retrying after 3 minutes” in logs.


Here are my usual test files:

safe cat safe://hygoyeymnywejbhhx6jkw5nhsd19ggtgxs4q6kwongyeo5jk9ap5x1e1p9h > 1.png
safe cat safe://hyryyyyjhxs65mpjyt9ihttd54b67fap7uq6s3cmbrjb5gswb3anwurkxyy > 1.flac
8 Likes

Woohoo… go on Armin.

instant png download, circa 1 minute download for the 8 minute song. We’re on to something here if this remains stable :wink:

18 Likes

Please share detailed download times, with size, and seconds. So we can compare it with others and get some overview about speed improvements. Thanks:)

9 Likes

Looks like an early finish for me from work today😂 well done team and @sharontannahill for keeping the cogs turning :clap:t2::clap:t2::clap:t2:

9 Likes

Yes! Great work team, I’ll be testing this soon.

6 Likes