Test 12 gave us a lot of valuable information and exposed several bugs. We addressed the following issues, both to fix the problems we knew about and to improve the usefulness of the logs, and are starting an updated test network today: Test 12b.
- We reduced the amount of log messages, as they were far too detailed and created several gigabytes of log files over the course of just one day. In particular, Test 12 was logging every single message that was being sent.
- On the other hand, we added a bit more information to the logs in some places, to get a closer look at the points where we still suspect issues. For example, there were several
UnexpectedStateerrors and we now print what state that peer is actually in (connected to us, through a tunnel or not, in the routing table or not, etc.). Similarly, when we receive a message from an “unknown peer”, we now print the message itself. And in addition to the node’s name, every line contains the current section’s prefix.
- The hole punching mechanism is creating lots of sockets and can reach the file descriptor limit of the operating system. We disabled it completely and send only static information (IP and port) to the other peer for now. This is just a temporary change, of course, to help us debug other parts of the system that are unrelated to hole punching problems.
- Because of that and since we saw a lot of tunnelling in the logs, indicating that a lot of nodes had failed to establish direct connections, this network will reject any nodes that do not accept direct TCP connections from the outside. These connections can be direct, port forwarded or obtained via UPnP.
- We improved the logging for resource proof on both sides: It now indicates more clearly if you were not able to join because of ongoing churn, as opposed to failing the challenge. And the nodes themselves print the status of their current candidate periodically every 60 seconds. The log level for rejecting candidates was lowered.
- More logging was added for the routing table recovery mechanism that periodically restores consensus in every section about the network’s structure.
- The timeout for retrying if you failed to join was increased to five minutes.
- The network now tries to balance itself: if one part of the network already consists of much more sections than another part, it won’t accept new nodes until the other part catches up. That further limits the number of vaults that can join per minute, but it should increase the network’s stability.
- In some cases,
SectionSplitevents were not passed to vaults. That didn’t have any consequences but printed an error message every time.
Test 12b will be a quick test network. We’ve taken our best guess to try to address the problems with Test 12, but it’s possible that there might still be issues with Test 12b. If this is the case, then we’ll have a much better idea about what caused the issue, since we improved the logs in places where we suspect issues might happen. If it turns out that the problem is fixed, then we’ll continue to let Test 12b run, at least until we release the next set of updates.
Also, lots of people who were able to join Test 12 (with SAFE Vault) might not be able to join Test 12b. The burden of relaying on other nodes was bringing the overall performance down, so we made changes in Crust to block nodes from joining if they do not accept direct TCP connections from the outside. As hole punching / further protocol support gets added in Crust, the number of people “directly connectable” will increase vastly. For now, with this updated network, only people who either have a TCP port forwarded and specify it in their Crust config file OR people who have UPnP enabled will be able to participate as a vault.
If you see this message, it’s because the network is not able to reach you. You’ll need to forward a TCP port or enable UPnP on your router.
If you forwarded a TCP port on your router (see this website for more information on how to do this), you need to specify the port in the
safe_vault.crust.config file (which is located in the same directory as the
safe_vault binary) in the
Here are the download links for Test 12b: