Certainly true in general terms, however - after launch I think its a reasonable assumption that many folk, not just in this forum, will be uploading 24/7 for a while. We all have stuff we want to see on the perpetual web and are happy to pay for that. So I think stress-testing with many many uploads is reasonable to attempt to mimic what will happen in the first days and weeks after launch.
1 Like
I’ve gone slightly ‘off piste’ and done something beyond what you were wanting to run this test for. I’ve started multiple ‘safenode’ from 1 AWS Instance. I saw that the RAM and CPU wasn’t much at all for running a safenode - about 300MB. So I started another one. Then another…
I have 10 running on this AWS T4g.medium (ARM, 2 Cores, 4GB RAM) and it is by no means stretched. Not sure that would be the case if there were a lot of writing on the network or if the number of records per safenode weren’t limited to 1024.
It took less than an hour for all 10 nodes to be ‘full’ in terms of having 1024 records on them and between 250MB and 287MB on each. However, there is 1 that seems to have failed. It only has 258 records stored and 89MB. Looking at the log it only had entries for the first 7 minutes and nothing after that. It has these errors that I’ve not seen in the others:-
[2023-05-29T16:10:15.971880Z WARN sn_networking::event] MsgReceivedError: ReceivedResponseDropped(RequestId(12))
With this as the last one:-
[2023-05-29T16:15:54.480247Z INFO sn_networking::event] Connection closed to Peer 12D3KooW9vaN3N1khU58jcbkLGtcvE8Giu3QSLWdBmdQjR6Js4eh - Listener { local_addr: "/ip4/172.30.1.153/tcp/43669", send_back_addr: "/ip4/3.95.181.231/tcp/44008" } - Some(IO(Custom { kind: Other, error: Error(Closed) }))
I ran all the nodes in ‘screen’ so I can see if there are any errors displayed and this node has nothing.
Attached is the full log from that node in case it is useful.
safenode.log.zip (12.7 KB)
6 Likes
I am failing to get connected with the port forwarding
willie@gagarin:~/.safe$ RUST_LOG=safenode=trace ./safenode --log-dir=/tmp/safenode --root-dir=/tmp/safenodedata --port=12000
Removed old logs from directory: "/tmp/safenode"
Starting logging to directory: "/tmp/safenode"
Node is stopping in 1s... Node log path: /tmp/safenode
Error: We have been determined to be behind a NAT. This means we are not reachable externally by other nodes. In the future, the network will implement relays that allow us to still join the network.
Location:
safenode/src/bin/safenode/main.rs:235:36
no logging either, which is strange. Have I made a daft error?
willie@gagarin:/tmp/safenode$ tail -f safenode.log
tail: cannot open 'safenode.log' for reading: No such file or directory
tail: no files remaining
This is my port forwarding setup. Do I have this correct?
The UDP is redundant now, yes?
3 Likes
Looks good to me.
btw, uppercase -F would be the equivalent of --follow=name --retry whereas lowercase -f will stop when the log rotates.
You probably know that but it may be helpful to someone.
2 Likes
No I was not aware of that -F flag, so thank you for mentioning it
Very helpful 
2 Likes
If you can’t see the log file I wonder if you have the same thing as me?
- Ran a node
- cd to /tmp/safenode to tail the file with another shell session
- killed the node
- exited the tail -f
- ran a node again
- tried to tail -f the file
- it says there isn’t one
I realised that the safenode removes the directory. My other shell session was stranded in a directory that no longer exists. Linux doesn’t handle this brilliantly and just says there is no file. You can tell if this has happened by doing cd … which will also fail because the … is no longer there! But a full path cd to somewhere else will still work.
2 Likes
Not shure what I am doing so just tried with limited succes (?).
Windows from home with a fixed IP adress. do not understand how to set the network adress.with the export comand as seen in the screenshot. shure more people have simular problems connecting…
2 Likes
Yes the Windows equivalent for export would be useful to know.
1 Like
Maybe env?
2 Likes
Also set works IIRC
2 Likes
Yes, up running now for almost an hour after replacing EXPORT with SET:)
230mb in tmp directory
[2023-05-30T12:24:23.348521Z INFO safenode::network] Node (PID: 31288) with PeerId: 12D3KooWCMbPzVHJ4VpbvswE8sVKRfsusZiDyfFsRyPDdEyCP6z1
but faild to upload
safenode.log.zip (404.3 KB)
7 Likes
I think you may need double quotes around your full path since it contains a space, else it will treat it as another argument to safe CLI (invalid).
2 Likes
Tested again:
but anyway glad to get a node up in the tests for ones
Seems to be 1000 of ways using powershell/CLI/CMD so feels better after beeing a part of it after many atmpts
1 Like
dude you are having whitespace problems in your path names… see the folder structure… “egna project” and the error line only beginnging to complain about a path name named “project…”
this it industry will never learn from its half century old problems and the like. and they dont teach new people on best practices
no use that we have utf whatnot and character sets encodings and all when mere command line parameters and whitespace ruin everything just the same.
this industry seriously sucks.
2 Likes
It is “set” in windows - been around since PCDOS days. Also reused from DEC mainframes
x> set env_var=var
And yes for the filename in the command it needs quotes because of the space. CMD sees the space as separating two parameters to the command
Having said that the set value doesn’t want quotes because it will include the quotes when setting the environment variable
So now was tested nat, and how far or when will be tested payment? Thx
2 Likes
Powershell 5.1 / 7.3+:
[Environment]::SetEnvironmentVariable('SAFE_PEERS', '/ip4/206.189.29.202/tcp/38379/p2p/12D3KooWN7AJe7wxVckZ6SF3ybYXVgetQ9hkYWHCHLokpSRC31W2', 'Machine')
Note: Depending on the program using this environment variable and how its launched (under a service as a child process, or just a normal invocation under the logged in user on Windows OS), it may or may not require a restart on Windows OS to have the new environment variable show up in the PEB context of the user program.
1 Like
I’ve been poking around a bit and thought I’d share some of my learnings.
I was wondering a couple of things:-
- how many connections to other nodes are made from a safenode?
- what the size of the network is?
I figured there would be a way of parsing the log file to these but I realised the first one is fairly easy. ‘netstat’ has all you need really.
If you run:-
netstat -p | grep safenode
You can see the network connections that safenode is making to other IP addresses.
You will get output like this:-
tcp 0 0 ip-172-30-3-146:52706 159.65.56.79:46143 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:35590 172.30.1.197:40693 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:59096 ec2-3-95-181-231.:45187 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:50440 167.172.56.211:38457 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:38850 10.131.0.3:35283 SYN_SENT 11333/safenode
tcp 0 0 ip-172-30-3-146:47560 ec2-54-90-148-47.:34989 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 138.68.151.253:36460 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 152.67.100.216:36790 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 172.30.1.197:34490 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 188.166.146.13:49130 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:55766 172.20.20.22:65352 SYN_SENT 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 172.30.1.153:40848 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 138.68.166.11:58058 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:35674 172.30.1.197:42495 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 206.189.29.202:60246 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:58212 192.168.0.192:12005 SYN_SENT 11333/safenode
tcp 0 1 ip-172-30-3-146:52358 10.16.0.6:35283 SYN_SENT 11333/safenode
tcp 0 0 ip-172-30-3-146:46820 138.68.151.133:38269 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 172.30.1.153:36602 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:42790 176-58-124-107.ip:38473 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 138.68.151.132:59580 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:60172 209.97.183.87:41925 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:54632 138.68.151.93:43469 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 138.68.144.233:38612 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:40890 ec2-3-95-181-231.:35377 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:37300 172.22.240.1:65352 SYN_SENT 11333/safenode
tcp 0 1 ip-172-30-3-146:59710 gen-065-030-036-2:12005 SYN_SENT 11333/safenode
tcp 0 0 ip-172-30-3-146:52668 148.25.87.109.tri:16012 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:55366 10.16.0.6:36765 SYN_SENT 11333/safenode
tcp 0 0 ip-172-30-3-146:53836 172.30.1.197:42573 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 172.30.1.153:56222 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:56058 m5-242-169-12.cus:65352 SYN_SENT 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 172.30.1.153:48684 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:33758 188.166.144.67:45655 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:35754 138.68.151.223:46803 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 ec2-54-90-148-47.:42452 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:36544 192.168.1.100:12001 SYN_SENT 11333/safenode
tcp 0 1 ip-172-30-3-146:51968 10.131.0.3:36765 SYN_SENT 11333/safenode
tcp 0 1 ip-172-30-3-146:60388 82-181-223-188.bb:12001 SYN_SENT 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 138.68.152.226:45846 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 138.68.179.78:59094 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:50780 static-47-207-67-:12001 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:45738 159.65.95.30:36335 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 h-155-4-232-199.A:57616 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:51568 172.30.1.153:40235 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:44688 mail2.claritypart:12000 ESTABLISHED 11333/safenode
tcp 0 65 ip-172-30-3-146:39981 138.68.151.254:39124 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 172.30.1.197:60006 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:35080 188.166.144.67:45655 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:36360 142.93.34.220:44093 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:46920 ec2-54-90-148-47.:37833 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:55534 192.168.56.1:65352 SYN_SENT 11333/safenode
tcp 0 97 ip-172-30-3-146:39981 134.122.106.248:51566 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 188.166.151.219:59012 ESTABLISHED 11333/safenode
tcp 0 1 ip-172-30-3-146:55534 192.168.56.1:65352 SYN_SENT 11333/safenode
tcp 0 97 ip-172-30-3-146:39981 134.122.106.248:51566 ESTABLISHED 11333/safenode
tcp 0 0 ip-172-30-3-146:39981 188.166.151.219:59012 ESTABLISHED 11333/safenode
The string in the 3rd column will be your own network address which in this case is an AWS style FQDN starting with ‘ip’ because this machine is an AWS Instance.
The string like ‘188.166.151.219:59012’ is showing the address of the remove safenode that yours is trying to connect to and the port.
In the above output you can see that some of the connections show as ‘ESTABLISHED’. They will be the live ones of your safenode connecting to other nodes.
The ones that say ‘SYN_SENT’ are your safenode trying to connect to an address that is inaccessible. This is the thing that this testnet was all about trying to determine the handling of. Look at this one for example:-
tcp 0 1 ip-172-30-3-146:38850 10.131.0.3:35283 SYN_SENT 11333/safenode
That address of 10.131.0.3 is only ever going to be accessible on the same network as the machine running it so it is useless my safenode trying to connect to it.
So it looks like my safenode has 45 working connections to other safenodes:-
sudo netstat -p | grep safenode | grep ESTABLISHED | wc -l
45
If you are running multiple safenodes it will show the output for all of them. You could limit it to just one of the ‘safenode’ processes by running:-
netstat -p | grep <process number>
where the process number can be determined by running:-
ps -ef | grep safenode
you will get output like this:-
ps -ef | grep safenode
ubuntu 11333 11018 0 May26 pts/2 00:10:22 safenode --log-dir=/tmp/safenode --root-dir=/tmp/safenodedata
and it’s the first number shown in the output that is the process number.
—
The next thing was wanting to estimate the size of the network. Looking at the safenode.log file there is a line like this added periodically:-
safenode.log.20230530T213247:[2023-05-30T21:28:01.005527Z INFO safenode::network::event] identify: adding addresses to routing table peer_id=12D3KooWJk7Hg2uZoF6xaVpms3Q6RJmqHLzmycNUoztGadh72rhP addrs=["/ip4/3.95.181.231/tcp/43265"]
adding addresses to routing table peer_id
So this seems to be a node being added. However, this message appears multiple times in the same log file something between every few seconds and every few minutes. So it’s not really a node being added to the network but your safenode connecting to it. But it seems that every connection to it will cause an entry to the routing table and an entry in safenode.log.
So the things to do is look for unique entries of this node being added after sorting so that .
A new logfile is created every so 5000 lines and sometimes it will just be a few minutes between new log files or more than an hour when there is less network activity. But it appears that the log files last long enough for a lot of network connections to happen. So if we look for:-
‘adding addresses to routing table’
and count the unique entries of just the peer_id and the string part we can get a rough estimate of the number of nodes in the network:-
grep 'adding addresses to routing table peer_id' safenode.log | awk {'print $10'} | sort | uniq | wc -l
But the file safenode.log could have only just been rolled so maybe do it on one of the older ones eg:-
grep 'adding addresses to routing table peer_id' safenode.log.20230530T025714 | awk {'print $10'} | sort | uniq | wc -l
‘uniq’ outputs unique lines but we need to do a ‘sort’ otherwise unique lines are only removed if they are adjacent.
Doing this against a few log files I see between roughly 20 and 30 unique entries for peers in the log files. I’m guessing the size of the network is between those values.
I could be way off track with this and a single node doesn’t have entries for all the safenodes in the network but I think it can’t be smaller than this.
This is all just for fun at the moment because I’m sure a full network won’t have communication with all the nodes and even the size being determinable would be a threat. Maybe ultimately the size will be unknowable!
I also ran this to see if there was a pattern over time by looking at all the log files I had for this node because it has been running for a few days:-
for i in /tmp/safenode/safenode.log.* ; do echo $i ; grep 'adding addresses to routing table peer_id' $i | awk {'print $10'} | sort | uniq | wc -l ; echo ; done
I found a period on Monday when there were a lot less peer_id entries which puzzled me. I realised this was when I’d been adding more nodes. I think this will have caused a lot of log file entries on the node I was already running (and probably other people’s as well) so the 5000 lines of the log file were used up really quickly before there had been as much communication with other nodes.
–
Edited to add ‘sort’ before ‘uniq’ in all the commands courtesy of @vort
and therefore a correction to the numbers found
and the day on which I added more nodes - Monday instead of Sunday.
11 Likes
I wonder if Safe Network is targeted at globally accessible nodes only.
If answer is “yes”, then there is no need to even try connecting 10.*.*.* - addresses from reserved ranges can be filtered out without making connection attempts.
Your results are wrong because you have mistake in your command.
uniq removes only adjacent duplicates.
To remove all duplicates, you need to sort lines first:
grep 'adding addresses to routing table peer_id' safenode.log.20230530T025714 | awk {'print $10'} | sort | uniq | wc -l
Which will give 10-20 times less entries.
2 Likes
2 Likes