Thanks, Spandan! It was a pleasure.
Great interview @ustulation! It was great finally getting to hear you and I have to say, it was nice to get a refresher on the XOR address space. Some great insights in this episode but a big take away was the birds eye view of the network. Fantastic stuff!
One thing I’m curious of is where uTP fits in. Haven’t heard that talked about yet. Maybe a good topic if there is another episode on Connections in Rust.
I enjoyed the discussion on connection tunneling and the complexities it introduces.
Accompanied my commute as always - excellent stuff guys!
@ustulation While listening to the podcast I thought of something I had not heard commented from the devs.
I had noticed that a number of people were connecting via VPNs and thus not using the NAT of the home router since they were tunnelling through it.
So in effect there is a percentage of the results that were more testing (effective) internet facing connections rather than NAT connections.
This will have skewed the results in my opinion and is an important consideration. How many from China were VPN connections?[edit removed china talk since that is all anyone focuses on]
Hmm - I naively assumed that the localizing was ip based and that vpn from China would then ‘not be a Chinese connection’ because the ip of the vpn would be based somewhere else in the world
Is Hong Kong behind the GFC? It is still a special part of China I thought.
Wouldn’t the Chinese VPN users appear to be outside of China though from the point of view of the test, presuming they set their VPN to go through a location outside of China?
The Chinese seem to try to change, bit by bit, that special status of Hong Kong (with a new bridge however, not a wall ). Partly because they are economically less relevant as in the past. China has now other cities/areas with a good economy, like neighboring Shenzhen.
However, for the time being, I see you can pick a server from Hong Kong, but not from (the rest of) China by at least 1 big VPN service provider.
Anyhow the talk of the Chinese is a distraction from the real question of all those who did use a VPN, to what extent did they skew the results. Whether the Chinese using VPNs appeared inside Hong Kong or outside China is immaterial to the real question.
I know in my case it only really worked with a VPN. Without the VPN, you could say “forget it”
I wanted to be difficult so I ran the crust test from behind two routers, three standard firewalls, and a virtual machine. Not sure if that skews the results??? So I wan’t too surprised by the symmetric NAT warning from Crust…
hehe thanks ! I heard it once it was published (i generally find hearing my recorded voice embarrassing for some reason ) and thought there were a few places i could have been more precise. Like in one of the places i say routing invariant requires 100% connectivity and on hearing it back i thought “ah mean in a section and with neighbours, not the entire network ofc” though i think ppl would have got it anyway and much later i did say a bit about disjoint sections in some other context and not everybody’s connected to everybody - so an inadvertent saving grace maybe
uTP is the layer that would add reliability and congestion control over UDP. So takes UDP closer to TCP. It’s difficult or impossible to use vanilla UDP in many cases where you require re-ordering detection, congestion control, connection-orientedness etc. So you need something on top of it to add those - uTP/UDT/QUIC/RUDP etc. are those “stuff” adding various bells and whistles to different degrees.
Once you NAT traverse via UDP one would normally wrap that into a uTP etc., and then use it for proper communication.
It shouldn’t skew much i would guess. Whether VPNs or not the peer will need to know your public IP allotted to you, does not matter from which IP pool - whether by your router/ISP/VPN. So that is the IP that’s taken into account while trying to determine a region. If the region says Country X for e.g. it does not really matter if you were originally in X or you VPN’d through X. What it says is that traversal through a router in X is successful/not-successful.
If you see, the address translation will be done by the VPN provider too. I am guessing VPNs even if you are connected to one currently, are not going to allow unsolicited communications (from ppl you have not contacted before) just like an ISPs router. So exact same mech are happening in that you are forcing a translation at the VPNs router to allow the connection to happen. I would believe VPNs could cause holepunching to fail rather than assist it due to multiple layers of translations that are going to be done by involving them than without (they are usually more complex depending on how much they want to annonymise you or bounce you around).
VPNs role would be more pronounced in cases where a country bans an IP though. So say you know my IP and that’s banned by your ISP. So obviously you can’t holepunch or direct connect to me (basically you can’t do any connection with me even if i were a public endpoint). Here if you go via VPN then you have better chances of success as your ISP sees you going to the VPN IP instead. Given the short amt of time the test ran for i don’t think any IP involved in the test would have been blacklisted by the govt/ISP for VPN to make any positive impact.
So give all those point ^^^ i think VPNs would have had an overall negative (though not much) impact on the test.
The issue is that you are not testing the home router but a well engineered VPN end point that may not even have address translation. Some do and some don’t. And then you are just testing maybe a couple of different router designs for connecting to the internet rather than the many different home routers out there.
So a few VPNs is fine but if many used the VPNs then I am wondering if it might have skewed things a little. Anyhow thought I’d ask. That was my question/concern about skewing.
Yes that is reasonable.
But then there are those of us who live in countries where the government records our internet accesses, at the ISP level, and so on average have a higher use of VPNs to negate this data snooping. Even Nord is advertising to Australians the use of their VPN to the ordinary Australian, so expect a higher use in AU. I could not get connections without the use of VPN.
Now SAFE will allow us to not use VPNs since the data gathering done by the government is useless to them and thus we no longer need VPNs which end up slowing things down a little.
Ah that’s true - so instead of multitude of home routers you are testing VPN’s routers etc.
Yes but connecting to the Internet (why which i assume you are meaning a publicly reachable endpoint) is very different than connecting to a peer. It’s not like if you connect to Google through VPN, then VPN will allow Yahoo to contact you. There will still be translation or at the very least filtering. Holepunching maybe should be more defined like punching through the router filtering or something because as you say Translation might not always be involved. So yes holepunching is still forging connections through the routers of VPN providers, but we might not be testing user routers if many are just converging to same VPN router.
That why one symmetric NAT router with random port/ip allocation at VPN causes failures for all going through the same VPN (which also means there’s some translation or mapping going on there)
Yes and the basis of my question.
I do understand the rest of the testing and testing VPNs is also useful and needed. Just the lack of home routers being tested. I was a (qualified) network engineer at various times when I was not being a real electrical engineer
Thus you might get a small number per type of (home) router being tested but a significantly larger group behind a couple of VPN designs. Thus a potential skewing of results.
Also VPNs work well for crust. Thus potentially we have a slightly higher success rate that might not be indicative of how people will connect their nodes to the safenetwork. I personally would not like to need a VPN so I could run my NODE.
Na that makes sense - hopefully not everyone used VPNs; well we have pretty varied results and good distribution of successes/failures so at-least thinking not everyone went through just a few same VPNs - as long as the VPNs or their bouncing ppl around were varying that’s OK too. That’s why a community wide test is good to get an idea of the average real world case. I suspect a few might have also done something with the firewall that might have affected the test - but that’s the idea, see what’s happening in practice and what are the results that come naturally instead of asking ppl to do something special that would cause improvement/degradation.
Some mundane comments from me:
After dev conference I was always thinking that “hey, this bright Spandan dude, don’t think I ever saw him on the forum”
and also, at other times I was wondering about that ustulation guy with the mighty beard style “was he not at the dev con? I must have missed him”.
So things fall in place, and world is more understandable again
I suspect this community may not be very representative of the real world average. Many people are here because the exact opposite is true. People here tend to care more about privacy, understand technology better, and probably use a VPN a lot more often than the average surfer.
My thoughts too.
@ustulation, maybe if there is another test or in alpha3 a couple of fields are added where the user when starting their node (or crust test) are able to enter their router’s brand & model and maybe if specially configured router/firewall for he likes of @jlpell
If they are using a VPN then brand is vpn company and model is the end point.
If they are not sure of their router/vpn then enter unknown in to the fields.
This would allow you to do some real statistics and know where work is needed in order to improve connectivity and to know the mix of connection VPN/home-router/business-router. For instance if you found ASUS routers or opensourced router s/w gave problems then you could work on those to improve things if needed.