Rough plan for new RUDP API and long term future for RUDP

Features
1

https://groups.google.com/forum/#!topic/maidsafe-development/KfVo-d5Eyx8

David suggested that I post this internal memo onto this public list so the community can see our technical deliberations on the upcoming future of the Maidsafe platform in public and decisions are transparent.

I should explain some context to help readers not versed in this field: RUDP is one of the oldest parts of the code base, and it was originally designed and written by Chris Kohloff, the same engineer behind Boost.ASIO (Boost.Asio - 1.56.0) which is the de facto high performance networking library for C++, and is expected to be standardised as the C++ 17 networking implementation. As a result, its pedigree is second to none.

However, Chris designed RUDP before any of the rest of the Maidsafe design was finished, and he aimed it at a target which ended up never existing. Today’s Maidsafe mostly misuses RUDP and it more just happens to work at all than work well. In particular, the rest of the Maidsafe codebase is threaded, while RUDP is not; RUDP was never designed to tax much more than a single CPU with a 10Mbit network connection let alone multicore CPUs with Gigabit ethernet or even the 4G mobile networks imminent; RUDP’s present design is woefully power inefficient, and blows away the battery on a mobile device in almost no time because low power consumption wasn’t considered; congestion control - this being the algorithms which let RUDP coexist well with other traffic using the same network - never worked well, and has been disabled for a long time now; and the lack of peaceful coexistence with other network traffic makes Maidsafe traffic appear like a DDoS attack, which invites the wrong sort of attention from network providers such as ISPs and mobile networks; lastly we also have a resource management problem in that currently RUDP will accept as much data as you feed it, even if the network will never catch up, which eventually causes us to exhaust RAM.

The below memo sets out a rearchitecture of RUDP to fix all of the above problems. A key theme is “zero copy networking”, you can read more about this at Zero-copy - Wikipedia. Also, reading that ancient-but-still-useful site the C10K problem at The C10K problem is worthwhile, just replace C10K with C100K on modern hardware.

Any thoughts on the below are welcome.

Niall

So rudp will need to be rewritten?

…

will project safe still be live this year?

5 Likes
2

From the very end of the message:

I would also suggest that RUDP might be made into a Boost library, and submitted to Boost. That might actually elicit some free help as a zero copy RUDP implementation based on ASIO ought to be extremely popular with many. And besides, help in maintaining it into the future would be useful.

Sounds fair that some assistance could be rendered by Boost…I hope they get many clever people to chip in.

They’ve done the right thing going public, if it slows release…so what, RUDP needs to be awesome or it could severely affect network performance by the sounds of it.

Let’s not beat up on the dev team with this setback…agreed? It’s admirable that Ned spoke up about the weakness of the protocol, it would be an easy option to stay silent and move onto the next project.

The project will be stronger for it, I just hope they get the outside support.

5 Likes
3

yeah, i would like to know if the current rudp will be used for go live and be updated over time or will the “middle” layer of rudp need to be written before going live. If the latter, how long will it set back the beta release?

My opinion doesn’t matter; who cares. Just would like to know the answers.

4 Likes
4

As far as I can tell from the quoted part, it means the current SAFE network (using the current RUDP) may not work well on mobile and some other devices.
That’s fine. If it can work acceptably on some devices, let’s use those devices first and aim for mobile in a future version when RUDP gets rewritten.

4 Likes
5

Janitor - I agree given the assumptions you make.

More than likely this will impact on investor confidence and Safecoin price and awaken a troll army. Ultimately general confidence in the project will be affected. These are the realities I think. The devs seem best placed to make what is essentially a judgement call. I would caution against taking a “so what” attitude though, although I fully understand your feelings. I would give the devs all the time in the world, I’m just cautioning that delays could ultimately damage the project, so the decision should not be taken lightly - look at Mastercoin etc.
I’d say delay if necessary, but don’t do it lightly. It would be helpful if a dev could hazard a guess at what kind of delay we are talking about and confirm Janitor’s assumption.
Cheers

2 Likes
6

Hear, hear!

Having some experience in this field, let me just add one more thing: if the first release get pushed back because of RUDP, then I bet there would be additional delays because more things will be found not to work (well).

I read some dev posts on Google Groups and I can tell you (despite that recent comment elsewhere on this forum that MaidSafe isn’t a complex piece of s/w, that’s both true and false at the same time) the complexity of this s/w isn’t fully appreciated. It’s not complex in the sense that its components work are known technologies, but when you put a bunch of such fresh technologies together in a novel & unique way, crap happens.

I’ve no doubt v1.0 will be released with a long list of limitations and that’s how it’s suppsoed to be. Only poorly QA-ed and badly documented products do not have one. I understand that MaidSafe is not supposed to be that kind of product.

EDIT: Security is the only part that must be done as good as humanely possible. The rest needs to provide reliable performance in at least 2-3 desired configurations. If only Linux systems that are permanently connected to the Internet work well, that’s okay. If need be call it a limited release.

7

So with:

RUDP was never designed to tax much more than a single CPU with a 10Mbit network connection let alone multicore CPUs with Gigabit ethernet

and

RUDP’s present design is woefully power inefficient, and blows away the battery on a mobile device in almost no time

Your average DSL connection with home Desktop/Laptop should not be a concern for now…but with:

the lack of peaceful coexistence with other network traffic makes Maidsafe traffic appear like a DDoS attack, which invites the wrong sort of attention from network providers such as ISPs and mobile network

That sounds like a situation that should not be allowed to make it into a Beta release. I dont think you would want to incur the ire of the ISP community straight up.

1 Like
8

No this wont affect the beta release in any way. What you will read is hard Engineering talk and continually. We will never be happy until its perfect, but we will release for sure. On the dev list you will see us discuss many issues directly like a recent refactor of encrypt (and currently drive) to make it incredibly solid in the manic tests we run, but at a cost of file size limitation. This is overcome with a custom mapped allocator for the STL container we will use there (vector).

In terms of RUDP, what we have is way good enough for the network, what we will have will be much closer to perfection, but thats a few months or perhaps a year away. It wont slow us down. Routing vaults and all will get the same treatment as we go along. The important thing is a solid future proof solution.

We are looking at zero copy libraries, offload to gpu and much more. These will all be great, but not required to begin with. These things would be targets for any Engineering team that want the most efficient solution. So no panic folks, this is all good and we are looking to put much more of the internal design chats on line for everyone. If there were a problem it would get posted in the community for sure in plain English first.

Be ready to read a lot more of these things, and remember Google change 50% of their code base a month (see here), software dev is change like this. We just want to be transparent and at the same time future proof where we can. So yes look for perfection, but release.

What we have in place is probably some of the most efficient c++ code around, its very strict and tested with everything imaginable. We only get there with looking continually at much better ways and implementing them. Not by delaying though, that’s no real option for us just now.

For instance last week and the start of this week was transitioning workflows to Atlassian Jira (which they have provided a 2000 user license for us FOC). That again is a change, does not mean we were not managing, just now we can manage better s the team grows, especially the pods.

On mobile we are starting a team on Sep 1st specifically for mobile dev, so that will have soem serious focus now, with Viv of course leading the charge there. Mobile will be a big area, that is one reason we went for huge packet loss capable rUDP, Niall has done amazing work there.

rUDP 2 will be something we move to boost and is practically a rewrite (some small parts remain). It wont mean we wait on it though @D

9 Likes
9

Yes that was an old congestion control algorithm that would not work anyway as it would prevent hole punching (NAT traversal). We dont do that anyway and limit connections to 64 for vaults and 8 for clients. This is way less than many other systems and ISP friendly :slight_smile:

In terms of threading etc. Chris Kohloff implemented some fancy re-entrant code (BTW Chris is the designer of ASIO and helped us a lot with what he did for us) which means that getting rUDP on a core and giving it that one thread looks pretty good. In some tests faster than mutual exclusion code. So thats all cool too :smiley:

Betcha we will frighten more folk as we move along, but the proof will be in the steady improvements we will make as the years go by. We core dudes need to work fro our safecoin after all :slight_smile:

5 Likes
10

haha all good, thanks for taking the time to clarify.

1 Like
11

Pleasure Chris. I doubt many security communities will get this level of transparency, this should get even more open as we move along.

3 Likes
12

Agh I was just about to go to bed … but I need to clean up my own mess first before it gets blown out of proportion …

Yes RUDP needs to be rewritten. Maidsafe brought me onboard five months ago precisely to do exactly that task. Before I could get started though, I had to invest a not much fun five months into kicking the old RUDP codebase repeatedly so it would be fit for TestNet, and if needs be, a beta launch.

Some of the many and mission critical problems I have fixed over the past five months include:

  1. I spent the first three months or so sanitising RUDP and Routing for all race conditions using the clang ThreadSanitiser. That was quite tedious.

  2. Simultaneously I built out an automated build, merge and test solution based on Jenkins at https://ci.nedprod.com as I knew there would be a lot of 24 hour soak testing coming. I keep my own forks of RUDP and Routing to not disturb anyone else’s productivity, and every night Jenkins merges the Maidsafe copy with mine and soak tests them on Linux, BSD, Windows on x86, x64 and ARMv7 for about four hours. It also runs a clang thread sanitiser pass and a valgrind pass to prevent any surprises creeping back in.

  3. I added packet loss simulation about six weeks ago as a precursor to getting into congestion control, but discovered much to I think all our surprise that RUDP didn’t handle packet loss at all during connection bootstrap, and simply killed the connection on the first packet lost. This was despite RUDP being extremely resilient to packet loss once connected. That meant even tiny amounts of packet loss whacked RUDP’s ability to connect significantly, and that generated problems for TestNet stability and creation times. So I diverted to getting connection bootstrap to retry before anything else.

  4. I think it was only last week I got connection bootstrap to be 100% reliable under packet loss - it was slow going as I was highly distracted by the recruitment drive which brought on two world class engineers with a third possibly coming soon, however equally when you’re down to less than 1% failure rates that means the CI has to iterate 100 times per failure found, and the more you fix the longer it takes to get another failure.

  5. I then started into a new SYN cookie implementation as the old one was disabled, as the new bootstrap retry code made RUDP highly vulnerable to session hijack and being DDoS’d through packet flooding. This new DDoS implementation is currently 99.2% working, however 0.8% of the time it is falsely tripping due to a long standing ancient bug in RUDP where somehow it rarely tries opening some connections twice, and the DDoS protection code thinks it’s a connection hijack attempt and kills it. Once this part is fixed, this vastly improved RUDP v1.0 will be ready for engineering review and being merged into the mainline code.

And that fairly solid RUDP v1.0 is what will power the TestNets, so no slowdown will be caused by our subsequent rewrite of RUDP into a totally new RUDP v2.0. In fact, our next few weeks will be spent making RUDP v1.0 use the v2.0 API and adjusting Maidsafe code to fit, that way everyone is singing from the same hymnsheet. RUDP v1.0 is a brittle piece of code with poor performance, but after five months of work on it by me it is now good enough to ship in a product. Why we need to replace it anyway is (a) maintenance costs long run (b) performance, both in transfer and power consumption (c) RUDP is not a core competency for Maidsafe, it should be donated away to someone else to maintain (d) the current code drags down the rest of the Maidsafe codebase which was written using a very different idiom.

Just to make my point very clear here, Maidsafe management spotting this coming crunch a long way out - over a year ago. They then took advance measures to make sure it will not be a problem long before it will become a problem. All that is on track. Maidsafe’s management is very solid, this sort of early action to ward off coming car crashes is exactly what separates successful startups from the majority. And I say that as an Affiliate Researcher in a leading Complexity Research Institute, indeed I was only discussing with its senior researchers the geopolitical consequences of the Maidsafe vision last night.

Niall

12 Likes
13

RUDP v2.0 will use a very similar, but not quite identical, wire format to RUDP v1.0, mainly because the new wire format enables zero copy throughout. We hope to be finished RUDP v2.0 before the product ships, but if not a compatibility layer for v1.0 RUDP testnets would not be difficult. RUDP declares its version during connection bootstrap, so the wire format can be easily upgraded.

Niall

4 Likes
14

Every night I run the unit test suites on an ARM Tegra K1 board. This is representative of mobile parts hitting the market this year. It works very well, but does suck down 15w of power.

That was me who made that claim I think. I didn’t claim it isn’t complex, because it is, I did claim the technologies used are conservative. Little in here isn’t from the 1990s or earlier, apart from the language and toolsets.

Right now RUDP v1.0 is so slow it wouldn’t threaten anyone. This is because in the absence of congestion control, its maximum transfer rate has been capped very low, at no more than 2Mb/sec per connection, and usually less than that again. We could dial back up those settings, and indeed as people can recompile our software themselves they could do that themselves to get far higher speeds. This is our worry, because if enough people make a nuisance of themselves, ISPs will simply block all RUDP packets.

Indeed very true. Right now all the encrypt and decrypt sucks down a lot of battery because it’s done by the CPU. Offloading that to the stream compute (OpenCL) on next gen phones will hugely reduce that cost. And RUDP v2.0 is being designed to deliver data straight from the network interface into the graphics and video hardware with minimal CPU involvement. Compare this to RUDP v1.0, which currently copies data at least three times between the network card and it leaving RUDP for Maidsafe code to use. Every one of those memory copies costs battery, latency, and slows down everything else on the device due to LL cache loading.

Niall

2 Likes
15

You’re right. I couldn’t find the post so I wrote from memory. Sorry I misquoted you.

1 Like
16

This troubles me, isn’t that bandwidth for Vaults directly related to being the first to deliver a chunk and thus get a SafeCoin farming attempt? If so, the incentive to recompile with less conservative settings is extremely high.

Is there any update concerning RUDP 2.0? Any chance it’ll be ready for release?

17

It is likely three layers, the lowest being crux (see GitHub - maidsafe-archive/MaidSafe-CRUX: Connected Reliable Udp eXchange ) this may well be in for launch (fine balance), but unlikely the upper layers will make it. Its a lot of work, but being really well tested and documented as it will be posted fro a boost library and their standards are extremely high (a good thing).

The intention for us will be a plain UDP header with fully encrypted (we can AES at over 300Mb/s so OK as we can also parallelism that) payload and as we also currently use variable port addresses then security is very high off the bat. We do have to be aware of dos attacks though as we need to decrypt all incoming packets, we do have firewalling / filtering capabilities though so at least this is minimised.

4 Likes
18

The bottommost replacement layer, CRUX, is proceeding well. It should, eventually, be able to switch out the underlying transport protocol, so if your ISP blocks one transport it’ll autonegotiate another.

The middle layers have returned to the drawing board. I had been working under the assumption of a zero copy stack with crypto, hashing and message reassembly offloaded to the GPU via OpenCL, but last week I found the time to do out some benchmark prototypes and was quite surprised at how little benefit we’d get, especially with CPUs with dedicated offload hardware for crypto and hashing where the CPU is unbeatable and then some. Such CPUs will be commonplace in both mobile and desktop within three years, so it is a poor cost benefit return.

I concluded, with regret, that the zero copy stack via the GPU wasn’t worth the disruption. Shame, as we would have kept user data entirely in the kernel such that Maidsafe code never even saw it, making exploits to steal user data extremely hard.

What we’ll probably do instead is far simpler than the original plan, and the timescales should adjust accordingly. Essentially we’ll probably have the CPU do what the GPU would have done. CRUX only provides reliable datagrams, so the middle layer between Routing and CRUX merely needs to scatter and gather messages into datagrams whilst simultaneously doing crypt/decrypt. In theory, shouldn’t be more than a few weeks of effort, the harder part will be deciding whether it’s worth making that part a Boost library or not.

I’ll be jumping on board Routing to help out for a bit this week before christmas vacation. I think it’s going well overall, and less refactoring usually means less timetable slippage. I am also glad we didn’t embark on work with little cost benefit, a startup can’t afford such opportunity costs.

Niall

3 Likes
19

Some recent update on their quic protocol of the google/chromium folks.

1 Like