Just a brainstorm, not sure if it’s a good idea but am looking for comments and criticisms.
Fast Ephemeral Routing
Summary
Routing efficiency can be greatly increased by reducing the amount of data sent in the response route. This would reduce the necessary requirements for running a node.
When requesting a chunk, the section responsible for the chunk responds with metadata that allows the client to fetch the data starting much closer to the chunk location, reducing the number of nodes that need to route the full data chunk.
Motivation
The current routing design responds by sending the full chunk data through every hop on the response route. This adds work to the routing nodes and delays the response to the client.
The proposal changes the response to contain only an Ephemeral Voucher (EV) that can be used by the client to (almost) directly request the chunk, saving the chunk data from having to traverse the entire route.
This design also improves the performance of caching.
Detailed Design
Ephemeral Vouchers
Responses contain EV data, which are a type of metadata that allows the client to know which nodes can supply the chunk and how to make a request for the chunk to those nodes. The process is:
- The client requests a chunk.
- The network routes the request to the closest section via Secure Message Delivery (SMD) and Reliable Message Delivery (RMD) via nearest neighbors.
- The elders nearest that chunk generate and store an EV, which contains
- A list of IPs:Ports of elder nodes in the section (to keep one degree of separation between the client and the adult nodes holding the chunk).
- The name of the chunk being requested
- The one-time ephemeral access code that allows the client to download the chunk
- The expiry time (only for use by the elders)
- The elders route the EV response back to the client using SMD + RMD via nearest neighbours.
- The client receives the response and uses the data in the EV to contact one or more nodes in the elder list.
- The elder checks the request has a valid ephemeral access code and has not expired, then passes it on to the adult nodes holding the chunk.
- The chunk is returned via the elders to the client, avoiding using nearest neighbours.
This has several speed benefits
-
Only metadata, which is lightweight and fast, is passed along the full route. The slower chunk data is only sent over a single hop, skipping almost the entire route.
-
The client can request from a single elder listed in the EV, and only make additional requests if that response fails or is too slow. This prevents the client from receiving multiple redundant responses via RMD. The client can choose between multiple parallel requests for speed with extra wastage, or single serial requests to reduce wasted data transfer but at the mercy of the speed constraints of the particular elder selected.
-
Any action taken by the client receiving the EV allows the elders to reward based on speed to the client (on the assumption the client acts on the first seen response).
Caching
Any node along the request route which has the chunk in their cache can respond to the client with their own EV. This means the client will be able to receive the data faster, and any lagging responses from further along the route would only contain metadata and can be discarded without having to include the entire chunk data in every discarded response. This greatly reduces waste and duplicate data.
Privacy
Routing is meant to assist in privacy by scrubbing IP addresses on each hop.
For this reason the client does not directly download the chunk from the node holding it, but instead acts at one-degree-of-separation away with the elders.
Privacy may be further enhanced by including extra encryption of the metadata so elders are not aware of the particular chunk being requested (only which node to send the encrypted metadata to), but that is not detailed here to avoid complexity of the EV mechanism.
Optimisation For Message Size
Routing must currently deal with message sizes ranging in the order of bytes (eg requests for data) up to thousands of kilobytes (eg full chunks being routed).
When EVs are used all messages are of roughly the same size, in the order of hundreds of bytes, which makes it easier to optimise the flow of traffic through routes.
Efficiency Analysis
TODO Equation for expected EV message size, with example result of calculation
TODO Equation for expected total bandwidth per chunk in current style routing with RMD, with example result of calculation
TODO Equation for expected total bandwidth per chunk using EV, with example result of calculation
Deployment
This is an extension to the existing routing and does not need to replace it. All the current work on routing is still required for EVs to function correctly.
EV Retention Period
Elders must retain EV information about each request in order to complete it. If the client never requests the chunk the elder should discard the EV, but how long should the elder wait until they discard it?
The elder needs to resolve the conflict between holding on to the EV for a long time and consuming memory vs discarding the EV too early and being unable to send data to the client.
Once the voucher has been used it can be discarded. This is to prevent the client repeating the request and putting excess load on the elder. The EV acts as a type of rate limiter, however the elder could possibly implement their own rate limit rules.
Drawbacks
There’s some possible complexity of trying to deploy this on an already live network, since nodes that have not upgraded would not understand EV requests. This might be an interesting thought experiment about how other potential changes to messages and routing might be possible in the future. BIP9 is worth looking at for historical context.
Alternatives
Use the current routing as-is.
Maybe the client could open their own ‘temporary server’ and allow the node holding the chunk to directly push it to them? This would probably mean the client IP is exposed to every node on the hop so seems unworkable from a privacy perspective. Maybe the first hop in the request route (ie closest to the client) could act as the recipient for the chunk and then push it to the client? The best way to ‘close the loop’ and ‘eliminate hops’ is an open question.
Maybe the need for EVs can be eliminated and clients can simply request closer to the data by some less time-dependent mechanism?
Maybe having many simultaneous connections for every client is adequate to reduce the routing distance? In a large network with long section prefixes this may not be possible.
Unresolved Questions
Which gives the best result: a) slightly more hops with a lot less overall data transfer or b) slightly less hops but a lot more overall data transfer? (See Efficiency Analysis above, lots of TODOs, any help is welcome).
The best way to referee elders to ensure they do not discard EVs too quickly leading to poor experience for clients.
Should this include a way for clients to clear EVs from elders so polite clients can help clear unused EVs from elders without needing to return the data?
Should this be used for PUT as well as GET? Seems like it’s possible.
Can EV spoofing be used as a kind of DOS?
What privacy implications are there? How private is the current routing and what changes does EV bring to privacy? What risks to privacy are introduced or mitigated by EVs?