Race for the right to a Farming Attempt? What's in the code


#1

Okay, time to get this straightened out. If it’s already been clearly answered, cool, but I really want he answer as to what’s in the code. So perhaps @neo, @dirvine, @Viv or others can clarify.

It has been represented that there is a “race” to see which data holder delivers up a chunk fastest, as determined by the chunk being received by the requestor, and only then is a Farming Attempt awarded to the serving-up vault.

From my understanding of how the network works, this could not be so for several reasons:

  1. For this to happen, each returning chunk would have to have metadata attached as to which exact vault served up that particular chunk at that time. This would be a potential security problem, not to mention all the added traffic, as per further points below.
  2. The Client receiving the chunks would then have the burden of assessing the winner of the “race” and then have to send a message back to the Data Manager group responsible for that chunk, noting which specific vault won the race. This would be additional traffic for EACH unique chunk of a file.
  3. Only then could a request be sent as a Farming Attempt on behalf of the “winning” vault, to determine if a safecoin is available at the targeted safecoin address.

I can see even more potential for additional traffic having to be generated to do it this way.

I suspect that the simplicity is that if a vault serves up a valid chunk in response to a request, the math-“lottery” is run to see if it gets the chance to submit a Farming Attempt. If that is successful, only then does it get to submit a Farming Attempt, and perhaps get lucky and earn a safecoin.

Might more than one vault earn a farming attempt or even a safecoin for the same chunk? Sure. But then, each did serve up a valuable chunk. This is the most atomic action of a vault, done repeatedly and often. The Farming Rate is the lever. Determining winners and losers or all these “races”, with all the developed traffic, can’t be the way it’s done.

Am I wrong?


#2

I don’t think it works like this. I think it will be fastest to return the chunk to the group that looks after the chunk, not to the end user. So no need for metadata.


#3

If I were to speculate, I’d say that a UDP packet would be sent to the data manager group immediately following the sending of the requested chunk of data, and it would actually be that UDP packet race that becomes important. That packet could hold the transaction number and the managers compare receiving times and declare a winner.

This would limit traffic to a minimum of a single packet, and keep the client from having to do additional work.


#4

Okay, but how does the group determine which of the vaults was successful without differentiating them? I may have the exact mechanics wrong, and I definitely don’t get the nuts and bolts behavior at the protocol level, but something has never made sense about this “race” thing. An authoritative clarification by @dirvine or @Viv would really be helpful.


#5

We have not implemented that part yet. There are many options.

  • The vault closest to the client address (not fastest but closest)
  • The client asks for the data from the first vault to respond with the Ok

Many more, but we do not have farming code in place yet. The notion is that the group will select the vault to send the data and it must deliver it in a short time. If it cannot it will be punished. So it may not necessarily be the quickest, but quick enough. As I say there are a ton of options there and none to difficult at all so no worry about it for us just yet. This all goes into place with test safecoin.


#6

Thank you, David. That really puts things in the perspective for those trying to explain “how it works”. General is best till details are finalized. :sunny:


#7

Not to overcomplicate , you would want the declaration logic of a winner to consider DTU Data Tranmission Unit size as this will be will be important to consider in the UDP uTP source/destination pair up to ensure decent service once the vault has won, so just being fastest to respond and nearby IMHO is not enough. I can say (based on working for small “c” cisco as far back as 1989 when they were 12M US and heading to their first 1B US in sales) generally routers on the Internet like to work in 4096 packet size to keep things quick…, so the type of transaction might be something to consider before deciding a winner , is it a record, chat strings, database record, picture, video stream, data BLOB, a bit torrent stream, etc… and how does that fit into the Vault’s bid response in its effort to supply disk space given the download bandwidth on the vault side of things…, and what DTU size is offered over IP by the UDP/uTP port making the offer to store, and does the vault have the CPU cycle reserved and Memory reserved to facilitate the download at a level accpetable to the App/person looking to store something on the net… maybe the App/person doesn’t care and its a best efforts (no SLA) , but then again maybe just maybe the App/person has rules that pick an SLA of parms to best fit what the App/person wants… given what the App/Person is doing on their end (they dont want to be resending,they have short time window, minmal resources, are running out of mobile power, etc…, just saying… I hope this is useful… What I would like to see is primary, secondary, exceptions use cases matched to test cases and metrics as part of a phased release Acceptance plan that is out in the Forum for feedback by all , for this very important part of the MAIDsafe network… I am happy to help out in this regard. Awesome project best R2


#8

For Utp and Tcp we use a stream approach, so the whole data item has to be delivered, regardless of MTU etc, Several packets (for Utp) may arrive concurrently from several nodes, but the first to send all the pieces will win. Utp has a congestion control (sliding window) similar to Tcp except of course it is datagrams as you say.

Yes I agree 100% with this and I see more of this happening as we go on, the RFCs were/are a good start and recently using forum threads to dig into some points during the last missing pieces/algorithms helps a huge amount. You will see @mav hammering some algos with simulations and ideas, it is gold for us really. Can seem like we are missing stuff or not thinking about everything, but that is cool. We don’t think of everything and neither will the whole community, but will think of more with more heads. so yes for sure we will try and make sure to be as open as we can and more so nowadays as the last pieces drop into place.


#9

OK thanks for the clarification re: UDP and TCP ( fyi- Van Jacobsen crank back/up can be problematic and sometimes needs to be turned off :slight_smile: ) , Sounds great. :slight_smile: David @ cisco in the early days we used a feature list to accumulate all input from developers, testers , customers, sales people, channel etc…, monthly and asked them each to priortize the list, HQ product management collated the info and then composite feedback re- ordering of that list which drove the company for 5 years to the top of the heap we really did listen to our users and customers, they always came first in those days, the detailed RFCs made their way to the IETF after internal dev/test/QA sessions together with biz management, where the Draft RFCs were used by cisco to re-shape the battlefield continuously (BGP comes to mind with Greg Satz driving that one in the BGP2 to BGP3 transition for cisco), personally the Python community model works better for me, you need a benevolent dictator to be the trimtab on the rudder, to fine tune priority into a sane roadmap and architecture, to avoid what can be endless consensus forming debates, which can really slow progress, I 'm sure you will navigate this well, having someone own each RFC as the sheppard so to so speek is a good thing… best R2