No Dev here 
. Watch the videos. There are more of them. What I get from it is that the DHT get’s updated really really fast. And cause of constant checks, a close group would notice an offline node in milliseconds or seconds at max. I’m not gonna poke David because it’s weekend and he actually explains it in these videos. Takes some time to really get it though. It’s quite technical stuff.
2 Likes
Oh hey, I just noticed this. It’s incorrect because we’re not looking for losing the same chunk as one specific chunk, but we’re looking for any chunk that has two copies lost at the same time (basically, the birthday problem.)
By the way, I don’t believe chunks would be spread evenly. Things like this tend to be organized more along a log-normal (or similarly heavy-tailed) distribution (yes, “citation needed”):
.
The problem with heavy-tailed distributions is that they will make it hard (probably impossible!) to estimate how much of the data would be stored on the majority of the nodes compared to the (possibly very) few nodes in tail, very far from the average (i.e. 4-5 monster nodes may end up holding 50% of all the chunks.) I’m not saying that’s likely to happen, but it’s also hard to exclude this possibility. But I’m digressing.
So, the chances that (~no~chunks~are~lost~) CORRECTION: we lose a chunk are probably more like this:
P = 1 - (1-x^c)^n
Where x is the probability for losing a copy, c is the number of copies, n is the number of chunks. x^c is the chance that all copies are lost, so 1-x^c stands for “not all copies were lost.” We have n independent chunks to consider, thus the ^n part to get “none of the chunks had all their copies lost”, but that’s the opposite of what we need, so we turn it around (1 - ...) to make it “the chance that we have at least one chuck that had all its copies lost.”
The only thing we need to know is the probability for a vault to go down before a loss is discovered and rectified. As we’ve seen, that period is freakishly short (good design, I may say; respect) so this doesn’t seem to be an issue in the end.
Unless we end up with a fat-tailed distribution for the vaults, where the majority of the data would be stored by just a few humongous servers. Probably unlikely?
This does not seem to account that even when a chunk is “lost”, it does not remain lost. The system will always attempt to restore to at least 4 chunks (two permanent and 2 backup)
This does not seem to account that all chunks have to be lost at the same instant in time.
Even if we are able to give a probability to this, the problem is defining the equation to work out how likely this is to occur at the same time.
The most likely time that data loss will occur is when the network is very small and there is a major outage in the US or UK/Europe. The likely cause for data loss is somehow nearly 1/2 the world wide web goes down and additionally countries are isolated. But then when everything is back David has said that the network knows and will will try to recover vaults rather than reset them when they join the network.
1 Like
As I wrote just below that (in the other thing you quoted from me), x refers to the probability of loss for the (“average”) time between a loss and its getting rectified.
Ah OK, sorry. Any thoughts on actual figures?
Yea, I figure it would would work that way; why throw away data?
So the remaining questions:
- Is it possible to single out individual servers that store chunks? For example, to destroy them via non-SAFE related methods (e.g. zero-day exploits targeting the kernel.)
- The possibility that a few monster vaults (a huge number of vaults running on the same hardware) would store a disproportionate number of chunks; could that be a problem?
Ridiculously small? We’re talking about multiple events that happen once a few years to coincide within seconds. 
I say it’s a really good design. Nodes within close groups, as I understand, monitor each other, which raises the robustness very high with little overhead. If all but one nodes go offline in a close group, the whole network would still learn about it.
Am I correct with this?
Once a file gets chopped it’s evenly distributed in xor space around the globe.
After that copies of chunks are collected around the most accurate and shortest xor path for the most requesting vaults of that particular chunk.
The far away chunks or non requested chunks go in offline mode till some nearer xor path to a vault for the offline chunk is needed ?
I suppose if you developed a piece of malware that targeted vaults. If it only triggered at a certain time then it may take out a large number of vaults before being discovered
Other than something similar I am not sure how. ISPs targeting safe packets will be extremely difficult due to the protocol being resistant to that.
It is unlikely that a vault will ever be large enough. The resources required by a vault will limit the number of vaults on any one piece of hardware
Because of randomising across XOR space, most vaults will receive chunks at approx the same rate. If it was linear address space then clumping would be significant, but in XOR space it effectively is a double randomness and clumping very much reduced.
If the network proceeds as expected then there will be so many vaults that a “super vault” would be extremely rare because it would likely take years to fill up and just one cpu/network reset on the H/W or its local net would put it back to square one. It may happen but unlikely to hold a statistically significant amount of the network data.
Would even a massive 1PB vault ever hold 2 copies of any one chunk? A 100PB vault (if that were possible), would that hold 2 copies of any one chunk? Once the network is of significant size then even those would not be a worry. To ever fill a single 100PB vault the network would have to be very large indeed. (over 10,000PB at least)
1 Like
Thats a future feature. Archive vaults. And these may be very large, But for the time being Archive vaults are a long time away and only when the network grows large enough to require them
2 Likes
Malware to take out the server if it has a chunk. Or delete just that chunk. On command, remotely controlled. Somebody is working on an seL4 unikernel with Rust bindings. Sounds like something to look into.
So you say it’s unlikely to get the “basics” (IP address) for where a chunk is physically stored? I assume it because you suggested a more passive kind of attack.
Let’s say a single datacenter.
If I ran a million vaults on a single piece of hardware (or in my underground datacenter with a dozen lines) I could cover a decent segment of the XOR space. And would make me RICH 
Which is why I think some will actually do that. We are talking about equality and freedom and love, but if there’s money in it, big corporations will pick it up and take it back to the datacenter. I don’t think that’s a problem, and I don’t actually believe all servers would go down in such a datacenter even if something goes wrong.
It’d probably make you go bust, since you get paid only on GET requests, not on PUT requests.
Ultimately you’d get GET requests, of course, but you’d have to have them coming at a higher rate than the cost of financing your vaults.
I bet that farming won’t pay for professional farmers (who would acquire specialized resources for farming).
No two chunks would be saved to the same vault. That’d be a bug.
You may end up with an empty vault if you had more than 10% of the total capacity (or whatever the right percentage at which the system couldn’t give you more chunks is).
Related to all those copies, how many are there? I can’t believe that every chunk is copied 16 times. A 6.66% efficiency?
No just a blunt object. Hope that the malware takes out enough vaults to kill all copies of some chunks.
If they captured 70% or more of the network, then whose network is it? Once the network is large enough then that is unlikely to be enough. May lose a little by chance.
And they are competing with 100,000’s of “home” users running vaults on their computers that are running anyhow and have a reasonable amount of spare resources. Basically profitable farming will be if you pay nothing extra for running the vaults.
Running a datacentre solely for vaults will be unlikely profitable when competing against a much large storage capacity of the “home” vaults.
And then to run a datacentre scale farming “rig” when the network is relatively small and get say 1/10 of the farmed coins at $0.025 per coin is not going to make anyone rich apart from the equipment suppliers and the electrical companies.
[quote=“janitor, post:32, topic:6781”]
I bet that farming won’t pay for professional farmers (who would acquire specialized resources for farming).[/quote]
I’m still to come across something that can be done efficiently in a small scale but can’t be done more efficiently on large scale.
You wouldn’t run one vault on the same hardware, but possibly dozens. If you have a datacenter, even millions.
Sweet, exactly as it should be
Amazon, Rackspace, etc always have spare capacity, too; it’s part of their business. What stops them from using it, if they can make money (even if it’s just some change) and they can just discard what they need for something more profitable (i.e. paying customers)? They do business automating deployment, so it’s a no-brainer for them.
How about the fact it’d be a great way to lose money and paying customers by providing free service at a fraction of the price they could get if they simply offered a slightly better discount for their regular services.
I don’t follow. “We have free capacity. Here’s a simple way to make money until we need to claim it for something better.” Anything illogical about it?
It’s illogical to help grow your rivals.
It’s even more illogical to not make money off of your rivals. If you make money using them, are they still rivals though? Lots of philosophy there 
Joke aside, I’m fairly sure a lot of the SAFE services will be running on VMs in Amazon datacenters. Many copies of chunks will be stored as S3 objects, too.
Edit: Idealists tend to think everybody else is an idealist. Corporations aren’t; they are selfish and pragmatic, so if SAFE takes off, they will just ride the wave. How quickly they jump on will only depend on how quickly they realize its significance.
1 Like
I mentioned that you would:
- Lose money uploads (you have to pay for network ingress cost up front)
- Lose money on operations and hardware (same as above, you have to pay for those up front)
You’d have negative cash flow for a long while.
If you have spare network and disk capacity (say, 15% unused), you could simply cut your price 5 or 10% and fill it up.
If you can’t get 10% more customers when you lower your price by 10%, you should get the hell out of that business.