Who said you would need to upload? You just sit there and provide storage. You’re one of the nodes storing chunks for others.
Not if you’re Amazon: you have it as part of your business.
You can’t fill up all your extra capacity, you keep it for failover and growth. Now, if you can use that capacity for something that you can dump at any moment, you’re making free money.
It’s related to archiving as I understand and isn’t relevant for the moment then, but my question is if implemented doesn’t the archiving thing makes this whole topic irrelevant because if a total close group (big big datacenter) goes offline the file would searched farther away around XOR and a new path with groups(maybe little further/slower) is created from he fetched archived file…
You’d need to accept uploads from others. That would be your ingress. You can’t serve data if your vaults are empty.
The way it works is you need to have them filled up and then you get paid for GET requests (when/if you get them).
Right, but you aren’t making anything until you fill it up as I mentioned above. But you must pay for bandwidth, operations and h/w anyway. Ultimately you’d get paid for GET requests, but not more than other providers (most of which runs vaults on crappy home hardware which costs a fraction of your hardware).
Another thing is you’d not only have to come online and have your vaults filled up, but also stay online for a while to gain reputation to get data and later read requests. I don’t know for how long, but again, it’ll cost you dearly in power and cooling before you see any SAFE coins being credited to those vaults.
I honestly don’t get what you mean. Let’s say I’m Amazon, and I keep 50,000 vaults worth of free capacity (though I don’t measure it as such just yet) to cover for unforeseen demand related to my cloud services business (failover, new customers, etc.) I hear about this new awesomeness called SAFE Network so (being the future thinking company I am) I provision those 50,000 vaults I can afford, then I sit back. Chunks start flowing in, because some of those thousands of vaults are bound to be close (xor space and all) to some of the chunks that are looking for a place to crash. When somebody needs them, I get moneyz. If a new customer comes in, I throw away some lousy vaults, because more moneyz. THE END 
Look at AWS GovCloud (US) Product Details - Amazon Web Services, they now get paid 1 cent per GB of requests. I assume that’s the price they chose carefully and all that, so their systems are optimally loaded at that price. Now you’re saying that amount is somehow wrong but they can’t change it (i.e. cut the price) so they’re simply stuck with excess capacity. I think someone would have been fired already if that were the case.
You’d have to make at least (or around) $0.0312/GB (the price of Reduced Redundancy Storage for up to 1TB/month from URL above) for this to make sense. There was a post in which I calculated these scenarios, but let’s say you create a 1TB vault and sit pretty. Normally you’d make 0.03*1,000 GB = $30/month.
Now you’re saying it’s easier to make money from that idle storage by standing up a bunch of SAFE vaults than to lower the price to $0.03/GB (from 0.0312) and take some customers from your competitors. How can you make $30/month off 1TB of SAFE vaults? If that becomes possible, I’ll buy 30 large HDDs and retire.
“Optimally loaded” is NOT “fully loaded.”
A single storage rack in Amazon’s data centers stores multiple petabytes. Let’s say they have only 1% free space. And that’s where I stop.
It’s called optimal because having it fully loaded isn’t desirable even if you can make some money from it.
There’s no point in arguing about this, we’ll see when SAFE gets launched. If you want to place a bet, let me know.
Individual, unstructured blocks with a maximum size of 1 MB, that can be evicted at a second’s notice don’t have to add to the “fullness” of a drive, if it was organized with that in mind.
One could even modify a file system so it would lend its free blocks as opportunistic storage space for SAFE. Which is a pretty decent idea, if somebody is willing to pick it up 
If we needed some space (e.g. for a new file), the FS would just do its job, but it would also compile a list of the chunk copies that got deleted in the process, and then report them to the storage manager nodes (or whatever they are called) at the first opportune moment.
The Chunks are obfuscated before they’re send to a Vault. So the person owning the Vault doesn’t know what’s inside. He will only see random Chunks.
Chunks are spread evenly. That’s because the Hash of a Chunk brings it to the datamanager with an address close to them. And Hashes are quite random. So with 1 million Chunks in the network, they’re spread quite evenly over the datamanagers. And a Chunk with 2 copies lost at the same time isn’t really a problem. The network will notice and creates 2 extra copies as fast as it can.
Another thing to notice is caching. Popular files might see 20 or 30 copies around the network in cache. So to create a real problem with storage you need something like this:
A file that hasn’t been touched for some days. Where 3 of the same Chunks are lost all at once under a couple of seconds. That would be a serious problem. This could only happen IMO if we have centralized datacenters that make up x% of the total network (say 1% and higher) and their power/connection is gone in an instant. At that same time all 3 copies of a Chunk need to be stored at that datacenter.
Let’s say I upload Star Wars XI to the network because I’m a jerk and I don’t care about all the sweat and suffering that took making it. The “link” for this file gets public and everybody starts downloading it (now let’s pretend there’s no caching, so we only have our 4 copies to destroy.) Basically: the chunk ids are public, correct? Could the malwarized hounds of the RIAA (or whatelse) still not target them?
In other words, the obfuscating you’re talking about: is that another layer on top of the already opaque chunk ids?
I think I wasn’t clear about what I was talking about. I was considering the potential loss caused by single events.
Chunks are spread evenly between vaults doesn’t also imply that vaults are spread evenly by location. For example, you think you’re cool because you run a shiny little vault on your laptop, but the guy next door is a total freak and he filled up his bedroom with a small server farm and he’s running a thousand vaults. If your laptop falls in your soup that’s less of a problem for the network than if your neighbor’s house burns down.
No, they can’t target them. Sorry that I can’t provide a link yet 
. I’ll try to find. Nobody is able to target any of your Jedi Chunks™. The only thing one could do is request them (GET). If more people do this, than more is cached, so the file becomes faster. What I’ve understand from David is that the datamanagers will obfuscate the Chunks. So the Vaults have no clue. And the Vaults use their own encryption as well. The key’s are in RAM-memory, so gone when you turn of your computer. Now your the RIAA, you want to find these damned Jedi Chunks™. Where do you start? How do you know which Vault holds what? Maybe you’re a datamanager and see 1 Chunk come by. But as a datamanager you don’t know the IP-address of the Vault as far as I know. So now you’re still in the dark.
Yes, this is the datacenter problem as I said, indeed, not random in physical space. That’s right. There seems to be a weak spot in that. Don’t know what the Dev’s think about that. But I guess they’re busy. Not gonna poke them.
Edit:
Wow, that’s clever; the RIAA (though I’m confident they are the music business vultures, but anyway) would have to install their malware on both the vaults and the data managers, and not only that, but they would have to be in touch! My Jedy Chunks™ are safer than I expected 
I don’t think it’s very serious as long as there aren’t just a handful of really big players commanding the majority of the vaults. This is why I started thinking about Amazon, Rackspace, and other cloud services starting to utilize their free capacity for this purpose. Because then what if one day they just dump all vaults because why not 
The degree of physical centralisation affects the probability of an event occuring where a significant percentage of network data abruptly goes offline, but it does not directly affect the probability of data loss when such an event occurs. So I think they should be threated as two separate questions, though of course they both matter in the end.
Let’s rephrase the question in the OP: What are the odds of losing all copies of a chunk if suddenly 1% (in terms of capacity) of the network goes offline?
Obviously, any copy has a chance of 1% to be in that lost 1%. The network tries to keep 6 copies of every chunk, and it considers 4 the bare minimum. For simplicity’s sake lets settle on an average health of the network, where 5 copies of a chunk are kept. The risk of losing all copies of any particular chunk then is:
0.01^5 = a chance of 1 in 10 billion (or 10 milliard for non-English Europeans and others)
Since most chunks will be 1 MB, this means that if that lost 1% consists of a capacity of 10 petabytes (i.e. total network capacity is 1 exabyte, a million terabytes), the probably of losing 1 chunk is 1.
Now if we lose 10% capacity at once, then the picture becomes very different… We’d likely lose about 100K chunks if total network capacity is 1 exabyte. This doesn’t take into account caching or a hypothetical data recovery protocol when most of the lost vaults come back online, but it’s still pretty scary in my opinion. One major natural disaster at the “wrong” location could result in data loss, and in my view even merely one lost cat picture would significantly tarnish the perceived reliability of the network.
I’ve proposed it before, but I think it would be a great help if we’d add a parity chunk to every file’s datamap by default. For the cost of merely one extra chunk of data per file (so between 4 KB and 1 MB), it becomes possible to recover any lost chunk of that file, as long as no more than one chunk was lost. It’d exponentially reduce the probability of data loss (on a file level). In other words, it becomes acceptable for any file to lose one of it’s chunks.
Noooooo; probabilities don’t work like that. For one, you need to lose just over 75% of your data (for 4 copies / chunk) to get an actual literal 1 for the probability of losing a chunk. On the other hand, you would have a much higher probability for losing one much quicker than you’d expect.
The problem with your calculation (0.01^5) is that you’re computing the chances for one particular chunk, but what we care about is whether there is a chunk (any chunk) that gets lost. And that requires a different formula, for which we also need the number of chunks on the network.
Let’s stick with that exabyte, and let’s say we have 10^12 chunks, 5 copies each, then compute “the chance that we have at least one chuck that had all its copies lost” for different percent of sudden losses:
EDIT: Actually, this formula is not correct because it doesn’t take into account that losing 80%+any amount of the data means there is at least 1 chunk that had all 5 of its copies lost; but I think it’s on the right track.
I did that, but I didn’t make that step explicit so I guess it was easy to miss. One chunk copy has a chance of 1 in 10 billion to be lost in my example, so if 10 billion chunks go offline (that’s the 1 exabyte being the 1% lost), then the probability is 1 to lose one. 1/10000000000 * 10000000000 = 1
That’s what I’m saying though: it doesn’t work like that. We have a (1 - 1/100) probability for not losing a chunk during a 1% loss. We can’t just multiply probabilities, because we’re not supposed to be able to end up with greater than 1, but we would. Instead, we need to raise this number to the power of how many chunks we have, and thus we get the probability of not losing any of those chunks. Then we can just do a “1 - this” to get the probability of losing at least a chunk. At least that’s how I understand these kinda things work like.
All this calculations do not consider that, though not at first, archive nodes also will exist.
@Tim87 you will find that he was saying that if you took one chunk then the probability that all the copies were in the 1% loss is 0.01^5. Then if you work this out for every chunk the probability is 1 (100%).
I like the concept, but one issue is that as the file grows larger the time required to recover one chunk increases at a v.much faster rate than chunks * the_time_to_recover_a_1_chunk file
I still would argue that this is done at the APP level (even if in launcher/client/NFS) so that the user decides which of their files requires this extra protection. After all there are some files that I would not. That library of my “funny pictures” gathered from my travels over the web are not important enough that I store an extra chunk per file for the 100,000 photos. But my family photos I would be willing to store an extra chunk for the 1000 photos.
That does not seem to make sense. You are taking the probability that a particular chunk is in the 1% loss then raising that to the power of 1E12 (number of chunks in network)
What does that even mean??
It would seem you are taking the chance of NOT having all 5 chunks of a particular chunk in the 1% (1-(.01^5)) and then saying that 1E12 of those exist
Of course to have 1% of the network storage in one facility when the network is say 1E12 chunks (1E10 chunks or 1E16 bytes) is in my opinion very unlikely.
Basic economics and keeping the shareholders happy would exclude any commercial datacentre from attempting this. The ROI for commercial farming is simply not going to enough, especially when it is more profitable to hire the resources out to customers.
That only leaves government or a malicious “rich guy” who is not after profit but disruption. That is a hell a lot of money to make me lose a “cat video” or two. And who is to say that I did not store two copies of my vid. In other words when the network is 1E12 chunks in size the network can be considered useful and thus the ROI on that disruption is not enough to reasonably attempt.
Maybe we need an app that can do par blocks across files (they exist already) and mount my SAFE drive and simply create a set of par blocks for each group of files. Yes it may cost me 1% more in PUT costs over my total PUT cost to store all those par files, but then even your 1% unlikely situation will not worry me. Except the cost to reconstruct. Hell I could even store the par files encrypted on a local drive. That means my personal info & family photos are safe from prying eyes. The par files cannot reconstruct the files without the SAFE drive mounted, so they are useless to prying eyes encrypted or not.
TL;DR
I am more worried by war disrupting the network than any commercial entity. But then without the SAFE network I may have nothing. So something is better.