Feasibility of datacenter farming (and the risk of farmer centralization)

I’m continuing this from comments here. Although there are several topics related to the economics of farming, this one is specific to datacenter (hosting or co-location) farms and the potential for centralization of farming.

My claim is that it is going to be economically unattractive (probably impossible) to do that simply because

  • the colo farmer would be paying the market cost of online storage, so clearly any other cost (vault and OS management, costs of converting SAFE to/from hosting expenses, etc.) and profits (should there be any) would have to come at the expense of the MaidSafe user
  • because 4 copies are made and some (@niklas) said they’d also use RAID protection, that means that each MB-sized chunk will require over 6MB in disk space (1MB * 4 for MaidSafe replication + RAID overhead).
  • the colo farmer would have to compete with home farmers who have crappy network quality but are able to serve nearby users faster than the colo guy from his cluster in Virginia and - being hobby farmers - are mainly looking to offset their cost of h/w by sharing a fraction of their HDD space with the network. Essentially their cost would be zero. (Semi-professional farmers’ cost would be close to the cost of h/w).
  • colo vaults may actually lose money on every GET request (Google and Amazon charge $0.01 per GB).

Recap:

  • colo farmers’ with protected (RAID) will have the cost of capacity 100% higher compared to home farmers
  • their other advantages over home users are none-to-minimal
  • colo farmers will not be able to compete against smart small scale hobby users
  • a lot will depend on how the Project decides to reward SAFE farmers. Since the Project is in favor of decentralization, I wouldn’t bet that MaidSafe algos will redistribute value in favor of large scale professional farmers
3 Likes

Using cloud storage for farming would be a waste, it’s heavily overpriced for this purpose. A large scale farmer would either rent storage servers or buy them and colocate them.

The home farmers would still have to worry about power costs.
And while they have low latency to local users - so does the big-time farmer. Who has the additional advantage of being able to serve multiple users at a time. Even if you have a relatively fast (10Mbit/s) uplink as a home user, that’s where things start to slow down for you.

That’s $10/TB, bandwidth can be had for cheaper. As I said, cloud storage is not the best choice of service for this kind of application.
In any case, it comes down to how much each GET request pays.

Okay, so we agree that renting is out.

True, but they’re insignificantly higher than the cost of making the disks spin (and as I mentioned in many cases the computer would be running all the time anyway - as long as the earned SAFE are worth more than the cost of powering the system, since all cost has already been sunk).

  1. I ping my telco and I get 3ms. I ping Google and I get 10+ ms. I doubt you’ll beat me in my area.
  2. But even if you do, implications for MaidSafe (and therefore financial implications) aren’t at all clear. Let’s assume my latency is X and yours is X-1 ms. Who wins GET requests? Does the winner make any profit? You can’t possibly know until you find out who MaidSafe devs decided to reward. They can flip some 0 to 1 and make me a winner. They can flip another 1 to 0 and make you win but stil not make money.

I agree. However home farmers don’t pay nuthin’.

IMO more important is who MaidSafe decides to favor. Even if each GET pays enough for you to be profitable, if the s/w decides to pass it to me although I’m a bit slower, then the pro’s may not get enough of those requests although they pay nicely.

Why would MaidSafe give me some requests although I’m a bit slower? The proximity matters. But if I am competing against a pro located in my city then either all “local” requests would go to him.
Will MaidSafe make this a winner takes it all type of market? I don’t know, I’m only speculating that as a project they seem to favor decentralization so they may tune the algo to randomly spray the requests according to some rule that would prevent centralization.
If they don’t discourage decentralization, then local pro farmers may do okay vs. their pro competitors without local presence.

I think what’s happening inside of s/w will be key.

Personally I haven’t decided whether to farm or not. I have a minimal hobbyist setup at home, but I could buy and bring 24-48 TB of capacity online within 12 hours. Because I don’t need that much storage myself, I’m delaying this purchase until it becomes clearer how the algo will work (cost of GETs doesn’t concern me as long as it’s enough to pay for the HDD spread over 24 months and power because my cost will be not higher than that).

Normally one may think the cost of GETs for colo guys would have to be competitive, but if the algo rewards response time and bandwidth at the expense of cost (which was indicated on several occasions), then my low cost wouldn’t help me. Are you thinking that will indeed be the case?

Something we both don’t know and which will be important is your vault-hit-ratio. If promotional campaign with freebies is implemented and successful, early farmers may end up with a disproportional amount of random garbage in their vaults that noone will ever request (which is why I have been consistently arguing against freebies and promotions and in favor of automated deletion of stale content). Nobody knows and nobody can predict how many requests one would actually get. It’s going to be wild.

My expectation of what will matter:

  • Algo (or “project values”, as decided by MaidSafe)
  • Vault hit ratio (freebies, applications, etc.)
  • Price of GETs
1 Like

“The cloud” is out, but I never suggested to use that anyway.

For file transfers, throughput beats latency. Even if you’re only transmitting 1MB at a time, a 100mbit/s uplink with 100ms of latency wins a 1mbit/s uplink with 1ms latency. That said, I haven’t studied MaidSafe’s network quality rating algorithm either.

The way I understand it, the pro will get a higher chance to receive PUT requests. And of course, when the pro’s drives are all filled up, further PUTs will have to go to other people (unless the pro expands).

One thing that’s for certain is that the algorithm will reward fast, stable vaults. As I explained above, this means more traffic for people with more bandwidth in core locations. Whether that’s enough to make a profit over bandwidth costs depends on the ratio of Safecoins you get for a GET vs. Safecoins you have to pay for a PUT and how that relates to fiat. We’ll just have to see.

I agree, I’m a bit concerned about old junk data filling up vaults. However, I would also argue that if you start farming early, you have a high chance of getting bootstrap data that applications on the network use all the time (resource lists, start pages, configuration data…).

1 Like

Well I didn’t study it, but I read the forums and I recall that it was repeatedly mentioned that responsiveness (and no, I don’t know if that means for the entire chunk, if which case it’d be a mix of responsiveness and throughput) will be rewarded but also that low-spec h/w will be okay. Quote of a quote:

The basis that I’m using to conclude these will work is from having consumed almost everything @dirvine has said on interviews, etc. He’s said that the client should run in background on most any computer without much notice. He’s said the client will run on systems with “terribly low” resources. Therefore, I have to assume that it will run well on a dedicated computer with a competent 64 bit processor with a lean OS.

Also another by Fergish

I’ve said before, and David Irvine has agreed, that rigs such as yours will not hurt the network, and if they work economically for you then there’s no downside.

URLs: Best Safe Node hardware and Will there be farming pools on SAFE network? - #5 by happybeing

One thing that’s for certain is that the algorithm will reward fast, stable vaults.

I agree, but you need to realize that MaidSafe, er, the community, will decide what fast & stable means.
For example, it was stated by Project members that reboots and even hours of downtime won’t be punished and you can also see (admittedly, I’m quoting indirectly via @fergish) that crawling-slow home vaults won’t be put out of business either.

Values determine policies, policies determine the algos, the algos decide who gets how much.
In my environment responsiveness and stability standards are different, but that’s because things don’t work the same way as here (i.e. no need to drive decentralization, for-profit enterprise, etc.).

If the aim is to achieve a reasonable degree of decentralization, crappy farmers need to be allowed to survive (and comparatively speaking, you and other from the Monsanto family won’t be rewarded for your superior harvests, so you may as well get some non-GMO inputs for your next harvest) :smiley:

Gee, there’s a lot of sneaky bastards on this forum! See this:

I’m sure we are not the only ones. (By the way, that’s why I earlier said I’d decide whether to farm later. Make sure you check that whole page, by the way).

Because there’s so many variables I think it’s going to be very difficult to create a good plan prior to beta.

Over the last summer I spent dozens of comments fighting for a laissez-faire approach to everything, but as I said above if the goal is to fight centralization, things will have to be micro-managed and that will make investment in dedicated professional facilities risky. For example, consider this: do you think that MaidSafe will lock their algo during the beta or tune it every few months (like Google tunes its index)?
Of course they’ll tune it as long as they have to and they’ll have to all the time (once a quarter or once every 6 months?, in the beginning maybe on a weekly or monthly basis).

I’m concerned about that so I’ll start very small but be ready to expand. I think farming at home up to 24-48 TB of capacity should.be doable considering my environment (including network bandwidth and latency).

People,

This question is important to me and it is all a bit confusing. I am moving house now - out of the middle of Sydney to a small country town, Cowra, 5 hours West of Sydney but paradoxically the house there has a faster Net connection (still ADSL2+ but closer to the Telstra exchange) than where I am now. I am also hoping that at some stage in the future that I will have access to the new NBN fibre network. I am in Cowra for some time probably and I have been thinking seriously about buying a lot of hardware to do farming for MaidSafe as part of the package of projects I will be working on. I am also thinking of making use of a lot of Photovoltaics to help power everything (LOTS of sun in Cowra) so I am interested in the possibility of “server farming” (not necessarily SafeCoin farming) in my own premises anyway.

Of course SafeCoin farming only becomes viable if there is some good reason to do it eg I make enough SafeCoin to spend on things that are available and useful to me in the MaidSafe network. I think MaidSafe, SafeCoin and maybe other cryptocurrencies are going to be a big, important part of the things that I am interested in and am going to be working on for the foreseeable future - if the more I am involved (hardware for farming, apps for deployment etc), the more I can be a “self-employed” citizen of this new economy, the better I will like it.

So, if I can start to see actual measurements of usefulness of various configurations of hardware, it will make it much easier for my current planning processes so I can start budgeting etc. As a generalisation, I would prefer to be a “full-time” citizen / participant of this new economy, not just a hobbyist farmer - but obviously it would be dumb at the moment to commit to start setting up tens / hundreds of TBs of hardware processing if it is actually going to lose me a lot of money (Australian $) for little return in SafeCoin.

I realise it is still early days and how these algorithms are going to work is still being decided and maybe they will even need constant tweaking once the system has gone live but I would appreciate some clarity about what configurations of hardware would be sensible. At the moment, it looks like the safe route for me is to just upgrade my Linux server to the latest, fastest machine with a spare 4TB drive to allocate to MaidSafe?

Thanks for any clarifications possible at this stage.

Regards,
Phil.

The safe way is always to just use capacity that would otherwise sit idle. Got a few hundred gigs free on your home server? Farm on those. I suppose adding a 4TB drive wouldn’t be too much of a risk either, could always resell it or use it for something else if farming turns out not to be your thing.
And if you make enough money with that, you can always decide to go bigger.

At this point, the MaidSafe network is not ready for farming yet; we cannot try out what works - so this is the best advice I can give you.

Very well, as I’m sure you’re aware, pretty much every other cryptocurrency is generated through proof-of-work computations. All you need for them is hardware, cheap/free power and a place to store and cool those machines. Bandwidth-wise you could do it on dial-up.

Great discussion here.

Loving it

1 Like

@phillip_rhoades at one of the links I referenced (Best farming hardware) you can see that small ARM-based farms “should” be fine (assuming the algo sends them some requests despite their slower response time and throughput compared to x86-based competitors). I have one such small system that I will try out during the beta.

In case of most altcoins, Proof-of-Work @ home seems difficult because those coins are unstable and the investment is usually dedicated (i.e. unlike with MaidSafe where you can use the HDDs to store your own data, compute-specific hardware can’t be readily repurposed for something other than Proof of Work mining).

In that Testnet to Mainnet Transition Plan post you can see that you’ll be able to try your setup during the beta so that will be a time to give it a try with a small (maybe 1 HDD?) setup and see whether it could be economical for you, and if so add more equipment prior to the mainnet launch.

@janitor,

That all makes sense - thanks for that. As well as lots of old hardware that I have that could be used (I do a lot of BOINC stuff already) I will be buying new hardware too - but like you say, I can see how it goes early and add new (or old) stuff as appropriate. I guess what I had in the back of my mind was that if did actually become worthwhile to buy a rack of stuff (as opposed to desktop stuff) then it would be good to know that earlier rather than later but it doesn’t make too much difference really . .

P.

The ideal from a global perspective will I believe be if the entire network ran at least cost, maximum efficiency, which would mean entirely on existing spare capacity. We might expect that this is what MaidSafe are aiming for, though no one can predict how close to the ideal it will be.

This means that in an ideal situation, dedicated farmers will be squeezed, and will need to be very efficient to compete with casual farmers who have little or no marginal costs.

Obviously there may be speculative profits from Safecoin price rises and volatility, but if we ignore those, I suggest that investing in hardware to farm is a high risk venture.

Being a part of SAFE and helping it succeed, or earning by adding value to the platform through contributing to the code, apps, or content are probably sounder options IMO.

Get working on those apps, or forming teams! :slight_smile:

1 Like

Glad to hear that an experienced forum member feels that way. @niklas, are you reading this? :wink:

I think I’ll stick with my plan to start small scale, casual home farming until it becomes clear whether our expectations (in terms of the algo’s “personality”) have been met, then decide the next steps.

The beta period will be fun (even testnet3 could be) as farmers start figuring out the algo…

1 Like

Yes, and I would beg to differ. A 24/7 farmer increases overall network speed and data availability. They can also quickly add lots of capacity. I’d say the best outcome is a healthy mix :stuck_out_tongue:

I suspect the bulk of the people who will jump on big time Safecoin farming, Bitcoin miners, won’t be too concerned about that. They’ve shown a tendency to take risks.

I have plans for that already, just waiting on a proper API :slight_smile:

2 Likes

I expect they won’t be ignored. The early farmers are the early farmers because they see great potential in MaidSafe and most likely expect SafeCoin value to increase significantly in the future.

Personally, I’m going to start farming with just my current systems. If I like the results I can easily expand in a very short time frame.

2 Likes

smoothing the transition from subletting artificial scarcity bandwidth to mesh

Think of an entity with nearly unlimited computational resources like the NSA-- if the main network gets attacked and corrupted at any time by data centers with billions of VMs pretending to be individual nodes with average network specs-- remaining online for a year or two, gaining reputation, then shutting them off, could a large amount of the data be disappeared? I think that adding in a channel feature in the MaidSafe network might be a good idea, but it will start fresh – say a 32-byte identifier with the default being all 0s for the original MaidSafe network-- and ABCDEFGHIJKLMNOPQRSTUVWXYZ01245 could be a different MaidSafe network with it’s own currency, etc-- allowing people to f’ork’ their own by setting identifiers if the main one gets corrupted.

wouldn’t there be huge safecoin incentives to keep the data up and running though?

it would be shooting yourself in the foot in a huge way to shut off your own money making nodes like that

Some people-- the people in many places of high authority and power are not concerned with money. They have more than enough, and consider any means of free publication a threat to their nesting of power. And it’s easier to collect the coins, then dump them for cash and then attack the network.

You’re right, it’d be a completely pointless attack.
Millions of compute hours would have to be invested.
Absolutely nothing of value would be gained in return.

1 Like

The idea that a government doesn’t like something, and their response is to subsidize it massively for 2 or 3 years so that they have the opportunity to sabotage at a later date it seems pretty far fetched to me.

Particularly since the subsidy makes it cheaper to use - and would result in wider adoption and their share diminishing, and their likelyhood of damaging the network diminishing as well… They would need to throw exponentially more and more resources…

If you water the weeds you will get more weeds…

2 Likes