Pre-RFC Suggestion: Adding a memory persistence model to SAFE Network

I was looking over safe_vault code and conversing with some interesting individuals on the forums and it seems from the codebase that it would not be so tricky to add a memory persistence model to SAFE in the same way that the disk operations work (i.e create an alternative chunk_store to begin with which is in-memory specialised).

What I envision is two distinct types of memory, Transient and Persistent, where the persistent model is introduced initially for those who want a faster distributed disk array. Persistent memory functions similar to the way disk persistence works except it uses main memory as a resource rather than disk, similar to memcache or AcidState. Transient memory would be memory used for ‘compute’ runs once the CPU becomes a resource.

I’ve thought about the CPU as a resource but have not come up with a good enough model yet which is why I’m trying to focus on memory instead of a general compute model. :slightly_smiling:

In terms of value and how it would affect farming, once could draw a comparison of the price per mb of HDD space and main memory and use that as a basic guideline to begin with.

Would it be worthwhile if I work on creating a more comprehensive RFC for this?

4 Likes

I’m not clear on the motive for this, so can you explain a little about the differences between this and current vault, and why they might be useful (use cases might help).

I’m also wondering about the differences between this and the existing cache mechanism.

1 Like

The idea appears attractive, but it also requires resources so IMO it should be judged against resources required.

With memory cache you get “instant” access to data, but from the network there’s no difference, so while you may save (say) 0.2 ms in IO time, you’ll still have to wait 20x longer to get the data over the network, which is where cost/benefit becomes dubious.

If this were to be implemented in a trustless network, I wonder how you could actually check whether people are not lying about their RAM. I think it’d be very expensive to check and verify this.

A simpler and better measure would be pay for (low) latency so that you don’t have to care whether data is in RAM or on SSD or whatever works, but even that is very complicated. It would pay better to serve data from HDD in a fast data center than RAM data from Siberia.

1 Like

The current vault stores data on disk using the standard file I/O operations provided by Rust. This is great but disk accesses / writes are an order of magnitude slower than memory accesses / writes.

What I am proposing is to implement an in-memory persistence solution ala AcidState in order to have the option to persist data in memory instead as well as a ‘Transient’ memory solution for a compute layer.

The in-memory persistence solution can be seen as a faster / more expensive type of storage for those who want a more performant access / write time. Most big websites these days persist in memory anyway i.e Facebook.

The transient memory solution is a type of reserved memory which is to be used for computations rather than persistence. It can be seen as less permanent than the memory based solution because it will only be required for the duration of said computation.

Edit: Is the current caching you are referring to the LRU cache? I think this is related to network caching, thereby improving latency in the general case across the network. It is different in the sense that a memory persistence solution is how fast data can be accessed / written to from the farming nodes rather than how fast it can be transferred across the network.

1 Like

@janitor Network latency is an issue but I think you are underestimating how slow data reads are from disk relative to main memory. If you are accessing mem it becomes a lot more feasible to have bulk I/O operations rather than just get / put. I think it needs to be tried in order to assess what type of performance difference there is.

I’m not sure I understand what you mean about lying about RAM. If the vaults access to your RAM is the same as the disks, you check whether there is available memory, if not you don’t get paid for it. A check could be put in place in order to estimate how fast read / writes are to that persons RAM and get priced according to this metric. Similar types of verification would be needed for inclusion of CPU resources I’d imagine.

@et4te yes regarding LRU cache.

I think you can eliminate the “lying about memory issue”. If data is PUT using a “fast access premium” PUT fee, it tells vaults they can earn more by serving it quickly. It won’t matter whether they do it using RAM, disk or light crystal - the fastest vault wins the associated premium farming reward, so vaults have an incentive to keep the data in memory, or on faster disk, or whatever - they must compete on speed, to earn the higher rewards.

This does beg the question though - do you have to limit the number of “qualifying” vaults for this new class of storage (not good allowing it to end up only on slow vaults)? In which case it undermines the security model. However, you may not need to do that, because so long as there are enough vaults to ensure that most chunks end up on at least one fast-vault (which would almost always win) probably meets the performance aim, probably without compromising anonymity or data redundancy significantly, this may not be much of an issue. I don’t think that works though, because unless a vault knows it’s going to receive more fast chunks, it isn’t going to create an incentive to fast track them. So I think the first question of this paragraph is going to be answered “yes, you have to store fast chunks on fast vaults”, which as I said, I think would undermine the security model. :frowning:

1 Like

I made a specific example. By how much is my estimate wrong?

If you read a file from someone who’s on the opposite side of the world and has a DSL connection to the Internet, it won’t be efficient even if the chunk is in RAM.
Secondly, your “bulk” operation must be sent to several hosts, so you’d have to ensure all of them have RAM disks, and when you read back the slowest chunk you get will determine the maximum speed. Because each chunk has several replicas and you can’t choose which will serve data, you’d have to put all copies on a RAM disk or otherwise have a very smart caching system which outs maybe one replica in a RAM based vault and the rest on cheaper SSD or HDD vaults.
And then when you read you’d have to rewrite the network caching mechanism to not have the built-in caching layer slow you down.
There are way to many problems here.

Writes always go to RAM, even if you write to disk.
You would have to check that the disk is actually a RAM (and SSD, if you wanted to encourage that tier too) based disk. But even if you did, because all writes go to RAM, you as the user wouldn’t benefit from writing to a RAM based disk - you couldn’t tell the difference between it and a write to any HDD based system with 2+ MB of free RAM that would buffer the write.

Another problem is you want to reward read speed, but it doesn’t matter how fast reads are on the host. What matters is how fast the client gets the chunk and by then you can’t even tell whether the chunk was sent from a RAM or HDD based vault.

The Safe network isn’t designed to be low latency, high performance network. It was designed to run on cheap hardware, be resilient and private/anonymous. In order to make it low latency the optimal way would be to not have intermediate nodes between users and vaults, but that would ruin the anonymity.
I get your idea, it’s good but I don’t think it applies to this network. It would be interesting to have a P2P network storage system with smart caching, but it would have to be designed for that kind of workload, I think.

2 Likes

This “fast access premium” PUT fee is a very good point since it removes the need for distinct implementations of the chunk store for varying memory models. One could in this view use the exact same store and in light of this the ‘persistent’ memory model would be redundant.

I see what you mean with the qualifying vault reductions being problematic. I don’t think this has to be the case though as long as you don’t constrain exactly how efficient you need the accesses to be. In this scenario having a premium option would not be a good idea. The slower vaults should be used in the same manner the faster vaults are once faster vaults are not available.

I think you have successfully upgraded my view on things from ‘Lets have fast memory access’ to ‘Fast network access + PUT / GET competition for resources should be fast enough’.

@janitor Yeah I was wrong about this because I’m not used to a completely decentralized world so my current notions of reality and importance are somewhat unreliable. :stuck_out_tongue: You rightly could just use a ram disk and incentivise access / write speed.

This discussion has been very educational and I am now starting to believe that the only infrastructure left to be put in place for a ‘compute’ operation to be implementable is the CPU being made a resource.

I think that despite the fact that SAFE isn’t designed to be low latency or high performance, it still has the capability to compute distributed computations in an ‘efficient enough’ manner to be useful. The most efficient route would entail not having any security at all and clustering high performance computers together I would imagine but this doesn’t mean that the SAFE network cannot do the same thing securely, reliably and with a performance tradeoff that is acceptable to most users.

1 Like