Appendable Data discussion



So it is more calling the current functionality for what it is: mutable, bot only in that things can be appended, not changed what is already written.
I’m more open for practical reasons atm: e.g. functionality freeze to get alpha 2 done, instead of ‘ideological’.


Is there a mayor technical reason for not having cheap temp data store or it is just pure ideology? I am trying to imagine how could a web with dynamic data operate on this but I can’t find a solution. If every attempt to write some data change costs exactly same as 1 MB of all time storage than I can’t imagine even simple forum to operate on this. Let say I write this comment, with few sentences, I pay 1 MB fee, someone likes it, he pays 1 MB fee just for liking it. After few minutes there are costs equal to store few GB of all time data just to store some temporary status of some counter. Not talking about when client wants to render the web he has to download whole chain of changes, iterate it analyse and render. Small forum like this we are using here would cost insane amounts of money just for updates of daily likes, not talking that downloading few years history chain would require long minutes and enormous CPU load to parse it. What am I missing? Will this network storage really end like this? Who will than use it and why?


For a start is give CRDT data types which is of course a valuable asset to the network.

I do not think that will be the case though? I agree if the network charged 1Mb for every 32Byte entry in an appendable data item the network would likely fail.

I think you are assunming every entry to appendable data (which will be 32Bytes) will cost 1Mb, it wont.

In your example for instance, the likes could be a single appendable data item where people add a like by inserting a 32Bytes entry to the appendable data item. There are more ways to handle much of this of course.


Thank you for explanation. I got really scared because I thought minimum data size written is equal to 1 MB block price. If pricing is granular enough like 32 Bytes, than it is not such a problem. Appendable database grow in time can be solved by recalculating state every now and than by web owner from appended data. It is not user friendly DB at all, but can be handled. I take your CRDT data type argument, it makes everything much simpler.


As they have the right to do so.

I see your point, but I think it’s up to the app builder if they want to implement history or not. For my own data I want full control. For a business it’s up to them if they want to keep history(an stay transparent) or not. At the end businesses who choose to keep history will be regarded as more reliable. Give websites with history a green mark in the browser or so. It will then go natural I think.
The network should provide the tools, choice is upon us how to use them.

As developer I would like to see just 1 datatype, and then be able to set a flag for making it mutuble or not and for keeping history or not. Some other thing I was always thinking of is an expiry date property, but I guess this is like swearing in church here :sweat_smile:


It can be argued that is exactly why we are where we are, profit-seeking will always overtake good morals. So perhaps forced freedom is good?

If the network only replaced servers to provide exactly what folk have done with servers, then are we not painting black over black?

If we say, the network will never lose data, but then it does as folk delete it we are in weird place.


Slightly in parallel to all this, but related:

Placing effort in a SQL API, or at least set as goal to facilitate such implementation by design, would be a HUGE benefit for adoption.
I’ve tried to scan the sentiments at maidsafe for this before, but got tumble weed. I think it is a very important question, regardless if the answer is that technological limits won’t allow it, and that you have some rational for it.

While rdf/solid is very interesting and useful in its areas, it is not replacing SQL, it cannot. All databases have their limitations, and thus are crap for some uses, but SQL is like the most general of all types.
I feel that it is a bit ideologically narrow minded to not acknowledge that business people simply will not leave excel and SQL, no matter how wonderful academy and data scientists think some other solutions are. (I started out as data scientist, so I am saying this out of own experience).

I feel that in the vision for the network, the large majority of business has to be included, small and big, as to maximize utility of the network, so that we can ensure the more niche uses as well (what academy and ideologically motivated people need and would want).
At least I feel it is risky to not consider it, to rely on the success through niche uses. Maybe those are internal questions, but I think it is a very interesting discussion for the community as well.

I think the question can be more nuanced.
If we say network will never lose data when you store it as ‘non-deletable’, then we are not in a strange place when data stored as ‘deletable’ is deleted.


I agree this is a valid consideration.

I am not sure what you mean here, do you mean nothing we do will matter or we should provide excel?


You mean with ‘forced freedom’ in this context that you won’t be able to choose to delete the data on the Safe Network?
You can only force to a certain extend, I think.
If it is easy to fork the Safe Network and add the delete option (and there is demand for it), it will happen eventually.
Edit: with fork I of course mean the Safe Network software, not the data already on it. You start with a new, empty network.

–>Exactly: if one can verify if the data can be removed or not, that should be enough I think.

A lot of if’s above, I know.


Well, I might interpret it through my own experiences and lense (most likely), but there’s always a risk in this.
If it is a network for the world, the majority of people, then it must be made very accessible to them. The people building it are a very tiny minority of extremely intelligent people, and … from my own experience … it can be very hard to see just how different it can be in needs and capabilities.

So, I’m definitely not saying nothing we do will matter, not seeing how what I said implied it but I’m sorry if I was. Just observing that there’s a risk here.


Much of this is not so obvious. Say for instance I store a video that has millions of hits. All good, but then I delete it? the video is gone. So we can then say, OK forget de-duplication, allow many different copies of the video with different chunks.

Then we say, no make one of them no-delete so we are good there, but then the no-delete chunks need to be different from delete chunks, so more types. Then the data map also needs to be non-delete, so linked to chunks which must also be non-delete. Then chunks become linked on the network, where at the moment they are not. So folk know which files no-delete chunks belong to. Then vaults can track the chunks they hold and what those files are (many will scream they host XXX ).

I just say this to show, nothing is impossible but much is very subtle.


Ok, let me correct it:
If it is easy possible to fork the Safe Network and add the delete option (and there is a lot of demand for it), it will happen eventually.


I’m a little surprised by the reaction to Appendable Data. We already published a proposal to store data in perpetuity (See #8 here SAFE Network Fundamentals: Context) and I thought we had this discussion with the community, but maybe it got missed somehow or there was a miscommunication from our end. Maybe we should work on better communicating potentially controversial issues in the future.

What’s to stop anyone from downloading “your” data and copying it? You can safely assume that everything that you’ve ever posted online is sitting somewhere on corporate and governmental servers. The illusion of being able to delete your data is just that, an illusion, and perhaps we shouldn’t encourage the fantasy notion that data once shared publicly can ever be taken back.

The challenge is in making sure that private data always stays private and does not get shared with the wrong people. However, once you’ve made the conscious decision to share something, it is impossible to reverse this, even on the SAFE Network. Note that this will still be different from the current internet where your data is already perpetual but you do not choose whether your data is private or not in the first place, and every action you’ve made on the clearnet is logged – forever.


It certainly needs some explanation, that’s for sure! The ramifications of this are massive and I feel it’s been sort of waved through without due consideration of all the pros and cons. It may have been discussed in depth within MaidSafe but what emerges into the outside world is a bit vague and waffly, at least that’s how it feels to me. Of all the arguments for SAFE it’s the one I’m least convinced about.


I think part of the miscommunication is the reasoning why data can’t be deleted.
I think more people agree with the practical aspect: a lot of change on the current software.
But the ideological argumentation is less convincing. You could also turn it around: if other people can copy your data, like it is now on the clearnet, why the need of only immutable data?
And also: if it will become clear that storage capacity/bandwidth doesn’t grow that quickly, some deletion could be useful.


Can somebody outline SAFEcoin anonymity with appendable data? That was going to be one of its killer features, and now I feel like that anonymity may have been removed.


I think it’s worthwhile noting what the Network fundamental say about perpetual data:

All public/published data on the Network will be immutable and available on the Network in perpetuity.

Perhaps we need to better define what we mean by public/published, to bring some clarity.


I can understand logic for permanent public data, even if I do not agree with that. But I don’t buy any logic for not deleting private data. Private is private, so nothing is lost if I delete my own data. If I can’t delete my private data, than I will forever have there temporary junk there. Sure I can destroy my private keys, but it forces my to copy all non junk data to new private account. There is 0 logic to force permanent data for private data. The only argument for permanent only data I accept is technical difficulty. Ideological reasoning always fails when it meets reality.


Nail on the head, in my opinion.


I’m not certain on the scope of perpetual data WRT Safecoin, perhaps others can clarify, but it shouldn’t effect the anonymity part either way as:

The Network will always ensure that the User has the ability to send transaction messages and posts with a temporary and single-use ID that is not linked to any known identity on the Network.

So, only the sender and receiver are aware of each other’s ID, and that ID could be single-use and throw-away (should the user choose), then would satisfy your desire for anonymity?