Appendable Data discussion

appendable-data
immutable-data
mutable-data

#262

If an account is compromised will we be able to transfer ownership of and access to our data to another account. Any emergency migration system in the works?


#263

Not sure what I am missing, but I don’t think this is any change.
Say you have your private stuff as data maps. Where are you going to keep these datamaps so that they can be “killed”?
You need to keep them in the network, or outside the network and bring them in from unsecured storage. If you have them in the network, you might have them encrypted. But shall you encrypt everything with one key each, because when you want to “delete” one thing, maybe you don’t want to delete the other. So, 1 encryption key per datamaps. And where are you keeping these keys? It just goes round and round.

Or you can have it all under your login, so when you want to delete, you need to forget the login, as well as copying all that you want to keep to a new login.

Or am I missing something?


#264

David answered that before when I asked him. They are in yet another data type (blobs) in your account data, which can be deleted. They also could be in ADs if you put the datamaps there, now sure how or what circumstances yet.


#265

I am guessing that each file has it’s own datamap, so when a user ‘deletes’ a private file, then the datamap is deleted locally.

As a hobby gardener I see the importance of removing weeds as my garden has limited space and the weeds utilize nutrients that my plants need … translating that into data terms with my private storage, I think old unneeded files distract attention and focus (brain nutrients!), so IMO deleting is important for me anyway.

Is there any way that the network itself could be made aware of the removal of a users local data map and then use that ‘event’ to actually delete the data? I wonder if this could be used to create a further ‘refund event’ where the network pays the user some small amount in exchange for the freed-up space … sort of an additional means of farming, but in this case, farming space instead of data or put another way, ‘directed pruning’.

Any thoughts on this idea?

I’m of course just talking about private data here.


#266

Not really as the network has no idea about how to link chunks to actual files. You would need to present it with the data map and that would be a security issue as we do not know who is running the vaults, they may just copy the stuff.


#267

Right. I think I finally get it now. Thanks for your patience.

So to sum up, in effect, there would need to be something akin to a whole new meta layer added to the network which would have extra overheads in processing and security … so the trade-off for those overheads is no deletion and an ever-growing amount of data … which we hope will be dealt with through technological innovation.

As for user temp data, in the future, some sort of un-mapped non-network data could be added to the user account ‘zone’ later if such were to appear useful.

I use encfs on my local machine to securely store passwords, ssh keys, etc. Perhaps something like that could be thrown into the safe-network code for local temp data - just use existing libraries so not much code to be written? While space may be an issue on some devices, it would still give an easy option for local secure storage for those not savvy enough to install encfs (or similar) themselves and could help to shore up the whole of the Safe Network offering.


#268

@TylerAbeoJordan
And how would the network know if you’ve sneakernet the datamap to another and received payment to “transfer” the file. Thus everything the network knows is that you “may” be the only one with the datmap but in fact there are at least two with the data map and both have legitimate rights to the file


#269

I would think that having MDs with true delete is easy enough. Delete could be simply zeroing the data and can be rewritten over rather than an append done. Sounds easier than the other. If you encrypt the data in the MD then noone can read it anyhow.

Maybe temp files could simply be files that you delete the data map to. And if you don’t share the file (datamap) then it is an effective true delete.

Remember that storage is still got plenty of future physics & biological paths to follow so it is unlikely to slow down. Phones with 1TB storage, imagine putting those chips (+controller) into a 3.5 inch form factor device. You could have nigh on 50TB in a 3,5 inch form factor. The reason they don’t do it is mostly the cost is too much compared to spinning media (14TB in 3,5 inch form factor at this stage)

So not deleting chunks is unlikely to be an issue really.


#270

This is all super interesting - and difficult to grasp.

I think the discussion would benefit from some kind of “Alice, Bob, close friends, public…” kind of presentation peppered with accounts, keys, ID’s etc.

With my current knowledge, I see SAFE as a very good platform for at least:

  • Storing data privately
  • Private messaging
  • Anonymous operations
  • Publishing very well thought information
  • IOT communications

My fears revolve around the management of personal publicity. Now only the global top elite (in regards of data-ownership) is able to run analysis on individual people and it is in their interest to create such a social media that produces the data for them. With SAFE everybody is guaranteed to have equal access to all public data. But what you can do with that data is still very unequal, and will reflect the power dynamics of the World. Those that have the best analysis tools will benefit most from the public data and they have incentive to create the kind of social media environment that produces that data to them (while producing it to everyone at the same go). The idea that people would become more cautious in their behaviour - I don’t see that happening as long as the functionality and fun of the social media is at the same level it is nowadays.


#271

Nopes. That is wrong. There is nothing, in current proposal, that can be deleted.

So, again, the datamaps needs to be stored somewhere. If it is on your own disk, well, that is basically the same - with regards to security and robustness - as if you had all your data there. The benefit of the network is nullified.

So you store them encrypted in the network, but then you need to go between these two ends:

  • granularity; encrypt each datamap, so that you can forget that key as to “delete” it.
  • Or you can have it all under your login (that is something you can actually remember, as opposed to datamaps), so when you want to delete one thing, you need to forget the login, as well as copying all that you want to keep to a new login.

#272

This has been a massive discussion to follow, and I believe that I finally grasp it. Funnily enough AppendableData is about what I thought SAFE would be at the start but this discussion has raised interesting points. I’ve also thought many times about the possibility of a ‘rubber hose attack’… the endpoint is likely to be the weakest of the network. One advantage that hadn’t previously occurred to me would be that an attacker, even with control of your account, could not delete your files as you could merely use a previous version to recover them. So they may be copied, etc. but there is no way for you to lose access to them. Hardware security like Fido-U2F on e.g. a ledger seems almost a requirement for the network, especially if you use the deniable second layer.

I buy that increasing storage capabilities will exceed the growth of data on the network. Particularly if publishers are smart about how they use immutables and the pointers to them to minimize the size of updates.

I would think that having MDs with true delete is easy enough. Delete could be simply zeroing the data and can be rewritten over rather than an append done. Sounds easier than the other. If you encrypt the data in the MD then noone can read it anyhow.

I definitely do see some advantages/usecases for having a fully mutable scratch pad, a la truecrypt container. Some of these use cases are highly relevant to some core SAFE missions… such as for things like email or deniability… imagine a datamap to documents or safemails about government corruption: the map could be shared and then deleted, with no way to prove it was you, even with an account compromise which would otherwise reveal you through AD versioning. Simply delete your account password and various governments will put you in jail for refusing to divulge. And what if you spam someone’s safe inbox with incriminating stuff? Wouldn’t you want to be able to delete mails like that?

It could be relatively easy to implement, particularly if it is of constant size, e.g. always a 1 full MB chunk reserved on creation. Updates could then take the form of binary diffs / zeroing with no need to save versions. Perhaps something for later, but then again if it is a small container, then storing each new iteration should not be stressful for space. I do think though that part of SAFE is that you aren’t scared to put things on it or use it to its full potential. Immortal things are immortal, but there will be bad actors who use use the network in ways difficult to fully map out, and having some control over what others try to send to you via apps etc. seems more comfortable to me.


#273

You mean something like Rc (reference counted value), you count the number of references to the data, if it fells to 0 you can remove the data. There’s are problems with that:

  1. you can have links in others systems, like the current clearnet referencing some images or scripts on the safenet (like a package manager eg npm)
  2. not even the safenet can see those, if the references are encrypted

#274

For simple device only local tmp data, as i already have written: use an encrypted local partition, which key is randomly generated. On logout you would “forget” the key, and make the data unaccessible. As long as you are logged in there will be a connection between your device and your safe account, so there is nothing lost by storing tmp data locally. Storing that data on the network would be relatively speaking really slow, as network is slow compered to local file access.


I guess, what he’s saying is, that the data is separately encrypted (not self encryption), thus you would be able to change the encryption key (your safe account password), making the old data (encrypted with the old password) not directly accessible.


#275

We did look at that previously, the list could be billions of entries watching a chunk for popular data. So the management is larger than the thing you are trying to delete to save space. Each reference will be min 1/32 of the chunk size, then managing that data is again hard, the vaults need to check the person saying delete is actually the person who uploaded it. Then you can have an attack where I somebody just adds themselves as a reference and deletes their own id. So no delete ever, but more management of the chunks etc.

Hope that helps.


#276

Well David said that account data is different.

The ADs are replacing the MDs. Nothing in the announcement of appendable data said anything about replacing these blobs types.

Argue it out with him.

But he did say that the blobs can be deleted, wiped clean.


#277

Nor can it see sneakernet transfer of the datamaps.

Nor maps kept on external devices and perhaps shared (ie sneakernet)


#278

Yes this is true. The account blob is entirely different. It is the thing you download and decrypt to login.


#279

Yep, that’s what I was suggesting and what the encfs library could easily allow. Although I think deleting the partition when finished would be cleaner.

I’m sure there are other alternatives too. and @neo’s idea of using MD’s in the account blob would work too, but maybe slower - although would be less temporary (and would travel between devices) which could be a plus for some things.


#280

I’ve never seen this other account blob.
From what I have seen, the account blob is today an MD, and if MD is replaced with AD, then that would be immutability with current proposal.
But if it is truly like this, that there is yet another type of data, that can be deleted, well then there are some more factors to it all… Can’t say yet how I would recompute that info. but I’ll be back :slight_smile:


#281

Self auth - quick overview.

You add one password - it encrypts stuff.
You add another it derives the location of the encrypted stuff.

You pay a bit of safecoin to store in that location and store your blob there. (you cannot store any old stuff, it is restricted and not shareable).

Go to any computer with those two passwords and you are logged back in with your keys and root dir all there for you.

So that is it. The stuff, is a blob, obviously it is things like keys and such, but also at least your root data map, which you may have several. So you have a bunch of data that is accessible from that root dir. If you then kill this block or just don’t ever request it then all the data from that root is inaccessible from anywhere, unless it was public. In that case it is all still there. All your private data though is gone.