RFC 55 - Unpublished ImmutableData

This means that a private/unpublished file is available if I can find that data map in either my vault or in a packet being hopped though my node (or in my cache).

For published files then this is fine since its published. But what if I publish a file but only share the file with my closed group. Then others may see the file by chance without any of the group exposing the datamap due to the unencrypted datamap stored in vaults.

Obviously the risk is there for one of the group to release the datamap, but this unencrypted storage of the datamap increases the risk by a lot since vault owners may trawl looking for datamaps.

1 Like

No you definitely should not find an open data map anywhere, unless it is open shared data. so if you as an app dev stored a data map (it’s data) in the open you would be making a huge error. The data map is your key to your data (all data). You will have loads of these, you can do a myriad of things here and you would in normal work. So 2 data maps concatenated is a file etc. etc. If you have say a million files, you end up with a single data map, that is stored on your account packet, which is encrypted at the client, so no data map should be published or seen in hops etc. Of course if somebody publishes a data map, then anyone can read the data (like a web site etc.)

Yes, any bad person could take private data and publish it if they wish, we cannot stop that. This is regardless of data map though, they can just publish. But a data map should not be stored unencrypted unless it is for public data.

4 Likes

But you agreed just above to @happybeing that they are not encrypted

So I am confused. Is the datamap encrypted or not

Even if the datamap is stored as part of the file you said above pointers are not encrypted. And in the case of datamap stored (self encrypted) with the file the pointer to the data map is not.

I must be missing something

3 Likes

This happens automatically when uploading data - doesn’t it…?

By circumventing the api - are we talking about not using the c api you provide or not using the c api ‘the way it’s meant to be used’?

If data maps as object would be encrypted then you’d always need to enter a password to retrieve the data… So they (as pure data maps) cannot be encrypted

The question (imo) is the data map encryption/storage in the account automatically handled by the api or do devs need to set additional flags for doing it right?

2 Likes

Yes a file goes into self_encrypt and out comes encrypted chunks of the data and a data_map. You put these data maps into other containers, like directory structure and these are then fed into self_encyrpt etc. So a data map itself is not encrypted, but they are encrypted by using them, i.e. when you put them in containers etc. as the container is then self encrypted.

The account will hold an unencrypted data map which can (should) be the root of all your data, but that account is already encrypted by users inputting their passwords for logging in.

8 Likes

Not by self encryption they are not, but when you gather them somewhere then they are concatenated and treated as a whole file, if that makes sense?

3 Likes

Does this mean that when a datamap is stored it is always encrypted because it is always stored in an immutable data (which is encrypted in the client)?

4 Likes

Yes, so if it ends up in a chunk/file it is encrypted by self encryption. Users only really need to end up with a single data_map and this is encrypted in their account packet (so by the user and encrypted in the client as you say).

9 Likes

Thanks for explaining all this David (when I’m keeping you from your sheep :sheep: :dog2: ) and it’s also helpful to understand how you are thinking about the APIs, and what can / would best be enforced by vaults. Very reassuring.

7 Likes

I think a picture/matrix will really help here to understand all the different datatypes. Well, at least to me.
Some volunteer out there? Shouldnt be that much work

7 Likes

This is a bit like capability based access control, just without all the good parts… It would be just a small step to things like: “read access right to signer with public key X” (and this credential signed by owner key, which in turn would validate the request). At that point, we have delegation. Extend this to mutable data and other forms of checks, and we hava full-blown capability based access control (including delegation to apps, not just users) without all the brain damage of ACLs.

8 Likes

To maintain the plausible deniability of vault operators, I’m not sure that the storage of data is all that matters. If the data is coming in unencrypted and is then just encrypted for storage, it seems that operators could be compelled to implement some filtering on the data that they can see before encrypting it.

3 Likes

Hmm - okay but all immutables have data maps - and for fetching an immutable I only need the xor address - right? So if the immutable is published with without restricted access on that xor name a unencrypted data map is probably stored …?

Ps:

I assume

This is the behaviour for the nfs api

Pps: Cross-citing from the dev forum:

So if encrypted the data map is encrypted by the app - but to have a public accessible e.g. picture atm ‘standard way to do it’ is to use an unencrypted data map located on the corresponding xor name for it…?

1 Like

If the chunk is part of a file you need the data map to decrypt it, but yes just the address allows you to Get it.

I need to ask @Krishna @bochaco or @joshuef perhaps. I am not as familiar with front end (yet), but these guys are all over much of this stuff at the moment.

2 Likes

it’s not like it would be difficult to encrypt all data maps and when referencing one (e.g. in an xor link) to include the decryption key too - it just would make the reference xorName+key instead of just xorName :man_shrugging: :wink:

It’s just a convention that would need to be agreed on :face_with_monocle:

4 Likes

Not only that, but bad actors could use it in an attempt to gain knowledge

2 Likes

Doesn’t this undermine the automatic DOS safety of safe through caching?

(sorry if that already was raised somewhere +just something that popped to my mind and thought I’d drop it here)

And what precisely is the advantage of unpublished data over published immutables (with encrypted data map)?:face_with_monocle: when /why would I want to use this?

Shouldn’t because only the owner can request that chunk, and once approved then the cached entry can be supplied. David says the chunk address is the hash(chunk hash + ownerID) and so is just an XOR address.

1 Like

Sure - but if I know the xor address of a unpublished immutable I can just sent many many requests for the file with a false signature where I just do a bit swap somewhere then it would not be a lot of work for me but I would keep the network busy because it always needs to really check the signature and tell me I’m not the owner :man_shrugging: … So classic denial of service I would assume…

3 Likes

[If we’d only have immutable, appendable and (unpublished) mutable then immutable +appendable can be cached - certain access rights would be managed through encryption and sharing of public/private keys - and mutables could get a special (fee costing) non cached get for cases where the mutable entries need to be really up to date - we’d have the dos safety through caching and apps are incentivised to not do unnecessary get calls for data they just mutated but work with a local cache instead if possible (only do a full get on keys that have/might have been changed) - as random thought about dos safe data structures - not thought through though … ]

1 Like