RFC 55 - Unpublished ImmutableData

[neo]

@happybeing

the datamap is not encrypted means that for any immutable data, there is a map of pointers to chunks, which are the self encrypted data of the file. Or, in the case of small files (<3Kb?) the datamap itself will contain the encrypted data of the file. No other metadata is included in the datamap (so no filenames, dates etc).

@dirvine

Yes, this is correct.

This means that a private/unpublished file is available if I can find that data map in either my vault or in a packet being hopped though my node (or in my cache).

For published files then this is fine since its published. But what if I publish a file but only share the file with my closed group. Then others may see the file by chance without any of the group exposing the datamap due to the unencrypted datamap stored in vaults.

Obviously the risk is there for one of the group to release the datamap, but this unencrypted storage of the datamap increases the risk by a lot since vault owners may trawl looking for datamaps.

[dirvine]

@neo

This means that a private/unpublished file is available if I can find that data map in either my vault or in a packet being hopped though my node (or in my cache).

No you definitely should not find an open data map anywhere, unless it is open shared data. so if you as an app dev stored a data map (it’s data) in the open you would be making a huge error. The data map is your key to your data (all data). You will have loads of these, you can do a myriad of things here and you would in normal work. So 2 data maps concatenated is a file etc. etc. If you have say a million files, you end up with a single data map, that is stored on your account packet, which is encrypted at the client, so no data map should be published or seen in hops etc. Of course if somebody publishes a data map, then anyone can read the data (like a web site etc.)

@neo

Obviously the risk is there for one of the group to release the datamap, but this unencrypted storage of the datamap increases the risk by a lot since vault owners may trawl looking for datamaps.

Yes, any bad person could take private data and publish it if they wish, we cannot stop that. This is regardless of data map though, they can just publish. But a data map should not be stored unencrypted unless it is for public data.

[neo]

@dirvine

No you definitely should not find an open data map anywhere,

But you agreed just above to @happybeing that they are not encrypted

@happybeing

the datamap is not encrypted means that for any immutable data, there is a map of pointers to chunks, which are the self encrypted data of the file. Or, in the case of small files (<3Kb?) the datamap itself will contain the encrypted data of the file. No other metadata is included in the datamap (so no filenames, dates etc).

@dirvine

Yes, this is correct.

So I am confused. Is the datamap encrypted or not

Even if the datamap is stored as part of the file you said above pointers are not encrypted. And in the case of datamap stored (self encrypted) with the file the pointer to the data map is not.

I must be missing something

[riddim]

@dirvine

If you have say a million files, you end up with a single data map, that is stored on your account packet, which is encrypted at the client,

This happens automatically when uploading data - doesn’t it…?

By circumventing the api - are we talking about not using the c api you provide or not using the c api ‘the way it’s meant to be used’?

@neo

they are not encrypted

If data maps as object would be encrypted then you’d always need to enter a password to retrieve the data… So they (as pure data maps) cannot be encrypted

The question (imo) is the data map encryption/storage in the account automatically handled by the api or do devs need to set additional flags for doing it right?

[dirvine]

@riddim

If data maps as object would be encrypted then you’d always need to enter a password to retrieve the data… So they (as pure data maps) cannot be encrypted

Yes a file goes into self_encrypt and out comes encrypted chunks of the data and a data_map. You put these data maps into other containers, like directory structure and these are then fed into self_encyrpt etc. So a data map itself is not encrypted, but they are encrypted by using them, i.e. when you put them in containers etc. as the container is then self encrypted.

@riddim

The question (imo) is the data map encryption/storage in the account automatically handled by the api or do devs need to set additional flags for doing it right?

The account will hold an unencrypted data map which can (should) be the root of all your data, but that account is already encrypted by users inputting their passwords for logging in.

[dirvine]

@neo

But you agreed just above to @happybeing that they are not encrypted

Not by self encryption they are not, but when you gather them somewhere then they are concatenated and treated as a whole file, if that makes sense?

[happybeing]

@dirvine

Not by self encryption they are not, but when you gather them somewhere then they are concatenated and treated as a whole file, if that makes sense?

Does this mean that when a datamap is stored it is always encrypted because it is always stored in an immutable data (which is encrypted in the client)?

[dirvine]

@happybeing

Does this mean that when a datamap is stored it is always encrypted

Yes, so if it ends up in a chunk/file it is encrypted by self encryption. Users only really need to end up with a single data_map and this is encrypted in their account packet (so by the user and encrypted in the client as you say).

[happybeing]

Thanks for explaining all this David (when I’m keeping you from your sheep ) and it’s also helpful to understand how you are thinking about the APIs, and what can / would best be enforced by vaults. Very reassuring.

[nevel]

I think a picture/matrix will really help here to understand all the different datatypes. Well, at least to me.
Some volunteer out there? Shouldnt be that much work

[JoeSmithJr]

@dirvine

In these cases the validation is at the elder group of the holding section (datamanagers), the reason being the request goes all the way through signed. So on receipt the sig is checked, if that is the owner then the reply is sent to that owner ID.

This is a bit like capability based access control, just without all the good parts… It would be just a small step to things like: “read access right to signer with public key X” (and this credential signed by owner key, which in turn would validate the request). At that point, we have delegation. Extend this to mutable data and other forms of checks, and we hava full-blown capability based access control (including delegation to apps, not just users) without all the brain damage of ACLs.

[drehb]

@drirmbda

Vault owners need all data to be encrypted to avoid liability for storing (fragments) of data

@dirvine

I think as we launch we would see much more of safe client libs ending up in vault code.

To maintain the plausible deniability of vault operators, I’m not sure that the storage of data is all that matters. If the data is coming in unencrypted and is then just encrypted for storage, it seems that operators could be compelled to implement some filtering on the data that they can see before encrypting it.

[riddim]

Hmm - okay but all immutables have data maps - and for fetching an immutable I only need the xor address - right? So if the immutable is published with without restricted access on that xor name a unencrypted data map is probably stored …?

Ps:

I assume

@dirvine

and this is encrypted in their account packet

This is the behaviour for the nfs api

Pps: Cross-citing from the dev forum:

@bochaco

No, I just mean that if the ImmD you are trying to fetch is encrypted by another app, the call to app.immutableData.fetch(iDataAddress) fails because it cannot decrypt the ImmutableData, therefore you don’t get the Reader object

So if encrypted the data map is encrypted by the app - but to have a public accessible e.g. picture atm ‘standard way to do it’ is to use an unencrypted data map located on the corresponding xor name for it…?

[dirvine]

@riddim

• and for fetching an immutable I only need the xor address - right?

If the chunk is part of a file you need the data map to decrypt it, but yes just the address allows you to Get it.

@riddim

This is the behaviour for the nfs api

I need to ask @Krishna @bochaco or @joshuef perhaps. I am not as familiar with front end (yet), but these guys are all over much of this stuff at the moment.

[riddim]

it’s not like it would be difficult to encrypt all data maps and when referencing one (e.g. in an xor link) to include the decryption key too - it just would make the reference xorName+key instead of just xorName

It’s just a convention that would need to be agreed on

[neo]

@drehb

it seems that operators could be compelled to implement some filtering on the data that they can see before encrypting it.

Not only that, but bad actors could use it in an attempt to gain knowledge

[riddim]

Doesn’t this undermine the automatic DOS safety of safe through caching?

(sorry if that already was raised somewhere +just something that popped to my mind and thought I’d drop it here)

And what precisely is the advantage of unpublished data over published immutables (with encrypted data map)? when /why would I want to use this?

[neo]

@riddim

Doesn’t this undermine the automatic DOS safety of safe through caching?

Shouldn’t because only the owner can request that chunk, and once approved then the cached entry can be supplied. David says the chunk address is the hash(chunk hash + ownerID) and so is just an XOR address.

[riddim]

Sure - but if I know the xor address of a unpublished immutable I can just sent many many requests for the file with a false signature where I just do a bit swap somewhere then it would not be a lot of work for me but I would keep the network busy because it always needs to really check the signature and tell me I’m not the owner … So classic denial of service I would assume…

[riddim]

[If we’d only have immutable, appendable and (unpublished) mutable then immutable +appendable can be cached - certain access rights would be managed through encryption and sharing of public/private keys - and mutables could get a special (fee costing) non cached get for cases where the mutable entries need to be really up to date - we’d have the dos safety through caching and apps are incentivised to not do unnecessary get calls for data they just mutated but work with a local cache instead if possible (only do a full get on keys that have/might have been changed) - as random thought about dos safe data structures - not thought through though … ]