It would be desirable to do this especially if chunk sizes are to be increased. But even at 1MB so many files would fit in a single chunk I assume there must be a very good reason for not doing this.
I just can’t think of one!
Most websites, most user created documents etc would fit in a single chunk. This would reduce storage costs and network load by around 60%.
From my understanding from discussions around that time the change was done was
small files at the time where upload unencrypted
solution was to allow self encryption work on as small as possible. Is it 3 bytes or 15/18?
part of the increased “quantum” protection is to split the file up so no one chunk contains the whole file
You could perhaps, for websites, have an object that reads a file and splits it up into separate file. Like if your site has many images then some images are packed into one file (zip?) and the object unpacks the file when retrieved.
You could have a custom client that creates one chunk only for files less than max chunk size and just upload that.
This was as data was being uploaded in the clear. Now it’s all encrypted essentially (unless < 3 bytes).
As a default this seems sensible to me.
If you were to manually encrypt the data eg, you could have that happen. Probably that could be another client-side option (this is all client side). To allow for such optimisations down the line
Are you saying it would be hard for SE to encrypt/decrypt the data automatically when storing a small file in the datamap? I’m not clear how SE works exactly, so not sure if it just isn’t feasible.
If it is feasible, it seems like a very desirable feature to include as a seamless feature of SE. But maybe you’re saying that’s not possible?
Well - I guess the 3 encrypted parts can always be glued together and put into 1 chunk - so it is possible for sure
I think he is saying it’s not on the roadmap as of now but can easily be done later on because it’s client side and not affecting network operation at all
… But then again he can just speak for himself =D…
Do we have any data on uploading large zip files that then get broken up into chunks to get some real test results to ensure downloading that chunked zip file actually stitches together and works? Double encryption gives me the willies.
Same Question to validate large unencrypted compression files?
Hmm, it’s where/what you decrypt with that’s the thing.
I think essentially it’s out of scope for self encryption (it’s no longer self encrypting here).
So you want some other means of encrypting and managing keys there. There vault / store of your own data could well do this, and this is what’s imagined to happen down the line.
And just to underline, there’s nothing inherent in the network blocking us here, it’s client side data organisation alone that needs to be standardised really.
My point is that it would be good to have this in the first API to keep the file/datamap format standard. So the client API just generateds the datamap as needed according to file size.
Apps then don’t need to handle special cases, do their own encryption, worry about keys, know about datamap format etc.
@joshuef suggests it’s about security, so there could be an issue there idk.
Security could though be handled by a flag: always self-encrypt or for small files store inside the data map.
That’s cleaner, gives massive early benefits and also simplified more efficient apps.
Doing this in the API later is tricky because you end up with two file formats and no clean way to know which you have.
What will happen in practice will then depend on app devs, but most likely they will use SE as is so we lose the benefits for some time, and make it hard to change later. And any apps built up to that point will probably continue to do things the old way because reasons.
My take here is:
the API could be implemented in a way that dramatically reduces storage costs and network load
there may be use cases that require the option (default?) to always self encrypt
this won’t be implemented now because it is not seen as high priority
implementing it later may fall by the wayside because it won’t be able to be done cleanly
I always knew that a client could do this itself, but that wasn’t my question because ad-hoc doesn’t deliver the benefits - to do that it needs to be in the API, and probably from the start.
providing vastly cheaper and more efficient storage options in the API (as previous reply) using a common datamap format. (I agree this is out of scope for SE, but that’s not the issue. The idea is to have this in the storage API.)
versioning in the datamap would allow this to be added later. @joshuef does the datamap format cater for this? If not I think that at least should be a priority.
support for storing data to datamap style API from memory. Currently the API for this requires a file, which is inefficient and will lead app devs into bad practice (writing to and leaving data on the local device for example). There’s an old issue open for this but no replies yet AFAIK.
My second point was wrong as this would break the xor address for the datamap which is a proxy for the file address.
This also means that if this doesn’t go into the API from the start it can’t added later or the same for will have different addresses between one version of the API and a later one.
That pushes this enhancement into the app layer which would be very bad: killing universality which means to read some files you will need a particular app and not just any app using the API. It also means the same file will have different addresses according the the app which created it, and whether it used the API as-is, or a more efficient version built on top.
@joshuef For these reasons I don’t see that this can be done later as you suggest. It either needs to be in the API from the start, or it creates a host of problems that put the very large benefits out of reach forever or makes them app specific which reduces to marginal and only specific use cases.
Since the encryption/decryption key is derived from the hashes of the 2 other neighboring chunks putting all the chunks together results in giving away the key.
Separating those 3 chunks and sending them to each edge of the network with no apparent link between them is what makes the key secret.
No worries! I’m on my free time here, scrolling the forum at 1am for fun and no profit
The datamaps are not encrypted. Which is why we have the vault to keep them safe. Another rabbit hole!!
And with the content of the data map people can query the chunks and it’s basically the same as putting the chunks in it in the first place isn’t it? (or am I missing something?)
