Questions about Chunks

[anon40790172]

@dirvine was just posting some more info on the storage of Chunks and the Vaults.

About the security of Chunks in the Vaults:

There are a couple of things we have done/can do here. Keeping in mind data is not persistent and will take a lot of effort to recognise already, but going further.

A vault creates an AES random key in memory and stores all data encrypted (again).
A vault uses a double encryption method, so stores with an AES key derived from real name. Then when a chunk is requested, we xor request against all known chunks we have to see if we have it (slower). This was in c++ code, but arguably can still be detected, although much harder.

Is simple and perhaps initially we should do this (a hour would code this in place), but we would have to keep in mind republishing of data between logins would be then not possible at all. So this affects network restarting on a global outage etc. (2. has same side effect)

So it’s under consideration, but as ever there are some side effects of such policies. Nothing wrong with keeping prodding though.

I have some more questions about the Chunks in the network.

1. I’ve seen some work has been done on compression. Will compression be enabled on the network before it goes live? Is it fast? like, how many MB’s can be un-compressed per second? My winrar takes some time to do stuff like that. Does this slows the network down a bit, or does the compression actually speeds it up,even while you need time to de-compress?

2. When someone alters the code of a client, will it be possible to not use self-encryption, and just put plain files (like a 1MB .jpeg) into the network? Your close nodes wouldn’t have a clue, do they? And would it be possible in theory to find that .jpeg back in someone’s Vault?

3. When a Chunk is PUT to the network, it actually goes to the datamanagers that are closest to the address of the hash of the Chunk. But before it’s actually stored, what do the datamanagers to the file? Do they alter it with another layer?

4. What about dispersion, is that technology still being used when someone PUT’s or GET’s data from the network?

5. And when I PUT a 4MB file to the network (4 Chunks), my 31 XOR-friends confirm I can PUT. After they do, to whom do I actually send the Chunks? Just the 4 nodes who are the closest to the hash of my Chunks? Or all the 31 nodes? Or random or dispersal etc?

[dirvine]

@anon40790172

I’ve seen some work has been done on compression. Will compression be enabled on the network before it goes live? Is it fast? like, how many MB’s can be un-compressed per second? My winrar takes some time to do stuff like that. Does this slows the network down a bit, or does the compression actually speeds it up,even while you need time to de-compress?

circa 280Mb/s at moment. Can be benchmarked though, best benchmark whole process. This is possible just now though.

@anon40790172

When someone alters the code of a client, will it be possible to not use self-encryption, and just put plain files (like a 1MB .jpeg) into the network? Your close nodes wouldn’t have a clue, do they? And would it be possible in theory to find that .jpeg back in someone’s Vault?

No, but a client could put a structuredData item, it’s very size limited. No plain data is allowed though. Not on SAFE but routing can if others wish this in other networks.

@anon40790172

When a Chunk is PUT to the network, it actually goes to the datamanagers that are closest to the address of the hash of the Chunk. But before it’s actually stored, what do the datamanagers to the file? Do they alter it with another layer?

They do nothing to affect content or name, they have a bunch of internal operations though, I can name them all at soem stage, but getting deep for now, it’s all documented and there in code etc. Mainly record who has stored a copy, monitor them, ensure integrity etc. Then farming rate calculation and issuance of attempts plus a bit more.

@anon40790172

What about dispersion, is that technology still being used when someone PUT’s or GET’s data from the network?

Not at the moment, it will be an efficiency saving though soon. Won’t alter API’s or data so can be after launch.

@anon40790172

And when I PUT a 4MB file to the network (4 Chunks), my 31 XOR-friends confirm I can PUT. After they do, to whom do I actually send the Chunks? Just the 4 nodes who are the closest to the hash of my Chunks? Or all the 31 nodes? Or random or dispersal etc?

You do nothing, the network takes it from there. The ClientManagers will send to DataManagers and then on.

Whew I really need to get back to work now