Safe Network storage features

[dirvine]

Please see above

[Josh]

@dirvine

I hope this makes sense

I’ll be the dumbass that says no .

@JPL

Small chunks, or small files, which are likely to be text files and contain readable info

And that is why, we can see bits of other peoples data.

They are small and most likely insignificant but a while back I could identify the fact that I had some of @Southside’s data stored because I could see enough to know that it was scotcoin files.

So are you saying that this will no longer be the case? Not clear to me that you are.

Also not 100% sure that it is a issue as is, but I can see the argument against it.

[mav]

@dirvine

on the very small files not being encrypted. I think there is confusion there. It’s true they are not in themselves, but they are part of a directory and that is encrypted

Is this more accurate as ‘if they are part of a directory’? Small files uploaded individually would still fall through the cracks, right?

Scanning my node chunks on CloserNet there’s definitely plain text content being stored.
eg this chunk
d3e5372436d7fde0e02ea0379d4d2bf62eba5ecf9c7cdc2c8e46b7fc7b326453
contains plaintext:

{"rustc":12752801693322442712,"features":"","target":0,"profile":0,"path":0,"deps":[[2404243893358302026,"build_script_build",false,352219009932321881]],"local":[{"Precalculated":"1.0.98"}],"rustflags":[],"metadata":0,"config":0,"compile_kind":0}

Anyone else with a node on CloserNet have this chunk and can confirm?

It would also be very easy to work out which chunks are unencrypted datamaps (most are not but some are) and then have access to those larger files from the datamap.

It seems like allowing a layer of encryption before self-encryption would be useful. Maybe a flag --encrypt-with-secret <secret> could work?

It would be nice to pre-encrypt by default and have a --public flag to disable encryption, but then we need to deal with credentials and it gets very messy for ux. We can do pre-encryption manually with pgp/age/etc but I feel unencrypted data should never hit the wire; we should try to somehow incorporate this as default.

edit:
safe files download test.txt d3e5372436d7fde0e02ea0379d4d2bf62eba5ecf9c7cdc2c8e46b7fc7b326453
returns the same data as the node is storing, so plain text on nodes seems quite possible (likely?) to happen.

[Southside]

@Josh

most likely insignificant

except when they are not…

the data you could see was the string “scotcoin” in a small json file amongst other artifacts of a failed build of a failed project. The dir was the right size for the upload test at the time. Totally insignifcant - in this case but visible because the individual file was too small to be encrypted.
Are we going to need to pad every uploaded file so all data is encrypted?

[Southside]

@mav

Anyone else with a node on CloserNet have this chunk and can confirm?

[neo]

@dirvine

So this structure all goes through self_encryption and you are left with a single data_map. That is your root directory, and unwrapping that gives you access to the whole filesystem that is only yours.

If this completely true then all I need to do is search my hundreds of nodes I run looking for that “single data_map” and i have somebody’s whole filesystem available to read.

What am I missing as i’m sure that someone would have thought to ensure the “single data_map” (root dir) is encrypted.

[dirvine]

A few things to point out here.

@Josh

And that is why, we can see bits of other peoples data.

@mav

Is this more accurate as ‘if they are part of a directory’? Small files uploaded individually would still fall through the cracks, right?

Right now folk have no choice but upload direct files, however this should not be a use case on launch. For the reasons folk are saying.

We should note this is all client side, nodes will store whatever we ask of them if the data is valid (which does not include a check for encryption)

@neo

If this completely true then all I need to do is search my hundreds of nodes I run looking for that “single data_map” and i have somebody’s whole filesystem available to read.

This root_dir component would be one of the items in your 3rd data type I discussed earlier. Let’s call it account packet. This is a scratch area you create on “login” it’s a simple data type you can overwrite and can contain any information. It certainly would be encrypted by default. In here could be private keys, root_dir data_map and more.

Right now the client has had almost no love. We are uploading files directly one at a time without structure, but the world does not have a use case for this at the moment. All of our data AFAIK is collections handled through web sites or collections of data like music lists and so on.

I belive all files will be part of such collections. I am pretty sure they are at the moment (any file you have on a hard disk is part of a collection).

So the client will not change this for the SAFE network, it will just allow that to exist on the network in a decentralised secure manner.

EDIT
I should say the above is not currently how we do things. We are focussed on network side work and that above is client side, but a distraction for the devs right now. However, the time is very close where we need the client API solid and this is how it has to work. We cannot upload files directly as we currently do.

Directories will be registers pointing to subdirectories and these should be private to users and encrypted. If folk wish to share directories then it’s at these points that the register can be shared or completely open (the web). I think then we will see client apps getting lots of love and the functionality of the SAFE network becoming much more obvious.

Right now though it’s all node side work and that is going very will with everyone focussed on that side as well as the currency part. Data handling is not a huge deal on top of this, but it’s a massive impact on how folk will use SAFE for data.

[happybeing]

Thanks David. As I understand your response you agree that right now, uploading data using the CLI causes small files to end up stored in plain unencrypted form, readable by those with access to the nodes storing them.

But, you say there will be a default way of uploading data in which this is no longer the case.

Please can you confirm or correct what I’ve said above, because your explanations aren’t clear to me and I’d like to be 100% sure what is planned here so I can be accurate when describing what SN will deliver.

[dirvine]

@happybeing

Thanks David. As I understand your response you agree that right now, uploading data using the CLI causes small files to end up stored in plain unencrypted form, readable by those with access to the nodes storing them.

Yes, this is 100% correct. Right now we are focussed on node side work. This is showing the network storing data securely, in terms of reliability. The encryption part of the story is all client side and that is where the API has to be super simple and focussed on data collections instead of single files as it is right now.

The client side to “fix” data handling is much less work but will have a dramatic impact on the API. That is the switch from random single files (albeit with registers) to data collections (i.e. your hard disk on the network).

[loziniak]

@trycatch

I’m trying to figure out how one would implement a public directory of file sharing for example with these data structures?

One of possible solutions could be storing a collection of file addresses as a Register under a well-known addres, like for examlpe hash(xor(your-public-key, "file-sharing")).

[happybeing]

By “your hard disk on the network” I don’t suppose you mean something based on the amazing TreeCRDT which @danda was able to demonstrate?

[dirvine]

Yes, very similar. It is a tree based crdt really with registers being the CRDT part. so yes 100%

@loziniak

One of possible solutions could be storing a collection of file addresses as a Register under a well-known addres, like for examlpe hash(xor(your-public-key, "file-sharing")).

Yes, lots of things like this will work well. Same for Music, WebSite, Blog etc.

[happybeing]

@happybeing

By “your hard disk on the network” I don’t suppose you mean something based on the amazing TreeCRDT which @danda was able to demonstrate?

@dirvine

Yes, very similar. It is a tree based crdt really with registers being the CRDT part. so yes 100%

[Southside]

@happybeing

the amazing TreeCRDT which @danda was able to demonstrate

Link please cos I’m lazy???

pretty please?

[dirvine]

Safe Network Dev Update - August 27, 2020

Summary Here are some of the main things to highlight since the last dev update: The CRDT Tree repo has now been made public, and the crate has been published on crates.io Refactoring Routing and Vault codebase to make use of async continues to progress well. Routing relocation refactoring work and parsec removal will be combined into a single PR, with this now in the final stages of development. We have a behind the curtain sneak preview of the SAFE Network App ‘Action Menu’ UX. […

[mav]

@dirvine

it’s all node side work and that is going very will with everyone focussed on that side as well as the currency part. Data handling is not a huge deal on top of this, but it’s a massive impact on how folk will use SAFE for data.

Thanks for clarifying. It makes sense to focus on the node stuff. Client side stuff will improve very quickly once the backend is robust.

[trycatch]

@loziniak

One of possible solutions could be storing a collection of file addresses as a Register under a well-known addres, like for examlpe hash(xor(your-public-key, "file-sharing")).

So I’m playing around and trying to do something like that in this repo - is this pretty much equivalent? Seems to generate the same address every time:

    const HONEYCOMB_PUBLIC_KEY: &str = "98b0807cde78204259d89a7ca510fe41a763774117efbdb0134c125b0a730567e8fb7699a7f6c686e7e5a013bfe6078d";
    let app_name = XorName::from_content(b"safe://honeycomb");
    let honeycomb_pk = PublicKey::from_hex(HONEYCOMB_PUBLIC_KEY).unwrap();
    let address = RegisterAddress::new(app_name, honeycomb_pk);

But the end-user client can’t find and open it, it fails and creates a new one on every execution. Not sure if I’m doing something dumb or running into a bug :

cargo run --features local-discovery                                                              ✔ 
   Compiling honeycomb v0.1.0 (/home/tom/Projects/personal/honeycomb)
    Finished dev [unoptimized + debuginfo] target(s) in 3.83s
     Running `target/debug/honeycomb`
🔗 Connected to the Network                                                                                                                                Created end-user client with public key: "a64d149be856cef31cdac420ce5a9a0dc8e382668c1f97ae20905746365dfcc2419dce78927a5fc4607ca0be706bd989"
Honeycomb XorName: 15f06f(00010101)..
Honeycomb public key: "98b0807cde78204259d89a7ca510fe41a763774117efbdb0134c125b0a730567e8fb7699a7f6c686e7e5a013bfe6078d"
Honeycomb public RegisterAddress: "15f06fe5513e937db30997890d6a0a6094cc42437b628e0cc039d200d1f0158198b0807cde78204259d89a7ca510fe41a763774117efbdb0134c125b0a730567e8fb7699a7f6c686e7e5a013bfe6078d"
Register not found, creating one at RegisterAddress(15f06f) { meta: 15f06f(00010101).., owner: PublicKey(18b0..bf39) }
Successfully made payment of 0.000000011 for a Register (At a cost per record of NanoTokens(11).)
Successfully stored wallet with cached payment proofs, and new balance 99.999999967.
Honeycomb register address: "15f06fe5513e937db30997890d6a0a6094cc42437b628e0cc039d200d1f01581a64d149be856cef31cdac420ce5a9a0dc8e382668c1f97ae20905746365dfcc2419dce78927a5fc4607ca0be706bd989"
Owned by: "a64d149be856cef31cdac420ce5a9a0dc8e382668c1f97ae20905746365dfcc2419dce78927a5fc4607ca0be706bd989"
Permissions: Permissions { anyone_can_write: false, writers: {PublicKey(064d..4d3e)} }

cargo run --features local-discovery                                                      ✔  32s  
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `target/debug/honeycomb`
🔗 Connected to the Network                                                                                                                                Created end-user client with public key: "a81e2a7a29448cbdb7d9f86dbd89793b8e4eac9a9502082522cd1d6be456452eed61a0e4214c4c0a84bfa0bce1a2f2f5"
Honeycomb XorName: 15f06f(00010101)..
Honeycomb public key: "98b0807cde78204259d89a7ca510fe41a763774117efbdb0134c125b0a730567e8fb7699a7f6c686e7e5a013bfe6078d"
Honeycomb public RegisterAddress: "15f06fe5513e937db30997890d6a0a6094cc42437b628e0cc039d200d1f0158198b0807cde78204259d89a7ca510fe41a763774117efbdb0134c125b0a730567e8fb7699a7f6c686e7e5a013bfe6078d"
Register not found, creating one at RegisterAddress(15f06f) { meta: 15f06f(00010101).., owner: PublicKey(18b0..bf39) }
Successfully made payment of 0.000000011 for a Register (At a cost per record of NanoTokens(11).)
Successfully stored wallet with cached payment proofs, and new balance 99.999999956.
Honeycomb register address: "15f06fe5513e937db30997890d6a0a6094cc42437b628e0cc039d200d1f01581a81e2a7a29448cbdb7d9f86dbd89793b8e4eac9a9502082522cd1d6be456452eed61a0e4214c4c0a84bfa0bce1a2f2f5"
Owned by: "a81e2a7a29448cbdb7d9f86dbd89793b8e4eac9a9502082522cd1d6be456452eed61a0e4214c4c0a84bfa0bce1a2f2f5"
Permissions: Permissions { anyone_can_write: false, writers: {PublicKey(081e..deb8)} }

[storage_guy]

For Right to be Forgotten I believe it’s considered acceptable to delete the database entries, or delete the file, or delete the key to the object. However the data was stored without going down a layer to the underlying infrastructure below that. No one makes you find the bits and make them all 0.

That would be the equivalent of just not referencing the key anymore because Safe is the equivalent of the disk drive: dumb (no offence), the lowest thing you can address with an application and you build all the sense on top of it.

What you have to do about Backups and Archives in the world of Right to be Forgotten is a whole different ballgame and doesn’t have any easy answers because they are specifically designed to restore things to how they were were - not an edited version of that.

Right to be Forgotten has turned into Right of the powerful to deny what would otherwise be easily findable facts about what they did.