Everything is fine, that’s what tests are for 
8 Likes
whistleblowing. Sometimes it doesnt need a 50k article by Pilger (RIP) to get the word out.
Sometimes a whistleblower has to act fast and has only time to pass on the minimum info.
I can think of several other cases where a <1k text file can be useful/
1 Like
I am not sure that would help?
You upload a single file and then somehow publish the link?
It would be the same as uploading a directory with a single file in may respects I think. But less code and no special case etc.
3 Likes
Yes good point.
Forget I spoke
I guess the is a general case that it should be easy to upload arbitrary single documents and for them to be encrypted regardless of size.
That seems like a base API that a variety of apps might want, then using their own system to index access them without having to go through an indirection API (ie encrypted register or directory).
So you are correct to say you can do this using a filesystem like API, but it may not be an elegant basis for an app if it wants to organise individual documents differently.
Just thinking out loud. 
5 Likes
That would almost be like a disconnected git branch (I think). So the reader of that location (parser) would need to see it’s not a direct dir so we would need to differentiate though, not a big deal, but it is something else.
Are you thinking of a scenario that some app wants to index only certain files in a dir but not them all and should not have to read the dir they are in but somehow wants a separate direct connect link, which has to be readable by at least them (and unencrypted data_map) or similar ?
Me too 
5 Likes
Sorry to be vague. I’m not in the right headspace it seems to think up examples but I suspect there are a variety of business logic that would benefit from having a simple one size fits all API for storing arbitrary, let’s call them ‘packets’ rather than files, of data which they want secure by default, but also may want to publish.
So this is at the level of the current API, a simple way to store packets of data.
Maybe a single configuration packet. Maybe a database containing packets of different types (values, maybe too small, JSON or LinkedData, images, a wallet or private key IDK).
My point is it seems an imposition that apps treat their storage backend like a file system and don’t have access to a more basic API (which is secure by default).
2 Likes
My issue is, if any unencrypted data is uploaded no-one knows on whose device it will end up. It would be easy enough to find by someone setting up a script to search for common words, and once found it could be a very convenient stick to beat the network with.
It also muddies the pitch to node operators. “Don’t worry, all the data you’ll be storing will be meaningless, unreadable chunks,” becomes “Don’t worry, most of the data you’ll be storing will be meaningless, unreadable chunks, except for a few that may be readable.”
8 Likes
Oops, you’ve just exposed a fundamental ‘vulnerability’. There’s no way around that last point with the current design.
All encryption is done in the client so there’s nothing to prevent an app* uploading unencrypted chunks - unless you move the encryption to nodes, which breaks the content addressing and increases the load on the network by pulling it in from the edge, and also complicates keeping data in motion secure 
*You can make it hard, by ensuring the API encrypts everything, but people can bypass the API and talk to the network using their own library to implement the protocol (client-node messages).
So if someone wants to upload unencrypted 1MB chunks of anything I don’t see a way to stop them.
6 Likes
This sounds like you are talking about an ‘object’.
I’m interested in hearing about whether there will be a way of treating the network as a great bit object store like AWS S3. That could be key functionality. A lot of systems store their data as objects these days rather than ye olde files in filesystems.
2 Likes
I think if you look at the current web/world then every addressable file is in a folder, whether a web site or on disk. I cannot think of a case where files are individually addressable without some root? They can be sent as bytes, but from an addressable standpoint I suspect they are all in some form of folder? (as I type I question myself here)
The problem the network (any network) has is that a jpeg set of bytes has similar entropy to encrypted bytes. So if somebody subverts the API and uploads tiny jpegs then we cannot do much. Same for zip etc.
True, but there is a way. It’s clumsy and I don’t like it very much, but it works. The nodes can store data encrypted with an ephemeral AES key or similar.
When asked for the data they unencrypt it and give it back.
This means on reboot all data is useless (as the key is only in ram in a session). It also means the user of the machine cannot read the data. But a downloader will get the right data, so if unencrypted then thats what they get.
I am not sure it’s a vulnerability, to store unencrypted data would/should (MUST) mean subverting the API.
Even if clients totally encrypted the data but published the key then we cannot do anything about it. Even if they use self_encryoption and published the data maps, we are in the same position.
3 Likes
Network Royalties are described in some detail in RFC 0061.
Is this accurate? If it is, perhaps a bug there @bochaco?
Should be 15%.
6 Likes
Yes, I think you should! 
What about key value and database etc. Ultimately most (not all I suspect) packets, not files [cough] end up stored in a filesystem - like our chunks for example.
But we’re discussing how an app wishes to implement and interact with it’s backend logic, which may be using directories and files, database, URLs, hashes, key value etc.
Which is why SN built a register (key value) and content addressible implementation. Some apps will want to use that API rather than be restricted to a file based backend on top of it I think.
In this instance the vulnerability is reputational (hence my quotes) and potentially very significant legal liability for a node operator.
Having illegal content on nodes creates reputational risk for the network and potentially serious legal jeopardy for node operators (which could be weaponised for different purposes by different kinds of bad actor).
I understand why your solution is not liked but think it needs considering. I note that it also solves the issue that gave rise to this discussion.
6 Likes
These are not files, though, but chunks of data. So that makes SAFE a huge DB and not a file store does it not?
Or is it both? (we are straying from just having full addressable files at a xor address mind you?)
i.e.
It does solve that issue for sure.
4 Likes
Well, strictly (correct me as necessary!) Safe Network is a content addressable store of packets of data. Which also provides CRDT shareable objects called registers which are append-only, versioned key-value stores.
There’s a base API for interacting with those core data types. For an app, it matters not how those how those are stored, only the means by which they are put, accessed, and where allowed, mutated, along with the performance characteristics of those methods (such as cost, latency and privacy).
Using those core features many kinds of application logic can be built, including the filesystem like service inside the account packet or personal ‘Safe’.
I think of the personal Safe as an application and the question is whether to limit other applications to using an API which sits on top of that, or to allow them to use the core features directly.
I’m not really sure that’s what you mean because it seems sensible to allow applications to access the core features directly. But it seems to be what you are saying in order to avoid the problem of self encryption not encrypting all packets before uploading.
So I’ll leave it there in case what I’ve stated needs correcting before continuing.
5 Likes
First, I am delighted to be in a position for these chats. Not having to think of network implications is like having our hands untied 
I think we have strayed into 2 distinct, but related areas.
- File handling
- K/V type storage
For 2, I think you are spot on. Registers provide us with a fluid KV store that is CRDT, retains history and more. This will allow a huge amount of applications in the data handling fields.
For 1. We have the questions of how they are held on the network. I think a lot of the above convo MIGHT? show they should be held in a dir/folder structure and to keep that as simple as possible.
Then a further part of the covo, encryption (privacy etc.)
Here are some thoughts (Critique as hard as we can but default to simplicity where possible)
-
The files API should default to self_encryption and the dir type structure that ensures everything is actually encrypted.
-
The KV API should impose no default obfuscation, but give the user complete control over content (but each entry limited to 32bytes or whatever).
-
The KV store should allow encryption as an option.
-
The Files API will use the KV underneath but not expose it.
-
Register encryption for the FILES API will be by default and keys (AES likely) will be derived from
Hash(the users (owner) base secret key PLUS Register address) -
Register encryption for KV API may be similarly derived, but will be application dependent and an option for developers to create unique or group derivation and access etc.
7 Likes
I think my point is that it makes sense there be a simple API for putting packets of data to the network which an application indexes however it wants, perhaps storing their address in a register, or in another packet etc.
And the contentious point is whether those packets should, by default, be encrypted regardless of size.
It is confusing to say that storing a packet above a certain size using the self encryption API results in encryption, but below it does not.
However the indexes are stored, if apps have access to the self encryption API for storing content addressed packets, it’s going to cause problems (unless everything is encrypted by default).
If apps don’t have access to that, it’s a loss of functionality or at least a more complicated route, leaving apps to do it themselves (unlikely), or force them to only store packets using a higher level filesystem API.
1 Like
I see where you are coming from. My view is self encryption is incredibly powerful, but designed for files, so it is likely best kept for that. Large data items.
With dirs etc. all files can be sure to be encrypted properly via the files API. IFF we also encrypt those dirs (registers) (see below)
So then we have packets (names good so we should stick with it). So these can be anything, open or encrypted by any cypher. So the KV API should perhaps supply a different encryption API for keys and values. This is worth thinking about. If we link that encryption to BLS then we have group capabilities by default.
Therefore, I propose we have a separate KV OR register API (see above) that is aimed at encryption with sharing taken into account.
4 Likes
Without considering what you’ve just said too closely, sorry (!), I want to add that I’m keen to have a single unified - content addressable - API for all packets regardless of size.
It seems a design limitation that self encryption is seen as not able to deliver that, but perhaps that’s something we choose to live with.
Also, nit picking: nobody gets to say files are only used for “large data items” and get away with it.
3 Likes
As this goes away if nodes encrypt everything, perhaps we should conclude that discussion first unless it is off the table?
2 Likes