[RFC] Data Hierarchy Refinement

[jlpell]

@oetyng

This is a proposal describing a system where all data types are built of chunks, and together with decoupled metadata gives uniform data handling, for all types of data structures. This also solves the sustainability of the AppendOnlyData / MutableData types.

Nice work. There is a lot to dissect here after reading through it a few times. The only way for me to give feedback in a coherent manner is to pick through it line by line as a running commentary. Although crude it is the only way I can manage a response in the wee hours of the morning. Here goes:

@oetyng

All content is built up from chunks.

Good. I would go one step further to require that the “chunk” datastructure is a base unit used in the construction of ALL other datatypes in the hierarchy following an OOP construction by assembly approach, including metadata.

@oetyng

The names of the data structures are the following:

• Blob , whose structure is just a single blob of data.
• Sequence , whose structure is a sequence of data.
• Map , whose structure is a set of unique keys, and a set of values of data, where a key maps to a value.

Based on this RFC and past discussions in the other thread on Data Types Refinement you have convinced me that the term Blob is an absolutely horrible descriptor for what you are trying to accomplish. More on this below.

@oetyng

A Shell instance holds information about a data type, such as the actual structure with the pointer(s) to the chunk(s), name, type tag, ownership and permissions.

It is unclear here if you really mean data “type” vs. an instantiated data “object”.

@oetyng

To chunk or not to chunk

Just chunk it. Chunk early and chunk often.

@oetyng

The most recent design removes the hard-coded limit (i.e. entry size and count is now unlimited), but does nothing to solve the problem: a group of 8 nodes will typically have a very limited storage size, which will act much the same as a hard cap on entry size and count.

Interesting insight. I thought that if the local group of 8 did not have enough storage than the nearest neighbor search radius is expanded to include more than 8 nodes?

@oetyng

Making a Blob Public would today change the XorName of the data (as private XorName is hash(owners + contents) while for public it’s just hash(contents) ), i.e. require the data to move in the network .

I agree that there is an opportunity for improved deduplication here.

@oetyng

The idea of symlink is born

I like where this is going. Viewing SAFE as a big hard drive in the sky with analogous operations/functions to a common tried and true filesystem like ext4 or xfs will help speed development IMO since you already have a stable, working and well documented model of what you are trying to accomplish at a grander scale.

@oetyng

Metadata and chunks

The data stored to the network, i.e. chunks, doesn’t change. Chunks are added or deleted. What changes is metadata . So, a specific chunk, should only ever have one XorName . Basically, it is hash(chunk.payload) and so resides in one place in xor-space regardless of Private / Public scope.

@oetyng

The conclusion of this is the following:

Data has no reason to move around, other than section membership changes that would require it to be copied over to new members.

Not only in code, but even physically, what changes often should be separated from what doesn’t. This begs for metadata and data being separated.

If not careful with the definitions this could lead to some circular dependencies since the metadata needs to be stored somewhere too. Consider as an example the EXT4 filesystem where we have data blocks and metadata blocks. Regardless of data block type (meta vs. actual) they are all stored in fixed block sizes on disk (typically 4kiB to match hardware sector size). I view the EXT4 block on disk to be analogous to a SAFE chunk. This indicates that your metadata should ultimately be stored as a “chunk” too if you want to keep the logical consistency and benefits of assembling a well defined object hierarchy.

@oetyng

Terminology

Specific comments about terminology:

@oetyng

Gateway nodes : A category of nodes performing validation and acting as a barrier and gateway to further access into the core of System nodes .

• System nodes : A category of nodes performing core functionality of the system , such as storage and retrieval of data.

Nice. I like the differentiation here. This could also be generalized to N layers extending from core to boundary. A few synonyms that evoke different imagery for the case of N =3:

• “Gateway Nodes”, “System Nodes”, “Kernel/Core Nodes” for a computer reference.
• “Exterior Nodes”, “Boundary Nodes”, “Interior Nodes” for a spatial reference.
• “Frontier Nodes”, “Border Nodes”, “Control Nodes” for a geographical/political reference.
• “Peripheral Nodes”, “Passing Nodes”, “Principle Nodes” for a roles reference.

@oetyng

• Client nodes : The subset of Gateway nodes that connect to clients and manage their balances. Corresponds more or less to ClientHandler at Elders .
• Shell nodes : The subset of Gateway nodes that holds Shell instances, and validate access to them and the chunks they point to. Corresponds more or less to DataHandler at Elders .

The term Shell is not used appropriately here and also later in the document. In computing the term ‘shell’ is synonymous with a user interface that allows access to an operating system, its programs and services. It is confusing to equate shell terminology with pure data and datatype constructs unless you are specifically building to a shell program like the SAFE CLI. I do like the simple and self-explanatory definition offered by Client Nodes

@oetyng

Chunk nodes : The subset of System nodes that hold data in the network. Corresponds more or less to DataHolder at Adults . Should only care about storage and retrieval of chunks, and be oblivious to anything else.

Programming wise, if chunks form the base object in an OOP hierarchy from which other types are assembled, then your metadata should also be stored as chunks. This means that all nodes would store and retrieve chunks, but nodes dedicated to dealing with metadata would store and retrieve metadata chunks, and nodes dedicated to data would store and retrieve data chunks. For this reason I would recommend using the terms Data and Meta Data. To maintain continuity with previously employed terminology I suggest using Data Vaults and Meta Vaults here.

@oetyng

Blob : A structure of a single Pointer . (Usually used for a single large file, which is stored in a chunk or a set of chunks, hence only necessary with a single Pointer ).

You seem to really like this term, but it is not a good designation for what you are trying to achieve here. This is made more evident by the picture you drew below. Really what your are designating by your Blob and Sequence is an Unordered Set vs. an Ordered Set. The mathematical definition of an Ordered Set is essentially a Sequence (when duplicate entries allowed) so you’ve got that one. Blob on the other hand evokes no intuition of a Set. So why not just keep it simple and call it a Set. You could also extend the Set terminology to a Collection where duplicate entries are allowed.

@oetyng

Shell : Contains a higher layer representation of the data held by chunks, also all its metadata.

Everywhere else in the computing world this is called a File. The use of Files in the network is OK. KISS. “Everything is a File.” And like I said earlier, shell is usually reserved for user interaction with programs/services. Later on when SAFE has a computation layer, I could see “ShellNodes” as being the perfect description of an interface to this layer. These future ShellNodes would handle the running of SAFE programs and processes as part of a general SafeOS.

@oetyng

• ClientNodes(id) : The 8 Client nodes closest to the client_id .
• ShellNodes(id) : The 8 Shell nodes closest to the shell_id .
• ChunkNodes(id) : The 8 Chunk nodes closest to the chunk_id .

ClientNodes, FileNodes, DataVaults, MetaVaults.

@oetyng

The Shell holds the metadata

I would be happier if you replaced the term “Shell” with “File” in this large section.

@oetyng

Pointer holds either ChunkMap or XorName

Seems inconsistent to have specialized ChunkSet and chunk_map types. Wouldn’t it be preferred to build this higher level types from lower level ones? So this way a chunk_map is instead Map<Chunk>.

@oetyng

If the Shell size exceeds some pre-defined size, the Shell is blobified , i.e. stored as a Blob

This is a problem. Nothing in this world should ever have the indignity of being blobified at this high level of abstraction. I suspect that it would only get too large due to the owner history and permission history. Better to change these constructs from a Vec to a Sequence so that all of the #blobification can happen under the hood.

@oetyng

In the case of Scope being Private , we allow deletion of the actual data. By refcounting the chunks, and only deleting the actual chunk if the decrement results in 0.

Perfectly logical and follows standard filesystem practice. However, is there a chance for a security exploit here where the refcount could be decremented maliciously?

@oetyng

Storing data

If the content size is too small to undergo self encryption, client has to encrypt the data before producing the ChunkMap .
A wrapper around the self-encryptor could handle this case, to ensure that any data leaving the client is always encrypted, as per the fundamentals .

You forgot one of the best features that is possible with this approach. The chunks can be encrypted again by a subtype of Gateway nodes prior to being sent to a Data Vault (your ChunkNode) and decrypted when retrieved from the vault. These keys would only be known by the Gateway layer, not the Client nor the System Layers.

@oetyng

What we achieve

The above achieves the following:

All nice to see. I also like your data flow diagrams. They really help make things easy to understand. A lot of possibilities here.