Safe API - Registers

It was working on that that helped me come to my primary view. The discussion uncovered reasons for the other view. I can’t say which is best overall now, problems both ways so we need to suck and see.

7 Likes

Main thing is the discussion happens and means something, whoever is right. I think we have something powerful here and done correctly it will change our view on data/knowledge.

11 Likes

Could be. I kind of imagine some higher level wrapper over the low level register with warts + limits, eg.

I agree with that, we’re not going to hide any lower level API. But the pointers or no Q is about what that lower level API is doing for you…

So right now, @bochaco is fleshing out a simple file management using registers. The simplest we could imagine (ie, not attempting to be a fully fledged filesystem as yet). But basically getting us minimal metadata around files etc.

Our current thinking is to get that metadata in chunks, and we’re seeing how that’s going. (in both dev experience, and practicality terms).

The more we chat this through the more I think the simplicity of NetworkAddress pointers, makes a lot of sense. (It raises less Qs about magic numbers), and we tie the size to that of chunks and go from there. With that we could see what the impacts are, and make further decisions on this front.

5 Likes

Just wanted to add a couple of important details to the debate here.

Registers are currently stored in a Record just like the other data types on the Network (Chunks and Spends). Those Records have a fixed size limit that is configurable but it affects all other data types as well. This limit is currently 5mb.

Also we need to keep in mind that each Register entry (one entry = one version) comes along with its CRDT data and signatures so that’s an extra constant weight for each entry (roughly 500 bytes).

Smaller entry size means we can store more entries per register in total. Bigger entry size means we waste less space as each entry comes along with 500 bytes of data necessary to the proper functioning of registers.

Assuming the Record size is 5mb and the extra data per entry being 500 bytes:

  • If we say we want register entries to be 1024 bytes, that means each entry takes up ~1524 bytes of actual space. That means we can store ~3280 entries in a Register. That’s 3280 * 1024 = 3,358,720 bytes of actual entry data.
  • If we say we want register entries to be 32 bytes, that means each entry takes up ~532 bytes of actual space. That means we can store ~9398 entries in a Register. That’s 9398 * 32 = 300,736 bytes of actual entry data.

So with 32 bytes we can have 3x more entries per register. But with 1024 bytes we’ve stored 10x more data in the register as we’ve effectively wasted less space with the 500 bytes overhead per entry.

Now the question is, what do we want?

6 Likes

Are we confusing entry data with a key? A key can be unique (32 bytes is way enough) And point to any chunk (data) it wants, with no limit. Then chunks carry metadata/parsing rules or whatever and registers stay as key only pointers.

If registers were meant to store actual data they would be a terrible design.

2 Likes

Yes those would be the amount of keys we can store : 9398. Since each key is 32 bytes, 32 * 9398 = 300,736 is the amount of total (key) bytes a register can store. Then of course each of those keys can point to more data and so on. But the max amount of pointers or keys a register can ever store is limited to this currently.

1 Like

For those interested in Safe Register data type (mutable data) there’s a new example which extends the existing simple chat example by allowing you to inspect the structure of the Register as the example progresses.

Actually you can use it to inspect any public Register, but it works particularly well with the simple chat example for now.

5 Likes

How to History?

I created the above register_inspect example with the intention of showing the Register history which is advertised as a feature of the MerkleReg data type used to create the Safe Register data types but have been unable to see how to do this in a generic way. Hence it now inspects structure rather than shows history.

I’ve concluded that for all but trivial cases, such as a single value per Register being overwritten repeatedly, there is no way to provide a history without including logic specific to a particular configuration.

So for many real world use cases the application needs to provide a way to navigate the history within the Register, a bit like leaving breadcrumbs.

I’d like to be contradicted on this and shown a generic way to access history of any Register. So if anyone familiar with the ClientRegister for example, can explain how to extract history from say a tree structure (like a filesystem) without seeding the root nodes to give access to the ‘trail’ I’d be very interested.

Without ‘seeding’ the Register with a trail I don’t see how to access history for application specific purposes using a generic high level register history API.

To avoid limiting applications this may also mean leaving the MerkleReg exposed, as was necessary for me to be able to create the above example.

Can you comment @Anselme? Anyone else who’s looked at Safe Registers or MerkleReg?

2 Likes

Absolutely. 100% with you in this one and as you say there is no current implementation to do so.

My opinion is that the register API should include high level history related methods to allow users to view it. This will require reading data from the underlying MerkleReg but not leaking it out to the users to keep things simple.

Definitely should be in our TODO list.

3 Likes

I don’t understand the technical language but this seems important in the context of “setting managing the multiple entries for an algorithmic promise/set of values at a particular time” for example e.g. your genetic open ended algorithms on safe? …or am I missing the point. I think your filesystem time-machine style concept need further examination. :blush:

2 Likes