High-level datatypes library

I’m thinking about joining all emerging high-level datatypes (implementation and documentation) in one library, or at least keeping a list of them for everybody to choose from and not re-invent the wheel. A common design patterns repository.

We have:

What we can think of:

  • A chain of pointers, addressed with sequence-derived keys
  • An indexed chain (some idea of that)
  • live event queues @Champii
  • anonymous unix pipes
  • stack/queue data structure
  • …?

What do you think about it? And how could we create this thread a wiki for everybody to edit?

8 Likes

Good initiative @loziniak - it would be great if this kind of thing would get sponsorship from Autonomi IMO, not just app comps like IF. cc @JimCollinson

I have some enhancements to Archive and other data types:

For Archives, I have a PublicArchive and PrivateArchive types, and code which will return whichever it finds at an Archive address. This enables the History<Tree> type to read archives of either type and can be used by other apps to do the same, using the dweb::files::directory::Tree type.

This code has to handle the fact that the Autonomi API doesn’t know the difference, so will return success when deserialising the “wrong” type, and return valid but incorrect content.

For all data types, but currently only in the REST APIs, I provide a name based interface where:

  • each app has its own name space
  • each type has its own name space

So you can use a single owner secret for all your apps (provided each uses a unique app-id), and the app can just give each object a name when creating it, and can retrieve any object using its name or address.

You can see and try this out in the dweb REST APIs by running dweb serve and opening the Swagger / OpenAPI UI with dweb openapi-docs.

It would make sense to provide this slightly higher level API to the Autonomi data types, not just in the dweb REST API but also as a Rust library either dweb-lib or another. It makes sense to have this in dweb-lib and then use that for the dweb REST API but I don’t know when I’ll be able to get to that.

6 Likes

I love this idea !
I have some more high-level data types that I want to experiment with, like some live event queues, anonymous unix pipes, actual stack/queue data structure, and some more.
They are just concepts in my head for now, and I wish to find some time beside Mutant to experiment with that

8 Likes

Do you think a common repo for such things would be ok? Everybody from community could have write acces, but only peer-reviewed and accepted PRs could be merged. Also, we could agree to a standards and rules we would stick to – documentation, examples, minimal dependencies etc… Would you be willing to contribute code to such thing? This could be then used as a crate in our own projects.

Or just better if everybody has it’s own libs, and just publishes some pointer with a description, so that people could look into one’s code and take what’s interesting for them?

2 Likes

Would that work on other OSes?

1 Like

I do like the idea of a centralized (sorry for the swearword lol) repo, with high-level data structures and abstract concepts ready to be used, battery included.
There is also a lot of value to get inspired from what is being build for IF and I hope more people would like get involved
TLDR count me in

4 Likes

If anyone wants write access to safenetforum-community · GitHub just ping me.

6 Likes

For the pipes, in theory yes they are abstracted from the OS. So, you wouldnt be able to use it like actual pipes on windows like in the linux shell (or maybe you can now ? I didnt touch a windows in a long time) but you could still use them in your programs like actual pipes

2 Likes

The futures::stream for retrieving async streams of data may be one for the list. It isn’t a library yet, but probably should be.

5 Likes

I like to use zeromq for inner-/inter-process communication and to parallelize stuff without the need for using locks.

that can use different types of protocols

  • ipc → inter-process but sadly in python not working on windows (but that’s a python related issue if I’m not mistaken)

  • tcp → works OS-independent for local comms but uses a port … which is a bit annoying but not a real issue …

  • inproc → name-based inner-process-communication that works on all OSes …

it comes with communication patterns/recipes too

1 Like

I like the idea of this thread! Maybe we can go even further and also include things like best practices, pattern / anti-pattern, common pitfalls, etc. - building something on Autonomi is still largely uncharted territory at this point.

I’ll start with something:

Typed, hard to misuse keys

(Rust specific)

In Ark, the project I am currently working on, I have do deal with a number of keys for different roles / functions / responsibilities. To make things more interesting, I am also dealing with key hierarchies, data that is owned by one key but readable with another, key rotations …
Long story short, things could get messy easily. I ended up with type system I am really happy with and is really clean to work with:

I use zero-sized marker types to define a context / purpose, eg:

pub struct ArkRoot;
pub struct HelmKeyKind;
pub struct WorkerKeyKind;
...

Base Key types with common functionality (excerpt):

#[derive(Zeroize, Debug, Clone, PartialEq, Eq)]
pub struct TypedSecretKey<T> {
    inner: SecretKey,
    ...
}

impl<T> TypedSecretKey<T> {
    ...
    pub fn public_key(&self) -> &TypedPublicKey<T> {
        ...
    }

    pub(super) fn decrypt<V: for<'a> TryFrom<&'a [u8]>>(
        &self,
        input: &EncryptedData<T, V>,
    ) -> anyhow::Result<V>
    where
        for<'a> <V as TryFrom<&'a [u8]>>::Error: Display,
    {
        ...
    }

    pub(super) fn derive_child<C>(&self, idx: &TypedDerivationIndex<C>) -> TypedSecretKey<C> {
        ...
    }
}

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct TypedPublicKey<T> {
    inner: PublicKey,
    _type: PhantomData<T>,
}
...

Most notably here, decrypt and derive_child are not public - they are supposed to be used by more domain specific functions that are public, eg:

pub type HelmKey = TypedSecretKey<HelmKeyKind>;

impl HelmKey {  
    pub fn worker_key(&self, seed: &WorkerKeySeed) -> WorkerKey {
        self.derive_child(seed)
    }
    ...
}

Type aliases are really great here. The fact we can have an impl for a specific type alias and basically treat it like its own type feels almost like cheating. I can easily make sure certain operations are only possible for/with certain types and its enforced by the compiler. Its much easier to reason about and virtually eliminates an entire class of crypto-related bugs.

An example with typed encrypted data:

pub struct EncryptedData<T, V> {
    inner: Ciphertext,
    _type: PhantomData<T>,
    _value_type: PhantomData<V>,
}

pub type EncryptedManifest = EncryptedData<WorkerKeyKind, Manifest>;

pub type WorkerKey = TypedSecretKey<WorkerKeyKind>;

impl WorkerKey {
    pub fn decrypt_manifest(
        &self,
        encrypted_manifest: &EncryptedManifest,
    ) -> anyhow::Result<Manifest> {
        self.decrypt(encrypted_manifest)
    }
}

pub type PublicWorkerKey = TypedPublicKey<WorkerKeyKind>;

impl PublicWorkerKey {
    pub fn encrypt_manifest(&self, manifest: &Manifest) -> EncryptedManifest {
        self.encrypt(manifest.clone())
    }
}

If you have a worker_key your can call the decrypt_manifest function and get a fully decrypted Manifest back:

worker_key.decrypt_manifest(&encrypted_manifest)

decrypt_manifest will not be available on any other key type or for any other encrypted data type. And this is fully enforced by the compiler and supported by any IDE. Neat!

This is just to give an idea whats possible. I’ve explored this concept much further in my project. More code here.

4 Likes

Fantastic guide, this is what types in Rust are for! I love the idea. Keys and derivation can get really messy.

Something like autonomi-book? I like this. Maybe for language-specific topics it could have a chapter for each language? Do you think if we created a repo for this, you could make a PR with what you posted here? Could you think about some guidelines for others, so that the whole thing would be easily digestible and consistent?

2 Likes

Is it possible to stream usual autonomi data, or does it have to be saved in some specific format? Would you like to add code for this to a common repo/library, or just make your own lib? Would there be any external dependencies for that besides autonomi and its dependency-tree?

2 Likes

Maybe it would even make sense to make a PR to autonomi crate with these?

Instead of app-ids we could use common data structures if we agree to some standards, like “forum” or “files-tree”, things like this…

1 Like

That sounds great. I’m all for it!

Sure thing

You can use all the normal autonomi data types with it, it’s just a typesafe wrapper. In fact, if you check the repo I linked to, I have higher level wrappers there too, for Pointer, Chunk, Register & Scratchpad.

No, not really. I use blsttc and zeroize, but they are in the autonomi dependency graph I believe. I have added Serialization / Deserialization to the base key types that uses bech32, but that could be omitted or made an optional feature.

edit: oops, sorry. I misread your second post - I thought it was also directed at me. Please disregard :sweat_smile:

1 Like

What about starting a companion crate instead - autonomi-util maybe?

I opened an issue about this for discussion with Autonomi and other devs, share here, but nobody has commented.

That’s a different use case. It useful for each app to be in uniquely identifiable so named data created by the same user with different apps doesn’t clash.

This is super. Funny I saw it just now, I posted today about extracting these things out of the code I’m on.
Great ideas in the thread, happy to join in.

5 Likes

I was thinking about autonomi-hld or autonomi-highlevel, so that we don’t end up with too much diversity there, utils can be anything… Do you think it makes sense? Are there any utilities you are thinking about, that would not be high-level datatypes?

No problem! This could be directed at you, perhaps this could also be somehow abstracted-out and useful for others as part of a crate, then it applies as well hahaha :slight_smile:

You mean this one: feat: Unify PublicArchive and PrivateArchive in a single type (for discussion) (edit: there’s also forum thread) ? Perhaps there were too few of us to have an opinion at all in the topic :slight_smile: I still don’t, because I still haven’t used Archives.

Sure, there are data specific to a particular applications. But do you think it’s unapplicable to common formats?

So, we count you in, right? Then I’m setting up a repo now, and everyone, who makes a PR, will be added to the team. Or right-away if you give me your GitHub IDs, I may not have them all…

4 Likes

Yes, absolutely.
There’s a lot that we can sort out, like everyone is mentioning with patterns etc, and identify needs and use cases and where things can be consolidated. We are many now who are doing very similar things in more or less different ways.

As already said, at least we can all document the specific patterns we use so that newcomers have a good overview of things.

I’m busy the whole day and the weekend, but my head is full of ideas :smiling_face:

6 Likes