I’m starting to color outside the lines here, but I think what I’m wanting to do is a valid use case of the network. I originally thought I would want an appendable file, but after @Traktion mentioned using a scratchpad full of pointers/addresses, I realized that was much better for my use case . The problem is, scratchpad types are encrypted automatically and there isn’t a builtin method to prevent that encryption step. After digging through the codebase it looks like it would be possible to work around this. The Scratchpad struct looks like this:
pub struct Scratchpad {
/// Network address. Omitted when serialising and
/// calculated from the `encrypted_data` when deserialising.
address: ScratchpadAddress,
/// Data encoding: custom apps using scratchpad should use this so they can identify the type of data they are storing
data_encoding: u64,
/// Encrypted data stored in the scratchpad, it is encrypted automatically by the [`Scratchpad::new`] and [`Scratchpad::update`] methods
encrypted_data: Bytes,
/// Monotonically increasing counter to track the number of times this has been updated.
/// When pushed to the network, the scratchpad with the highest counter is kept.
counter: u64,
/// Signature over the above fields
signature: Signature,
}
So to write an unencrypted scratchpad to the network:
build the Scratchpad struct by hand
put the unencrypted data you want to write in the encrypted_data field
write the newly constructed Scratchpad to the network with scratchpad_put
to retrieve the data, call scratchpad_get, however, because the Scratchpadencrypted_data field was written to the network unencrypted, you can just read out the data directly from the encrypted_data field without a key.
Kind of hacky, but I think it will work. Does anyone see any downsides to this approach?
And a side note to the Autonomi folks, would you be open to adding an optional argument or additional method to enable unencrypted scratchpad create/update operations? I’m happy to code it up and file a pull request if this is one of those “if you want it, code it” situations, but I wanted to make sure I’m not doing something that the team would be philosophically opposed to.
The concept of enforcing people to provide a encryption key is an anti feature.
I want to store (for the p2p chat) on a fixed network address publicly readable data (the same information multiple times encrypted with the public keys of my friends so only they can read it) containing my current ip address and used port for a hole punched connection.
I know this can be achieved by putting the data into chunks and then using a pointer to point to them… But the ip address changes at least daily over here and the last ones are of no use for anybody ever again… Just updating a Scratchpad would be a perfectly fine approach imho and wouldn’t require constant payments for data that is temporary of nature …
Can’t we just share the keys if we want it public?
I was studying the APIs a bit ahead of AntTP integration and it looks like the wallet keys can be very separate from datastore keys.
So, I was thinking to share the keys and let many people update the data. Each use must still pay to deter spam, etc.
May be different with scratchpad, but the pay once nature is a bit unusual.
Edit: I think multi key signing probably plays a role with data owners wanting to vet data too, etc. Maybe a side question though. The dervived keys looks powerful for lists etc too.
Last time I checked it was only possible to get to the content of the Scratchpad through the private key… Not the public key… Not because it would make a lot of sense but just because of the functions exposed to python… Which are ofc very limited and missing in part… So it possibly doesn’t reflect the rust situation…
So either I’d share my private key for all my friends to read (and change) the content I want to share with them… Or we are in the pointer /graphentry situation where every update costs and which generates insane amounts of junk data
EDIT:
okay - just had a discussion with chatgpt (and Claude about the autonomi codebase) - the issue (and the misunderstanding from you and me) is that BLS keys are no encryption keys. it’s a signature concept … so you get multisig out of the box and a very nice signing tool. But BLS keys are signing keys by default.
maidsafe did expand the standard bls implementation by threshold_crypto ( GitHub - maidsafe/blsttc: poanetwork/threshold_crypto using supranational/blst for sign+verify ) to enable public key encrypting and private key decrypting (in multisig land everyone just decrypts with their keys and then the resulting shares can be combined again to get the fully decrypted content without any of the participants exposing their private key / the combined private key being exposed)
…the downside to this is that it’s a one-way street and only public key encryption + private key decryption is possible - not the other way around (as in e.g. RSA keys)
it indeed is a technical limitation with regards to the keys … and maidsafe seemingly not wanting to allow people to store publicly readable data in a scratchpad
Edit2:
@dirvine / @qi_ma if you don’t want to allow publicly readable data (on nodes) through the official libraries … how about allowing the content of a scratchpad to be simply AES encrypted via the public key? not perfect but then people can use their own way to encrypt data inside on top too and it might be better than just unencrypted? (oooooor with an AES key that needs to be shared out of band - people need to share the public scratchpad address anyway; just adding a key info to it wouldn’t change anything for the public scratchpad creator but then someone just storing a pad wouldn’t be able to access the content)
For scratchpad, yeah I hear you. The pay once thing and mutable data make it tricky to work with as a public store.
Going a little off topic…
For append only data structures it may make sense though. E.g. for graphs, the descendents could be addressed from derived keys, but each entry is immutable. So, anyone could append, but no one could delete. A bit like defining a sequence of write-once pointers.
As it would cost to append, spam would be discouraged. The app reading the sequence of graphs could filter for spam and/or there could be a blacklist maintained by owner/mods.
if that’s an issue then it’s nothing being resolved by introducing read-limitations of the content … someone will release a “autonomi-on-steroids” that just allows public data on scratchpads … because we hear nothing about the apis I already was tempted to see how hard it is to do it myself and exposing it to python …
I do think unlimited free writes is risky. It could encourage everything to pivot around scratch pads, with the cost of everything else increasing to cover the load it creates on the network.
The economics on such things can be delicate, especially while upload fees are so high due to gas prices.
sure - but then everyone can edit it … I want to be the publisher of the mutable data - the others are meant to be readers
only cleartext
say I have 3 good friends
loziniak
Traktion and
happybeing
i have the public keys of them and can just put my contact info 3x (encrypted with each public key) into one scratchpad for contacting me … I don’t want to share it with all the world I just want to share with my friends …
…okay … maybe it would be possible with 3 pointers which I then all need to update on every ip change …
…maybe possible but super annoying …
the upside of the pointer approach would be that my friends don’t know how many other friends I have - they cannot see when I start blocking them because then suddenly only their info doesn’t update anymore
maybe the pointer approach indeed is the right way to go funny - I need to check if I can just write the info encrypted into a pointer
ps: still I think public readable scratchpads are of value - it might just be possible by accident that my info fits into a pointer …
I was looking at the source code and docs and there is a new_with_signature call, which leaves encryption to the caller.
Create a new Scratchpad without provding the secret key It is the caller’s responsibility to ensure the signature is valid (signs Scratchpad::bytes_for_signature) and the data is encrypted It is recommended to use the Scratchpad::new method instead when possible
So, create/put/get… repeat, seems like normal usage, if you want unencrypted data.
Would be nice to use the update call without encryption, but it doesn’t look hacky to avoid it.
Oh it’s just the private key for that one Scratchpad… But still sharing write access when you want other people to read data you publish doesn’t make sense at all…(and doesn’t necessarily lead to a good outcome… Depending on trustworthyness… One friend being super funny could write the Scratchpad counter to its max value and make it read only +break the whole thing…)
I’m digging into this in Colony. I’m able to create/geta publicly readable scratchpad, but I am unable to update the scratchpad. I tried duplicating the scratchpad_update() function from the client library, but I was foiled by the network and config private fields in the Client struct. I looked around for a while, but I haven’t been able to figure out a reasonable workaround. I looked at @happybeing 's dweb, but it looks like he ran into similar problems because dweb calls scratchpad_put again whenever there is an update, which I don’t want to do. The point of the scratchpad is to update for free!
We’ll see what they say. Did any of you figure out a different work around for this one? This would be a killer feature that has a lot of applications. But maybe it is just too good…
can be retired - content is replaced with a tombstone value and counter is set to u64::MAX, so it cannot be updated any more (the closest thing to deleting content on Autonomi), and
is convertible to a regular Scratchpad.
Feel free to use it as you see fit or just to get an idea how to implement it yourself. Updating works fine, btw and no need to modify the Autonomi client API to use it. Are you increasing the counter before uploading the updated Scratchpad?
Thanks guys. I figured a scratchpad_put() would take a payment because that’s what I saw in the code, but I never thought to try it to see if that payment would be zero I’ll write this off as one of those ‘late-night-over-thinking-everything’ events.
I’ve got to say, I’m thoroughly impressed with your rust foo. I like this type a lot. I’ll dig more into this tonight, with a side of copilot to explain the bits that are over my head Thanks for sharing!
Really appreciate the positive feedback! I’ve written a bit about typesafe keys here if you’re interested. And if you really wanna go down the rabbit hole, you can look at the EncryptedManifest type I built for Ark:
A Scratchpad-backed encrypted data type that requires four specific public keys for encryption, but can be decrypted with any single one of their secret-key equivalents. All enforced by the compiler and with minimal boiler plate code. Any valid key type automatically has a decrypt_manifest function that accepts the ciphertext and returns a decrypted, deserialized and validated Manifest instance. It’s almost impossible to use incorrectly.
I bumped into some current limitations of Rust’s type system when implementing all of that. Nightly would have helped, but I want Ark to build on Stable, so I wrote some small macros to keep the boilerplate low.
For all the things that are great about Rust, is still has a fair number of rough edges and limitations, especially when trying some more advanced stuff. I can’t count how many times I’ve come across some problem whose solution has been in development since 2018 but has completely stalled. I truly wish the Rust Project would get some paid, full-time PMs that simply go through all the old, open issues and push them to completion.
. The problem is, scratchpad types are encrypted automatically and there isn’t a builtin method to prevent that encryption step. After digging through the codebase it looks like it would be possible to work around this. The Scratchpad struct looks like this:
I’ll write this off as one of those ‘late-night-over-thinking-everything’ events.
Thanks for sharing!