Account Packet

[loziniak]

Hi, what is this Account Packet all about?

Community Dev Alliance Roll Call

Plus folks, there is gonna be a huge push to update the docs and APIs for the data types (inc account packet as we need something to start any app). It’s not unnoticed at all and the team is being built up a bit, plus with libp2p getting in shape so can we. There is some big things happening in the background to help with launch and exchanges and so on as well. So expect incoming now and over the next few short weeks.

dirvine

[joshuef]

It’s a data type to allow for read/overwrite of a vec of bytes by an owner.

It is conceived initially as how we would store/sync eg:

• a list of files uploaded
• wallet activity
• other user space conceits we might want to easily sync between machines.

It’s addressed by the owner, and requires a signature to update (from the owning BLS key).

We previously (a long while ago) had a verison of this with the login user flow (and some basic browser interactions w/ WebIds etc).

[neo]

@joshuef

It is conceived initially as how we would store/sync eg:

I gather this is also to be used for the account record? If not then ignore the rest of my questions

With this new info, I re-ask my question from a while ago

Will the data there previously be retrievable? Mistakes happen and sometimes people want to revert to previous. Or account logged in and owner walks away and sibling/friend/whomever decides to mess with them and deletes stuff, and owner comes back to a wrecked account. Be nice if they can just rollback the damage.

[dirvine]

Currently this is a CRDT based counter managed packet. So if you screwed it over and ruined it then it’s likely gone.

The similar argument could be somebody gets wallet access and spends all your money.

I am not sure it’s a problem for us to solve right now, but instead get the API as solid as we can. Then look at ways devs can use these account packets to also have backups in place (many ways to do that).

[loziniak]

So it’s basically like a register, but without history, yes?

[Traktion]

A vector is usually a list of something. Certainly in Rust: Vectors - Rust By Example

Maybe this is a replacement for using registers (with history)?

From my experiments with registers, they aren’t very usable as lists. An append only list/vector would be a very useful data type to have.

It would be great if they could be append only, with permissions to be owner append or anyone append. Maybe only owner able to delete too.

Then we could have, say, a list of forum posts (by anyone), with a moderator to delete bad posts (owner), etc.

Or a DNS type system for name/xor address lookups, with similar rules.

[loziniak]

Vectors of bytes in Rust are just one of many ways of storing arbitrary binary data. Could be text, could be files, could be anything. Registers give the same functionality if I’m not mistaken.

[happybeing]

Registers are in some ways more flexible but harder to use. You can use them to hold a sequence of pointers to xor data such as a file or chunk of metadata.

awe shows how to do that and I understand that David supports this kind of use so I expect it to continue even as the API is simplified.

I believe you could do more than that, as in a sequence of directory entries, or a history of such, but that has yet to be demonstrated.

They can be write by anyone or just owner as shown by the examples (in sn_client I think).

I wasn’t aware that the account packet was effectively a rewritable store, so that’s an interesting addition.

[riddim]

@happybeing

I wasn’t aware that the account packet was effectively a rewritable store, so that’s an interesting addition.

We have mutable data that is not append only? Oo

Can it be set to be publicly readable? And just the write being limited to the owner? Oo

I really want to get that api!!

[happybeing]

I’ll have a little bet that it isn’t part of it yet.

I noticed that it went from API this week to basic API next week, and I read stuff into such things when there’s so little to go on.

[loziniak]

@happybeing

Registers are in some ways more flexible but harder to use. You can use them to hold a sequence of pointers to xor data such as a file or chunk of metadata.

I’m not sure what you are trying to say here. There is a method, that does just that – writes some arbitrary data to a Register. Sure, it has size limitation, but it does not have to be a pointer.

@happybeing

David supports this kind of use so I expect it to continue even as the API is simplified

I hope writing arbitrary data will still be possible as well.

[dirvine]

@happybeing

awe shows how to do that and I understand that David supports this kind of use so I expect it to continue even as the API is simplified.

100%. I think registers are complex by requirement (concurrent updates and merging etc.) but we can have seemingly simple API’s there that allow them to be treated.a bit like vectors @riddim but we need to keep in mind concurrent means conflict and those conflicts need to be resolved at the app. Otherwise it’s a network wide agreement on the correct next value (block etc.).

The Account Packet is a scratchpad really, so any key has one, it’s a simple credit counter with data in it. The data is app defined. So we can use it as

• Login packet (use a password to derive location and keys etc.) that holds your root dir, wallet, key pairs and more
• App defined cfg location (so any app can use this to keep state and config info)

And I am sure much more. It’s basic though and will lost history, so it’s not for public data, instead private data, non publishable data and configuration management.

[happybeing]

@dirvine

seemingly simple API’s there that allow them to be treated.a bit like vectors

This isn’t there yet David. I am able to do this because I had provided a PR which exposes the MerkleReg so I could use that to obtain a history.

The ClientRegister and Register APIs only have very basic functionality, plus access to the MerkleReg which when it was added Joshuef said would probably be removed later when they got to working on the APIs.

It will be interesting to learn the status and features of the account packet.

[dirvine]

@happybeing

This isn’t there yet David.

Yes, we need to add that or at least show in docs how to do it. The fear is folk think it’s a vector and ignore the fact it is crdt. What I mean is no matter what we do, concurrent updates produce conflicts and the underlying type needs to handle holding conflicts until the client app merges those conflicts to a new HEAD.

@happybeing

I am able to do this because I had provided a PR which exposes the MerkleReg so I could use that to obtain a history.

Yes, that was neat, I think that is what I am referring to above. It’s seeing it’s not a vector but can be made to look like one (to users at least). So users may see. linear history but app devs have to handle the fact that underneath it cannot be, so there is more work to handle conflicts.

We are debating internally and the issue we have is basically this

• Different nodes can and will hold different updates (until they merge this must be the case)
• As merges happen they all hold the same data (if no more updates happen)

This is eventual consistency and is very powerful. But it means our limit on register size is practically unenforceable as different nodes will have different views of when the register is full.

So we are likely gonna have to switch to:

• Allow registers to grow
• Pay for each entry (which is a must)
• Limit entry size (say 100B plus signature) - to prevent folk trying to hold actual immutable data there

So it’s closing in to see what we need and a load of your input has forced us to look deeper here. Not a huge issue to fix, but an important one.

@happybeing

It will be interesting to learn the status and features of the account packet.

That should be in place very quickly as it’s a simple data type. I think it will be very flexible, but these types can never be public, to prevent abuse.

The data types seems to be falling into place now and Benno has been working hard on the API, so we should see some dramatic improvements this coming week, I would hope.

I am fully bought into app devs requiring all the help we can give them. As we are resource limited in terms of responding to all issues and forum calls we can at least get the docs and tests updated to allow devs to have a solid foundation to build upon.

[happybeing]

Sorry to add to the list but examples will be vital, and for Registers in particular non trivial, worked and explained.

These are areas the community could help with, but we needed the dev forum for that - and were using it for this until a few months ago we were told it was likely to be shut down.

I’m not sure I will rejoin that effort because of quite a few things, but if I do I don’t expect i will be contributing here, on sub topics. I think that’s going to be another avoidable mess. I guess you read my proposal and so am waiting to see the response.

[happybeing]

I don’t think it’s wise to try and fix these issues on the fly. It time is not taken I fear we will see a repeat of the chaos and loss of momentum and opportunity as has happened with the beta rewards program on Discord, but for the whole project post launch.

Taking the API, there are things that have been known about and I expect forgotten and plenty not yet uncovered. Not just in the API, but it’s a good and current example.

What you’ve picked up now isn’t the whole story and quick fixes to that can just create more problems down the line. Joshuef acknowledged this month’s ago wrt Registers. But it’s clearly much more than the that.

Maybe to illustrate this, or to help with the quick fix, here’s something to add into the mix on Registers.

Creating a Register currently takes over a minute (which I opened an issue for a while back but has had no response).

Reading and writing to a register is much quicker but if the delay in creation is due to payment, and writing takes much longer than currently this will make Registers far less useful which would be terrible.

[loziniak]

@happybeing

Creating a Register currently takes over a minute

Gosh, I thought it was only me…

[happybeing]

I doubt anyone on the team knew this either, despite being raised in an issue. They just haven’t had time to do what’s needed.

[dirvine]

@happybeing

Sorry to add to the list but examples will be vital, and for Registers in particular non trivial, worked and explained.

The documentation tests should be examples of how to use these.

@happybeing

These are areas the community could help with, but we needed the dev forum for that

This is exactly where pull requests are valuable to allow community members to directly help with improving docs and doc tests.

@happybeing

but we needed the dev forum for that

I am keeping out of the dev forum discussion, It seems to have overheated to a significant level

@happybeing

I guess you read my proposal and so am waiting to see the response.

Do you mean it’s dev forum or bust kinda. thing? I am not sure I have read the proposal, but happy to if you can link it please.

@happybeing

I don’t think it’s wise to try and fix these issues on the fly.

I am not sure. I feel the API should be published and hammered in real time by app devs. I also feel it should be app devs screaming they want something and the API champion pushing back as much as feasible to ensure we don’t have an API that is too sloppy… I mean this in as nice a way as possible as devs will mostly want the exact API call they want to make life easier, but that can be very much detrimental moving forward. So we must find the minimum API that allows full access to types and how to use them and so on.

@happybeing

Reading and writing to a register is much quicker but if the delay in creation is due to payment, and writing takes much longer than currently this will make Registers far less useful which would be terrible.

For sure create is payment based, a minute seems like a bug though, But registers are CRDT so can be local first. Therefor we need to find clever ways of handling payments and so on. Getting the stuff out and used allows us to see these issues.

I am not seeing this as quick fix, but I do want to find such issues quickly. If we need a redesign of an area then it’s a pause for sure, but let’s get the API out and used by devs to find what we need to.

Let’s see where we get in the next week or 2 as I feel we will know much more.

[riddim]

@dirvine

Let’s see where we get in the next week or 2 as I feel we will know much more.

Would be nice to get something to test - api for coin tx including metadata 2oukd be cool too