Transactions - a chained type to replace CRDT Registers

Problems with:

  1. It is a driver for centralisation. To keep the app working you have to accept the changes from the author, so it is a weakness that will be exploited by the unscrupulous.
  2. Hits the problem with downloading the blockchain.

I think both to lead to centralisation which I, er, thought was a bit of an issue. :man_shrugging:

5 Likes

I am not in agreement about centralisation unless itā€™s coded that way. I am all for flexibility.

Registers as is has all the chain type behaviour, its just not implemented yet, but when they get to 1Mb then they stick unless we chain them. So chaining them is like chaining transaction, but in bunches instead of individually.

Now registers are CRDT DAGā€™s in the back end. The means we cannot chain them. Thatā€™s a pretty hard issue to get over. But essentially if we could get registers working properly they would also be a chain.

So with these Transaction types we can transfer stuff with them (like currencies) or transfer ownership of a thing (so ETF like) or use them as something like old school pointers to directories and such like. ( I think this pattern may prove to be wrong in a decentralised world mind you, anyway) If we use these as say updating pointers to content like web pages/directories etc. then it looks more like a chain.

So letā€™s dive into each to see how it helps (or hinders)

  1. Currency
    -The content is a value

    • The app checks the value is positive and all outputs are positive and == value
    • The app checks parents (back X parents) for uniqueness and the correct outputs lead to here
    • The app can if needed check back to known transactions or back to genesis.
  2. ETFā€™s or digital assets

  • The asset is provably belonging to a claimed owner iff
    • The owner publishes some statement signing it saying itā€™s his
    • The app checks the outputs are not spent (i.e. the outputs have not been transferred, i.e. the output name is not also a transaction on the network)

If the asset was something like title deeds, you can see current owner easy and also go back in time. So like blockchains proving a hash, we prove the hash but also the actual data itself.

  1. Links to changing data
    • web sites or document sets (folder/directories/metadata)
    • People do a doc or web page with a Transaction in there
    • If the web page updates and links to new content then thatā€™s cool, no chain lookup, but they can go back in time

These are examples of using the transaction type. It will not achieve everything IMO, none of our data so far will. So like you are meaning I suspect @happybeing stuff like twitter likes or relationships might be more difficult to do on the network (registers or transactions), but I suspect it may be we just have not fond the correct design pattern OR we need another data type for that.

10 Likes

Will saving a file use 3 PUTs (transaction, metadata, data) then? And also 3 GETs? This looks like performance problem to me.

Does one transaction take one record, and one xor address? This means, that ā€œpay once, update many timesā€ feature in Registers is gone here, right?

Could the content field be bigger? Given the network protocols overhead/latency, it could probably be happily 500 bytes or even 2kB, without any significant effect on speed, but allowing to save more data in-place, like file metadata etc., and spare additional PUTs/GETs.

Will the address of the transaction be selected by user, like with registers? Is it only genesis, or every transaction?

Maybe instead of just parent, we could save a parent, a 10th ancestor, 100th, 1000th etc.? Then going back to genesis would not be a problem, and itā€™s an O(logn) complexity, so quite good perhaps?

What happens if client detects bad transaction (verification fails)? When node is doing check in registers, it reports the bad node, but what is the process with the client and transactions? Does client also report a node, that broadcasts broken transactions? Is it the same process?

I cannot understand the outputs part. Is it like when transaction T1 has outputs T2 and T3, that means T1 is a parent of T2? How is that possible, when we donā€™t know the future? Or the outputs maybe could be not only transactions, but any arbitrary data?

Can we perhaps not deprecate registers until new datatype proves itself stable?

8 Likes

It depends how you wan tot save a file really, I you want to use a full metadata type approach then you would have more info to store, but we can do that in many ways, this does not change that part IMO.

Yes, here it would be pay per update. That is one drawback for sure.

Then we get ā€œStuffingā€ happening and with registers that is already an issue. Folk could store whole files in a register etc.

No, and like registers itā€™s the name of the owner public key. No difference there

Itā€™s an app decision, so yes you could.

Nodes do not currently detect bad node in that way. Here all checks are client side. Nodes will not accept client accusations though in either case.

Like registers internally, it creates a DAG. In your example we donā€™t know the future really, but we know the children of T1 are T2/T3 but they are not used yet, but when they are we will know their children.

The proposal depricates registers, so they are marked depreciated to allow devs to test transactions and switch over.

Every sentence was another question there chap :smiley: :smiley:

5 Likes

Although, that can also help to reduce spam. If the person has only worthless things to say, they will be unlikely to pay! :sweat_smile:

8 Likes

Iā€™d like to be able to store file name (255 bytes), modification timestamp, size and a xorname of content. What other ways do you have in mind? Could be possible to describe in one sentence?

I see this as a feature, not an issue :slight_smile: Everything that folk could do is a gain for me, meaning more possibilities. If this is an issue, has it been discussed earlier perhaps? Iā€™d be happy to read more about it.

I donā€™t quite get what this all means.

Could I do it in transaction structure somehow (outputs maybe?), or you simply have on your mind just a separate chunk which Iā€™d point to with content field?

Ok, so does a client notify others somehow about bad transactions? Is there any mechanism of collectively dealing with them (like in blockchain not confirming bad blocks), or each client has to verify everything on their own?

Ok, I think I get it now, thanks for clatifying. Seems just like in blockchain, I forgot the exact mechanisms.

Iā€™m at ease, knowing the transition will be smooth :slight_smile:

Lots of questions, thatā€™s true. Thanks for answering them. Surely more will be clear when thereā€™s more documentation and the code, but perhaps knowing what the doubts are, will help making it more clear in the future :slight_smile:

4 Likes

You will see our FS part of the API developing, this will allow metadata etc. but itā€™s all in chunks and referenced by links. That is the best way to keep things decentralised IMO. All open for debate though, but this new network needs new models as well.

So metadata is separate from files. Even in a. filesystem or your disk. So we have metadata and separately bytes (content). The metadata can be set by different apps for different use cases as well as the directory structure.

Why not then just allow registers to grow to any size and leave it at that? Itā€™s not feasible I think.

Registers are stored at the name of the owners pub key. Itā€™s not stored wherever you want. So you create a key pair and that key defines the name of your register or transaction

Do yo humean store checkpoints in some way? I think you have options for sure. A simple one would be a parallel transaction chain that you somehow agree to. So the parallel chain updates every XX of the primary chain and acts as a checkpoint. That would be similar to a register approach which is really a. bunch of transactions and possibly forks. In transactions you can ban forks, but in registers you need to accept them, that is why currencies would not work in registers.

We donā€™t know yet how people will do that at their app, but the work on bad node detection and removal will be a very very long road. What we have is sybil avoidance which makes sybil nodes impotent, but does not remove them or identify them.

Itā€™s more like a myriad of chains that can be blockchain like in terms of chain selection etc. but apps are free to use any mechanism there as well as cross connect these (As above).

Yes I hope we can always do that.

No worries, I am lacking energy these days, so itā€™s not easy at time, apologise if I sound quick at answers, itā€™s not intentional.

Yea you would kinda hope so :smiley: I think it will take folk a while to digest and perhaps longer to design with this. Itā€™s a new paradigm for people, but I think a powerful one when you get into the weeds of it. The data type could hardly be simpler. So itā€™s like inventing a brick and all the questions folk can have about itā€™s use case when logs we every bit OK for housebuilding :smiley: :smiley:

9 Likes

Thatā€™s not what I had in mind. I was thinking chained data of any kind. For Twitter that would be tweets, for Facebook posts and comments etc.

Thereā€™s another issue though (and Iā€™m not arguing for Registers here, just illustrating): with a Register I always have the latest value. So to visit a website I query the Register at a fixed address and immediately have the most recent value (metadata for the most recently published billion of the site). No chain to scan.

For websites going from a fixed value (Transaction chain genesis) to the most recent value will probably remain a manageable number of hops. But for other applications it will become ridiculously insufficient without somewhere to pick up later points in the chain. And thatā€™s where people can and will centralise.

So, given the Twitter type service and app on Autonomi, I assume the app knows the genesis address of some chain. The chain entries enable it to find the genesis of various feeds. As each feedā€™s chain grows, finding the most recent data (far from genesis in each chain) gets impractical unless you can pick up later points from somewhere. And thatā€™s a point of centralisation isnā€™t it? That seems like an ideal opportunity to create lock in.

Transactions will be useful as is, but donā€™t they need a decentralised way of getting to the most recent data? There a way to do that? If so, how does it cope with forks?

Registers were a good conceptual solution to all this, but as noted Iā€™m not arguing for them. Iā€™m trying to highlight what appear to be missing capabilities in Transactions

7 Likes

With forks you mean duplicate entries as well as a tree structure emerging from multiple outputs being used - right?

If we use more than 1 output even without duplicate entries/forks it gets very hard to tell what the head really isā€¦?
Maybe one could create some form of metadata as a scratchpad that gets updated on chain changesā€¦?

So centralised but since you can append to the tx only if you have the private key itā€™s a somewhat centralised object anywayā€¦ (ofc gets difficult if scratchpad is meant to be private read onlyā€¦ Difficult to discuss this stuff without knowing the specs of the different data typesā€¦)

1 Like

The implementation has been merged so with your new Rust skill you could take a look! You can generate the docs yourself until they get pushed to a new version of autonomi on crates.io. For ex, clone safe_network and do cargo doc -p <PACKAGE-NAME>. I havenā€™t looked myself yet so donā€™t know if they have their own package.

3 Likes

I get what you are saying here @happybeing what I am suggesting is this.
In web sites or whatever you use these types to link to other data. So that web page is always the same, the content does not change. If you want to check content back in time, then all is well.

I think where you are coming from is (correct me if wrong). is that the web site ROOT is a register or transaction. So this is where registers gave us a wrong answer as they would fill and your website would freeze at that point. i.e. registers were not giving us a web site root that you could change forever as it would actually freeze at some point.

I do understand this part and appreciate the input.

So we need to have a mechanism where we have some ā€œthingā€ that allows changing forever and the Transaction would do that, but mean lookups for each version until the latest.

Like registers really, in registers you still have forks if there is concurrent access form different users/devices etc. The app dev has to choose how to select the branch they want to progress. A way to do this is apply the next link to be a child of both forks. So the dev gets to decide on the merge rules there.

Anyway back to the issue I think you are discussing.

I agree, but again need to point out registers did not do that and did fork internally.

I think we need to consider this part deeper, not in a. registers verses transaction thing, but as a new design using them or a new type.

I donā€™t want to knee jerk a design here, but I think it would be a terrific discussion if we all get our heads together here to come up with something valid, but we need to be laser focussed on simplicity as itā€™s too easy to imagine quick fixes like a growable list (List) crdt type and these also grow to potentially enormous lengths, although they are simpler than a CRDT DAG as registers are.

3 Likes

I donā€™t know if we should discount traversing from genesis here either unless we see the impact is seconds of time (i.e. over 1 or 2 seconds for say 1000 entries)

Also web sites that link to your site may not link at genesis, but at the point in time they want to. The users can see your site at that point in time or go back or forward in time to see future or historic versions and so on.

/brainstorming mode on :smiley: :smiley:

i.e. youā€™re site comes up at version XX and while folk read that, the app/browser has identified there are newer versions and ask if you want to see latest (or history) and so on.

4 Likes

A sledgehammer approach to the old web would be using a scratchpad type here, but browsers should perhaps state this site is not ā€œcontent secureā€ or something, i.e. it could change without preserving history.

I would love if we could avoid that and have a new paradigm here though.

3 Likes

I meant metadata that could be created by anybody by crawling the tree so that not everyone needs to crawl it and can jump directly to the point of interest

HEAD: [xor address of last tx]
V0.0: [xor address of first tx (a bit redundant)]
V1.0: [xor address]
Rebranding to autonomi: [xor address to tx somewhere in the middle of the chain ]

But ofc youā€™re right and the history of this metadata would not be recorded

2 Likes

What makes you think we could get below this?

Even with small chunks being transferred here possibly you only know the next tx - so you need to iterate over them and cannot get them as a bunchā€¦? So youā€™re expecting 1-2ms from query to delivery per tx?

I am hoping we can get very low, but yes, youā€™re right, even at 100ms itā€™s too slow.

2 Likes

Iā€™m not arguing for Registers. Iā€™m using them to illustrate what I see as a problem with Transactions.

Letā€™s focus on the problem - how to get the most recent data efficiently (or at all!) without centralisation being so much simpler or efficient that it becomes the norm.

To be clear again. None of my comments have been arguing for any solution - because I canā€™t see one.

I see a crucial problem, which you obviously see as important too, and I donā€™t see a way to solve this, including with Transactions as they are.

BTW the size limit problem was known many years ago. Maybe as far back as 2015 but whenever it was raised the question was not taken seriously. That was long before CRDTs. So to find there is no solution to it now is a shock. Was there in fact a workable solution in mind for any of their predecessors? If so perhaps one of those should be brought back alongside Transactions. Together they could work very well.

I agree that forks could be handled with a simple rule. Itā€™s not the same as a CRDT though I think, just to be clear again Iā€™m not arguing for that. Iā€™m just pointing out that a simple rule for a chain isnā€™t - I think - quite the same as using a CRDT is it? Or am I wrong about that?

1 Like

No worries Mark, we are in brainstorm mode here nobody is wrong or right :smiley: :smiley: However

Itā€™s actually exactly the same. Registers handle concurrent writes by forking internally (in the DAG), If you are updating them properly and not concurrently then you would see behaviour like a single list that grows (no forks), but internally they do fork. So the outcome is the same here in both types.

Can we frame this as a specific issue and not to total issue. So this is where we want a static address with changing information. Transacitons (or registers) achieve what I set out above (not static addresses) and here we are looking at a specific use case of a static address that does not change but points to changing content.

I am not sure about this, from 2015 to just recently the massive fight was about routing and consensus. The whole team to a person were on that. I was very opposed to it, but this was the focus. Even until recently the data types have not had the attention they needed, until now. Now we can focus on those.

So back to the issue of a static address with changing content. Here we are pushing further than any conventional key//value store or CDN etc. What we are looking for is perhaps not even Transactions or Registers but something else.

I wonder actually if a new topic should be done to dive into this one. I am not for just getting a solution like the old web mind you, I hope we can be more creative and efficient, but this is one to dive deep on.

5 Likes

I think weā€™ve had different ideas abou the web on Autonomi and how links should work for a long time so not much point arguing this.

My approach relies on a fixed address/identifier for a site, plus additional data to identify a resource and version. The existing web has the first two, and using awe I demonstrated a way to have all three even using existing web tooling. If you lose the first element (fixed identifier) you will break existing tooling and may also encounter fundamental problems with links for example. If you detail a practical scheme we can look into that. Either way I think itā€™s a big loss if everyone needs custom tooling to create simple websites.

Having said all that I believe Transactions will be easy for me to implement that same approach as in awe, and that they will work as well as Registers but without the size limit. So thatā€™s good.

Linking to unverifiable data as you seem to suggest above isnā€™t tenable IMO, so I would stick with using the genesis Transaction. Which in turn means that some use cases such as social networks are not going to work without either a decentralised add on data type, or centralised alternatives.

Are you seeing my point in that we donā€™t currently have a fixed identifier we can change though? We are not losing anything here. Registers donā€™t give us that when they fill up.

I am not sure I am getting what you mean here. Are you meaning using a scratchpad as a fixed address? I agree that is not verifiable in terms of history etc. but it is in terms of ownership as itā€™s signed by a key that the public part can be published etc. I donā€™t like it though as itā€™s too much like the old web, but it may be the best we cna do in the short term, I am not sure though as I feel we can do better and I am keen to discuss and figure stuff out.

If we come up with a data type that is agreeable though then the network can and should adopt it IMO. As a community we can make that happen for certain.

I am not arguing here, but I am keen we find a good solid and simple approach that not only works, but is not able to be sidestepped to introduce a non verifiable history as scratchpad would do.

So right now we can have transactions they give a history and future of any key (so any data). We also have scratchpad, non crdt static addresses that can hold up to 1Mb of stuff.

With the we can build websites with static links etc. but they would resemble the old net and potentially lose links to data (although folk can copy them and keep them valid in their own timeline or whatever, a bit like Internet Archive.

So I feel we need to be considered here and I am happy to investigate all angles of what we have and how we can use it and then if necessary come up with another data type to handle situations where we are forced to.

So a real deep think about this and how we handle perpetual data and hopefully not using old web type approaches, although we cna do the latter today and they will work functionally, but we would potentially lose the history and I think thatā€™s a loss, but not total as the data is never lost, just the possible links to it.

2 Likes