Appendable Data discussion

[Toivo]

@anon26713768

I am struggling to understand how deleting something once it has been made public changes that copies have most likely been made if it is even of the slightest interest to another person.

Do you really think so?

If we had Facebook-equivalent in SAFE Network, I don’t think all my posts would be copied to somewhere. Why would they? Who would?

[dirvine]

@Toivo

How do you think this affects the will to publish in SAFE Network?

Publishing will be forever, so set up a wewbsite and that is done forever, no renewal etc. People will like that and I feel there is an appetite. People who want to remove published stuff though will find it hard to see the hard fact that they cannot, unlike today where we think we can. This is more honest.

Folk think they delete a facebook post and it is gone, it is far from gone, it is already in the databases of many marketing companies and more. Or perhaps you can tell google to invoke right to be forgotten, that only means they remove you (do they) from the index, all the data is still there

So today folk are fooled into believing you can delete published stuff, but it is not true, worse though the original data is perhaps out of the reach of the common folk and in the hands of the powerful folk, lots of them and lots of peoples data.

It goes further, say I have a pic of you or a video that is produced by an AI, its indistinguishable (soon possible) from the real you. Then what do we do

Anyhow we make everyone’s published data and private data last forever. Only if you want to delete your private data then you can. You do this by throwing away the data map and nobody can ever read your data again. It is that simple. Data map for public data is public, so you cannot just delete that unless we had signatures to show who put it up there to delete it etc. Then you have all sorts of edge cases and anonymity issues, never mind losing deduplication, caching etc.

[Toivo]

@dirvine

Folk think they delete a facebook post and it is gone, it is far from gone, it is already in the databases of many marketing companies and more. Or perhaps you can tell google to invoke right to be forgotten, that only means they remove you (do they) from the index, all the data is still there

Yes, but still on my experiential level the situation is that if I write something slightly stupid, or make embarrassing typo I can remove or edit my post, and no one relevant in my social sphere will get back to it. Maybe it is somewhere in Facebooks databases, but so what? It was nothing major. If something (edit:someone) digs it out from there, I don’t care that much. But at this moment I want to hide my mistake, and I can do it.

Of course it would more mature to just admit your mistakes, but let’s be honest, how many really are? I see this as a barrier for adoption. Your promise is factually correct history, and I can see how that would really boost the image of a company etc. but the inablility to mask your small mishaps will drive everyday folks away.

[dirvine]

@Toivo

Your promise is factually correct history, and I can see how that would really boost the image of a company etc. but the inablility to mask your small mishaps will drive everyday folks away.

These two things are mutually exclusive

@Toivo

Yes, but still on my experiential level the situation is that if I write something slightly stupid, or make embarrassing typo I can remove or edit my post

I agree with this as well (speaking as a typo king )

[Toivo]

@dirvine

Data map for public data is public, so you cannot just delete that unless we had signatures to show who put it up there to delete it etc. Then you have all sorts of edge cases and anonymity issues, never mind losing deduplication, caching etc.

Of course if it is technically impossible I can’t argue against it.

But I see huge value for the user in the possibility to choose if some public data will be up forever or not. And also I see “You will be accountable forever” not very appealing.

[Traktion]

@JPL

Are there any examples of appendable-only CRDT applications in the wild outside of the logfile use case? A quick search tells me that CRDTs are used by Riak and Dynamo NoSQL Key-Value databases. The detail is rather over my head, but I can’t see the word ‘appendable’ there. Another one is OrbitDB on IPFS-logs, but again the use case seems to be logfiles.

Considering most serious relational databases have a append only transaction log, with the current state being the latest (head) of this, I think we can say there is much precedence. Taken to the extreme, the transaction log is never truncated and you can return to any prior state.

[dirvine]

@Toivo

But I see huge value for the user in the possibility to choose if some public data will be up forever or not. And also I see “You will be accountable forever” not very appealing.

I agree some folk might not like to be accountable forever, even if they think they currently are not.

Remember though in your earlier thing about typo’s, even this forum uses a keep mechanism but shows your latest edit s the current one. I can still go though your edits. So it does not harm this forum

also create a new id, twitter is a good example of accountable forver, but who is behind the account etc. ?

I am not sure it is black and white as we imagine.

[Mindphreaker]

@dirvine

If we end up in the future with actual Mutable Data (edit/delete) then that is all OK, but it has to be in a eay that other things like perpetual data cannot be damaged.

I understand your priorization here and appreciate that you are open for the addition of true editing in the future.

Beside other important things SAFE stands for freedom and for me this means freedom of choice. I strongly believe that we should give people the opportunity to do with their data what they want (in both directions!):

1. Give them the possibility to post something which can’t be removed (e.g. political critical post)
2. Upload something (privately) which you might want to remove/edit at a later point (e.g. private photos)

As long as the choice for immutability can’t be reverted at a later point I don’t see a reason why we can’t have both possibilities. Both have legit use cases.

In order to not violate the recently published network principles I would therefore suggest to change #8 to:

The network is capable of storing data in perpetuity.

[Traktion]

I am really glad this conversation is being had. The discussion about how MD is to be cached effectively had come up multiple times without a satisfactory answer, imo. This AD discussion, with each new element being immutable data, addressable in the same way as any other immutable data makes a lot of sense in this regard.

I wonder how the impression and reality of MD has become so wide for so long. This brings a lot of clarity and removes most of the doubt about scalability of MD (content can be easily cached).

Obviously small changes to large chunks is going to be relatively expensive, but apps can be designed with this in mind - splitting changeable and non-changeable data, storing delta to another location instead of clone with change, etc.

I think true temporal data storage may still be desirable, but that sounds like a future debate in light of this thread.

[Traktion]

@Antifragile

Those databases return actual state, not whole time transaction history. Nobody cares how db works internally if it returns current state fast and client does not need to have gigabytes of ram, network does not need to send whole backlog and results can be displayed without additional processing on client site.

From what has been discussed above, getting the latest data should be done via client libs without heavy lifting being done by apps. Much like how a relational database expose current state for queries.

The fact that you have the power to roll back to a prior version from the app is rather powerful though. Given that all immutable data is eminently cacheable, it should also scale well too.

It will be interesting to discover how the AD records themselves can be effectively cached too, as i haven’t read much about this.

[Traktion]

@Antifragile

I can imagine append only data as combination of appending of small changes and than storing actual state recalculated from those changes after let say X appends. All done automatically by network. So client needs to download only last checkpoint plus few changes and everything can be calculated on client site from that checkpoint by client libs, so devs will work directly with latest state of all data. This is append only strategy that acts like mutable data and is fast and easy to use.

I can imagine this too, but the network itself still needs to construct the final data before presenting it. This would require multiple GETs internally, as well as some reconstruction operations. In fact, storing the result as a new element in an AD would make sense.

Perhaps client libs could be smart enough to facilitate a combination of the above to keep an optimal level of performance vs price. It is common practice to do similar things in relational databases (summary table for results of slow/expensive query), so I suspect similar techniques for AD optimisation will evolve.

[dirvine]

@Mindphreaker

In order to not violate the recently published network principles I would therefore suggest to change #8 to:

I would say this is a place where I would fork and make it immutable ofc we call have that choice, but I do like “warts and all” approaches to things. That is just my personal opinion though of course and folk might not want that, even though they currently have it.

I do very much appreciate what you are saying though, but this hill is one to climb.

[neo]

@dirvine

Append only is a list of pointers (for us these can be ID’s as well).

ADs is the wrong term as well I see now. Its an ALD (append link data) object and the current MD is really a MLD object from what has been said.

@happybeing

I’ve read /skimmed all this and still am not clear what is being discussed

I am probably to blame for that since much of my discussions was based on the MDs as described when being introduced but it seems that is not what is implemented as an MD. These are link objects linking to the actual data. @dirvine is that right? I never went into delving into the current implementation or writing code against it so I took the discussions from a while back.

@happybeing

but if the record is public and permanent,

Except David said a record of ownership is not kept when its changed. So I can publish a lot of bad things then change ownership of my ALDs to you and then you seem like the bad guy.

For public published things ownership must be a part of the kept data otherwise perpetual data is broken on that point.

Imagine reconstructing history of events when ownership is changed. You cannot unless ownership history is kept. Imaging attributing KKK press releases to your neighbour (if you had one) by changing the ownership of those press releases to your neighbour’s blog site.

@dirvine

So rather than history being written by the victor we hopefully will have enough proof points for our descendants to write a better history than we have had, well we can hope so, but step 1 must be, do not allow published stuff to change.

But if you do not keep ownership details when changed then the victor only has to write their version of it and then change ownership to the loser and then anyone researching that historical event is presented with views favourable to the victor (but falsely)

@Traktion

I am really glad this conversation is being had. The discussion about how MD is to be cached effectively had come up multiple times without a satisfactory answer, imo.

Yes I learnt that MDs are really MLDs and not what I thought was being presented when MDs where introduced. And ADs are really ALDs (L === Link) The data is not stored in the MD or AD

[tfa]

@neo

current MD is really a MLD object from what has been said.

In fact it is up to the application to store objects directly in a MD or to store pointers to them. For example WHM stores “File” structures directly in the MD corresponding to a directory. This structure has then a pointer to the datamap pointing to content of file. So we have a mixed situation in this example.

[neo]

@tfa

So we have a mixed situation in this example.

Well then the issues can will still arise to varying degrees depending on the application. But still a double access to get data that is “stored” (well referred to as stored but not actually) in a ALD. Only if the APP stores the whole data in the ALD is it a single store (then its an AD I suppose)

[Traktion]

@tfa

In fact it is up to the application to store objects directly in a MD or to store pointers to them. For example WHM stores “File” structures directly in the MD corresponding to a directory. This structure has then a pointer to the datamap pointing to content of file. So we have a mixed situation in this example.

Interesting! So they can be/are used as general purpose indexes to immutable data items.

[Seneca]

@neo

Except David said a record of ownership is not kept when its changed. So I can publish a lot of bad things then change ownership of my ALDs to you and then you seem like the bad guy.

Ownership isn’t the same as authorship. Proof of authorship should always be based on a cryptographic signature, not on a public key record. As long as software devs understand this and properly implement it in their applications, there shouldn’t be any confusion about that.

[dirvine]

@Seneca

Ownership isn’t the same as authorship. Proof of authorship should always be based on a cryptographic signature, not on a public key record. As long as software devs understand this and properly implement it in their applications, there shouldn’t be any confusion about that.

Agreed, completely. Also though

safecoin can change ownership no prob as there is no data (no list of entries). Take a website pointer though it points to a series of different data maps. If there is then change of ownership of this website location, then we change the owners keys, no prob, even multisig is fine. However nothing stops us forcing that list entry to point to the ownership change data. Therefore the list can show ownership change at points in the sites history etc.

Of course many ways to do that, but as an example this is valid. The RFC should be great if it comes soon as everyone is primed to find bits to be used against the network, so this is perfect timing I suppose.

The authorship is very important as well and not something we have really answered yet. It is worth thinking more about.

[Toivo]

@dirvine

Remember though in your earlier thing about typo’s, even this forum uses a keep mechanism but shows your latest edit s the current one. I can still go though your edits.

Ok, I did not know that. Is the possibility to see earlier edits restricted to the administrators only, or can anyone see edit history of others?

@dirvine

also create a new id, twitter is a good example of accountable forver, but who is behind the account etc. ?

That is OK when you know in advance that you are wanting to hide your personality. I’m talking about the cases after the fact that you have published something that you realize would have been better to stay out of public. Now I hear you and other saying that once you have published something, you can never be sure that it can be hidden again. And you are right of course. But while you can not be sure if you manage to hide again (if someone else has seen and copied it), neither can you be sure that someone else has actually seen and on top of that copied it. Well, in the case of Oldnet platforms, you actually know that it is copied somewhere, but in principal that would not have to be the case.

Another thing, are you even planning to have accesses to data like “Shared with 20, which of 15 can only view and 5 can modify”?

[dirvine]

@Toivo

Another thing, are you even planning to have accesses to data like “Shared with 20, which of 15 can only view and 5 can modify”?

Initially it would be shared with 20, viewable by 20, changeable by 5, if that makes sense. Again though viewable by 20 likely means public in many cases.