Not at this time.
Maybe the scratchpad type can be overwritten, I don’t know for sure so don’t quote me
Private data only needs you to “lose” the datamap and it is effectively deleted since the datamap is never stored on the network
3 Likes
Persistent web cannot exist if people can just delete their web pages. Back to the current web and the rot that is happening all over the place. Also being able to delete allows the companies to demand deletion like they do on the current web. All the fake DCMA and copyright trolls that remove so much stuff illegally claiming copyright.
3 Likes
What part of deleting your copy of the private data’s datamap from your computer does not represent that file as being deleted?
Dedup means that people cannot delete the chunks since they may also be chunks in other people’s files. Dedup will save a lot more storage than allowing the deletion of private files by somehow changing the chunks as they are stored so dedup cannot happen
2 Likes
All you do is exchange one danger/problem for another.
Not all solutions will suit all people. Best to encrypt your private data before storing then. I am sure there will be an app for that
2 Likes
Do you have any thoughts on why Autonomi is tweeting about this problem as if it can provide a solution?
2 Likes
Unsure. Maybe they meant to say private data. I dunno
3 Likes
yes - the scratchpad datatype that has data and when you upload a new version it comes with a higher counter value that populates through the network and fully replaces the previous version of it; so history is gone with that data type and cannot be brought back
2 Likes
Because you upload a video as private, joe bloggs uplaods the same video as private. You delete yours and their’s is gone too.
Also complexity, also dedup will save so much space required to store data.
And people typically do not delete private data so you increase the data stored for what. So you can race home and beat the thief? Why were you uploading that sort of data anyhow? And why keep the datamap so insecurely. So make the world’s nodes suffer with more complexity more data to protect your insecure ways of storing obviously valuable keys (datamaps)
Hopefully a solution will come in time.
Without a delete option for private data, some security minded organisations & individuals won’t consider the network for sensitive private data, so adding it will bring more use cases & flexibility to the network.
But, perpetual public data is where the network is focused and that seems like a strong offering to begin with.
The kind of files people would want a delete option for generally won’t impact dedup at all, as they’ll be private data with potentially sensitive personal info on and will be unique files, so if private data ‘delete’ were implemented in a way that didn’t mess up dedup for public data, dedup could still save loads of storage.
If deletion were added and someone did upload a public file as private, it would need to be ensured that the chunk can’t be deleted (e.g. any ‘public’ flag for a chunk would trump any ‘delete’ flag so nodes don’t forget it).
1 Like
as mentioned in the pre-dev updated the vault storage got an upgrade to use an array of scratchpads to be able to use a largish amount of space that doesn’t have a history
1 Like
It will increase the amount of data stored. People who backup their disks will do it as private files. Millions doing it and the dedup will save a ton of data stored and make it so backups are automatically incremental type. Salting each private file loses this benefit of dedup. All those videos people have stored and backup and ones they d/l.
Use the scratchpad type for those files that will need deleting. Leave private files as part of dedup and save 10-30% of the storage and make incremental backups automatic
3 Likes
using an array of scratchpads will be significantly more expensive than using chunks … because scratchpads are planned to be smaller
so it makes sense to use chunks for data where you don’t care/want them to persist … and to use scratchpad only for stuff where having a history would be very inconvenient …
4 Likes
People might want to store sensitive private info on the network so they have it available on the go, and choose to share it with specific people, e.g. medical records to share with medical professionals or researchers, or ID info to share with specific parties but still in the control of the user.
Even if someone is very careful with keys & stores them securely, knowing that any slip up could cause records to become permanently public with no possibility of deleting if they felt keys might be compromised still seems like a significant risk.
1 Like
Also for stuff that is not CRDT or concurrently updatable. So these are really for private, personal and small (or expensive) amounts of data
3 Likes
Is ‘salting’ necessary to implement delete from the private files, or might there be ways for uploaders to ‘flag’ chunks in a way that doesn’t interfere with self encryption and therefore dedup effectiveness?
And then what happens if a node is offline when the deletion occurs and comes back shortly after. It then causes those chunks to be replicated again.
Its not just simply deleting chunks, this network is designed to be resilient and resist loss of data. You remove that and persistence is at risk due to ability to not replicate some chunks over others.
Deleting your privately held copy of the data map is the most secure way to delete a private file. Deleting the chunks is unneeded and the chunks could end up being replicated back again because there are so many copies on more than 5 nodes. Back to square one and another redesign after a year of redesign. Remember nodes are keeping chunks that they are “closer” to but not one of the closest 5.
To have deletion of chunks themselves requires a redesign of how chunks are stored and the way they are kept in nodes. It turns a very simple nice design into a spider’s web of when a chunk is or is not also part of another file (requires knowledge of file now or at least linkage), when or when not to replicate a chunk that others think should be deleted but others think it should not because of missed messages.
Lets bring back sections and consensus mechanisms to vote on whether that message to delete needs to be remembers for days just in case a node missed the signal.
1 Like
Just to expand on the chunks reappearing again.
At the moment the replication process requires no prior or kept knowledge for replication. There is simply a closest node in its normal processes finds out a chunk it is closest to is not stored so it gets a copy and stores it. Replication at work
To delete a chunk requires ALL nodes that hold that chunk (closest, closer, caching) have to all agree that the chunk is to be deleted which also requires no hiccups in messages sent.
Then any one node that is holding that chunk could end up causing the chunk to be stored by all close/closet nodes again. So then you need some state to be kept that lists the chunks that they should delete and keep deleted. Also some consensus that agrees that this was not a malicious request to deleted the chunk. An attack vector is to have a bot farm just keep requesting any chunk they are storing to be deleted from the network. Slowly they end up deleting whole sections of the network as churning keeps giving them chunks as the data redistributes.
It is not just a simple lets delete chunks because they are told to
To ensure a chunk stays deleted and not end up being replicated again requires a lot more complexity than what saying just delete the chunk suggests. And it’d be real bad if you claim the network securely deletes your private file chunks and all and then a day later they check and a lot of the chunks are still there due to replication.
Far cheaper in design cost to have an APP that does the encrypt for you so that the private file can never be read if you never want it to be even if the datamap leaks. Medical data, or sensitive data, or …
1 Like
You may think it’s unneeded, but some people will avoid using the network for sensitive use cases where they want to know they can delete files and have it removed from the network.
So, while it’s not essential, it would add to the flexibility and appeal of the network in the maket for some use cases.
Of course if there’s no way of doing it that addresses the required challenges without a massive redesign then it shouldn’t be considered a priority in the near future, but given the likely appeal in the marketplace, it’s worth considering if there are ways of doing it e.g. with simple flags that can be signed by uploaders and easily verified by nodes to check whether a chunk could legitimately be forgotten or not.
1 Like
It is possible that the majority of data will be private. Backups, private media kept by everyone, company backups and data.
salting will mean that so little of the network data will be dedup.
Also this was discussed so long ago and the team current feels deleting private data chunks is not on teh agenda.
Also read my explanation of how the current simple design of the network will case the chunks to reappear anyhow. So just delete the data map from your local storage and use an app that encrypts the data with your secure key so that when you delete your datamap there is no way for the file to be read even if the data map is leaked.
Then people will selectively use it for sensitive data like medical records etc and save the dedup to help save on upload costs and node operator’s disk space.
And again you ignore the very simple solution of an APP to encrypt sensitive data first. Saves redesign of network to reintroduce tables of chunks, keeping state, consensus, etc. The replication will resist deletion of chunks and would require all that consensus/state keeping/etc
why are you ignoring that even large amounts of data can be deletable by simply using an array of scratchpads? Oo