SAFE Network - TEST 7

Certainly keeping an eye on this one to see whats causing this issue. Hopefully @Krishna_Kumar has some info on this soon.

This is unfortunately a bit more involved issue where it ties into a few other restrictions. It’s something guys from safe_core are currently looking into and should hopefully result in a few RFC’s to refine some of the ideas here.

So yeh moving on from the cryptic para ^^ that no one prolly understood, this is whats going on with this one.

It’s not the counter itself from Launcher thats at fault here, but currently there are no delete requests actually getting made to the network. So right now a “delete” operation triggered via launcher is more of a “forget” operation than a “remove from network” operation.

To understand ^^ need to know a bit about how data is currently structured in the network. So we know files result in chunks and a DataMap(index list of chunks that make this file up) and before someone mentions “we dont always create chunks when storing a file”, yes if a file is smaller than 3 * MIN_CHUNK_SIZE we store no chunks and we just post the content of that file into the DataMap itself. Chunks get stored as ImmutableData in the network. DataMap gets stored in the parent DirectoryListing(another index list which contains all the DataMap(files) / DirectoryListing(sub-folders) inside this folder) which is StructuredData. So thats the basics covered.

So for the forget operation → all that needs to happen to “delete a file/folder” is to remove the corresponding DataMap /DirectoryListing from it’s parent DirectoryListing. This occurs as a POST operation to the parent DirectoryListing and thus why we do not see it getting logged as a DELETE request/count from launcher. User’s folder hierarchy from that point is as expected but the data in the network still exists and isn’t actually removed.

Right now, network only supports actual delete(“remove from network”) operation for StucturedData from Vault code. Trying to delete ImmutableData would just result in a UnknownRequestType. It should probably also tell the client this error which it doesn’t do right now, but back on topic. ImmutableData delete isn’t supported right now cos the network doesn’t keep track to which clients stored what data to know when to delete the same. StructuredData on the other hand has this feature built in with the data type itself knowing who “owns” it at any given time. Remember the cryptic para and what we mentioned about RFCs, so this is something one such RFC needs to look into to being able to know “when” it’s safe to delete immutable data. It’s may not be quite as simple as just keeping a record of all clients storing this piece as it may lead to massive meta-info that would make churn relocation infeasible, so needs some tolerance levels and also why/how it benefits the network to keep track of this.

Right now, there are also no “refunds” of any kind and no actual “motivation” for users to delete the actual data apart from clearing their own directory structure which the forget operation accomplishes. It does result in “abandoned” data which resides in the network not needed by anyone but vaults are monitoring and relocating with churn etc. So another RFC needed where this is one area that it needs to cover to motivate users to actively “delete” data by maybe offering a refund and at the same time help the network not deal with “abandoned data”.

There exists a final point to also consider about making DataMap’s Immutable in the network and stored separate to the parent DirectoryListing and just have the DirectoryListing reference the DataMap by its name/some identifier. This helps work around small files not triggering Put’s since they got POST into the parent DirectoryListing as now even a small file will trigger a PUT for it’s DataMap + possible chunks instead of just possible chunks triggering PUTs. It has added benefits to being able to reference files directly than needing to wrap them in a DirectoryListing to share/reference with others ofc provided its public and unencrypted/…

Soz abt the long answer, but hopefully the cryptic para conveys some meaning now.

14 Likes