Topic Review
The purpose of this topic has been to discuss potential filesystem implementations that can support a locally mounted FUSE drive, and to try and develop and underlying standard API for filesystem storage on Safe Network that encourages use of a universally understood API for applications storing data.
This is tricky because of:
- the ambitions of Safe Network (particularly versioning and concurrent editing)
- the need to define what those ambitions mean in this context, including functionality, APIs and user experience
- the challenge of implementing filesystem conventions (e.g. POSIX style features) using the available Safe Network datatypes
This is compounded because much remains undefined. So this topic has been an attempt to clarify these areas and try to arrive at a useful way to proceed, and ideally the outline specification for a FUSE based filesystem, associated APIs for Safe Network for this and other applications.
Topic Status
We haven’t yet decided on a suitable set of capabilities. I attempted to define an ideal, but in discussion, some areas (e.g. versioning) mean different things to different people which means we don’t have a clear target.
Ideal Feature-set
Here’s my earlier summary which leaves concurrent editing to one side:
In the above, versioned refers to the whole filesystem though it is apparent that is both challenging and not necessarily seen as a feature for the Safe Network APIs, but perhaps something for the application level.
Personally I believed that versioning should be across whole datasets to ensure coherence within the history, but it isn’t clear that this will be feasible in a filesystem using MaidSafe’s preferred approach or Register APIs. Instead it may become an add-on in the application layer, much like taking a snapshot or backup is now, rather than something that “just happens” because you are storing data on Safe Network.
Implementation Difficulties
We’ve discussed two attempts to use the expected pointer-only Register API to implement a filesystem. One attempts versioning of the whole tree, and another versioning of each directory independently both of which strengths and limitations.
Versioning of the whole tree makes sense for a wider range of applications, where a history of multiple documents across multiple directories can be correlated together and backups or snapshots of the filesystem are an inherent part of the Safe Network filesystem APIs rather than requiring separate application logic. However, without a suitable TreeCRDT it would be hard to handle conflicts due to concurrent edits, than with directory-only versioning.
Using independent versioning for each directory would make it easier to merge conflicts and to support review and modification of the results of merging within a single directory. It is not clear that merging of hard-links would be feasible with this design, which would probably require a specialised form of CRDT (such as a successor to the TreeCRDT as suggested by @danda). It would also mean that features such as backup or check-pointing of a directory tree (or the whole drive) would be an application level function requiring storage of the state of the entire directory tree (but excluding the files themselves). So versioning of file trees becomes would become a user/application level issue as with contemporary filesystems, rather than an inherent feature of a Safe Network filesystem API (which was my aim).
Register API
I’m still concerned that there may be unanticipated problems with the pointer-only Register design, and also if use of registers prevents the ability to create and access the full structure of the node tree. The latter would for example prevent implementation of the whole tree versioning design referred to above, and perhaps other creative ways of using the only mutable data type available at least for now.
What now?
There are lots of uncertainties here, both in what is useful to aim for and how that might be realised in a Safe filesystem for a FUSE mountable drive.
I’m sure that a mountable drive can be implemented whatever route is taken, but it seems too early to proceed further on design or implementation with so many things being unclear. So I’m leaving this outline of the areas that I think need to be clarified.
It concerns me that we don’t have a definition of what we are aiming for within the Safe APIs, or what will be feasible based on examples of how to deliver it.
So I think it will be valuable to have concrete reference designs available for key use cases (such as versioned websites and mountable drive), to demonstrate the suitability of proposed Register and filesystem APIs. And from those, a clear understanding of what will and won’t be possible from both a user perspective and from a developer perspective.
With regard to perpetual versioned public data (such as websites, document collections, libraries and archives) I’m concerned in case simplifying the Register APIs in fact pushes too much responsibility in to the application logic (for publishing and viewing). I think if the Safe Network supports those functions implicitly it is much more likely to be successful in delivering perpetual versioned public data across multiple applications and data types, which seems important.
Finally
Even if there were complete clarity it would be a lot of work to tackle the filesystem APIs and implement a FUSE filesystem. Even moreso with versioning and merging concurrent edits. But with much uncertainty I’m going to pause rather than proceed with a design or continue trying to clarify (because it is consuming and arduous).
So I’m going to return to Solid related things for now. I’m more confident that LDP or indeed a graph-store can be implemented on top of whatever filesystem or APIs are available, and that versioning can be handled in a LinkedData API layer (even if it isn’t implicit in structured/trees built with Register APIs). I’m less sure about concurrent edits which may also rest on an improved TreeCRDT but can proceed without that for quite a while.