Safe filesystem API for a FUSE implementation in Rust

Development
61

The APIs I outlined support this:

  • std::fs and some of std::io on top of…
  • a POSIX style API with underlying extras including versioning (tree rather than container) and concurrent editing
  • the core Safe API for RegisterCRDT and files

FUSE would be an app on top of the POSIX layer. Your extra types sit alongside that layer or within the lowest core APIs.

My concern has been that the lowest level above won’t support the layers above it. In one instance by limiting the ways that history data within a register can be structured and interrogated, in another by limiting Entries to pointers (even a type field could help a lot), and the lack of a capable TreeCRDT that can merge trees and even hard links.

2 Likes
62

I think you mean POSIX filesystem when you say filesystem here.

I am hearing POSIX repeatedly and agree it’s important. However windows is not POSIX and many others are not. We seem to be saying POSIX its all we need. It just feels a bit like stick with the existing world and all is well.

Could the same arguments not be made for using DNS and HTTP and forget any new protocols there. I cannot see a difference in the debate from that and it worries me that we seem to be saying POSIX it must be POSIX as that is what devs want. Again I claim iOS/android and many others moved away from that.

The argument that devs expect X so we must do X feels like Henry Ford being told he needs a hay carrier for the front of the car :wink:

This is my concern, we seem fixated on a linux and possibly osx route of old desktop apps here. I feel the world is moving on fast and we would be looking backwards and would as well offer an http / dns protocol at the network core.

Anyway the conversation is definitely valuable, but I am less opinionated on this one for sure. I just don’t want us to get to a place that cements us into a corner. I know you don’t either.

Again, the debate has to be what goes into core and what is outwith core and how can that be implemented and test the core API

2 Likes
63

No, I mean filesystem. I have focused on POSIX as the most well defined, non-proprietary and most widespread implementation outside of Windows which is a bastardisation that has crept towards POSIX over the years. So POSIX is a useful aim for a filesystem in any system, though not necessarily achieved in practice.

You agree POSIX is important.

I think there’s a big difference between file storage and DNS and HTTP. I also think there are benefits in supporting, or rather leveraging legacy standards (such as HTML and the web), and particularly in supporting Linked Data Platform (i.e. LDP, which is a RESTful interface normally over HTTP). LDP is stepping stone to Solid and Linked Data which is one of the ways we might improve interoperability of applications, or at least improve semantic understanding and portability.

None of that support prevents further innovation. It means that innovation has to improve on what is there already. I’m sure it can, but so far all the discussion of alternatives is still vague. We’ve seen mock-ups of UI/UX and so on, but the specification, APIs and road-map for those features is unclear at this point. It would be great to put them into hands of early adopters and see the benefits, and how people (developers and others) respond to them.

I see value in supporting LDP/Solid for the same reasons I advocate for a filesystem interface: ease of migration of existing apps and ease of adoption by people. In the case of LDP, those people will be developers of systems using Linked Data, and users of apps that will be more easily adapted to Safe Network if it supports an LDP style interface such as the one I demo’d with praise from you and Tim Berners-Lee back in 2018. I see POSIX and LDP / Solid as useful enablers for adoption for the same reasons, though there are also differences.

We do have different takes on this obviously, but I don’t see it as sensible to foist new - still unrealised and undefined paradigms - on developers and users, at the exclusion of what is familiar, works, and will encourage adoptions by users and developers.

You seem to fear that providing a familiar filesystem will prevent innovation which is a valid concern, but so is insufficient adoption. I don’t share your fear that providing something useful will prevent adoption of superior ways of managing data from being developed or adopted. I don’t deny they will impact each other.

I can only speak about Android, but it does support a filesystem interface. I use that regularly to transfer directories of files (such as photos, downloads etc) for backup or use on other devices, for storage and viewing on my phone. Some Apps also store proprietary data in databases on the same filesystem (the same way apps do on PCs), enabling backups of their data from one filesystem to another. The same for Android app package files for side-loading etc… I’d be surprised if Apple doesn’t take a similar approach for its devices, but I don’t know.

I think we’re repeating ourselves without making progress. You are repeating your views and I’m repeating mine.

I’ve also summarised my perspective on title of the topic above. I hear your concerns but don’t share them. I guess you don’t find my position convincing, but by all means check if iOS has a filesystem, and accept that Android at least up to v11 (which I use) has a filesystem and a tree-based file manager that lets users browse downloads and apps present file selector dialogs for a range of functions, and to support transfer from one filesystem to another.

This is the type of thing that a capable Safe drive can maximise and a sub-standard Safe drive/filesystem can undermine. It doesn’t have to be POSIX, but why not aim for that, if doing so maximises usefulness to users and developers, enables more people to try and adopt Safe Network more quickly? Maybe we can’t have hard links, so be it.

I see a capable filesystem as doubly beneficial: first it accelerates adoption which is vital to a viable network, and second it exposes more people to the benefits, and to the innovative ways of managing data that you and others can envisage beyond what people use now.

3 Likes
64

Just a reminder that I presented a sketch above for using the existing sn_fs (tree crdt) without need for any changes in safe-network code.

I have quite a bit on my plate at present, but I’m hoping to find some spare time, probably no earlier than March 15, to hack on that idea a bit. I could be wrong, but it feels like it should be pretty simple to adapt the existing brb_node_snfs p2p example to use safe files put and safe files get for persistent log storage/sync, which replaces the brb/p2p stuff in the example and provides persistence and write versioning as well. The FUSE stuff is already written. So I think a simple, totally unoptimized prototype could probably be made in less than a week, that would be good enough for playing around with and sharing some files.

8 Likes
65

With regards to “new types of filesystems”, I think my favorite I have tried is actually just a flat searchable collection. With good, fast searching/indexing, not the perpetual uselessness that is locate on unix.

For example, I use gmail quite a bit for sending myself notes. Sometimes I add extra keywords/tags in the subject or body.

I love that years later I can search and get near instant results. I didn’t need to worry about filenames or directories. gmail has labels but I don’t even bother with those – that’s work… I just leave everything as flat collection in the inbox.

I would think one could replicate that on safe network with a FUSE mountable filesystem that provides only a single directory/container and some sort of label scheme, esoecially for media/binary content. A local search-engine would run periodically that indexes all the files. meilisearch is a well-known rust search-engine that might be a fit.

my $.02

7 Likes
66

This is where I keep going back to. What must the core network provide. I think we are bouncing past that point here. Right now it’s the only answer we really need to find. Then posix or any other FS imaginable or not can all happen and SAFE need have no opinion on it.

3 Likes
67

Let’s say we have website about mushrooms, and there is a photo of mushroom, with a text: “This mushroom is edible.” Then later it is found out, that it was a wrong photo, and the photo is changed to another, presenting the right mushroom. Now how would we achieve a web, where it can be proven, that wrong combination of text and image was up for a while, even though it is now correct?

What is the place / layer / app that put’s the text and image together as a website? I think the versioning of the combination of image and text should be done there. Even if we would like to force websites to adopt a system where the whole of the page is versioned, can we do it? As long as the “publicity” is not baked into network I don’t think we can. Even if registers would permit the versioning of these sorts of combinations, can it force them?

To have a majority of Safewebsites apply versioning of the whole site, we would need that to be an easy and convenient go to solution. I think the alternative, the possibility to state that: “Nooo… I didn’t really say that!” is more lucrative to almost everybody. I think we would like to tie everybody else to the things they once said, but keep ourselves free of that commitment. And since the decision to publish is on us, I don’t see folks adopting the stricter perpetual versioning policy. The looser version, where just the separate pieces are perpetula, is already scary enough. :thinking:

1 Like
68

Some will, some won’t, and that’s a useful signal for anyone consuming their content.

Websites that adopt an open, auditable approach will have higher reputation and integrity than those that erase or rewrite history.

On Safe it will be hard to hide that because it will be so easy for others to generate independent histories of public data.

So there’s an incentive to play things straight.

4 Likes
69

That’s exactly what the topic is attempting to do: help define a useable core API that makes it possible, ideally easy, to build what is a core claim for the network: perpetual versioned data that can be edited concurrently.

To do that I’ve said I’d like to see use cases worked through to show how the APIs will deliver that. The is a particularly useful use case.

I despair. It’s statements like that which make this so arduous for me, make me leave this incomplete.

70

Sorry about that, but let me be more clear. IMO you are saying SAFE must provide a POSIX filesystem interface. Not at core or app level, but just must provide it. Also you are saying this is what devs will want as they are used to it, so that makes sense.

I am not seeing how you are despairing if we are trying to figure out what the core network must provide as it’s API.

We seem to be speaking of some API and it’s POSIX, but not clarifying the core network API. If you read the above convo it looks like you are arguing POSIX has to be what SAFE provides.

So right or wrong, the point I am making in the above post is that we keep bouncing past this Pont and that point is what does the SAFE core API look like.

3 Likes
71

Not so. I don’t think it would be a bad idea at some point, but the reason I picked this up and was trying to work out how to build it myself as part of developing a Safe FUSE drive, was because I could see it wasn’t going to happen unless I or someone else did it.

The only things I’ve argued for MaidSafe to do are to ensure the core Register API supports the aims I’ve stated for that drive, and the APIs I think I’ll build on top as part of that project and to support others building other kinds of app.

I would have preferred MaidSafe to be doing this and not take it on myself, but I’ve not been asking you to do it. So I was hoping that MaidSafe would see value in that, but have not been expecting any development from you.

After reminding myself and others about Syncer, everything following that in this topic (and in the Registers topic), was me trying to figure out how to build on the API MaidSafe are working on, and feed into that before it is set in stone.

I’m moving on from that endeavour for reasons stated, and think I will find it easier to work on something else, possibly LDP / Solid.

What’s been arduous is putting in so much effort to understand, design and feed back while you seem to have been arguing against something I’ve not been saying or expecting MaidSafe should do - without saying that’s what you won’t do! And pushing back on things needed in the core to support it, rather than showing me how the core API can deliver the things I’m advocating for, or convincing me that it can’t or shouldn’t be supported. I’m not saying you don’t do that at all, but getting there has been slow and difficult because being at cross purposes keeps moving the discussion away rather than focusing on on what we are apparently both trying to achieve.

I’ve been trying to do this in two main ways:

  • asking how this or that can be achieved using the Register API, and the same for your proposed folders implementation, and
  • using the project of a capable FUSE drive to test your and my ideas on how to implement filesystems, so that we can try to ensure the APIs support this use case, and the particular key capabilities that it aims to deliver (perpetual versioned trees with concurrent editing).

I’m advocating for the same with other use cases, but this is probably one of the more challenging, and already enough for me.

I had the same feeling in the Registers discussion, where you made statements that you never clearly justified despite many back and forths, and later made one pretty disheartening/rude comment about it (in a reply to Anselme) about register entries holding anything but a pointer being “terrible design”.

You’ve still not shown how your statements on that are justified, so again I just gave up. I accepted that I’ll have to work with pointers alone regardless of how much more difficult it makes what I’m trying to build. I can show how it hobbles the design of a filesystem using the approach I designed (that can track versions of a tree rather than just a directory) while using less chunks than yours, but there’s no point me spending time doing so. Your design is good too, but it has different capabilities.

This isn’t personal for me, and I am sure it isn’t for you either. I’ve been around long enough to know what you care about and I hope you’ve seen enough of me here to know that I am also committed to seeing this through, whatever our differences of opinion or approach. If I didn’t believe this I wouldn’t be so frank about all of this.

I’m content to leave this now, no hard feelings, just weariness. I have learned as well, which to be honest is one of the main reasons I like to participate.

4 Likes
72

Delighted to read this para :muscle:

If we don’t fight about API’s then they’re not worth having, and they would be too large soft and ultimately useless. They need to be as small as possible but providing the ability to build anything, BUT, and this is the big BUT, they should make perpetual data and versions simple and default while making things that destroy either of these obvious and difficult to do.

I hope you don’t find my responses rude, as I am sure you are aware that’s not my bag. I do like to question things with clarity of thinking. That may appear very direct, but I find that valuable. I also hope you see that I don’t try and make things personal or fight to have a particular scheme a must have as I don’t believe in that, until all avenues are exhausted. Innovation is that hard and does mean challenging our dep beliefs and first principles, in my mind.

5 Likes
73

Thanks David. I don’t mind discussion but I wouldn’t characterise this as fighting about APIs. I don’t believe I have the right view, even if I come across like that. I’m putting a point and happy to have that challenged with explanation and justification, and vice versa. That’s why I spend a lot of time trying to explain what I think and why about the issues we’ve been debating. I know we are both trying to make this work, but think we either have different styles of communication, or maybe you don’t have enough time to spend explaining your reasoning in detail etc. Whatever it is, I have found that onerous. You aren’t rude in my experience so I didn’t respond to that comment at the time, assuming it to be unconscious and not meant as it sounded to me.

So no worries here, and thanks for your confirmation.

6 Likes
74

Right now, I am very time limited with this stuff but don’t want to miss inputting to the convo. Again apologies if I am too direct and short. Folk will know me and like you know I am just distracted.

6 Likes