Code: FileSystem - Unlimited Size (working on Mock Network)

[oetyng]

Update 2019-02-23

The filesystem code was separated from SAFE.DataAccess (which also had the code that is now in SAFE.DataStore repo) and the filesystem code can now be found in a new repository: https://github.com/oetyng/SAFE.Filesystem.
Some bugfixes have been made, and some more tests passing as well as some refactoring and cleanup of code.

I have installed Dokan and implemented the IDokanOperations interface (from Dokannet) as SAFEDrive class.

This means that when you run the application, a drive is mounted and can be found in Windows File Explorer.

Right now it supports creating dictionaries, open them and create new dictionaries within.

And we’ll see a notify icon in the tray:

To use this you need to install Dokan, instructions can be followed here:

Installation

But, the SAFE.Filesystem code is not really mature for testing at that level yet. So, just putting it there for your information.

How Dokan works

Dokan library contains a user mode DLL (dokan1.dll) and a kernel mode file system driver (dokan1.sys). Once the Dokan file system driver is installed, you can create file systems which can be seen as normal file systems in Windows. The application that creates file systems using Dokan library is called File system application.

File operation requests from user programs (e.g., CreateFile, ReadFile, WriteFile, …) will be sent to the Windows I/O subsystem (runs in kernel mode) which will subsequently forward the requests to the Dokan file system driver (dokan1.sys). By using functions provided by the Dokan user mode library (dokan1.dll), file system applications are able to register callback functions to the file system driver. The file system driver will invoke these callback routines in order to respond to the requests it received. The results of the callback routines will be sent back to the user program.

For example, when Windows Explorer requests to open a directory, the CreateFile with Direction option request will be sent to Dokan file system driver and the driver will invoke the CreateFile callback provided by the file system application. The results of this routine are sent back to Windows Explorer as the response to the CreateFile request. Therefore, the Dokan file system driver acts as a proxy between user programs and file system applications. The advantage of this approach is that it allows programmers to develop file systems in user mode which is safe and easy to debug.

I can say that starting to use Dokan is dead simple. You just run the installer, then you implement the IDokanOperations interface, set it up with a few lines of code in the entry class, and then it’s running! Or as they write themselves:

To make a file system, an application needs to implement IDokanOperations interface. Once implemented, you can invoke Mount function on your driver instance to mount a drive. The function blocks until the file system is unmounted.

The devil’s in the details though.

It’s been some struggle to understand how exactly the methods on the IDokanOperations interface are supposed to work. It’s not running very smoothly, and it’s a lot of calls back and forth to the various methods, you have no chance of knowing what is going on and why, or what you are supposed to do.

For example CreateFile being called 8 times when it starts, and then 10 times when you enter the drive, and then 27 (!!) times for creating one folder?? It just doesn’t make sense. What is the purpose of all that you think.

So the struggle continues I really hope that I have just not understood it properly, and that it’s good stuff at the core. I’ll have to trawl the community forum and start asking some questions about this.

We have come this far at least!

For this to work on something else than Windows I will need to take another approach. But I’ve just wanted to test and see how well the SAFE.DataStore would work as basis for a filesystem.

[upstate]

Astounding update! This is going to be great!

[JPL]

Is there anything that the technically challenged can do to help at this stage @oetyng or is this still a conversation between you and the MaidSafe devs?

[oetyng]

@JPL

Is there anything that the technically challenged can do to help at this stage @oetyng or is this still a conversation between you and the MaidSafe devs?

Nothing yet unfortunately, thanks for asking! Right now, there’s not anything for MaidSafe devs to do here either. It’s a battle between me and Dokan now I need to look at more examples, and dig deeper. I am going to implement the interface again based on another example, and see if I can get it straight.

As soon as it is possible to create files and write to them, I’ll holler and provide some instructions for how this can be tested

@upstate

Astounding update! This is going to be great!

Thanks! I was very happy myself when I saw the drive in the explorer

[dask]

This is some impressive work here, I’ve been following your ideas from your first “infinite file structure” thread and am definitely excited to see where you can take it.

[ravinderjangra]

@oetyng

This means that when you run the application, a drive is mounted and can be found in Windows File Explorer .

I think it’ll be very helpful for the users to store and retrieve the data on the SAFE Network. Also, I see you are using .NET Core 3.0 so basically a cross-platform CLI app can be built and used on different desktop platforms. Maybe different code can be used for mouting the drive based on OS.

[oetyng]

@ravinderjangra

I think it’ll be very helpful for the users to store and retrieve the data on the SAFE Network. Also, I see you are using .NET Core 3.0 so basically a cross-platform CLI app can be built and used on different desktop platforms. Maybe different code can be used for mouting the drive based on OS.

Yes, I think this can be very good.
As far as I can see, Dokan will only work on windows, and for mac and Linux I would need to use FUSE. Now, Dokan has a wrapper for FUSE. What this means in practice I’m not 100% sure of yet. But I’ve assumed that it means that if you have code that is already built on FUSE, you can use the Dokan wrapper to also have it running on Windows. So, that’s what I meant when I said I will need another approach to run on mac and Linux, since I have code based on Dokan, and there’s no FUSE wrapper of Dokan that I know of.

So, the best thing would have been to focus on FUSE from start. But I want to be productive and try things out, so I’m focusing on Windows now (and evolving a general idea for the non OS dependent design), and leaving the rest for later

I have found a much better source to use btw, and my issues from previous post I can now confirm are because of not getting the previous implementation right. So I’ve got a very good foundation now for next step.

I’ve started to explore options for performance and reliability. This means some new interesting approaches to data storage. I will try make some more progress before I go into more detail here though.

[bytes]

I thought I’d share some feedback which might help, although this thread is a bit inactive lately. Nice proof of concept, by the way.

The approach of s3backer is sort of similar, but more narrow in scope (and filesystem-agnostic). It uses fuse to create a virtual file of any size you want. The file contents are actually stored in (configurably-sized) chunks in an Amazon S3 bucket. That’s all it does / needs to do. I’m sure a “safebacker” tool could work the same way.

Linux (and other operating systems) can use a file as a block device, formatting any filesystem on top. xfs, reiserfs, ext3, ntfs, fat, etc. This approach could be worth looking into. I think it would be a lot less work (not necessary to “reinvent the wheel”/filesystem). There’s a lot of mature filesystems out there with experienced and active development teams - work that can be leveraged. I’m not as familiar with Windows, but I’m sure it’s possible to do this - for example, VeraCrypt lets you create/mount a virtual encrypted disk from a file.

One thing handy about having flexibility of which file system is in use is that it satisfies the requirements of users of all sorts of platforms. For example, Windows users who want to back up their files (including the ACL file permissions) can do so to their virtual NTFS drive. Similarly, Linux users who have file names and permissions that NTFS doesn’t support, aren’t stuck having to work around quirks and lose metadata.

[oetyng]

Hi @bytes, thanks from the feedback.

This POC was just a first small reconnaissance.
I then began new one with a better starting point.
So this topic is not the current head.

The current project is here: Release: SAFE.NetworkDrive on Windows v.0.1.0-alpha.1

It is now totally different. As it is event sourced, with the actual file system in memory, the data being stored to network is becoming agnostic of platform as well (that is what is being worked on now).

So the filesystems are only projections, and this means you can build on the same stream (same drive) from Windows and Linux. If and what compromises that will lead to in the mapping is to be seen.

This is a distinct approach from the one you suggest. Check it out.

I started with Windows since that is what I am fluent with and I have thus made fast progress with the concepts. Linux and OSX will take more time.