Code: FileSystem - Unlimited Size (working on Mock Network)


#21

Update 2019-02-23

The filesystem code was separated from SAFE.DataAccess (which also had the code that is now in SAFE.DataStore repo) and the filesystem code can now be found in a new repository: https://github.com/oetyng/SAFE.Filesystem.
Some bugfixes have been made, and some more tests passing as well as some refactoring and cleanup of code.

I have installed Dokan and implemented the IDokanOperations interface (from Dokannet) as SAFEDrive class.

This means that when you run the application, a drive is mounted and can be found in Windows File Explorer.

Right now it supports creating dictionaries, open them and create new dictionaries within.

And we’ll see a notify icon in the tray:
notifyicon

To use this you need to install Dokan, instructions can be followed here:

But, the SAFE.Filesystem code is not really mature for testing at that level yet. So, just putting it there for your information.

How Dokan works

Dokan library contains a user mode DLL (dokan1.dll) and a kernel mode file system driver (dokan1.sys). Once the Dokan file system driver is installed, you can create file systems which can be seen as normal file systems in Windows. The application that creates file systems using Dokan library is called File system application.

File operation requests from user programs (e.g., CreateFile, ReadFile, WriteFile, …) will be sent to the Windows I/O subsystem (runs in kernel mode) which will subsequently forward the requests to the Dokan file system driver (dokan1.sys). By using functions provided by the Dokan user mode library (dokan1.dll), file system applications are able to register callback functions to the file system driver. The file system driver will invoke these callback routines in order to respond to the requests it received. The results of the callback routines will be sent back to the user program.

For example, when Windows Explorer requests to open a directory, the CreateFile with Direction option request will be sent to Dokan file system driver and the driver will invoke the CreateFile callback provided by the file system application. The results of this routine are sent back to Windows Explorer as the response to the CreateFile request. Therefore, the Dokan file system driver acts as a proxy between user programs and file system applications. The advantage of this approach is that it allows programmers to develop file systems in user mode which is safe and easy to debug.

I can say that starting to use Dokan is dead simple. You just run the installer, then you implement the IDokanOperations interface, set it up with a few lines of code in the entry class, and then it’s running! Or as they write themselves:

To make a file system, an application needs to implement IDokanOperations interface. Once implemented, you can invoke Mount function on your driver instance to mount a drive. The function blocks until the file system is unmounted.

The devil’s in the details though.

It’s been some struggle to understand how exactly the methods on the IDokanOperations interface are supposed to work. It’s not running very smoothly, and it’s a lot of calls back and forth to the various methods, you have no chance of knowing what is going on and why, or what you are supposed to do.

For example CreateFile being called 8 times when it starts, and then 10 times when you enter the drive, and then 27 (!!) times for creating one folder?? It just doesn’t make sense. What is the purpose of all that you think.

So the struggle continues :joy: I really hope that I have just not understood it properly, and that it’s good stuff at the core. I’ll have to trawl the community forum and start asking some questions about this.

We have come this far at least! :slight_smile:

For this to work on something else than Windows I will need to take another approach. But I’ve just wanted to test and see how well the SAFE.DataStore would work as basis for a filesystem.


Code: AppendableData as infinitely expanding DataTree, over doubly linked MDs
#22

Astounding update! This is going to be great!


#23

Is there anything that the technically challenged can do to help at this stage @oetyng or is this still a conversation between you and the MaidSafe devs?


#24

Nothing yet unfortunately, thanks for asking! Right now, there’s not anything for MaidSafe devs to do here either. It’s a battle between me and Dokan now :slight_smile: I need to look at more examples, and dig deeper. I am going to implement the interface again based on another example, and see if I can get it straight.

As soon as it is possible to create files and write to them, I’ll holler and provide some instructions for how this can be tested :slight_smile:

Thanks! I was very happy myself when I saw the drive in the explorer :slight_smile:


#25

This is some impressive work here, I’ve been following your ideas from your first “infinite file structure” thread and am definitely excited to see where you can take it.


#26

I think it’ll be very helpful for the users to store and retrieve the data on the SAFE Network. Also, I see you are using .NET Core 3.0 so basically a cross-platform CLI app can be built and used on different desktop platforms. Maybe different code can be used for mouting the drive based on OS.


#27

Yes, I think this can be very good.
As far as I can see, Dokan will only work on windows, and for mac and Linux I would need to use FUSE. Now, Dokan has a wrapper for FUSE. What this means in practice I’m not 100% sure of yet. But I’ve assumed that it means that if you have code that is already built on FUSE, you can use the Dokan wrapper to also have it running on Windows. So, that’s what I meant when I said I will need another approach to run on mac and Linux, since I have code based on Dokan, and there’s no FUSE wrapper of Dokan that I know of.

So, the best thing would have been to focus on FUSE from start. But I want to be productive and try things out, so I’m focusing on Windows now (and evolving a general idea for the non OS dependent design), and leaving the rest for later :slight_smile:

I have found a much better source to use btw, and my issues from previous post I can now confirm are because of not getting the previous implementation right. So I’ve got a very good foundation now for next step.

I’ve started to explore options for performance and reliability. This means some new interesting approaches to data storage. I will try make some more progress before I go into more detail here though.