Release: SAFE.NetworkDrive on Windows v.0.1.0-alpha.1


#1

Web30 celebration by announcing the alpha of SAFE.NetworkDrive on Windows!

Alpha version with WPF GUI and console app:

Event sourced virtual drive, writing encrypted WAL to SQLite, synchronizing to MockNetwork and local materialization to in-memory virtual filesystem.

  • Multiple local user accounts.
  • Multiple local drive configs per local user account.
  • One SAFENetwork account per drive.
  • Encrypted local data.
  • Adds local user if not exists.
  • Add/Remove drives.
  • Mount / unmount.
  • Remove local user.
  • Tray icon.

login
drive_unmounted
menuitems notifyicon


add_drive

Prerequisites:

Dokan driver (Dokan_x64.msi at https://github.com/dokan-dev/dokany/releases/tag/v1.2.2.1000)


Dotnet core runtime 3.0 ( .NET Core Installer: x64)

To compile the source you need Visual Studio Preview (4.2 will work fine).
Visual Studio RC works as well but apparently, to get Visual Studio to use .net core 3.0 as target, it needs a global.json file (can be created by running ‘dotnet new globaljson’ as per https://docs.microsoft.com/en-us/dotnet/core/tools/global-json).

Some referenced assemblies will be missing in the release -> either grab them from the zip of binaries and recreate the packages folder structure, or better yet grab the source for those assemblies from https://github.com/oetyng/SAFE.AppendOnlyDb and add those projects to the solution (and update references).

How to use it:

Make sure you have Dokan installed as well as dotnetcore 3.0 runtime.
Compile the app from source.
Then run SAFE.NetworkDrive.UI.exe or SAFE.NetworkDrive.Console.exe.

Known problems:

  • Renaming or moving the first folder on the drive will result in a BSOD (blue screen of death) from a page_fault_in_nonpaged_area.
    This is likely a problem in the dokan driver. I’ve made them aware of this, but I’m not expecting it to be fixed anytime soon.

Beware that there might be BSOD:s in other cases as well.

Also, good to know that there are some user experience related things that has been left unpolished in this version.

Coming up:

Detailed description of the inner workings of SAFE.NetworkDrive.


#2

Any chance we’ll get this on Mac in the future? :grin: This looks awesome @oetyng! If anyone wants to share some vid of it in action that’d be really cool. Happy 30th to the Web!


#3

I hope so! Getting in to the nitty gritty details of implementing FUSE will take me some time, and I will not start with that quite yet. But I’ve hopefully got a good architectural idea here to base it all on. I will be trying that out more, refining it and complete some things still needed, and then I would guess, I’ll head over to Mac and Linux!


#4
  1. How does one compile the app from source (or get the source code for that matter)?
  2. What does EB stand for? I’m not familiar with that memory unit.
  3. Does, or will, the size of the drive increase proportional to how much safecoin one has to spend or space one has purchased with one’s safecoin? I realize this is an alpha on a testnet I’m just thinking ahead here.
  4. Can files be deleted or removed from the SAFE Drive and space thereon be freed up like a typical thumbdrive?

#5

Most likely Exabytes


#6

Thank you for the gift @oetyng! I translated it in Bulgarian:


#7

Congratulations and well done. Good work!


#8

The code is linked to at the top of OP. The source is here, and compiled with Visual Studio: https://github.com/oetyng/SAFE.NetworkDrive/archive/v0.1.0-alpha.1.zip
There are binaries provided here: https://github.com/oetyng/SAFE.NetworkDrive/releases/download/v0.1.0-alpha.1/SAFE.NetworkDrive.UI.zip
https://github.com/oetyng/SAFE.NetworkDrive/releases/download/v0.1.0-alpha.1/SAFE.NetworkDrive.Console.zip


#9

It’s a really challenging task @oetyng and it feels good to see the result of your hard work. This tool is easy to use and will be helpful for all the users.

For Mac and Linux GUI, you can check https://www.mono-project.com/docs/gui/gtksharp/.


#10

Apps like this will guarantee faster adoption of SAFEnetwork on launch.

Excellent work!

I look forward to storing my home OS folder on the SAFEnetwork.


#11

Great work @oetyng, looks very impressive :clap:

We now have all three platforms covered, if not with the same drive! :slight_smile:

I assume you’ll be using C# rather than Nodejs here, but if my experience with fuse-bindings can help give me a shout - the low level interaction should be the same and there are some things it may help you to know about in advance (which ops must be implemented, and that FUSE is addicted to getattr for example). You can of course find this online or work it out pretty quickly yourself no doubt, but I’m happy to help.

If you want, it should also be easy for you to get, or help me get, SAFE Drive working on Windows (which I don’t have atm). That would give you a live example of what FUSE does (I have everything going via debug(), which provides decorated logging to the console, so you can select and filter what you are seeing).

Good luck, and don’t pause now! :slight_smile:


#12

What is an exabyte and how big is that in say GB or TB?


#13

#14

Great work @oetyng! Good to see more options on this front!


#15

Hi @Blindsite2k, good questions!

  1. I think @drehb answered pretty well. But I’ll also say that I am deliberately leaving the accessibility a bit rough for now, because it is an unstable alpha, so there can be blue screens and stuff like that. So those who try this out now, it’s better they are a bit more used to that. But naturally, if you’d like to try anyway we can help you set it up.
  2. As @Nigel said, Exabyte. Very large number. The reason it shows 2,0 TB free of 7,99 EB, is because it has no conception yet of how much total space there is, so it is using long.MaxValue bytes (9223372036854775807) minus the current process 64 bit VirtualMemorySize (which showed about 2,0 TB on my VM for some reason), which gives 7,99 EB.
  3. This could absolutely be a possibility. It’s quite open for suggestions here. It might be configurable as well how we want it to be displayed.
  4. You can delete and move around files and folders just like on any drive. The data that has already been stored to the network will always be there, but your drive will consider it erased just as if it was a local drive that you removed something from. There are some more details on limits and file sizes for practical usage with current version, but I’ll explain more in the details of the inner workings that’s coming up! :slight_smile:
    EDIT: And you’ll always have as much space, on any single drive, as you can pay for, until the network runs out of space :slight_smile:

#16

Wow, that’s great @dimitar, thanks!

Thanks @ravinderjangra, yes that might be a good start. Cross platform GUI is a bit of a quest. I have also been looking at Xamarin Forms 3 for this: https://blog.xamarin.com/xamarin-forms-3-0-released/
It will add full support for desktop apps on Windows, Mac OS and Linux (current status here: https://github.com/xamarin/Xamarin.Forms/wiki/Platform-Support).
It’ll still be some time before I have the backend ready for that :slight_smile:

Thank you Mark! Yes I was thinking the same! :smile: Together we now have those platforms covered.
I’ve been planning to get in contact with you about FUSE as soon as I get ready for it. I’m sure there are plenty of things we can discuss regarding the general problems also.

Thanks everyone for the comments!


#17

Well if it does everything you say it does then it certainly has the potential to become one of those go to apps everyone uses. Could certainly replace the need for external hard drives and thumb drives in a lot of ways though one would probably want those for backup and archival purposes in case the net went down. Have you tested the data transfer speeds this gives? I mean if it takes me an hour to copy an mp3 file then that kind of defeats the purpose of having mass space.


#18

The inner workings - Overview

Storage architecture

Event sourcing

Being an event sourced system, every change to the filesystem is recorded as an event.
These are encrypted and stored in a local SQLite database, as an append only log a.k.a. WAL (write ahead log), and subsequently, in a transaction, applied to an in-memory representation of a filesystem. This means that as you work with data on the drive, read and write it, it will be manifested in the in-memory filesystem. This also leads to some limitations that we will cover further down.
One of the reasons for storing into a WAL, and building the current state as an in-memory filesystem, is to get minimal latency when working with the network drive; the aim is to make it feel as snappy as if it was a local drive. Another benefit of this WAL approach, is that after initial connect, you can be offline without noticing any difference, and your changes will be synched to the network as soon as connection is back.
The event sourcing, and the perpetuity of the data in the network, also means that you would be able to reconstruct your drive as it has looked at any point in history, by just replaying the events, change by change - i.e. restore it to a previous version.

Event synchronization

A background job is detecting activity on the drive, and as soon as you leave it idle for a few moments, it starts synchronizing the events and the content to SAFENetwork.
If your machine was to go down, you won’t risk losing any changes, as the WAL is kept encrypted locally, and will continue synchronizing to the network on next start.

The events are stored into StreamADs (appendable data) of the SAFE.AppendOnlyDb project recently presented.
If the written content of a file is larger than what fits into a slot in a StreamAD, it will instead be stored as immutable data, and the datamap stored to the StreamAD.
The SAFE.AppendOnlyDb is an infinitely expanding data structure, which uses common indexing techniques to allow you to get good access times of your data, even as it grows very large.

As you use a drive, events are produced, and a history of all the changes builds up. Any time you connect to the network, from any device, you will download this log - without the actual data - and build up the folder and file hierarchy locally. Using a technique called snapshotting, this will be a rather small amount of data and fast download, regardless of how long time and how many changes you have applied to your drive. It would take a very large folder tree with a huge number of files, to make this initial synchronization notably slow. (But naturally, the limit to how large this folder hierarchy can be - without the actual file content remember - will be bound by how much working memory your machine has.)
The actual content of files are downloaded on demand as you access it, whereby it is cached in-memory while you use it. (Cache eviction is still on todo-list.)

Merge conflicts

You might be guessing by now, that by choosing this strategy, we have traded speed for complexity, because when the WAL is asynchronously uploaded to SAFENetwork, any changes you (or a team mate, family member, etc. etc.) might have incurred on the same drive from another device, might lead to a conflict, which isn’t detected until only after you have happily continued your work as if the changes went through just fine.
This is a big area which will probably need most of my focus from now on. First of all identifying all compatible changes that can be merged automatically. Second of all, identify and implement a strategy for dealing with other conflicting changes that cannot be automatically merged. This is not a new problem, on the contrary, it is a quite common problem today. So there will be plenty of resources to dig through, and then see how it most sanely can be applied in this situation.

Drive data handling

As a LocalEvent is produced, it is encrypted into a WAL entry, which is stored in a local db file (one per drive). Asynchronously, this log will be worked down and uploaded to SAFENetwork, in the form of a NetworkEvent. Unless you shut down the application before the last entry has been synched to the network, all local drive data will be wiped as the application is shut down.

Security and configuration data

There is currently a convenience approach to this, and there are rooms for improvement.
You create a user on your machine, by providing a username and a password. The password will be used to encrypt the user and drive configuration that you store locally on the machine, as well as the data in the local WAL db.
In your encrypted configuration file (one per user), you will store the SAFENetwork credentials to each drive.
It’s certainly possible to go about this in some other way, for example use several drives per SAFENetwork account, or not store the network credentials locally, etc. I’m fully open to ideas and requests, as to craft the solution that is most desirable for personal or collaborative use.

Performance

My initial experience with this alpha version, is that it actually does feel very snappy, thanks to basically being an in-memory drive. The write throughput to the network will primarily be restricted by your upload bandwidth, and secondarily CPU and local implementation details, which I hope are sufficiently optimized for practical usage, but could surely be improved upon otherwise. This is also something we will find out in better detail as it is being used.

Limitations

The SQLite local database for intermediate storage of WAL entries, has a limit of 1 Gb per row. I have currently not implemented splitting of larger files into multiple rows, so for now it can only handle files up to 1 Gb. But this is a priority feature, so it will soon be able to take larger files than that.
Being an in-memory filesystem also presents some challenges and limitations to how large files can be worked on at any given time. The available RAM will restrict how large files can be handled at any time, and currently also how large proportion of the filesystem you have accessed during a session, since there is currently no cache eviction. This is also a priority feature, as to not put a limit on how much content that can be accessed during a session.
Also, it is currently only working with MockNetwork (which is storing SAFENetwork data to a local file). Naturally it will eventually be possible to configure which (real) network to connect to.

Cloud drive and file system framework

I was able to find a cloud drive abstraction, and a good implementation of IDokanOperations, with related tests, that I could use as a base. CloudFS project: https://github.com/viciousviper/CloudFS and https://github.com/viciousviper/DokanCloudFS. It is virtual drives over various cloud storage providers, and was an excellent template for my work. It is much more generic than I had need for, as it is supposed to be able to handle any additional implementations of cloud storage providers. I’m just interested in one :slight_smile:

I have refactored this mentioned code, and used it in some new ways. It sits on top of the storage architecture described in previous section (the event sourcing, WAL synchronization, local current state as in-memory filesystem, etc.). I have cleaned up a lot of unused functionality and updated the code base to fit with the newest C# code features and my personal coding style. There’s still a few parts of unused code to clean up. I can probably also do some architecture improvements and simplifications, since it was written to be very generic, and SAFE.NetworkDrive does not aim to be generic. There’s no other storage provider needed when you have SAFENetwork :wink:

Deeper dive

I’ll post this for now. It would probably be nice to go even further into the implementation details, with code examples, as well as some visual representations, but I’ll do that in another post in that case.


#19

so just to be clear the purpose of this app is so you can save/retrieve files on the SAFE network and it will appear/act like a thumb drive (with impossibly large capacity)? Sounds like just the kind of thing we need to bring SAFE to the cavemen like me that don’t know much about coding and are used to self explanatory boxes :wink: On these forums we are a minority but I assure you the majority of people in the world need good interfaces like this, not an open engine and a wrench.


#20

That’s right! You start the program, in your explorer the drive(s) will show up like any other drive, and behave more or less exactly like it.

I completely agree. Most often it should be no more, than the simplest way it can help people do what they need to do.