UPDATE: By late February 2024 this discussion has moved away from an approach using Syncer and looked into various ways of providing a filesystem on Safe Network for a FUSE based mountable drive and the potential Safe filesystem APIs and the features they might support.
I’ve summarised the status from my perspective in the following reply:
This is a reminder of an earlier discussion and some experiments I did using the old Safe CLI / APIs (remember Safe NFS?) and a very interesting but still experimental Rust project called Syncer. See:
Syncer has some very interesting goals and is structured in a similar way to Safe Network: hashing of files as blobs (chunks) which live on a remote server but are cached locally on disk, using
rsync to retrieve blobs as needed when reading the filesystem, and to push mutations back to the server. A POSIX like API is used to access this cache, and on top of that is a FUSE based mountable drive.
The project has been dead for a long time but the author claimed very good performance with very similar goals to Safe: a massive virtual drive allowing access to files as if they were held locally.
I had some success in getting Syncer to mount a local drive connected to a local Safe Network testnet based on the old Safe CLI and Safe NFS API. Those experiments are summarised in the topic above. A problem I had with this was that Safe NFS had no concept of directories, but I understand they will be present in the new FoldersAPI which is just being explored, and being based on CRDT registers (see notes on early draft PR).
It might be worth taking a look for synergies here in the Syncer codebase and the Safe Network ‘backend’.
Here’s a diagram I came up with trying to summarise how Syncer worked.
(image from this reply).
Here’s the author’s summary of the design:
I wonder if this could be revisited and provide a shortcut towards a capable Safe Network Files API (more POSIX) which can be easily used to create a mountable FUSE drive and API level support for numerous other projects.
A limitation of the Syncer code is that it uses the high level FUSE API, which uses temporary inodes and so lacks support for hard links.
For a Safe Network filesystem based on the TreeCRDT, the handling of inodes across multiple concurrent user systems was a problem, and while some stochastic solutions were considered they had their flaws (e.g. some risk of inode collisions in a large filesystem).
This limitation is I think solvable using the Register APIs, because inodes can be kept in the local system as temporary identifiers for the life of the disk mount, and the Safe CRDT implementation can use 256 bit content-hashes with no risk of collision.
Each client device will manage local inodes (that never cross the Safe APIs), and the underlying FUSE implementation can map between inodes and hashes, to implement the low level FUSE interface on top of a Register CRDT based filesystem. Soft and hard-links and full POSIX style support become feasible provided the Register CRDT based filesystem implements support for symbolic-links and hard-links. I believe this is feasible by using multiple types of metadata chunk: directory, file, symbolic-link and hard-links.