The tool is probably not too useful, I’m not really sure what use I’d have for it besides the learning I got from the exercise. Building the tool helped me understand potential improvements and limitations of the library, and also helped me find where I had misunderstood certain things.
Developing this tool directly led to the self_encryption compression topic which ended up much more detailed than I had originally imagined.
Development also motivated the separation of sn_url from sn_api, which arose from the discussions in getting a safe:// url for a file (something needed for this tool). This conversation and process also had plenty of unexpected details.
I suspect this tool could indirectly lead to some future changes and ideas. It’s not really an end-user tool, I can’t imagine why anyone would need this, but hopefully it’s useful for engineers. The development process has been really rewarding (but not in the way I had originally expected).
Very nice tool!, I really find these type of tools very useful, I mean as a dev/eng.
It seems so…
…I gave it a try with a file and got three Safe URLs (awesome!!), but I checked you are encoding/generating them with data type set to SafeKey which wouldn’t be correct, it wouldn’t be correct to encode them with data type Public/Private Blob either, so I was wondering if the sn_url should have a Chunk data type defined which can allow us to encode Safe URLs for chunks as you are doing?
I think that’d be quiet simple to implement and useful as you say, the $ safe xorurl command could print out all that same info as it already uses self-encryption and without the need to connect to Safe, so I think that’ll work quite nicely there, maybe behind a flag --chunks-info or something like that.
Ah, I see! I assume from this that clients store and access chunks only by their xorname, never by the xorurl (makes sense). I’m not sure if we need or want chunk-level safeurls. I would want to see some real example cases where it’d be useful (where xorname isn’t enough) before making any changes.
Do datamaps have a safeurl type (I guess they must)? I don’t show a url for the datamap because I figured it would be misleading if the data was private, in which case the datamap would be encrypted and the tool doesn’t have the encryption key (nor should it) so can’t create the content to derive the xorname and safeurl.
There’s basic_encryptor in self_encryption examples which works as a cli. You can build it
cargo build --example basic_encryptor
and it will create an executable in the target directory that can be used to encrypt like this:
./basic_encryptor -e /path/to/my/file
and decrypt files from a datamap+chunks like this:
IIRC this currently requires a connection to a safe network to function which is frustrating, it would be great to have it work offline. I’ve tracked down the reason before in a hope of making it work offline but to cause ended up being very deeply nested so I left it. Was a long time ago so might have changed now…
I’ve been considering adding an example to sn_url that will give the safeurl for a file (the url for the datamap is what most people would want I guess?).
Also have been considering something similar to the unix command tree that lists the safeurl next to each file. And something like tree -v that also shows the chunk details for each file too. But it’s all nice-to-have so who knows if I’ll ever get to it.
Small thing (maybe ties in with the self_encryption api) but I wonder if we can show a progress bar on safe xorurl my_large_file since it can take a long time for large files. It’s something I planned to add to the ui at some point too.
Yes local. Looks like safe xorurl <dir> is good enough for my intended use case - checking if a file exists on the network before paying to upload. ie use the locally generated safeurl to do a GET and see if it succeeds/fails. sn_api packs in a lot, it’s easy to forget some things!
As you say self-encryption builds the data-map using xornames and no xorurls, Safe URLs are only used from sn_api layer up (at the moment at least).
Thus, the idea of being able to encode a chunk’s xorname in a URL would not be necessary (not desirable either I guess) for self-encryption but just for end users or apps.
I’m not fully sure of a use case right now, I’m just thinking that the idea of XOR-URLs is to be able to have a URL for absolutely everything that is stored on the network, a chunk is a piece of content on the network which has its address and thus can have its XOR-URL. The nice thing about XOR-URLs is that I don’t have to tell you what kind of content I’m sharing with you, it’s all encode in it, if you have the right app, it’ll know how to fetch it and how to render and/or use the content.
There is a piece of logic in sn_client at the moment which was discussed to be part of self_encryption. This logic is about chunking even the data map if it’s too big, which means you could have a root data-map, and children data-maps, some of this is here.
That being said, the root data-map’s xorname is what we encode as the Blob xorurl, and that’s what we use to then fetch a Blob. The Blob is the abstraction we expose for users/apps, which virtually contains data-map/s and chunks to reconstruct the file stored on Safe.
Private Blobs/chunks also have their xorurls, I should be able to give you the xorurl of a private file we share (or any type of private content), the keys to decrypt it shall be shared out of band ofc., or for example if you create a private FilesContainer you’ll need the xorurls of your privately stored files (private data-map/s).
I should be possible I guess, I know self-encryption works by you providing the object which is responsible for the read and write of chunks, so from there you can use some closure provided by CLI…perhaps…
Currently sn_api uses sn_client for safe xorurl command, but I’m suspecting we may want to use self-encryption API directly to be able to do all that’s been discussed here.
