SAFE Network - TEST 7

Hi All,

Here is an interesting one, I just tried to upload a directory with a website inside it with a freshly created account, the directory had over 900 files inside but was only about 37mb in size. I tried to upload in one go and it failed with the low balance error. When I checked it had only uploaded 11.8MB but had used all 500 puts.

Maybe every file uploaded is a put even if it is v very small?

Cheers
Al

1 Like

In some of the next iterations , the directory listings should show us some info like how many files are in each ā€¦ that would be great

http://maze.savage.safenet

2 Likes

It is worse than you think: Files over 3076 bytes and below 3 MB cost 3 puts each. This is probably the case for most of your files (41 KB average).

By contrast files below 3076 bytes cost a lot less, because they are stored in the SD corresponding to the directory (one put when they are totalling 100 KB)

1 Like

All files have 3 PUTs because there needs to be 3 chunks. The exception might be the ones < 3076 since it is stored with one PUT into the SD. Also a PUT is needed for any SD created, no PUT for updating a SD.

4 Likes

No, putting a file < 3076 into an existing SD costs nothing. You can also modified it freely any number of times while it remains < 3076. I used this possibility extensively for my dynamic safe net sites on previous test networks.

I think you are both saying the same thing (correctly :slight_smile: )

3 Likes

+1 Would certainly like to see this available from demo app. Move apiā€™s are available from launcher. So should definitely be something that gets support soon.

Yes it is.

Not even part of the actual file chunks, just meta-data of the file. So updating just that is the efficient route and also lot faster than delete & upload.

5 Likes

I am thinking now that what I said is only true while directory size is below 100 MB. That was my case but it is not the case of @Frontrow_AI. Above this limit I am afraid that each added file cost 3 puts, even for small ones, unless the launcher optimizes the upload by computing first all the datamaps of included files and then writing the SD only once?

Correct, but at some stage the SD has to be created. so there is a PUT at some stage.

Sorry I had misread your post, I had read it as you saying it costs 3 puts for < 3076, thus why I posted. My Bad Sorry

2 Likes

feel free to slap me downā€¦

iā€™ve been on holiday for a week and things have moved on quickly in my absence, which is good i thinkā€¦

soā€¦

test 7 is running?

test 7 vault binaries can be grabbed from???

there are 2 test 7ā€™s?

which one should i run my vaults on?

how do i set which network they are on?

thought it best to just ask rather than read 1000ā€™s on postsā€¦ a good engineer doesnā€™t remember stuff, he knows where to look it upā€¦

rup

1 Like

TEST 7 is now live with the launcher and demo app. The network is run by Maidsafe on droplets. Give it a go :thumbsup:. We are waiting for new Vaults. In that case, the droplet network will be up (A) but we also test the Vaults (B). So think of 2 nice maps on your computer in the coming weeks. One to connect to the stable droplet network, the other to run your own Vault.

1 Like

ahā€¦ okā€¦ will wait for vault bins to arriveā€¦

cheers!

rup

Certainly keeping an eye on this one to see whats causing this issue. Hopefully @Krishna_Kumar has some info on this soon.

This is unfortunately a bit more involved issue where it ties into a few other restrictions. Itā€™s something guys from safe_core are currently looking into and should hopefully result in a few RFCā€™s to refine some of the ideas here.

So yeh moving on from the cryptic para ^^ that no one prolly understood, this is whats going on with this one.

Itā€™s not the counter itself from Launcher thats at fault here, but currently there are no delete requests actually getting made to the network. So right now a ā€œdeleteā€ operation triggered via launcher is more of a ā€œforgetā€ operation than a ā€œremove from networkā€ operation.

To understand ^^ need to know a bit about how data is currently structured in the network. So we know files result in chunks and a DataMap(index list of chunks that make this file up) and before someone mentions ā€œwe dont always create chunks when storing a fileā€, yes if a file is smaller than 3 * MIN_CHUNK_SIZE we store no chunks and we just post the content of that file into the DataMap itself. Chunks get stored as ImmutableData in the network. DataMap gets stored in the parent DirectoryListing(another index list which contains all the DataMap(files) / DirectoryListing(sub-folders) inside this folder) which is StructuredData. So thats the basics covered.

So for the forget operation ā†’ all that needs to happen to ā€œdelete a file/folderā€ is to remove the corresponding DataMap /DirectoryListing from itā€™s parent DirectoryListing. This occurs as a POST operation to the parent DirectoryListing and thus why we do not see it getting logged as a DELETE request/count from launcher. Userā€™s folder hierarchy from that point is as expected but the data in the network still exists and isnā€™t actually removed.

Right now, network only supports actual delete(ā€œremove from networkā€) operation for StucturedData from Vault code. Trying to delete ImmutableData would just result in a UnknownRequestType. It should probably also tell the client this error which it doesnā€™t do right now, but back on topic. ImmutableData delete isnā€™t supported right now cos the network doesnā€™t keep track to which clients stored what data to know when to delete the same. StructuredData on the other hand has this feature built in with the data type itself knowing who ā€œownsā€ it at any given time. Remember the cryptic para and what we mentioned about RFCs, so this is something one such RFC needs to look into to being able to know ā€œwhenā€ itā€™s safe to delete immutable data. Itā€™s may not be quite as simple as just keeping a record of all clients storing this piece as it may lead to massive meta-info that would make churn relocation infeasible, so needs some tolerance levels and also why/how it benefits the network to keep track of this.

Right now, there are also no ā€œrefundsā€ of any kind and no actual ā€œmotivationā€ for users to delete the actual data apart from clearing their own directory structure which the forget operation accomplishes. It does result in ā€œabandonedā€ data which resides in the network not needed by anyone but vaults are monitoring and relocating with churn etc. So another RFC needed where this is one area that it needs to cover to motivate users to actively ā€œdeleteā€ data by maybe offering a refund and at the same time help the network not deal with ā€œabandoned dataā€.

There exists a final point to also consider about making DataMapā€™s Immutable in the network and stored separate to the parent DirectoryListing and just have the DirectoryListing reference the DataMap by its name/some identifier. This helps work around small files not triggering Putā€™s since they got POST into the parent DirectoryListing as now even a small file will trigger a PUT for itā€™s DataMap + possible chunks instead of just possible chunks triggering PUTs. It has added benefits to being able to reference files directly than needing to wrap them in a DirectoryListing to share/reference with others ofc provided its public and unencrypted/ā€¦

Soz abt the long answer, but hopefully the cryptic para conveys some meaning now.

14 Likes

Thanks for the clarification. Really good post, viv.

I think we should stay that way, and let users know that there are two ways dealing with data.

Immutable data is impossible to delete, so in this scenario, it would be good for gps, scripting, machine learning, etc. I wouldnā€™t even bother trying to figure out how to delete immutable data.

Structure data would be good for comments / post or setting up goods ( prices fluctuate) so that users can modify or delete any time.

Please, donā€™t do it. I beg you!

My safe sites rely on this functionality. They are dynamic and my account will run out of free space after few hours if you do this.

Mutable data is a fundamental part of safe network. You cannot remove one of its applications like that, at the last moment.

Besides I donā€™t think that would improve performances because the number of gets will increase dramatically when reading a set of small files in a directory, for example a web site carefully divided (html + css + json + JavaScript)

2 Likes

re Launcher:[Logs] tab

While making actions that see progress, like rattling through a series of HTTP:400 urls where publicid does not exist and then the speed of progress is clear in the terminal, switching to the [Logs] tab in the GUI slows everything down something like a quarter speed, which perhaps is expected but perhaps can be improved.

Similar seemingly one-off effect on entering the [Dashboard] but perhaps thatā€™s expected if its requesting information from the network to populate the dashboard detail.


So, the sticky in the launcher is not to do with the Demo App.

All tabs in the launcher are responsive to the end of the scanning script and upto the point of logging in. Immediately after the login the launcher is stuckā€¦ I stuck with it. htop suggests a launcher process is chewing up 104% CPU when tabs are selected. Outside that selecting a tab, nothing extraordinary on the CPU useage.

Specifically:

Moving [Account] ā†’ [Settings] takes 9 secs
Moving [Settings] ā†’ [Account] takes 61 secs

Moving [Account] ā†’ [Dashboard] takes 9 secs
Moving [Dashboard] ā†’ [Account] takes 61 secs

Moving [Account] ā†’ [Logs] takes 9 secs
Moving [Logs] ā†’ [Account] takes 61 secs

Moving [Account] ā†’ [Help] takes 8 secs
Moving [Help] ā†’ [Account] takes 61 secs

Any other tab switch that does not involve [Account] tab works fine.

So, I wonder if itā€™s to do with the Authorised Apps element - though none are authorisedā€¦ or some other element in the Account tab. Itā€™s odd that the delay is not simply on entering the Account tab but also apparently on leaving it too.

One step forward perhapsā€¦ but why should that occur only after having given the launcher some work to do - albeit 2hrs? Through that script running, activity is through to the terminalā€¦ Iā€™ve not used the launcher while its running, in any extraordinary way.

It might be the proxy?.. I didnā€™t have patience to stick with it to use the Demo App and update the directory but I did get to the point of itā€™s having authorised the Demo App. The initial step in the Demo App appeared very delayed and I worried the proxy/launcher might cause a trouble uploading a file, so stopped there.

Iā€™ve updated the bug report with that detail too.


Out of 80 PublicIDs, fell 60 websites

http://dir.yvette.safenet

1 Like

Thereā€™s nothing set in stone here and weā€™re certainly not going to be implementing any of it until an idea runs through the rfc process, the whole point of which would be to challenge the idea and see what benefits it adds and why it should rather not be applied.

Donā€™t you see the problem elsewhere with this in the first place. If users are running out of storage cap, there are ways to address that than encourage more storage of data into DirectoryListing directly cos all thats going to do is flood the network eventually with multiple miniature requests and bottleneck the system.

This shouldnā€™t have anything to do with mutable data, unless you mean the case of files being less than 3kb which get mutated inline with POST requests. Again in that case, thats more like point above than ā€œmutable dataā€ in general.

and thats why they need to run through a rfc process :slight_smile: where improvements can get quantified and tested in scenarios to see actual usecases too.

3 Likes

Yes, this is exactly my use case. I have 2 examples in mind:

  • A small html file that is periodically mutated

  • A fixed html file loading a small json file that is periodically mutated.

Hello all, Iā€™m finally dipping my toes into maidsafe after years of watching from the sidelines. Downloaded the launcher and demo app and things worked great, but I canā€™t get to my or any of the websites listed hereā€¦ Should they open in a standard browser? my website is here: website.eric.safenet but i get a 404, what am i doing wrong?

4 Likes