Appendable Data RFC

https://reviewable.io/reviews/maidsafe/rfcs/167

Looks pretty good to me! This will be helpful in developing Project Decorum for sure.

I just wonder, why is for example HashSet<AppendedData> used instead of simply a Vec<AppendedData>? Isn’t the order of keys in a HashSet arbitrary? Or is that not the case here because it will never be resized?

Edit: One thing concerns me a bit, and that is that the cost to fill a Pub/PrivAppendableData without a white list with spam is zero. Public keys can be genereted en masse, so blacklisting won’t work against a determined attacker. In my original Project Decorum design, every reply costs a PUT and there is no kind of “reply list” that can be filled up. Spamming would thus be costly and would never result in complete denial of service.

14 Likes

To prevent duplicates.

This data list is required to be seen as “unordered” and any manual changes of ordering would probably be reset by the Vaults. Thats just a requirement to facilitate the union nature of accepting updates from multiple senders concurrently and treating them as valid updates and also allowing accumulation for GET requests.

Fair point. To prevent determined spamming white-list would be the only option currently.

In terms of cost for normal usage. PubAppendableData data holds AppendedData which itself holds DataIdentifier which needs a PUT by the sender to create. Again only in normal case. For someone trying to spam, they can just provide some random DataIdentifier to thereby cap the PubAppendableData itself without a whitelist.

unless

the APPEND rpc also routes through the MaidManagers thereby incurring a charge for the sender to append info. Worth discussing for sure :slight_smile:

10 Likes

Maybe restrict it to Data you own? Aka, Data you already paid for?

We can’t validate that for say ImmutableData which isnt “owned” by anyone in particular that the network keeps track of. Even with ImmutableData deletion support RFC its not always guaranteed like StructuredData for example

5 Likes

I think I’m in favour of that. Also, since the network would be paid as AppendableData fills up, we could look for a way to somehow automatically/seemlessly extend AppendableData when it reaches it’s 100 KiB limit. Would be a bit annoying if for example a forum topic locks up just because it’s AppendableData is full.

ImmutableData doesn’t know ownership. Edit: Ninja’d…

6 Likes

Yes, I really like this one. The way I get it someone can add a “pointer” to something like his reply on a blog. So I write a blog, I open an AD for everyone to point to their reply. From the reply-er perspective it goes something like:

I read a nice blogpost, I make a PUT to write a reply and I point to my reply in the AD offered by the owner of the blog.

If I just want to reply with some weird smiley I don’t have to PUT as someone else probably already did this. So I only have to add the pointer in the AD. To prevent spam is has to cost some money, otherwise people start to fill up AD-fields with whatever and you need an extreme long whitelist. For a forum where you need to become a member probably not a problem, but for an open blog it is. So adding something to AD should cost a PUT or half a PUT maybe to keep spammers away.

5 Likes

It is easy to do, rather than Post of AppendableData it becomes a Put, which may make sense as you are creating something, even though it is just a tiny thing.

7 Likes

Probably offtopic, but I’m trying to nudge a few people :wink: for pretty much this and not just for this type but for StructuredData too. I know @ustulation would very much like to have that in place too . Most of us agree with this being an annoying limitation and while it justified its annoying presence a while back, some of those requirements no longer exist cos say routing does a message split for transport itself regardless of the actual payload size from upper libs. Some points still do remain :expressionless: such as data concentration within a given group. There are answers, need some more bashing about and we should certainly be able to achieve it cos it is annoying and also creates loads of indirections forcing API’s to morph into linked lists to workaround these limitations.

10 Likes

Yip a next block on each SD/Appendable would be good, so then every SD etc. you create has a special link which always points to next block in chain. Sort of like a linked list, difference is that cannot be mutable AFAIK. So if an owner thinks that more than XX size is needed they create a next link item on the SD (link item is immutable, when it exists it cannot be amended or deleted), otherwise when it fills that is it.

That in itself should be an RFC though, there are many ways to do it.

4 Likes

Am I correct in observing that this is essentially a blockchain data structure on safe net? :slight_smile:

Question: Can the filters be changed after the original data has been created? I am thinking that a permanent, unmoderated, chain of data elements would be a powerful thing in some use cases.

Of course, blockchains cannot contain or link to large, on network, data items… But this could!

1 Like

From my understanding, it looks like the owner of the AppendableData is the only one who can actually delete entries from the AppendableData, while the closest someone can come to deleting an entry that they attached to an AppendableData that they don’t own is to delete the data that it points to and to leave a dangling pointer. Would it be helpful to add a mechanism for users to remove their own AppendedData entry from an AppendableData they don’t own, or is the current method of leaving dangling pointers (which could be cleaned periodically by the owner using a mechanism like the one for cleaning deleted_data) sufficient?

1 Like

We have left this open to app developers. So for instance to say this app X requires you have immutable data that parses in a specific way (say email or chat or forum post). As this is immutable it cannot be deleted (in this case) so I may delete your email/comment etc. but cannot delete your data and neither can you. This saves people posting stuff then deleting/ editing a post to make it appear like an answer is wrong etc. (you see it happening here).

Other app devs may not mind and allow structured data be pointed to, allowing editing capabilities.

Other devs may not even care what the pointed to ting parses as or have them as pointers to videos/pictures etc. who knows?

tl;dr We have made this extremely open to app devs to specify a fixed format/editable or not etc. but only owners can remove the links in the appendable data.

It seems the obvious next type to add here will be push aware data items for more real time chats in band (out of band chat is Ok regardless)

10 Likes

Is it possible to have lists where even the owner cannot change the history too? Immutable data lists/journals/logs, etc, being the use case.

Apps could achieve that, either through using ImmutableData or using a known invalid public key in the owner field of a Structured/AppendableData.

1 Like

Presumably, the reference to the structured data reference to the immutable data could still be removed by the owner though?

Can this still be done? I thought this functionality was pending removal or is it here to stay?

It’s not possible through the API at the moment, but this is from the latest update:

So I assume that the low level API will allow app devs to do so. Would be a pretty high low level API otherwise. :slight_smile:

1 Like

Here’s hoping! I suppose the devil will be in the detail of the low level API. I am looking forward to exploring these!

1 Like

I suspect much of these dynamic data types will evolve on a short period. That will be API breaking, but as @neo pointed out recently these perhaps could make use of the tag type more like a mask. Thereby setting the attributes of the SD type.

We were contorted the other day on whether the appendable types are specialisations of SD types or there is perhaps another base SD type and the other types inherit them etc. It may be the tag becomes a mask bit to state the attributes of a fundamental type? This is where we need folk to create apps and tell us

  1. Do they have the functionality they want
  2. How easy was it to get the functionality
  3. Is there a cleaner way to provide greater functionality with a simpler API.

Yes does mean breaking changes up to version 1.0, but I feel if these changes are considered and tested then API 1.0 will be widely used.

4 Likes

As an app developer, I really don’t care about that at this stage. Let’s take our time to make that API as good as we can.

That’s a splendid idea in my opinion (thanks @neo!), but would still like to have part of the tag_type bits to be open/arbitrary as it is currently (that’s probably the idea, but want to make sure).

2 Likes

Absolutely, first byte is mask or something along those lines.

2 Likes