SAFE Network Dev Update - January 30, 2020

I stoped trying to understand a while ago :joy::joy:

Now just a cheerleader and look for the comments to possibly understand whats happening

9 Likes

Personally I think it’s great we have non technical people so engaged in even reading updates. It means more than just tech savvy are wanting this and who better to test it?!

Love the update. Hooray for the great progress @maidsafe makes every week! :tada:

19 Likes

Interesting related page Large Triple Stores.

eg “LUBM 4400K: 1.08 Trillion triples were loaded, inferred and queried. The industry-leading results, and the details of the configuration and best practices are described in the white paper Oracle Spatial and Graph: Benchmarking a Trillion Edges RDF Graph”

8 Likes

Curious, are CRDT’s and DBC alternative approaches to a Safecoin implementation or meant to be used together to compliment each other? Really hungry to know more details about the relationship between the two.

So CRDT is a data type? That allows for eventual consistency (like PARSEC does) but without the need for PARSEC and therefore avoiding additional performance load? Obviously PARSEC is essential for coming to consensus for basic network operations, building trust/node age, etc but in the instance of Safecoin it may not be needed with the use of CRDT?

12 Likes

I have a request: would it be possible to include a section in the update that would allow non-technical people like me to make a realistical where we are at? Or is it impossible to do that? I understand progress ia made, but I cannot assess it qualitatively, so a section for dummies would be awesome.

I mean, sure, there ia a roadmap, but it gives any clue with regard to time, does it? On the other hand project ressources are highly critical and scarce, so if I understand things correctly the time until completion of a non-beta version should be forseeable.

I remember 2015 when I kindly asked for a roadmap with scales. Now we have a roadmap, but with no scale attached. Or am I looking at the wrong place?

P. S. : don’t stone pls

5 Likes

I work in software development and progress around here is hard to follow. More enjoyment can be had from understanding the concepts.

3 Likes

“Macho Man” Randy Savage is still there if you “imagine it”. New guy is “Old Dirty Bastard”.

Cheers have a good weekend and enjoy :crazy_face:

7 Likes

Is there a gh issue or something? i would like to learn how you can use crdts in an permissionless context. bounded counters seem to limit the quota each replica is allowed to transfer without contacting the other replicas, but it could still just ignore that quota without the other replicas noticing it immediately.


is this the crate you’re going to use: https://github.com/rust-crdt/rust-crdt/issues/61 ?

3 Likes

Good luck but I think the answer is No. That’s why I suggested disconnecting the project boards. It serves little purposethese days

3 Likes

The github projectboards seems to be out of sync with reality for few months. Some old tasks are updated but some are missing and some are stacked without any update even that there is activity according to weekly updates. I love them, and I think they are important. But if people come to let say https://github.com/orgs/maidsafe/projects/6 and see last timetable for November, it makes more harm than good. It seems abondoned.

8 Likes

Everyone knows where the project is at and the updates convey progress. The project boards only eat up scarce resources. Perhaps they can be picked up when resources aren’t an issue

5 Likes

We can make a CRDT permissioned, but permissioned by either the clients signature or the sections signature. There are a few nice mathematically (now thank god) guaranteed features and even some without tombstones (no infinite growth etc.)

They are a significant aspect of decentralised or distributed computing and since 2009-2011 they have become accepted. These respect the maidsafe designs from way back and allow containers to be “real” in terms of “this is the current answer, check back it might change”.

So if you look at dynamo / cassandra back in the day (you will see me talking about them a lot a while back) you can see the speed and efficiency improvements (especially latency).

So we have both an ABFT and many data types, our data types are good but forcing CRDT requirements makes them much better and in a great many cases not requiring ABFT/PARSEC for everything. So self validating on merge.

It really gives the Engineers goals to confirm on each type to give us a nice set of features.

These goals are
Idempotency (replay message/ops as much as you wish)
Associativity (a × b) × c = a × (b × c)
Commutativity ( a × b == b × a ) or (X% of Y == Y% of X)
[edited Cheers @tfa ]

If we ensure the types follow these requirements then concurrent access and updates are handled much better than stopping the world till we get agreement.

They generally work on grow only data, which most of ours is. However, you can use orswat to remove tombstones (the removed items etc.) and bounded counters to allow delete (spend) and add (deposit) in a non-negative manner.

I have mentioned these for a long time but they are new (not as old as maidsafe but interestingly original design and data chains depended on them) and not accepted as possible by many Engineers, mostly because they have never seen them.

Ultimately for us right now it allows us to remove any unknown errors we could have in syncronising writes to data. These save that by swapping sync with the merge of state of operations.

tl;dr More formality to data structures and possibly more. Giving us reduced latency and a very important thing I am keen on, what comes out of ABFT in a multi-part decision (such as add/delete) is consensus valid but not able to be applied, if that makes sense. However, what comes from a merge is valid. On occasion, you can have forks if you have linear requirements and this is more a linked list type structure.

so more formalised structures, not necessarily more work, just more correct work.

21 Likes

Good point. The boards are almost complete for now. lot’s of clean up testing and more testing happening. I will ask @StephenC to get us in line with these and we will try and come up with a way to show folk where we are in the process a little better. Right now it’s that last lap and we are rushing around quickly (but quick maybe a few months, who knows). This is a hard part to project.
Maybe a new testing section in the update, so we can show what we are testing and perhaps same with cleanup (which is vital for decent bug hunting).

23 Likes

(permissionless = as you don’t need permission to add your node to the network, everyone can add a node to the network, thus there needs to be a mechanic to catch wrongdoers cause you can’t assume everyone is going to be nice)

I think you would need a specialized crdt lib that adds checks to everything and has a failable merge function, so that you can catch wrongdoers when they try to game the system. I think that’s not the case for https://github.com/rust-crdt/rust-crdt cause it just assumes everyone is playing nice.

Also you need a point at which you can say that an operation was successful say when everyone in the responsible section saw it. But is it really that much faster than parsec? But as you said “more correct work”

3 Likes

Exactly, we are at somewhat cross purposes. We are saying the same thing, almost. But what I was meaning was that you cannot be part of the decision group until your node proves itself via node age. It’s subtle but yes it is fundamentally permission-less, perhaps there is a better way to say/describe this?

Yes, it’s really down to latency. Not all about speed mind you. I think like this, ABFT and family wait on strong consensus by a quorum. CRDT does the same but allows causal order as opposed to agreed order. So this means regardless of when you see an event occur ABFT will tell you it was in another order (potentially). CRDT does similar but does not wait on one decision before the next decision, that is where you get latency improvements.

Apart from latency, there is I believe horses for courses and sometimes you need absolute order and pay that price, but sometimes you need a more naturally defined order and that can be cheaper. I mean cheaper in terms of latency but more expensive in terms o thinking required. I believe though that thinking is not wasted and should also be applied even when you have abft/bft/parsec. So force thinking in a more meaningful way of how you apply operations to mutate stuff. So I see a nice feature of CRDT also helps strong consistency, otherwise, you get consistency on things that might not make sense so much. It’s hard for me to explain as the years have shown though :slight_smile:

12 Likes

Ahh, yes, ur right, ABFTs/PARSEC need some gossiping rounds before it’s comminting on an event order.

7 Likes

Wow, great update … the network is becoming so advanced that, when it’s released, it’ll look like magic! :smile:

magic-world

11 Likes

Formulas for associativity and commutativity seem to be switched.

But concretely what doesn’t that mean ? What are a, b and c ?

3 Likes

So are you big fan of his wrestling? Or do you just like his look?

Yes your right I switched them around. I will edit that.

This is why folk find it hard. It’s using such laws to create types that can satisfy these. Say a counter as the classic example. 1 + 1 is associative and commutative but not indepotent. so we are counting stuff here. So instead of using integers just use a set and the union is the + operator. The length of the set is the “count”. So 1 XorAddress plus another XorAddress is a set of 2 as long as they are unique.

So getting a count of elders in a group/adults in a section now can be made crdt like, so we don’t care about replay attacks, order issues etc.

The you can create OR-Sets to handle remove nodes (Observed Remove set) which is just 2 sets where one is add and one is remove. You have a rule such as last write wins/Add wins/Remove wins etc. and build from there.

The trick is with data in this form then it’s guaranteed to merge cleanly, as long as all pre-requsits are satisfied.

That gives us a lot of scope. But more importantly it allows us to look at data types/containers/functions and ask does it follow these requirements (associative etc.).

Making them so does not harm BFT type things, but can make data manipulation much simpler.

concurrency is the big one, so a social network feed/likes in forums and so on. They all work very will with these types of data

So back to what are a b and c. Well really they are operations or advances in the state of something that can happen at different times and possibly repeated, but if each node has such types and has even seen slightly different advances in state then all the nodes containers will merge. This guarantee is important and part of the larger insight. The data you get back from decentralised networks is likely not final and many times it will be transient. You need to know it is valid, but you can never tell if it is current. So you get a “best answer for now” and this is in fact the case with all of the methods, whether BFT or CRDT.

In this last part the method with the lower latency will give the most current (correct) answer, but it can feel like strict order actually would. It is counterintuative in many ways, but that’s just an interesting point.

Here @tfa There are a bunch of good videos on this subject, but these 2 are pretty good

15 Likes