Sharding and the SAFE Network

Sorry to ask if this is answered but i forgot about it and I am not sure if this is already implemented/solved- Sharding. Cannot find in the forum- Is this completed or?

5 Likes

I’d say sharding is inherent in the design, once multiple vaults are possible.

5 Likes

Sharding on blockchain


Sharding tries to solve the blockchain scalability problem (everybody needs to store all information/check all blocks) by building smaller groups that do stuff locally and then at some point publish the results back to the global blockchain.

Sharding and SAFE


The SAFE Network consists of linked sections/groups of vaults that cooperate to secure each other (through relocation of nodes; communicating node age; splitting sections when they become too large) and the data they store (storing data on trusted nodes with old age; punishing misbehaving nodes or kicking them out completely).

To ensure global integrity of all sections and that everyone has played by the rules to get where they are now, the datachain structure on SAFE logs certain events (so kind of like the blockchain does, but the datachain has a different purpose [unlike in blockchain sharding it doesn’t do the same as the sections (in slow) but only stores relevant events that define the governance structure on SAFE] and is event-driven so it doesn’t slow everything down like a blockchain does).

PS: oops - now that does sound a bit complex. I guess the details then are a bit complex in the end too, but the basic idea is simple: you just form groups that have different members (of different age), the old ones decide what’s OK to do and the young ones need to play by the rules, just like a small community. If communities grow too big they split up and to make sure reputational members don’t collude to do evil stuff, members get sent to other communities on a regular basis to shuffle the mixture.


TL;DR

  • Sharding on blockchains is 100% secure for the blockchain, but you probably need to trust the other people in your shard. (so you need to choose between on-chain but slow/expensive and fast/cheap but you lose the safety of the blockchain)
  • On SAFE the whole network is less resilient when it’s small, but when it has grown to significant size all sections are very safe for all participants (and performance is excellent because there’s no blockchain).

This is an editable post that tries to explain sharding and its relation to SAFE. Feel free to change&advance it in any way you want. (Please don’t hesitate to delete or add stuff if you think it makes it easier to understand/clearer.)

21 Likes

That’s a really nice explanation! I’m going to tweet it, but would you like to tidy it up / fix typos a bit before I do? I will also steal it for future use - and if you fancy making it into a post for dWeb Blog let me know - I think a piece explaining this would fit very well there. We could add a couple of diagrams, links to more info etc too. No worries if not, just asking! :stuck_out_tongue:

10 Likes

Sections no longer merge. It is a one way progression now… Grow, split, grow, split, etc. There is no going back, just like cellular division.

2 Likes

How can that be? And once I’m all alone in my section I’m the ruler of the universe Oo

And if we have a power outage we reboot with thousands of empty sections…?

You don’t happen to have a link to where this was communicated @jlpell ?

EDIT/PS: i deleted the merge-sentences in the post because they were [as can be seen by the explanation of david later on] simply wrong/outdated; merges don’t seem to be a realistic scenario in the current design of SAFE

2 Likes

Executive/Developer decision. It was a remark from David from memory

Like you I am not sure they should say never

3 Likes

You will never be alone. A section split is division by two. Min section size for example might be 128 nodes, so it needs to accumulate up to 256 nodes before it can think of splitting.

The section structures wouldn’t get flushed. The nodes would need to reconnect the same way they were before the reboot… More or less… Allowing for some losses due to redundancy protections.

hmhmmm - so if we felt the risk for instability of the network we’d need to reboot everything

(and merges could come with a later upgrade again i guess; in the beginning assuming a growing network should be a sane approach)

:smiley: i need to get some other things done right now - but i’ll get back to this thereafter :slight_smile:

3 Likes

Maybe this thread can be moved to a new topic @neo and we can have a joint effort at reworking the post to be as simple as this but iron out any inaccurarcies - ok with you @riddim?

3 Likes

Getting rid of merge capability simplifies things and makes a lot of sense. It was a good call and will make launch easier. At some point they could add it back but there isn’t any real need to.

2 Likes

Yes. All those sections contain data. If outage means you have to reduce so many vaults that you have to merge, than you lost for sure some data forever. It means dead network anyway. Merging is very complicated to code, and should be very rare event, so they were abandoned for now(maybe forever) as not required for releasing the network. For this reason sections should be really big before they split. There is efficiency problem with too big sections. As far as I remember, count of 120 vaults were mentioned as preferred section size for split. Simple heuristic is, to make sections as big as possible before split, so new sections are big enough to survive random disaster events.

4 Likes

Too late for me @happybeing, maybe the other @moderators can do this later, but good idea

3 Likes

i made this post editable by anyone (if i did it right) so please feel free to correct mistakes/add something/delete wrong statements

6 Likes

Couldn’t you recover the data from “archive” nodes (and external caches), the “real” nodes would need to store more replicas of small identification/meta/hash data, that way they could verify that the data coming from the archives is valid and also that it has the latest version (for append-only and mutable data).

2 Likes

hmhmm - i don’t think archive nodes will be part of the release version already - but they have been thought of / part of the concept for many years i think

2 Likes

Archive nodes are future dreams. Yes, probably yes. but archive nodes will not be there on network release.

2 Likes

I have had a go at tidying it up a little bit. Please check it’s still correct. Also, I don’t understand the above quote. Are you saying that the logging process within sections is slow?

One more thing, I believe that according to the current design sections will not merge anymore, is that right?

4 Likes

i meant that on blockchain sharding the blockchain and shards do the same just at different speeds/trust levels vs. on safe the data chain has its own purpose and does only what it’s good at (which is not identical to the operations done in sections)

1 Like

Sections have Elders Adults and infants. So up to around 200 nodes (min 100), 7 of which are Elders. So Elders are the managers and decision makers. They represent 3-7% of the network. All immutable data is stored on Adults.
To get to a merge we need to lose almost 90% of data holding nodes. So we cannot get to merge. The network infrastructure can lost a large proportion of nodes (up to 97% in some cases) and not need to merge, well before that is had to take action which is not merge as merge causes even larger problems (complexity, nodes need to store data for a larger area of the address and so on).

i.e. no merge for us is like saying when we lose 90% of the nodes the infrastructure would suffer.The network would not survive at that point in any case, so the complexity of merge is irrelevant if that makes sense?

13 Likes