Sorry to ask if this is answered but i forgot about it and I am not sure if this is already implemented/solved- Sharding. Cannot find in the forum- Is this completed or?
I’d say sharding is inherent in the design, once multiple vaults are possible.
5 Likes
Sharding on blockchain
Sharding tries to solve the blockchain scalability problem (everybody needs to store all information/check all blocks) by building smaller groups that do stuff locally and then at some point publish the results back to the global blockchain.
Sharding and SAFE
The SAFE Network consists of linked sections/groups of vaults that cooperate to secure each other (through relocation of nodes; communicating node age; splitting sections when they become too large) and the data they store (storing data on trusted nodes with old age; punishing misbehaving nodes or kicking them out completely).
To ensure global integrity of all sections and that everyone has played by the rules to get where they are now, the datachain structure on SAFE logs certain events (so kind of like the blockchain does, but the datachain has a different purpose [unlike in blockchain sharding it doesn’t do the same as the sections (in slow) but only stores relevant events that define the governance structure on SAFE] and is event-driven so it doesn’t slow everything down like a blockchain does).
PS: oops - now that does sound a bit complex. I guess the details then are a bit complex in the end too, but the basic idea is simple: you just form groups that have different members (of different age), the old ones decide what’s OK to do and the young ones need to play by the rules, just like a small community. If communities grow too big they split up and to make sure reputational members don’t collude to do evil stuff, members get sent to other communities on a regular basis to shuffle the mixture.
TL;DR
- Sharding on blockchains is 100% secure for the blockchain, but you probably need to trust the other people in your shard. (so you need to choose between on-chain but slow/expensive and fast/cheap but you lose the safety of the blockchain)
- On SAFE the whole network is less resilient when it’s small, but when it has grown to significant size all sections are very safe for all participants (and performance is excellent because there’s no blockchain).
This is an editable post that tries to explain sharding and its relation to SAFE. Feel free to change&advance it in any way you want. (Please don’t hesitate to delete or add stuff if you think it makes it easier to understand/clearer.)
21 Likes
That’s a really nice explanation! I’m going to tweet it, but would you like to tidy it up / fix typos a bit before I do? I will also steal it for future use - and if you fancy making it into a post for dWeb Blog let me know - I think a piece explaining this would fit very well there. We could add a couple of diagrams, links to more info etc too. No worries if not, just asking! 
10 Likes
Sections no longer merge. It is a one way progression now… Grow, split, grow, split, etc. There is no going back, just like cellular division.
2 Likes
How can that be? And once I’m all alone in my section I’m the ruler of the universe Oo
And if we have a power outage we reboot with thousands of empty sections…?
You don’t happen to have a link to where this was communicated @jlpell ?
EDIT/PS: i deleted the merge-sentences in the post because they were [as can be seen by the explanation of david later on] simply wrong/outdated; merges don’t seem to be a realistic scenario in the current design of SAFE
2 Likes
Executive/Developer decision. It was a remark from David from memory
Like you I am not sure they should say never
3 Likes
You will never be alone. A section split is division by two. Min section size for example might be 128 nodes, so it needs to accumulate up to 256 nodes before it can think of splitting.
The section structures wouldn’t get flushed. The nodes would need to reconnect the same way they were before the reboot… More or less… Allowing for some losses due to redundancy protections.
hmhmmm - so if we felt the risk for instability of the network we’d need to reboot everything
(and merges could come with a later upgrade again i guess; in the beginning assuming a growing network should be a sane approach)
i need to get some other things done right now - but i’ll get back to this thereafter 
3 Likes
Maybe this thread can be moved to a new topic @neo and we can have a joint effort at reworking the post to be as simple as this but iron out any inaccurarcies - ok with you @riddim?
3 Likes
Getting rid of merge capability simplifies things and makes a lot of sense. It was a good call and will make launch easier. At some point they could add it back but there isn’t any real need to.
2 Likes
Too late for me @happybeing, maybe the other @moderators can do this later, but good idea
3 Likes
i made this post editable by anyone (if i did it right) so please feel free to correct mistakes/add something/delete wrong statements
6 Likes
Couldn’t you recover the data from “archive” nodes (and external caches), the “real” nodes would need to store more replicas of small identification/meta/hash data, that way they could verify that the data coming from the archives is valid and also that it has the latest version (for append-only and mutable data).
2 Likes
hmhmm - i don’t think archive nodes will be part of the release version already - but they have been thought of / part of the concept for many years i think
2 Likes
I have had a go at tidying it up a little bit. Please check it’s still correct. Also, I don’t understand the above quote. Are you saying that the logging process within sections is slow?
One more thing, I believe that according to the current design sections will not merge anymore, is that right?
4 Likes
i meant that on blockchain sharding the blockchain and shards do the same just at different speeds/trust levels vs. on safe the data chain has its own purpose and does only what it’s good at (which is not identical to the operations done in sections)
1 Like
Sections have Elders Adults and infants. So up to around 200 nodes (min 100), 7 of which are Elders. So Elders are the managers and decision makers. They represent 3-7% of the network. All immutable data is stored on Adults.
To get to a merge we need to lose almost 90% of data holding nodes. So we cannot get to merge. The network infrastructure can lost a large proportion of nodes (up to 97% in some cases) and not need to merge, well before that is had to take action which is not merge as merge causes even larger problems (complexity, nodes need to store data for a larger area of the address and so on).
i.e. no merge for us is like saying when we lose 90% of the nodes the infrastructure would suffer.The network would not survive at that point in any case, so the complexity of merge is irrelevant if that makes sense?
13 Likes
makes sense - yes i didn’t expect it to be this extreme 
thank you very much for clarifying
5 Likes
linking previous topic about this
2 Likes