To be clear, this is not a public testnet announcement post!
Work continues in a couple of areas to link the features in this document together, and so a testnet announcement will not be this week.
We hope to have a public testnet ready for you to try out within the coming weeks. Thank you for your patience while we work through the remaining issues.
As always, a full update on progress will be provided in Thursday’s Dev Update post.
With a new public testnet inching ever closer, we thought it would be good to post a summary of some of the main innovations we’ve made recently to get us to this point. These details are mostly available via our usual weekly dev updates, but have been spread out over several months now, so we bring them together here in one post.
Will this be the Fleming release?
Not just yet. We view this as a testnet for the Fleming release.
As well as there being a couple of known gaps in some of the areas that are described below, for example not all data is split between sections yet, we fully expect to have to make tweaks to algorithms and address bugs or shortcomings identified through the much more realistic, varied, long term and intense testing that a public testnet provides when compared to our internal testnets.
This will be the first time many of these new features, and even refactored existing features, have been used in the wild - theory on paper and in code doesn’t always match up with the real world! We therefore expect that there will be several iterations of this testnet before we consider it feature complete and stable enough to officially say that the Fleming goals have been achieved.
See the Safe Fleming Network target feature set on our roadmap page for an overview of what the Fleming release will achieve, originally published in 2019. This is what we have been working towards all this time, with so many related and supporting features and updates required to achieve this.
What will be new/improved in this testnet?
There will be many new, improved or refactored parts to the upcoming testnet, probably the most notable being section splits, rewards, AT2 payments, accepting new nodes only when resources are required, and the section chain being fork resistant.
Note that we are close with two other features - lazy messaging in sn_node and one single (CRDT) data type with immutable policy, but we are not 100% sure at this point whether they will make it into the initial testnet. Perhaps a partial implementation of lazy messaging will be sufficient for the first iteration, while the data type change will only be included if it is complete in time and does not cause any adverse effects which would delay testnet release.
Some further details on all of these below.
Section Splits
Section splits mean that the Network can grow naturally.
As the section size grows and there are enough nodes to separate into two new sections, a split happens. This is quite tricky as the section chain splits into two, while the data held also splits with approximately 50% moving to the new sections.
As we balance nodes’ addresses — note that we still have more work to do here in a future testnet iteration — then the data should already be on the correct nodes in the vast majority of cases. However, some promotion will happen. This means the requirement of switching real chunk copies among nodes is reduced to a minimum. Mostly, it will only be the indexing map of the chunks, held among the elder nodes, which needs to be split and transferred among the now two elder groups.
To ensure the two split-out sections both contain enough capable nodes to handle requests and provide storage space, we enforce that both of the after-split sections satisfy the RECOMMENDED_SECTION_SIZE, and only then shall the split happen.
Not all data is split at the moment, we have not yet implemented the splitting of Map, Sequence and chunk metadata. These will follow in future testnets.
Wallets should be split, however we are still to see how this fares with higher traffic.
Rewards/Farming
Individuals who choose to supply the resources that the Network requires will now be rewarded with Test Safe Network Tokens in the upcoming testnet. In testnet iteration 1, these rewards will be paid to a node (by its new section) after each relocation of it. In an upcoming testnet iteration, the plan is to add a (smaller) payout on split to Elders only.
As this is a testnet, please keep in mind that these are only Test tokens, they will not be able to be traded for real Safe Network Tokens on launch of the real network.
Token Payments: The AT2 way
We’ve had Safe Money and its transfers, or payments if you will, integrated into our testnets for a while. We recently revamped the whole implementation with the AT2 protocol, which gives cryptographically verified transfers at theoretically higher speeds than blockchain-based technologies.
The introduction of shared signing of Elders makes these payments historically verifiable, and together with clients keeping their replicas, this would also allow for re-uploading and verifying a wallet even without it existing in the network.
This lives entirely apart from the consensus mechanism of the Network and is done one-on-one with the users, hence reducing the stress on the network.
It’s worth noting, that the client driven process can be frozen if there’s an attempt to make an out of order payment. The balance will currently thereafter be unusable. We’ll be addressing this in future testnet iterations.
Accepting New Nodes Only When Resources Are Required
A recent improvement to the Network, to be showcased for the first time in the upcoming public testnet, is that existing nodes now decide when they want new nodes to join the network.
This improvement is designed to help prevent Sybil attacks by virtue of the fact that we do not allow unlimited nodes to join the network at will; the danger of an unchecked number of nodes being that potential attackers could easily flood the Network, taking it over with hostile nodes.
Routing assumes the elder-nodes will track all the adult-nodes in the section, and when they detect the average storage capacity (or some other resource) becomes too low, they will flip a flag so the section starts accepting new nodes. The section elder-nodes should detect this at more or less the same time, so consensus can be reached.
Fork Resistant Section Chain
In order to prove that a piece of data was signed by a group of nodes that were once valid members of a section, even after those nodes are long gone, we have a “section chain” which is a list of section keys linked together with signatures. Previously, this chain required that the section agreed on which key to append to it next. If there was a disagreement on that, the chain could fork into two (or more) mutually incompatible chains which could break the section. This could happen, for example, at times of intense churn.
This problem has now been resolved by accepting section chain forks internally, but always presenting a unique and deterministic order of the chain blocks to the outside.
A new implementation of the section chain data-structure, which is now a proper CRDT, guarantees (eventual) consistency regardless of in what order the operations are applied, how they are grouped, or even duplicated. Even if multiple distinct keys are inserted into it, everybody will eventually agree on which one is the most recent one and thus who the current elders are (because each section key is uniquely associated with a single group of elders). And all of this is achieved without involving any complicated consensus mechanism.
Lazy Messaging
Note - Lazy Messaging across the board is not guaranteed for the upcoming testnet
A major hurdle in any decentralised network which will experience churn, such as ours, is keeping the nodes on the network up to date with the network’s current state. For example, who has joined and who has left? Or who are the current Elders in any given section? We have implemented lazy messaging to resolve this issue.
The lazy messaging pattern has existed for some time in sn_routing, we’ve now updated sn_client’s infrastructure communications to the pattern. Which means we’ll now get updates to section info as soon as we attempt to message an incorrect, i.e. old, section.
Lazy messaging works by slightly increasing the size of messages sent between nodes so they include some extra information on the Network’s current state as seen from different observers in space and time. If a node receives a message and it realises that the network state details in the incoming message differs to what it believed was the state, then it communicates further with the sending node to bring itself, or the sender, up to date with the correct network state, then the original message can be processed accordingly.
Lazy messaging is currently being implemented at the sn_node level, and will feature in a future testnet.
One single (CRDT) data type with immutable Policy
Note - Not guaranteed for the upcoming testnet, a nice to have which is very close to completion
Up until now we’ve been exposing two different Safe native data types, a Sequence (already being CRDT) and a Map (which we haven’t yet converted into CRDT). They both used to allow mutations to the Policy set for them, i.e. the permissions to access and mutate them.
After more research and tests made on the Sequence data type, we came to the conclusion that being able to handle all different types of scenarios of concurrent Policy mutations, combined with concurrent mutation of the data itself, was making the logic and code very complex. We also realised we were trying to go against the CRDT nature itself. This is why we moved away from that path and we now have a Register CRDT data type which doesn’t allow mutations on the Policy - once the content is stored on the network the Policy is enforced on each message and while the policy cannot be mutated.
This all means that all our client side abstractions, e.g. FilesContainers and NRS containers, are now based on this single CRDT data type.
While its not ready yet, it seems like it’ll make a big impression when it does arrive 
So data to follow is perhaps the best thing to say right now. We are hoping to get this cracked in days though. (also keep the API we have as much as possible)
2 weeks out is Thursday April 1st so let’s write that off… three weeks out is too far away for setting the stage.
