The Fleming testnet has dropped!
It feels very much like a decade+ of research and development, trial and error, blood, sweat and tears has borne some fruit today. We know you’ve been here for the ride and stuck with us, so enjoy trying it out and of course spread the word of what we’re achieving together here!
You can consider this Fleming testnet as the dry run, or the pre-flight checks for the huge milestone that is the Fleming release itself. This testnet includes all the major components that will form the core of the Safe Network.
We know we’re not finished yet. There are several finishing touches, tweaks and optimisations that we have planned before we can say that the Fleming release goals have been fully met, plus we know that getting a testnet out into the hands of the public will always throw up issues that we hadn’t anticipated, or haven’t been able to cover with our in-house testing. Hence the term testnet! So we anticipate several Fleming testnet iterations before we are satisfied enough to announce the official Fleming release.
What’s left after Fleming? Next up is the Maxwell Network, which will add in the ability for upgrades on top of Fleming, as well as some additional ways to prevent spam, and optimise performance.
You can check our roadmap for more details on both Fleming and Maxwell releases.
So what’s new/improved in this testnet?
We recently released a post here describing the new and improved features we were anticipating to be included in the testnet, and this has proved pretty accurate, with predictions that we wouldn’t need to fully implement lazy messaging across the board being the case, plus we held back the Register CRDT so as not to complicate matters and with a little work still to do there anyway.
As a recap, probably the most notable new, improved or refactored inclusions in this testnet are section splits, rewards, AT2 payments, accepting new nodes only when resources are required, and the section chain being fork resistant. We’ve also partially included lazy messaging, which note that we’ve renamed to anti entropy.
Some further details on all of these below.
Section Splits
Section splits mean that the Network can grow naturally.
As the section size grows and there are enough nodes to separate into two new sections, a split happens. This is quite tricky as the section chain splits into two, while the data held also splits with approximately 50% moving to the new sections.
As we balance nodes’ addresses — note that we still have more work to do here in a future testnet iteration — then the data should already be on the correct nodes in the vast majority of cases. However, some promotion will happen. This means the requirement of switching real chunk copies among nodes is reduced to a minimum. Mostly, it will only be the indexing map of the chunks, held among the elder nodes, which needs to be split and transferred among the now two elder groups.
To ensure the two split-out sections both contain enough capable nodes to handle requests and provide storage space, we enforce that both of the after-split sections satisfy the RECOMMENDED_SECTION_SIZE, and only then shall the split happen.
Not all data is split at the moment, we have not yet implemented the splitting of Map, Sequence and chunk metadata. These will follow in future testnets.
Wallets should be split, however we are still to see how this fares with higher traffic.
Rewards/Farming
Individuals who choose to supply the resources that the Network requires will now be rewarded with Test Safe Network Tokens in the testnet. Here are some information about how it will work in the first testnet iteration.
-
Distribution
At every section split, the sum of payments since last split is distributed over all registered node wallets of the section, proportionally to their age. Rewards are paid to nodes of age 6 or more, which means nodes have to relocate at least once before being rewarded. -
Minting
The minting of new tokens will be at most equal the amount of the payments.
The reward amount is thus the sum of payments + newly minted tokens.
If however (unlikely in this testnet) the rewards plus all managed in user wallets at the section would exceed the max supply of the section, the minting would decrease accordingly. If payments alone would bring the sum above the max supply, it as well would be decreased, even down to zero (when section managed >= max supply).
Decreasing the amount of rewards below what has been paid, is the same as burning tokens.
This way, an approximate max supply will be withheld over time across the network. -
Max supply
Max supply in a section, is inversely proportional to network size and is derived thus; global max supply of~4.3bndivided by number of sections (known by2^[section prefix length]).
As this is a testnet, please keep in mind that these are only test tokens, they will not be able to be traded for real Safe Network Tokens on launch of the real network.
Token Payments: The AT2 way
We’ve had Safe Money and its transfers, or payments if you will, integrated into our testnets for a while. We recently revamped the whole implementation with the AT2 protocol, which gives cryptographically verified transfers at theoretically higher speeds than blockchain-based technologies.
The introduction of shared signing of Elders makes these payments historically verifiable, and together with clients keeping their replicas, this would also allow for re-uploading and verifying a wallet even without it existing in the network.
This lives entirely apart from the consensus mechanism of the Network and is done one-on-one with the users, hence reducing the stress on the network.
It’s worth noting, that the client driven process can be frozen if there’s an attempt to make an out of order payment. The balance will currently thereafter be unusable. We’ll be addressing this in future testnet iterations.
Accepting New Nodes Only When Resources Are Required
A recent improvement to the Network, to be showcased for the first time in the upcoming public testnet, is that existing nodes now decide when they want new nodes to join the network.
This improvement is designed to help prevent Sybil attacks by virtue of the fact that we do not allow unlimited nodes to join the network at will; the danger of an unchecked number of nodes being that potential attackers could easily flood the Network, taking it over with hostile nodes.
Routing assumes the elder-nodes will track all the adult-nodes in the section, and when they detect the average storage capacity (or some other resource) becomes too low, they will flip a flag so the section starts accepting new nodes. The section elder-nodes should detect this at more or less the same time, so consensus can be reached.
Fork Resistant Section Chain
In order to prove that a piece of data was signed by a group of nodes that were once valid members of a section, even after those nodes are long gone, we have a “section chain” which is a list of section keys linked together with signatures. Previously, this chain required that the section agreed on which key to append to it next. If there was a disagreement on that, the chain could fork into two (or more) mutually incompatible chains which could break the section. This could happen, for example, at times of intense churn.
This problem has now been resolved by accepting section chain forks internally, but always presenting a unique and deterministic order of the chain blocks to the outside.
A new implementation of the section chain data-structure, which is now a proper CRDT, guarantees (eventual) consistency regardless of in what order the operations are applied, how they are grouped, or even duplicated. Even if multiple distinct keys are inserted into it, everybody will eventually agree on which one is the most recent one and thus who the current elders are (because each section key is uniquely associated with a single group of elders). And all of this is achieved without involving any complicated consensus mechanism.
Anti Entropy (formerly known as Lazy Messaging)
First of all the change of term here - we believe anti entropy is a more accurate term to use for what we are trying to achieve. You can think of anti entropy as a mechanism to get in sync, or in other words to reduce entropy in a system. We’ve noticed that it is also the term used in algorithms such as CRDT, BRB and AT2. Therefore, we will no longer use the term lazy messaging.
A major hurdle in any decentralised network which will experience churn, such as ours, is keeping the nodes on the network up to date with the network’s current state. For example, who has joined and who has left? Or who are the current Elders in any given section? We have been working to implement anti entropy to resolve this issue.
The anti entropy pattern has existed for some time in sn_routing, and we’ve now updated sn_client’s infrastructure communications to the pattern. Which means we’ll now get updates to section info as soon as we attempt to message an incorrect, i.e. old, section.
Anti entropy works by slightly increasing the size of messages sent between nodes so they include some extra information on the Network’s current state as seen from different observers in space and time. If a node receives a message and it realises that the network state details in the incoming message differs to what it believed was the state, then it communicates further with the sending node to bring itself, or the sender, up to date with the correct network state, then the original message can be processed accordingly.
Anti entropy is still in the process of being implemented at the sn_node level, but we seem to be at the point where we are working well enough with a partial implementation and so you can expect once this is fully implemented there will be another testnet iteration released.
No More IGD connection problems!
Who remembers not being able to participate in previous testnets over the years due to an IGD error:
Automatic Port forwarding Failed. Check if UPnP is enabled in your router's settings and try again. Note that not all routers are supported in this testnet.
With ZERO reports of IGD errors after 15 hours of the testnet being online, we can now safely say this issue is resolved! 
We’ve changed a few strategies in our qp2p crate since previous testnets. We now lean on the bootstrap nodes and echo service, rather than trying to contact the IGD gateway, and we haven’t seen an IGD issue in-house since. But with such a small in-house sample size we wouldn’t know for sure whether this had been completely resolved until we tried it in a public testnet - well the results are in and the evidence confirms that all IGD issues have been completely eliminated!
What doesn’t this testnet do?
-
As mentioned above, anti entropy has not been implemented across the board just yet, work to do so is ongoing and hopefully just a matter of days away from completion. Once complete and added, the testnet will be far more robust - until that point we are anticipating errors occurring in certain scenarios.
-
Byzantine Reliable Broadcast (BRB) membership stays in the background for now while we concentrate on assessing the various other new/enhanced features outlined above.
-
Not all data types are CRDT compliant just yet. Work to rectify this on a couple of underlying data types is ongoing and shouldn’t take too long to complete, test, and iterate the testnet to make it more reliable.
-
Currently we do not fully test remote section keys. This was disabled last minute as we had a hidden bug. It will be re-enabled imminently and ready for the next iteration.
-
There are still a few remaining known failure paths that exist in this testnet, i.e. chains of events which we know could bring the testnet down. These failure paths probably exist in every single decentralized network - but we’ve designed with these in mind so we can shut them off, and so have agreed the actions required to handle them for an upcoming iteration of this testnet. We believe these are not too likely to be seen in normal testnet usage.
-
Work also continues on client APIs. These are a work in progress at the moment, improvements are being made as requirements become clearer.
-
We’re working on several other changes to improve performance, reliability and stability, to the point that the testnet could cope with going offline, then being resurrected.
-
There is no compatible Browser release for this testnet, for now it is CLI only.
Keep an eye out for testnet iterations over the coming weeks as we fix bugs and make improvements until we are satisfied that all Fleming release goals have been achieved.
How to try it out!
Enough talking, let’s get you started…
Installing/Updating to Latest
Due to some recent changes potentially meaning that using the update function from older versions of the CLI, authd and node may not actually recognise the latest versions, particularly for Linux users, we recommend that everyone installs the CLI, authd and node afresh. Full instructions below.
Installing Latest
First off, lets clear out some old files you may have from previous testnets, local and public, to make sure they don’t interfere or cause confusion:
$ rm -rf $HOME/.safe
Now download and install the verified latest CLI binary via our install script with the following command:
$ curl -so- https://sn-api.s3.amazonaws.com/install.sh | bash
This script downloads the correct CLI binary for your OS (Windows, macOS or Linux), installs it in the correct directory, creating that directory if it doesn’t exist, and adds it to your system PATH.
You may need to restart your terminal window at this point for any changes to your system PATH to take effect. You can now confirm whether the CLI is installed and set up correctly:
$ safe -V
sn_cli 0.23.0
Next, you should install the Authenticator daemon and Node with the two commands below:
$ safe auth install
$ safe node install
You should now be equipped with the latest CLI, authd and node.
Note that you can find full installation instructions in our user guide:
Joining the Testnet
As with previous public testnets, we are hosting some elders and adults on Digital Ocean to kick off the Network. These act as hardcoded contacts which bootstrap you to the network, therefore you will need a network configuration file to inform your CLI which network to bootstrap to. We store and update these connection details on S3 for you to easily point your configuration to.
If you have connected to a previous iteration of our testnets then you may have a shared-section, or other, network profile saved on your machine, which should no longer be relevant or required. You can check what you have with $ safe networks which will list your saved network profiles. Note that old profiles are best removed now to avoid any confusion - they may be stored in now redundant config directories, and may use old names such as safe-cli and safe-vault - you can remove these with the $ safe networks remove <name of network profile> command, for example $ safe networks remove shared-section.
Now you have cleaned up any old network profiles, you can add a profile for the latest testnet which points to our S3 location, you can add using $ safe networks add:
$ safe networks add fleming-testnet https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config
Network 'fleming-testnet' was added to the list. Connection information is located at 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config'
Now you need to ensure you are set to use this fleming-testnet configuration that we have added, we can use $ safe networks switch fleming-testnet for this:
$ safe networks switch fleming-testnet
Switching to 'fleming-testnet' network...
Fetching 'fleming-testnet' network connection information from 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config' ...
Successfully switched to 'fleming-testnet' network in your system!
If you need write access to the 'fleming-testnet' network, you'll need to restart authd, login and re-authorise the CLI again
We now have our CLI, Authenticator daemon and Node components up-to-date, and we have the latest hardcoded contact details to bootstrap to the public testnet, so everything is in place and we’re ready to launch our node and add it to the Network. This is achieved using $ safe node join as follows:
$ safe node join
Creating '/Users/maidsafe/.safe/node/local-node' folder
Storing nodes' generated data at /Users/maidsafe/.safe/node/local-node
Starting a node to join a Safe network...
Launching with node executable from: /Users/maidsafe/.safe/node/sn_node
Node started with hardcoded contact: 161.35.36.185:12000
Launching node...
Node logs are being stored at: /Users/maidsafe/.safe/node/local-node/sn_node.log
Your node will now launch and attempt to connect to the public network. Keep in mind that it can only join the testnet if the Network is accepting new nodes at that time (see NOTE below). You can keep an eye on its progress via its logs, which can be found at $HOME/.safe/node/local-node/sn_node.log.
NOTE - please keep in mind that we have an anti Sybil attack feature in place now to only accept new nodes onto the testnet when resources are required, therefore you may be attempting to join the Network but your logs tell you that the Network is not accepting new nodes at this time:
Cannot start node due to error: Routing(TryJoinLater)
This is expected behaviour which will happen each time you try to connect until the testnet detects that resources are running low. The only solution here is to try again later.
See the “Accepting New Nodes Only When Resources Are Required” section above for more information on why we introduced this feature.
You can help to speed up the process by adding some data yourself - you don’t need to run a node to upload data! See the Do I need to run a node to participate? section below.
Before working your way through the CLI commands to perform various actions on the network, remember to authenticate, create and unlock your Safe. Following the steps in these links gives your Safe some test Safe Network Tokens to use. This means that there is no need to farm first to earn these rewards before being able to try operations such as uploading to the testnet.
You can even connect to the testnet in read only mode using CLI, i.e. once you 've installed CLI you can try fetching content uploaded by other users. For example, you can download the following image and open it locally afterwards:
$ safe cat safe://hygoyeye9mq5jmipo3we79wohue8hjyw55e6f8xyk1hquwy9hsxnk9tdsme > ~/safe-the-planet.png
Do I need to run a node to participate?
No, you can join with just the CLI and authd to experiment with data, tokens, etc. You can follow the instructions in the Joining the Testnet section above, but you don’t need to run $ safe node join - at this point you can just go straight to using the CLI as per the User Guide.
Further Information
Where are my node logs?
When you launch your node you should see the location of your log file printed on screen - this will be $HOME/.safe/node/local-node/sn_node.log. You can tail your logs with a command such as $ tail -f $HOME/.safe/node/local-node/sn_node.log
Where are my rewards?
If you’ve started a node you will have keys generated for you, stored in two files next to the logs of your node (see above). In case you want to access any rewards earned, it’s important that you copy these files to another location before starting a new node.
The keys can then be used as per the instructions of the CLI manual, for transfers and payments. We will provide a UX and app for easier use of these soon.
Where can I report any issues found?
As the test in testnet indicates, we fully expect that issues will be found given the volume and variety thrown at it in real world situations, something that is notoriously difficult to replicate in-house.
We hope that this testnet will be stable enough to stay online for ample time to allow as many people as possible to try it out, and for us to gather valuable information from it to analyse and troubleshoot anything found. From past experience, we know it is possible that it will break at some point, particularly as this is the first public iteration of this testnet and so much of this technology is new and untested in the real world.
If the testnet does go offline, we will fix, tweak, change and improve before putting it back online - we will indicate in the title of this post if it is known to be offline.
If you come across any issues in your testing, start by checking the Known Issues section of this post (see below) to see if it has already been acknowledged . If it has not yet been added, we would like you to report it in the Online Spreadsheet for Testnet Results and Issues - see the section below.
We will monitor and investigate reported issues as soon as we can.
Online Spreadsheet for Testnet Results and Issues
If you will be uploading data to the testnet and/or providing safes (nodes), please consider posting your data and results at:
SN Testnet Review
(Massive thank you to @VaCrunch for creating this spreadsheet 
)
Even if you don’t post your data, you can record any error messages and related information in the Error_Msgs sheet to help out the development team.
Ideally we can keep as much data, results and errors in this one location to allow us to more easily analyse it.
For those posting:
• No need to Sign In. The white cells are for your data entry.
• One row for each of your devices.
• Scroll down until you get to the first empty row.
• You do not need to use your Forum name for your ID, but please use the same ID for all your devices.
• Supporting/analysis tabs are at bottom of screen: Error_Msgs, Summary, Thumbnails, Charts, Matrix, Top 10, Map, Resources, Lists, and Comments.
• Use the View/Zoom feature at top to reduce the visible size of the spreadsheet to match your screen. 75% will be the best fit for most people.
• If you choose not to record the name of your country please select “Unspecified” at the bottom of the list.
• If you change the number of nodes (safes) you are running, just edit the existing figure, do not add another line for the same device.
Known Issues
Any reproducible issues that we, or the community, come across will be added to this section:
- We are seeing reports of performance issues, particularly around
$ safe files put ....operations with larger files/folders taking a long time. There has been no optimisation of write times this far so we are aware that speed will be slow. We will be working on improvements.
