Fleming Testnet v4 is LIVE!
Fleming testnet v4 is up and this one is for @oetyng who has worked tirelessly for many, many months now to design and implement Rewards for the Safe Network. Rewards is such a huge feature that has been in the pipeline for so long, and Edward only went and made it a reality 
With Rewards now in place, Edward is continuing his long road to a Digital Bearer Certificate (DBC) with some other team members - a significant feat, should we make that happen! So, Edward, not only have you climbed a mountain and shown us all how to do it, youāre now looking at even more ingenious ways to make Safe ā¦ well Safe(r), faster and even more powerful! The rest of the team, and Iām sure the community, thank you and appreciate your dedication and support. 
v4 Changelog
With the communityās help, we identified several issues with previous testnet iterations, and have been working hard to resolve as many as possible.
The fixes and improvements for v4 are:
1. Fix a data loss bug.
As the previous testnet grew larger we started picking up reports of data that we knew had existed no longer being found. We made finding and resolving this data loss issue our main priority for testnet v4.
We found a sneaky bug where newly promoted Elders were not updating themselves correctly with the data and metadata of their section. This meant that on every occurence of churn or splits, more and more Elders were online which did not contain all the data. This has been fixed in this PR, with new Elders now being brought fully up to speed to help support the network.
2. Measures to ensure nodes that are unresponsive are removed from the network.
Observations from the previous testnet highlighted some specific circumstances where unresponsive nodes were causing problems in the network. This PR was merged to catch Adult nodes who were chosen as Elder candidates, and then causing problems by not responding to DKG vote requests - this supplements the fix in the previous testnet which caught and evicted these nodes once they had been fully promoted to be Elders.
Two further PRs here and here were also merged to track an Adults responsiveness to requests for data, and if unresponsive, to vote them off the network.
3. Multiple CLI updates.
Some improvements and fixes in the CLI over the last few days, most notable is the widely requested addition of a $ safe keys show command to display any SafeKey, Public Key, XOR-URL and Secret Keys owned by your CLI - this was added here.
Weāve also removed the need for you to authenticate with authd while using the CLI. We found that this was causing some issues with Windows users in particular, and at this stage of the testnets it isnāt actually necessary, so it can now be bypassed when creating keys as follows:
$ safe keys create --test-coins --for-cli
After running this cmd you can now proceed straight to uploading files or transferring testcoins with the CLI. This also means that you no longer need to install authd - weāve updated the instructions in this testnet release post to reflect this.
Also now merged is this fix which re-enables the --dry-run feature for CLI commands. Weāve also improved the UX in a couple of areas, firstly here we have set the --to value of $ safe keys transfer command to default to assume it is a safe URL if it cannot be decoded as a PK, even if it does not begin with safe://. Secondly, we made a change here to remove the $ safe keypair command, along with the --pk arg from the $ safe keys create command - these no longer made sense with the current setup and were only causing confusion. Coin transfers to BLS public keys are also now supported by the CLI, introduced by this PR.
4. New Register data type.
Weāve now completed and merged the Register data type work in the sn_data_type and sn_node crates, plus updated various other crates to support this new data type.
There is still some work to be done before we deprecate the Sequence data type though, with associated APIs for Register still to be added in sn_client and sn_api, so youāll have to wait a little longer before actually getting to use this new data type, but itās reassuring that the the code is now there for the network to handle it.
As a reminder, using the Register data type along with a new traversal API for the MergleReg CRDT gives us the ability to walk back through the branching history of a register, as well as being able to query for any newer data thatās been written to the register.
5. Reduce the number of network messages sent per client query.
We havenāt spent much time looking for ways to optimise the testnets yet, thatās something weāll investigate once we are happy that everything is rock solid, but we did notice and implement a quick and simple update here which reduces the number of messages sent per client query. We donāt anticipate this making a huge difference, and have not benchmarked it ourselves, but itās only right to keep you informed of these changes as we go.
Further fixes and improvements for v4.1, added April 23 are:
1. Recognise when an Adult has been promoted to be Elder.
After the release of v4 we noticed that when an Adult was being promoted to become an Elder, the network was not noting that an Adult had therefore been ālostā and so not accepting a new node to replace that Adult. This was leading to sections with few, or even zero adults, and so causing several issues. This has been fixed here and here.
2. Ensure storecost is always > 0.
Storecost should be guaranteed to always be > 0. This was already the case during normal conditions, but v4 testing had led to some unusual conditions where we had seen storecost=0. Weāve added a catch-all fix for this here. Additionally we now log the input values to better track what those really were.
3. Improved logging.
We also made a change to node logging here to log more information around a nodeās current role, section prefix, age, and name, with every new routing event. This change should make it easier for debugging, and also just to provide more information for those interested in how their node is progressing.
What happens to previous testnet iterations?
All previous Fleming testnet iterations have now been taken offline, with all data removed.
This is to allow us to concentrate on any issues that arise from the new testnet iteration, and avoid any community confusion over which testnet they are connected to, or any errors as a result of contamination from previous configurations and data. We ask that everyone who attempts to use this latest testnet iteration removes their $HOME/.safe folder before trying to interact with this testnet, i.e. $ rm -rf $HOME/.safe/ to ensure there is no contamination.
All of the bootstrap nodes for previous iterations that we had on Digital Ocean have been destroyed, with new, clean D.O. nodes created for this testnet iteration - please follow the full list of instructions below to remove all previous testnet settings and data from your machine and to get you started with the latest testnet iteration.
Are We Working on Other Fixes?
Yes. Multiple other fixes and improvements are already in progress, some even from before we released testnet v1. One of the main priorities we have at the moment is completing the implementation of Anti Entropy across the board - we believe that this will resolve many of the errors that people have been seeing in their logs, and make the testnets even more reliable and scalable. We hoped to have Anti Entropy in place this week, but with all of the above fixes in place we decided this was ample for a new testnet iteration. This, along with other improvements and fixes, are in progress for future testnet iterations.
We also anticipate that the feedback from this testnet iteration will direct us where else to aim resources as a priority.
Where can I report any issues found?
If you come across any issues in your testing, start by checking the Known Issues section of this post (see below) to see if it has already been acknowledged . If it has not yet been added, you can post you issue in the comments of this post. You can also report issues in the Online Spreadsheet for Testnet Results and Issues - see the section below.
We will monitor and investigate reported issues as soon as we can.
Online Spreadsheet for Testnet Results and Issues
If you will be uploading data to the testnet and/or providing safes (nodes), please consider posting your data and results at:
SN Testnet Review
(Massive thank you to @VaCrunch for creating this spreadsheet 
)
This is a new version of this spreadsheet, with the versions used for previous iterations now locked.
For those posting:
ā¢ No need to Sign In. The white cells are for your data entry.
ā¢ One row for each of your devices.
ā¢ Scroll down until you get to the first empty row.
ā¢ You do not need to use your Forum name for your ID, but please use the same ID for all your devices.
ā¢ Supporting/analysis tabs are at bottom of screen: Error_Msgs, Summary, Thumbnails, Charts, Matrix, Top 10, Map, Resources, Lists, and Comments.
ā¢ Use the View/Zoom feature at top to reduce the visible size of the spreadsheet to match your screen. 75% will be the best fit for most people.
ā¢ If you choose not to record the name of your country please select āUnspecifiedā at the bottom of the list.
ā¢ If you change the number of nodes (safes) you are running, just edit the existing figure, do not add another line for the same device.
Known Issues
Any reproducible issues that we, or the community, come across will be added to this section:
-
We are seeing reports of performance issues, particularly around
$ safe files put ....operations with larger files/folders. There has been no optimisation of write times, nor any general analysis of speed so far, with all efforts concentrated on getting features in place and fully functional before we look at it from a performance perspective. -
We have removed
authdfrom the stack for normal users right now, instead focussing onSafe CLI. To that end, we have updated the CLI to be able to do all you previously required authd for. This means no GUI right now, mostly. The focus is on network stability and then speed. As we optimise for speed, the team will also focus on authd again, perhaps removing it (our goal) in favour of a simpler approach. So test as much as possible with the CLI for now. It reduces user error, reduces the potential bug surface, and allows total focus on the prize - a fully autonomous decentralised network.
Letās try it out!
Please read and follow the instructions below carefully.
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Installing/Updating to Latest
To avoid any pollution from previous testnet iterations settings and data, we ask that everyone removes their $HOME/.safe/ directory, then installs the CLI, authd and node again.
$ rm -rf $HOME/.safe
Note that Windows users may be required to install specific software before being able to install the CLI using the command below - see full instructions for Windows here.
Now download and install the verified latest CLI binary via our install script with the following command:
$ curl -so- https://sn-api.s3.amazonaws.com/install.sh | bash
This script downloads the correct CLI binary for your OS (Windows, macOS or Linux), installs it in the correct directory, creating that directory if it doesnāt exist, and adds it to your system PATH.
You may need to restart your terminal window at this point for any changes to your system PATH to take effect. You can now confirm whether the CLI is installed and set up correctly:
$ safe -V
sn_cli 0.24.0
Next, you should install the latest Node with the command below.
$ safe node install
Note - no longer a requirement to install auth, so steps removed for this.
You should now be equipped with the latest CLI and node.
Note that you can find full installation instructions in our user guide:
Joining the Testnet
As with previous public testnets, we are hosting some elders and adults on Digital Ocean to kick off the Network. These act as hardcoded contacts which bootstrap you to the network, therefore you will need a network configuration file to inform your CLI which network to bootstrap to. We store and update these connection details on S3 for you to easily point your configuration to.
If you have followed the instruction above to remove your .safe folder before installing everything again, you should have no existing network configurations - you can confirm this with the $ safe networks command.
You can add a profile for the latest testnet which points to our S3 location, you can add using $ safe networks add:
$ safe networks add fleming-testnet https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config
Network 'fleming-testnet' was added to the list. Connection information is located at 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config'
Now you need to ensure you are set to use this fleming-testnet configuration that we have added, we can use $ safe networks switch fleming-testnet for this:
$ safe networks switch fleming-testnet
Switching to 'fleming-testnet' network...
Fetching 'fleming-testnet' network connection information from 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config' ...
Successfully switched to 'fleming-testnet' network in your system!
If you need write access to the 'fleming-testnet' network, you'll need to restart authd (safe auth restart), unlock a Safe and re-authorise the CLI again
Now to get write access to the network, you can use the following command via the CLI before proceeding to uploading files or transfer testcoins with CLI:
$ safe keys create --test-coins --for-cli
We now have our CLI and Node components up-to-date, and we have the latest hardcoded contact details to bootstrap to the public testnet, so everything is in place and weāre ready to launch our node and add it to the Network. This is achieved using $ safe node join as follows:
$ safe node join
Creating '/Users/maidsafe/.safe/node/local-node' folder
Storing nodes' generated data at /Users/maidsafe/.safe/node/local-node
Starting a node to join a Safe network...
Launching with node executable from: /Users/maidsafe/.safe/node/sn_node
Node started with hardcoded contact: 161.35.36.185:12000
Launching node...
Node logs are being stored at: /Users/maidsafe/.safe/node/local-node/sn_node.log
Your node will now launch and attempt to connect to the public network. Keep in mind that it can only join the testnet if the Network is accepting new nodes at that time (see NOTE below). You can keep an eye on its progress via its logs, which can be found at $HOME/.safe/node/local-node/sn_node.log.
If there is no space on the testnet for your node to join, as of Fleming testnet v2, your node will automatically try to rejoin until it is successful, or until you kill the sn_node process.
NOTE - please keep in mind that we have an anti Sybil attack feature in place now to only accept new nodes onto the testnet when resources are required, therefore you may be attempting to join the Network but your logs tell you that the Network is not accepting new nodes at this time:
The network is not accepting nodes right now. Retrying after 3 minutes
This is expected behaviour which will happen each time you try to connect until the testnet detects that resources are running low. As of Fleming testnet v2 your node will automatically try to rejoin in a loop, you do not need to attempt to join again manually, or via a script. We recommend keeping a watch on your node log file to check if you have been accepted - $ tail -f $HOME/.safe/node/local-node/sn_node.log
You can help to speed up the process of the network needing new resources by adding some data yourself - you donāt need to run a node to upload data! See the Do I need to run a node to participate? section below.
Before working your way through the CLI commands to perform various actions on the network,
Following the steps above gives your Safe some test Safe Network Tokens to use. This means that there is no need to farm first to earn rewards before being able to try operations such as uploading to the testnet. You can now work your way through the various CLI commands to perform operations on the network.
You can even connect to the testnet in read only mode using CLI, i.e. once youāve installed CLI you can try fetching content uploaded by other users. For example, you can download the following image and open it locally afterwards:
$ safe cat safe://hygoyeye9mq5jmipo3we79wohue8hjyw55e6f8xyk1hquwy9hsxnk9tdsme > ~/safe-the-planet.png
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Do I need to run a node to participate?
No, you can join with just the CLI and authd to experiment with data, tokens, etc. You can follow the instructions in the Joining the Testnet section above, but you donāt need to run $ safe node join - at this point you can just go straight to using the CLI as per the User Guide.
Further Information
Where are my node logs?
When you launch your node you should see the location of your log file printed on screen - this will be $HOME/.safe/node/local-node/sn_node.log. You can tail your logs with a command such as $ tail -f $HOME/.safe/node/local-node/sn_node.log
Where are my rewards?
If youāve started a node you will have keys generated for you, and stored in two files next to the logs of your node. You can check the balance of the rewards public key with the following CLI commands:
$ safe keys balance --sk $(cat $HOME/.safe/node/local-node/reward_secret_key)
The keys can then be used as per the instructions of the CLI User Guide for transfers, payments and more.
____
/ \ \
/ /\ \ \
/ / /\ \ \
/ / /__\_\ \
/ / /________\
\/___________/
and 
and will see whatās afoot therafter).
. Thanks Edward.
ā¦ this may take several minutes
