Fleming Testnet v3 is up!
Fleming testnet v3 goes out to @yogesh and @lionel.faber who, at a second’s noticed, came into work on the fixes and improvements for this testnet iteration, immediately cancelling their public holiday day off. This level of commitment to other team members is the way we always want to work - have each other’s back at all times, of course give space for down time, and make sure we get these testnet iterations up and running with minimal stress. It’s the new and efficient way of working for us. The rest of the team, and I’m sure the community, thank them both and appreciate their support. 
v3 Changelog
With the community’s help, we quickly identified issues with Fleming testnet v2, released on 13th April, and have worked with lightning speed to resolve.
The fixes and improvements for v3 are:
1. Fix a data replication bug causing MissingSecretKeyShare log error
We realised on Tuesday evening that there was a data replication bug which was causing the MissingSecretKeyShare log errors that quite a few of you had observed in your node logs.
This was triggered when an Adult was commanded to replicate a chunk held by other Adults. It was using an internal cmd which assumed an aggregation scheme that requires a secret key share - however, only Elders have this secret key share. The adults were therefore unable to replicate the chunks, and reporting MissingSecretKeyShare to the node logs.
This also unearthed an issue with that flow, since Adults are not guaranteed to be able to communicate with each other.
We have now changed the behaviour so that Elders instead fetch the chunk to be replicated and distribute them out to the Adults.
2. Fixed dead Elders
In Fleming testnet v1 we observed a section become stuck in a DKG loop. We had thought that we had fixed the issue in v2, but again we were seeing some Adults which were not connectable, still making it through to become Elders - we identified a loophole in the test we were using to verify connectivity.
We have implemented two separate changes in sn_routing here and here which first of all kick any Elder node off the testnet if it cannot be connected to, and secondly closes off the loophole in the connectivity test we run by checking connectivity from a new endpoint, rather than a previously established endpoint.
Note that there could still be some rare circumstances where Adults which are not connectable to from other nodes make it through to be Elder, hence why we have implemented both these changes.
3. Fix stack overflow issue on Windows
Huge thanks to @Vort who identified and provided us with a fix which we applied here and here for a thread 'main' has overflowed its stack error he came across in Windows when he tried a put command via the CLI. Much appreciated, @Vort!
4. Do not overwrite existing reward key
We also made a change in sn_node to no longer overwrite your reward key when starting a new node.
We used to be reading from the key files, but at some point it was changed to instead look for it in a node.config that node tries to load from ~/.safe/node at startup. When none was there it generated a new keypair, overwriting any files already containing keys.
We’ve now changed this to ignore the config - until we have a better approach - and load from an existing reward key pk file or else create new. Next step is to allow for the CLI to read these keys, and then we can be looking at a smarter overall handling of these as well.
5. Modification to node joining logging
In v2 we updated the node joining process to automatically retry every 3 minutes. However, we had a couple of reports of this loop process exiting with no errors in the node log. We believe that the loop exits reported would have been caused by a separate error being reported when the node was attempting to join, and that error should have been reported in the node logs for the user to see.
We’ve made an update to sn_node which we believe should resolve this issue now. If the join loop stops then you should see a log entry with the specific reason why.
What happens to previous testnet iterations?
All previous Fleming testnet iterations have now been taken offline, with all data removed.
This is to allow us to concentrate on any issues that arise from the new testnet iteration, and avoid any community confusion over which testnet they are connected to, or any errors as a result of contamination from previous configurations and data. We will be asking that everyone who attempts to use this latest testnet iteration removes their $HOME/.safe folder before trying to interact with this testnet, i.e. $ rm -rf $HOME/.safe/ to ensure there is no contamination.
All of the bootstrap nodes for previous iterations that we had on Digital Ocean have been destroyed, with new, clean D.O. nodes created for this testnet iteration - please follow the full list of instructions below to remove all previous testnet settings and data from your machine and to get you started with the latest testnet iteration.
Are We Working on Other Fixes?
Yes. Multiple other fixes and improvements are already in progress, some even from before we released testnet v1. One of the main priorities we have at the moment is completing the implementation of Anti Entropy across the board - we believe that this will resolve many of the errors that people have been seeing in their logs, and make the testnets even more reliable and scalable. This, along with others, are in progress for future testnet iterations.
We also anticipate that the feedback from this testnet iteration will direct us where else to aim resources as a priority.
Where can I report any issues found?
If you come across any issues in your testing, start by checking the Known Issues section of this post (see below) to see if it has already been acknowledged . If it has not yet been added, we would like you to report it in the Online Spreadsheet for Testnet Results and Issues v3 - see the section below.
We will monitor and investigate reported issues as soon as we can.
Online Spreadsheet for Testnet Results and Issues v3
If you will be uploading data to the testnet and/or providing safes (nodes), please consider posting your data and results at:
SN Testnet Review v3
(Massive thank you to @VaCrunch for creating this spreadsheet 
)
This is a 3rd version of this spreadsheet, with the version used for previous iterations now locked.
Even if you don’t post your data and results, you can record any error messages and related information in the Error_Msgs sheet to help the development team keep these organised.
Ideally we can keep as much data, results and errors in this one location to allow us to more easily analyse it.
For those posting:
• No need to Sign In. The white cells are for your data entry.
• One row for each of your devices.
• Scroll down until you get to the first empty row.
• You do not need to use your Forum name for your ID, but please use the same ID for all your devices.
• Supporting/analysis tabs are at bottom of screen: Error_Msgs, Summary, Thumbnails, Charts, Matrix, Top 10, Map, Resources, Lists, and Comments.
• Use the View/Zoom feature at top to reduce the visible size of the spreadsheet to match your screen. 75% will be the best fit for most people.
• If you choose not to record the name of your country please select “Unspecified” at the bottom of the list.
• If you change the number of nodes (safes) you are running, just edit the existing figure, do not add another line for the same device.
Known Issues
Any reproducible issues that we, or the community, come across will be added to this section:
-
We are seeing reports of performance issues, particularly around
$ safe files put ....operations with larger files/folders. There has been no optimisation of write times, nor any general analysis of speed so far, with all efforts concentrated on getting features in place and fully functional before we look at it from a performance perspective. -
Windows users will see their terminal window stick when they run
$ safe auth start- it is important that they leave this terminal window in this state and do not close it down. The sn_authd process does start running in the background but will be closed down if you exit that terminal window. You can continue in a new terminal window/tab.
Let’s try it out!
Please read and follow the instructions below carefully.
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Installing/Updating to Latest
To avoid any pollution from previous testnet iterations settings and data, we ask that everyone removes their $HOME/.safe/ directory, then installs the CLI, authd and node again.
$ rm -rf $HOME/.safe
Note that Windows users may be required to install specific software before being able to install the CLI using the command below - see full instructions for Windows here.
Now download and install the verified latest CLI binary (v0.23.1) via our install script with the following command:
$ curl -so- https://sn-api.s3.amazonaws.com/install.sh | bash
This script downloads the correct CLI binary for your OS (Windows, macOS or Linux), installs it in the correct directory, creating that directory if it doesn’t exist, and adds it to your system PATH.
You may need to restart your terminal window at this point for any changes to your system PATH to take effect. You can now confirm whether the CLI is installed and set up correctly:
$ safe -V
sn_cli 0.23.2
Next, you should install the Authenticator daemon and Node with the two commands below.
Note for Windows users - there is a known issue where the $ safe auth install command seems to hang after entering it - please leave your terminal with this running and continue in a different terminal window or tab. Killing this terminal window will kill the sn_authd process:
$ safe auth install
$ safe node install
You should now be equipped with the latest CLI, authd and node.
Note that you can find full installation instructions in our user guide:
Joining the Testnet
As with previous public testnets, we are hosting some elders and adults on Digital Ocean to kick off the Network. These act as hardcoded contacts which bootstrap you to the network, therefore you will need a network configuration file to inform your CLI which network to bootstrap to. We store and update these connection details on S3 for you to easily point your configuration to.
If you have followed the instruction above to remove your .safe folder before installing everything again, you should have no existing network configurations - you can confirm this with the $ safe networks command.
You can add a profile for the latest testnet which points to our S3 location, you can add using $ safe networks add:
$ safe networks add fleming-testnet https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config
Network 'fleming-testnet' was added to the list. Connection information is located at 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config'
Now you need to ensure you are set to use this fleming-testnet configuration that we have added, we can use $ safe networks switch fleming-testnet for this:
$ safe networks switch fleming-testnet
Switching to 'fleming-testnet' network...
Fetching 'fleming-testnet' network connection information from 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config' ...
Successfully switched to 'fleming-testnet' network in your system!
If you need write access to the 'fleming-testnet' network, you'll need to restart authd (safe auth restart), unlock a Safe and re-authorise the CLI again
Let’s start up your authd process. Note for Windows users - there is a known issue where the $ safe auth start command seems to hang after entering it - please leave your terminal with this running and continue in a different terminal window or tab. Killing this terminal window will kill the sn_authd process::
$ safe auth start
UPDATE - having issues with authd? A new --for-cli flag has been added for $ safe keys create command. If you prefer not to use authd to get write access to the network, you can bypass it with the following command via the CLI (no need to install authd either) and then proceed straight to uploading files or transfer testcoins with CLI:
$ safe keys create --test-coins --for-cli
We now have our CLI, Authenticator daemon and Node components up-to-date, and we have the latest hardcoded contact details to bootstrap to the public testnet, so everything is in place and we’re ready to launch our node and add it to the Network. This is achieved using $ safe node join as follows:
$ safe node join
Creating '/Users/maidsafe/.safe/node/local-node' folder
Storing nodes' generated data at /Users/maidsafe/.safe/node/local-node
Starting a node to join a Safe network...
Launching with node executable from: /Users/maidsafe/.safe/node/sn_node
Node started with hardcoded contact: 161.35.36.185:12000
Launching node...
Node logs are being stored at: /Users/maidsafe/.safe/node/local-node/sn_node.log
Your node will now launch and attempt to connect to the public network. Keep in mind that it can only join the testnet if the Network is accepting new nodes at that time (see NOTE below). You can keep an eye on its progress via its logs, which can be found at $HOME/.safe/node/local-node/sn_node.log.
If there is no space on the testnet for your node to join, as of Fleming testnet v2, your node will automatically try to rejoin until it is successful, or until you kill the sn_node process.
NOTE - please keep in mind that we have an anti Sybil attack feature in place now to only accept new nodes onto the testnet when resources are required, therefore you may be attempting to join the Network but your logs tell you that the Network is not accepting new nodes at this time:
The network is not accepting nodes right now. Retrying after 3 minutes
This is expected behaviour which will happen each time you try to connect until the testnet detects that resources are running low. As of Fleming testnet v2 your node will automatically try to rejoin in a loop, you do not need to attempt to join again manually, or via a script. We recommend keeping a watch on your node log file to check if you have been accepted - $ tail -f $HOME/.safe/node/local-node/sn_node.log
You can help to speed up the process of the network needing new resources by adding some data yourself - you don’t need to run a node to upload data! See the Do I need to run a node to participate? section below.
Before working your way through the CLI commands to perform various actions on the network, remember to authenticate, create and unlock your Safe. Following the steps in these links gives your Safe some test Safe Network Tokens to use. This means that there is no need to farm first to earn rewards before being able to try operations such as uploading to the testnet.
You can even connect to the testnet in read only mode using CLI, i.e. once you’ve installed CLI you can try fetching content uploaded by other users. For example, you can download the following image and open it locally afterwards:
$ safe cat safe://hygoyeye9mq5jmipo3we79wohue8hjyw55e6f8xyk1hquwy9hsxnk9tdsme > ~/safe-the-planet.png
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Do I need to run a node to participate?
No, you can join with just the CLI and authd to experiment with data, tokens, etc. You can follow the instructions in the Joining the Testnet section above, but you don’t need to run $ safe node join - at this point you can just go straight to using the CLI as per the User Guide.
UPDATE - Now able to bypass authd for anyone with authd errors or hangs
A new version of CLI (v0.23.3) is now available which supports a new --for-cli flag for $ safe keys create command. If you prefer to not use authd to get write access to the network, you can just run the following command with CLI (no need to install authd either) and right after that start uploading files or transfer testcoins with CLI:
$ safe keys create --test-coins --for-cli
New SafeKey created at: "safe://hyryyyybbbo9e1ndkz8i3ghzs5j74p87hrhsmh8oc8nid76jiafpfewojay"
Preloaded with 1000.111 testcoins
Key pair generated:
Public Key = 210c3e89086ab9eb9372f6da7ba69fbc272cbe1e0c38aa3ef935c15a545209c0
Secret Key = fd61253273f1a280980b914f7338ca182f806a12c955b237c278b9200316912c
Setting new SafeKey to be used by CLI...
New credentials were successfully stored in /home/bochaco/.safe/cli/credentials
Safe CLI now has write access to the network
Make sure you upgrade your CLI to v0.23.3 with:
$ safe update
or simply re-install it with:
$ curl -so- https://sn-api.s3.amazonaws.com/install.sh | bash
Further Information
Where are my node logs?
When you launch your node you should see the location of your log file printed on screen - this will be $HOME/.safe/node/local-node/sn_node.log. You can tail your logs with a command such as $ tail -f $HOME/.safe/node/local-node/sn_node.log
Where are my rewards?
If you’ve started a node you will have keys generated for you, stored in two files next to the logs of your node (see above). As of now, these keys are not compatible with the CLI, but will be soon.
The keys can then be used as per the instructions of the CLI manual, for transfers and payments. We will eventually provide a better UX for easier use of these.
