Fleming Testnet v6.1 is LIVE!
We are excited to announce that Fleming Testnet v6.1 is now LIVE, following on quickly from last week’s v6 release.
Released versions:
- safe_network v0.2.17
- sn_api v0.29.2
- sn_cli v0.30.1
Fast paced, that’s how we can describe MaidSafe right now. So a lot is happening for us all.
@StephenC has been headhunted with some vigour and repeatedly offered a career opportunity that is basically too hard to resist. So his last day as a full time 100% dedicated MaidSafer is tomorrow. However, Stephen is our man and will be around for updates/testnets and more, when he can at least. So no au revoir, but a change in his prime focus. We wish him well and look forward to continuing our relationship with him.
We can’t say enough about Stephen; solid, dependable, supportive, great manager, a thoroughly decent person all around. So still here, but not nearly as much and not during normal working hours (but what are those in this team?).
We are delighted to welcome new Engineers Anselme Grumbach (@Anselme) and Chris Connelly (@chris.connelly) to the team 
Despite only being part of the team for a few days each now, they’re both already chipping in to offer value like we knew they would. Welcome aboard guys!
They are very soon to be followed by a face we know well, with Chris O’Neil set to rejoin MaidSafe in a couple of weeks time. Chris was part of Stephen’s QA team here at MaidSafe until late 2019, working as a DevOps Engineer, and is highly regarded by those who worked alongside him back then.
Following @bochacho’s promotion to Senior Engineer, and seeing how well that has gone, we are now also promoting @joshuef to Senior Engineer. Gab is all over the job of simplifying code and making sense of flows and much more, while Josh is a powerhouse of getting stuff done and getting it done now! He has been pivotal in getting testnets out and pushing every day to move forward. Even with a new child and all the extra duties that entails, Josh has been outstanding. The whole team will be very happy with this one. We already Have Dan, David Rusu and Edward all in senior positions with us, and Josh’s time has come to be recognised as a crucial part of the team in a critical role. You want this guy at your side in any project. A born leader who will never admit it.
So ups and downs, but never boredom and no bad news. Stephen is a huge loss, but not lost. We will see him often and he will remain part of the employee scheme that will see him well rewarded on launch, as he should be. That is not normal and we need to make an exemption for that in the employee scheme, but it’s an easy call. Good guys do win sometimes.
Change to the organisation of Testnet Release posts
First thing’s first - It’s been pointed out that it can be difficult to find support and information on a particular aspect of a testnet release, for example launching a node, and so to try to help with this we will split this topic into two - a release post for General and CLI Support, and another for Node Support. We’re hoping that this makes it easier to read through comments and support posts related to either of these.
You can see which post you are in, General and CLI or Node, by looking at the title of the topic. Both posts will contain more or less the same information in the OP for now - we can tweak that down the line to see how it works.
As per the topic titles, try to keep your posts in the correct topic to help others see posts specific to their interest. We hope the moderators will help us out with anything accidentally posted in the wrong topic.
This is the General and CLI Support topic, for the Node support topic please click here.
Continuing IGD connection issues
As highlighted in the original Fleming Testnet v6 release post, we have observed some IGD issues when connecting to testnets which we believe are in our IGD implementation. These issues can mean that trying to join the testnet with your node will sometimes incorrectly fail with an IGD error.
If you do receive IGD errors when you try to join your node, first thing to do is try a few times - the issue seems intermittent so you shouldn’t get this every time. You may however receive the usual “Retrying after 3 minutes” message but then it fails with the IGD error on one of the retries. Machines with a public facing IP, such as those on Digital Ocean or other cloud providers, should be able to connect or queue to join as nodes with no issues.
If you don’t have a public facing IP address, which the majority of us would not as we are behind a router at home, but you feel that you are advanced enough to set up port forwarding on your router, you can also give this a try. We can’t provide instructions for all the different variations of routers out there, but we will put instructions below on how to launch node when you are port forwarding.
If you are getting constant IGD errors and you don’t feel that you are advanced enough to set up port forwarding then please continue to use the testnet as a client. No matter whether you run a node or not, you can still follow the instructions in the Joining the Testnet section below and use the CLI as per the User Guide.
We are working hard to fully identify and resolve any IGD issues.
v6.1 Changelog
It’s been a busy few days since the original v6 release!
Note that we don’t expect this testnet to stay up for too long, it’s purpose is to test the fixes mentioned below, then it can be taken offline while we move onto the next stage. Here’s a summary of what we’ve improved since v6:
1. GETs should now be fixed.
There was a bug where due to incomplete knowledge of the network at a certain point, closest section could not be returned. This meant some GET responses were getting lost and not reaching the client. We have a workaround in place while we try to implement a more secure, AE focussed solution.
2. Client timeout has been removed.
With the live network, even successful but slow GETs were not being registered at the client due to a timeout, so that’s been removed (ideally will be set up in the CLI to be user configurable down the line).
3. AE for client debits.
It was observed that some Elders were out of sync with regards to the current transaction status of a client. For larger files (with many transfers), this would mean that after a certain point every attempt made was out of order. There is now a feature in place to update Elders when they respond with Operation Out of Order errors.
Note: we still do not have message retries enabled, so large files (>100mb) may still be corrupted by missing chunks due to this. But the next attempt at a transfer should now work where it previously could not.
4. Log rotation and flexi-logger implementation of async.
This should hopefully prevent any (possible) slow down. In the end it looks like this may not be that large of an issue, but it is one less thing to worry about now.
5. System file calls are now async.
Previously all operations there were sync.
6. We’re using the mono repo - safe_network!
We’ve merged 7 of our main core repos into one, streamlining CI and testing. The repos amalgamated together are sn_routing, sn_client, sn_node, sn_messaging, sn_data_types, sn_url and sn_transfers. You may see one or two more be added there in future, where it makes sense. Now every change to routing, messaging, etc, will be checked against a local network as part of the CI process, tackling what we were seeing as an increasingly common problem of a change in one of these repos breaking another, but this breakage not being discovered until well after the change was made.
The new safe_network mono repo is in use for the first time in Fleming Testnet v6.1. The existing separate repos are in the process of being archived, with us working through transferring over or closing PRs and issues.
Note that there have been numerous tweaks here as we got over some teething issues as we started using safe_network.
7. Some further message serialisation duplication has been removed.
This could account for increasing node memory usage, so we are removing any duplication wherever we come across it.
What happens to previous testnet iterations?
All previous Fleming testnet iterations have been taken offline, with all data removed.
This is to allow us to concentrate on any issues that arise from the new testnet iteration, and avoid any community confusion over which testnet they are connected to, or any errors as a result of contamination from previous configurations and data. We ask that everyone who attempts to use this latest testnet iteration removes their $HOME/.safe folder before trying to interact with this testnet, i.e. $ rm -rf $HOME/.safe/ to ensure there is no contamination.
All of the bootstrap nodes for previous iterations that we had on Digital Ocean have been destroyed, with new, clean D.O. nodes created for this testnet iteration - please follow the full list of instructions below to remove all previous testnet settings and data from your machine and to get you started with the latest testnet iteration.
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Are We Working on Other Fixes?
Yes, we’ve currently set a list of features which we would like to have as many of in place for v7:
- Integrate Dbc for payments and rewards (means ripping out current AT2 flows).
- Finalise message simplification (which will kill a lot of silly bugs and extra unneeded serialization).
- Integrate a faster threshold_crypto (thanks @mav).
- Formalise message flows for GET/PUT and mutate (along with costs).
- Switch on network versions in messages (to prevent people using the wrong version of x,y,z)
- Measure more accurately bandwidth use as well as mem issues (which will mean formalising logging a bit more).
These are not individually huge, but we do feel we need to regroup slightly to get them done right, and so expect at least a 4 week gap until we have enough of them integrated, tested and stable, no doubt along with various other fixes features and tweaks, and feel that Fleming Testnet v7 is ready for release.
Where can I report any issues found?
If you come across any issues in your testing, start by checking the Known Issues section of this post (see below) to see if it has already been acknowledged . If it has not yet been added, you can post your issue in the comments of this post. You can also report issues in the Online Spreadsheet for Testnet Results and Issues - see the section below.
We will monitor and investigate reported issues as soon as we can.
Online Spreadsheet for Testnet Results and Issues
If you will be uploading data to the testnet and/or providing safes (nodes), please consider posting your data and results at:
SN Testnet Review
(Massive thank you to @VaCrunch for creating this spreadsheet 
)
This is a new version of this spreadsheet, with the versions used for previous iterations now locked.
For those posting:
• No need to Sign In. The white cells are for your data entry.
• One row for each of your devices.
• Scroll down until you get to the first empty row.
• You do not need to use your Forum name for your ID, but please use the same ID for all your devices.
• Supporting/analysis tabs are at bottom of screen: Error_Msgs, Summary, Thumbnails, Charts, Matrix, Top 10, Map, Resources, Lists, and Comments.
• Use the View/Zoom feature at top to reduce the visible size of the spreadsheet to match your screen. 75% will be the best fit for most people.
• If you choose not to record the name of your country please select “Unspecified” at the bottom of the list.
• If you change the number of nodes (safes) you are running, just edit the existing figure, do not add another line for the same device.
Known Issues
Any reproducible issues that we, or the community, come across will be added to this section:
-
Although we’ve started making some optimisations, we still expect to see performance issues, particularly around
$ safe files put ....operations with larger files/folders. There has been very little optimisation of write times, nor any general analysis of speed so far, with the majority of our efforts concentrated on getting features in place and fully functional before we spend more time looking at it from a performance perspective. -
We have removed
authdfrom the stack for normal users right now, instead focussing onSafe CLI. To that end, we have updated the CLI to be able to do all things that previously required authd. This means no GUI right now, mostly. The focus is on network stability and then speed. As we optimise for speed, the team will also focus on authd again, perhaps removing it (our goal) in favour of a simpler approach. So test as much as possible with the CLI for now. It reduces user error, reduces the potential bug surface, and allows total focus on the prize - a fully autonomous decentralised network. -
We have discovered a bug with rewards not being paid out and therefore decided to disable rewards so as not to block the testnet release. We will not be fixing rewards or transfer based issues as it seems the DBC work will transform those processes to be simpler and more privacy-aware.
-
We believe there is a bug in our IGD implementation. This bug means that trying to join the testnet with your node seems to intermittently fail with an IGD error, unless you have a public facing IP address, or have set up port forwarding on your router and launched your node to reflect this.
If you don’t have a public facing IP address, which the majority of us would not, but you feel that you are advanced enough to set up port forwarding on your router, you can give this a try. We can’t provide instructions for all the different variations of routers out there, but we will put instructions below on how to launch a node when you are port forwarding.
-
It’s been noticed after v6 launch that large file uploads can hang as transfers get out of sync. A potential fix for this is known and already planned, as per here.
Let’s try it out!
Please read and follow the instructions below carefully.
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Installing/Updating to Latest
To avoid any pollution from previous testnet iteration settings and data, we ask that everyone removes their $HOME/.safe/ directory, then installs the CLI, authd and node again.
$ rm -rf $HOME/.safe
Note that Windows users may be required to install specific software before being able to install the CLI using the command below - see full instructions for Windows here.
Now download and install the verified latest CLI binary via our install script with the following command:
$ curl -so- https://sn-api.s3.amazonaws.com/install.sh | bash
This script downloads the correct CLI binary for your OS (Windows, macOS or Linux), installs it in the correct directory, creating that directory if it doesn’t exist, and adds it to your system PATH.
You may need to restart your terminal window at this point for any changes to your system PATH to take effect. You can now confirm whether the CLI is installed and set up correctly:
$ safe -V
sn_cli 0.30.1
Next, you should install the latest node with the command below.
$ safe node install
Before proceeding we’ll just make sure we kill any old sn_node processes which have been left running:
$ safe node killall
You should now be equipped with the latest CLI and node.
Note that you can find full installation instructions in our user guide:
Joining the Testnet
As with previous public testnets, we are hosting some Elders and Adults on Digital Ocean to kick off the Network. These act as hardcoded contacts which bootstrap you to the network, therefore you will need a network configuration file to inform your CLI which network to bootstrap to. We store and update these connection details on S3 for you to easily point your configuration to.
If you have followed the instruction above to remove your $HOME/.safe folder before installing everything again, you should have no existing network configurations - you can confirm this with the $ safe networks command.
You can add a profile for the latest testnet which points to our S3 location, this is done using $ safe networks add like so:
$ safe networks add fleming-testnet https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config
Network 'fleming-testnet' was added to the list. Connection information is located at 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config'
Now you need to ensure that you are set to use this newly added fleming-testnet configuration, we can use $ safe networks switch fleming-testnet for this:
$ safe networks switch fleming-testnet
Switching to 'fleming-testnet' network...
Fetching 'fleming-testnet' network connection information from 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config' ...
Successfully switched to 'fleming-testnet' network in your system!
If you need write access to the 'fleming-testnet' network, you'll need to restart authd (safe auth restart), unlock a Safe and re-authorise the CLI again
(Ignore the “…you’ll need to restart authd…” advice in there)
Now to get write access to the network, you can use the following command via the CLI before proceeding to uploading files or transfer testcoins with CLI:
$ safe keys create --test-coins --for-cli
New SafeKey created: "safe://hyryyyy8ogt4yxmtgqd5yj9kuim911yuchbz77mhcrsdqh6dc5noypi9nqa"
Preloaded with 1000.111 testcoins
Key pair generated:
Public Key = f0347407ae2670f604fd53aaff29026ce06fdeaf8c2586ee786cd8a006d7e276
Secret Key = 7c2839cfda20f3c0f7872372d20a28080c7713688f16d9199c1ec02a7b0d185b
Setting new SafeKey to be used by CLI...
We now have our CLI and node components up-to-date, the latest hardcoded contact details to bootstrap to the public testnet, and we’ve generated some test Safe Network Tokens that we can use later to upload data.
To Add Your Node To The Testnet
We encourage all users wanting to try connecting a node to try the usual method of using $ safe node join as follows:
$ safe node join
Creating '/Users/maidsafe/.safe/node/local-node' folder
Storing nodes' generated data at /Users/maidsafe/.safe/node/local-node
Starting a node to join a Safe network...
Launching with node executable from: /Users/maidsafe/.safe/node/sn_node
Node started with hardcoded contact: 161.35.36.185:12000
Launching node...
Node logs are being stored at: /Users/maidsafe/.safe/node/local-node/sn_node_rCURRENT.log
(Note that log files are rotated, and subsequent files will be named sn_node_r[NNNNN].log, with values starting at 00000 and up to 99999.)
BUT we’ve found during in-house testing that this will fail intermittently with an IGD error, when it shouldn’t. We’re working to resolve this now. You can retry joining your node multiple times if it fails unexpectedly for you.
The alternative/workaround for advanced users is to set up port forwarding on your router, then launch your node with:
$ $HOME/.safe/node/sn_node --public-addr <public ip:port> --local-addr <localnet ip:port> --hard-coded-contacts=[\"178.128.164.253:52181\",\"46.101.21.162:34546\",\"178.128.169.151:34948\",\"178.62.20.236:12000\",\"159.65.60.126:44800\",\"178.128.161.75:34431\",\"178.128.162.219:55531\",\"178.128.169.114:35708\",\"178.128.161.137:59232\",\"178.128.170.69:45043\",\"178.128.169.206:59730\",\"178.128.169.23:52762\",\"178.128.166.44:50687\",\"178.128.169.93:58040\",\"178.128.172.153:52125\",\"178.128.167.101:53725\",\"178.128.169.109:55504\",\"159.65.24.159:56465\",\"178.128.170.97:37173\",\"178.128.172.235:53286\",\"178.128.160.4:52049\",\"178.128.169.33:46106\",\"178.128.169.57:42902\",\"178.128.168.75:36398\",\"138.68.191.226:46181\",\"178.128.175.25:33290\",\"188.166.175.46:46503\",\"178.128.169.227:54173\",\"206.189.121.46:32982\",\"178.128.169.152:33365\",\"178.128.162.146:59775\",\"178.128.172.1:54636\",\"178.128.161.127:48264\",\"46.101.17.48:59649\"]
We’ve had success with this in-house, but it’s not for everyone.
We’re working hard in the background to resolve the IGD issue which should allow more people to join first time.
If the above steps are successful for you, your node will now launch and attempt to connect to the public network. Keep in mind that it can only join the testnet if the Network is accepting new nodes at that time (see NOTE below). You can keep an eye on its progress via its logs, which can be found at $HOME/.safe/node/local-node/sn_node_rCURRENT.log.
If there is no space on the testnet for your node to join, your node will automatically try to rejoin until it is successful, or until you kill the sn_node process ($ safe node killall). This is an anti Sybil attack feature which we have put in place to only accept new nodes onto the testnet when resources are required, therefore you may be attempting to join the Network but your logs tell you that the Network is not accepting new nodes at this time:
The network is not accepting nodes right now. Retrying after 3 minutes
This is expected behaviour which will happen each time you try to connect until the testnet detects that resources are running low. Your node will automatically try to rejoin in a loop, you do not need to attempt to join again manually, or via a script. We recommend keeping a watch on your node log file to check if you have been accepted - $ tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log
You can help to speed up the process of the network needing new resources by adding some data yourself - you don’t need to run a node to upload data! See the Do I need to run a node to participate? section below.
To Connect to the Testnet as a Client
Before working your way through the CLI commands to perform various actions on the network, following the steps above gives your Safe some test Safe Network Tokens to use. This means that there is no need to farm first to earn rewards before being able to try operations such as uploading to the testnet. You can then proceed with the various CLI commands to perform operations on the network.
You can even connect to the testnet in read only mode using CLI, i.e. once you’ve installed CLI you can try fetching content uploaded by other users. For example, try download the following image and open it locally afterwards:
$ safe cat safe://hygoyeyx768nkst7qqjjk8khjm67tr1n4otbctcts5j7dten43zae3ga83y > ~/safe-the-planet.png
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Do I need to run a node to participate?
No, you can join with just the CLI to experiment with data, tokens, etc. You can follow the instructions in the Joining the Testnet section above, but you don’t need to run $ safe node join - at this point, just go straight to using the CLI as per the User Guide.
Further Information
Where are my node logs?
When you launch your node you should see the location of your log file printed on screen - this will be $HOME/.safe/node/local-node/sn_node_rCURRENT.log. You can tail your logs with a command such as $ tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log
Note that as a result of these log changes, the log file that you will be used to tailing in previous testnet iterations has now changed name to sn_node_rCURRENT.log. You will also notice that the log files now rotate when they hit a capped size limit, i.e. the contents of sn_node_rCURRENT.log are copied over to sn_node_r00000.log when it fills and the current logs are clean again, next it will copy to sn_node_r00001.log, and so on. Searching for historic logs may therefore mean they are not in the sn_node_rCURRENT.log file.
Where are my rewards?
As per the Known Issues section above, in the build up to today’s release, we discovered a bug with rewards not being paid out and therefore decided to disable rewards so as not to block the testnet release. We will not be fixing rewards or transfer based issues as it seems the DBC work will transform those processes to be simpler and more privacy-aware.
____
/ \ \
/ /\ \ \
/ / /\ \ \
/ / /__\_\ \
/ / /________\
\/___________/
