Fleming Testnet v6 Release
Fleming Testnet v6 is LIVE!
Friday testnet anyone?
We are excited to announce that Fleming testnet v6 is now LIVE.
Released versions:
- sn_node v0.49.8
- sn_client v0.61.1
- sn_api v0.29.1
- sn_cli v0.29.2
Weâd like to dedicate Fleming Testnet v6 to @sharontannahill
Sharon has been an absolute rock for MaidSafe through thick and thin. We depend on her for so many things, and the work she does behind the scenes just doesnât always get the credit an increasing number of us realise she deserves. From administration, salaries, coin management, general organisation, and communications with exchanges and other 3rd parties, to name but a few of the tasks we rely on her for.
Sharon, this oneâs for you from all your Safe pals
IMPORTANT NOTE - IGD connection issues.
In the build up to todayâs release we have observed some IGD issues which we believe is in our IGD implementation. These issues can mean that trying to join the testnet with your node will more often that not fail with an IGD error, unless you have a public facing IP address, or have set up port forwarding on your router and launched your node specifying your internal & external IP and port numbers to reflect this.
We have a very small sample size within the company so we are looking to get some feedback from the community here on whether you can queue a node up to join the network, or whether you are receiving IGD errors.
If you do receive IGD errors, you donât have a public facing IP address, which the majority of us would not, but you feel that you are advanced enough to set up port forwarding on your router, you can also give this a try. We canât provide instructions for all the different variations of routers out there, but we will put instructions below on how to launch node when you are port forwarding.
If you are getting IGD errors and you donât feel that you are advanced enough to set up port forwarding then please continue to use the testnet as a client. No matter whether you run a node or not, you can still follow the instructions in the Joining the Testnet
section below and use the CLI as per the User Guide.
We are working to fully identify and resolve any IGD issue and hope to release a fix ASAP.
v6 Changelog
With the communityâs help, we identified several issues with previous testnet iterations, and have been working hard to resolve as many as possible.
Note that in the build up to todayâs release, we discovered a bug with rewards not being paid out and therefore decided to disable rewards so as not to block the testnet release. We will not be fixing rewards or transfer based issues as it seems the DBC work will transform those processes to be simpler and more privacy-aware.
The fixes and improvements for v6 are:
1. Only Accept âReachableâ Adults
There has been a major change to which nodes the network accepts to join as Adults - now only nodes which are reachable are accepted.
However, as detailed in the IMPORTANT NOTE above, weâve discovered a bug in our IGD implementation. Once this has been resolved you will see that not all nodes will be allowed to join the testnet, only those with correctly functioning IGD or a public IP.
How will you know if you have a correctly functioning IGD or a public IP on your node?
Once the IGD bug is fixed, when you follow the joining instructions below and attempt to run your node with safe node join
, if your logs, accessed with the command tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log
, report:
Unfortunately we are unable to establish a connection to your machine either through a public IP address, or via IGD on your router. Please ensure that IGD is enabled on your router - if it is and you are still unable to add your node to the testnet, then skip adding a node for this testnet iteration. You can still use the testnet as a client, uploading and downloading content, etc. https://forum.autonomi.community/
This means that IGD does not appear to be functioning correctly, and you donât have a public IP.
You do not need to take any further action as your node will not be running.
Weâre not sure at this point how many people this will affect, wider testing with this testnet iteration will give us an idea, once the IGD bug is fixed.
Can I still do anything with the testnet?
Yes! Everyone can use the testnet as a client. This change only affects those wishing to add a node to the testnet. No matter whether you run a node or not, you can still follow the instructions in the Joining the Testnet
section below and use the CLI as per the User Guide.
Will I ever be able to join another testnet if my IGD doesnât work and I donât have a public IP on my machine?
The short answer is yes. The long answer is, at the moment, it will take some manual configuration with different instructions for different people depending on your router. You would need to forward a port on your router to your machine, plus you would need to specify your machineâs IP:port when launching your node, which is not currently supported as an option when launching a node via the CLI.
We do have a sn_cli PR ready to test to support this change, but we feel that our resources should be 100% directed at the core testnet functionality right now rather than router configuration and other associated issues. This can be added, enabled and improved once we have stability with the testnets. In the meantime, if advanced users want to try port forwarding they will find some instructions below on how to manually launch their node.
2. Anti Entropy
As a recap, a major hurdle any decentralised network which will experience churn, such as Safe, must overcome is keeping the nodes up to date with the networkâs current state. Who has joined and who has left? Who are the current Elders in any given section?
Elders are by definition relatively dependable and churn amongst them is minimal, but Safe is asynchronous and communications between section Elders and external actors may be few and far between. The Elders in a section may have churned many times between contacts.
Itâs vital actors wishing to make changes on the network have an up-to-date âunderstandingâ of the section before they are allowed to do so, otherwise data could be written to Elders that no longer exist.
Entropy, or increasing disorder, is a characteristic of a constantly changing network. We cannot reduce it but we can wait for periods of zero change among Elders before allowing a change in their section. Anti-Entropy (AE) is the mechanism for doing this.
Even during periods of massive change such as a large global event, AE enables the network to inform actors attempting to mutate data that they must wait until the change is over before they can try again.
On Safe, the AE mechanism is applied to all messages that can mutate the network, such as a PUT request. It requires the requesting actor - be that a Client, an Adult or another section - to prove that the information they hold on the affected section is current. Until then they cannot make a change.
AE reduces the synchronisation issue to a series of questions to which the answer is yes or no. If the answer is no (your information is not current), you need to update your records and try again. There is no halfway house.
In this way AE forces all nodes that want to perform data operations on the network to prove they are up to date.
Weâve had AE implemented in sn_routing
for quite a while now, and more recently have been working hard to integrate it into many more of our crates, the biggest challenge being sn_node
. Kudos in particular to @yogesh for the many, many hours invested in making this a reality.
3. Out Of Memory issue
As you will know, Fleming testnet v5.1 seemed to be working spectacularly well until memory usage on multiple hosted nodes spiked and eventually led to those nodes dying, and so sections becoming unresponsive.
The reason for the memory spike was that the network was experiencing a huge explosion in the number of messages being sent across it, so weâve spent a lot of time since reviewing what generates all these messages and whether we could cut them down - the answer was yes, significantly!
Some examples of unnecessary network messages include the reserialising the whole message for every dst
update, or sending some messages to all Elders in a section when we actually only need to send to a subset, and also sending response messages, again to multiple Elders, when actually in many instances there is no need to send any response.
The sheer volume of messages was choking the system, we witnessed memory usage north of 1.5Gb per node. Our internal tests indicate weâve managed to cut that down to between 100-200mb.
4. Performance Improvements and Logging Changes
A nice side effect of the network message optimisations made with a view to resolving the OOM issue is that we expect that less network message traffic will equate to performance improvements. Weâve not gathered any stats on this, it wasnât our aim when we set out to tackle the OOM issue, but it should make a difference.
Weâve also reduced the amount of logging on our Digital Ocean nodes, which seems to have improved performance - we have noticed the tests running a fair bit quicker after this change. This is configurable so we can increase logging again to reproduce issues and gather more data, when required.
Note that as a result of these log changes, the log file that you will be used to tail
ing in previous testnet iterations has now changed name to sn_node_rCURRENT.log
. You will also notice that the log files now rotate when they hit a capped size limit, i.e. the contents of sn_node_rCURRENT.log
are copied over to sn_node_r00000.log
when it fills and the current logs are clean again, next it will copy to sn_node_r00001.log
, and so on. Searching for historic logs may therefore mean they are not in the sn_node_rCURRENT.log
file.
We still intend to look at overall performance and implement some ideas we have to improve the testnetâs efficiency and performance, but our initial goal is to get all the features working and the testnetâs stable, otherwise we will have a constantly moving target. It seems that as we go along improving stability and reliability, performance improvements are increasingly becoming a bonus side effect of this.
5. Account for Fluctuating StoreCost
Weâve observed during testing that sometimes the quoted StoreCost
can change between querying a section and sending a PUT. To account for this weâve added a StoreCost
buffer to avoid unnecessary errors here and allow the PUTs to go through. (Note: This should be resolved a little further down the line with DBCs and PUT contracts.)
What happens to previous testnet iterations?
All previous Fleming testnet iterations have been taken offline, with all data removed.
This is to allow us to concentrate on any issues that arise from the new testnet iteration, and avoid any community confusion over which testnet they are connected to, or any errors as a result of contamination from previous configurations and data. We ask that everyone who attempts to use this latest testnet iteration removes their $HOME/.safe
folder before trying to interact with this testnet, i.e. $ rm -rf $HOME/.safe/
to ensure there is no contamination.
All of the bootstrap nodes for previous iterations that we had on Digital Ocean have been destroyed, with new, clean D.O. nodes created for this testnet iteration - please follow the full list of instructions below to remove all previous testnet settings and data from your machine and to get you started with the latest testnet iteration.
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Are We Working on Other Fixes?
Yes. Multiple other fixes and improvements are already in progress, some even from before we released testnet v1.
We also anticipate that the feedback from this testnet iteration will direct us where else to aim resources as a priority.
Where can I report any issues found?
If you come across any issues in your testing, start by checking the Known Issues section of this post (see below) to see if it has already been acknowledged . If it has not yet been added, you can post your issue in the comments of this post. You can also report issues in the Online Spreadsheet for Testnet Results and Issues - see the section below.
We will monitor and investigate reported issues as soon as we can.
Online Spreadsheet for Testnet Results and Issues
If you will be uploading data to the testnet and/or providing safes (nodes), please consider posting your data and results at:
SN Testnet Review
(Massive thank you to @VaCrunch for creating this spreadsheet )
This is a new version of this spreadsheet, with the versions used for previous iterations now locked.
For those posting:
⢠No need to Sign In. The white cells are for your data entry.
⢠One row for each of your devices.
⢠Scroll down until you get to the first empty row.
⢠You do not need to use your Forum name for your ID, but please use the same ID for all your devices.
⢠Supporting/analysis tabs are at bottom of screen: Error_Msgs, Summary, Thumbnails, Charts, Matrix, Top 10, Map, Resources, Lists, and Comments.
⢠Use the View/Zoom feature at top to reduce the visible size of the spreadsheet to match your screen. 75% will be the best fit for most people.
⢠If you choose not to record the name of your country please select âUnspecifiedâ at the bottom of the list.
⢠If you change the number of nodes (safes) you are running, just edit the existing figure, do not add another line for the same device.
Known Issues
Any reproducible issues that we, or the community, come across will be added to this section:
-
Although weâve started making some optimisations, we still expect to see performance issues, particularly around
$ safe files put ....
operations with larger files/folders. There has been very little optimisation of write times, nor any general analysis of speed so far, with the majority of our efforts concentrated on getting features in place and fully functional before we spend more time looking at it from a performance perspective. -
We have removed
authd
from the stack for normal users right now, instead focussing onSafe CLI
. To that end, we have updated the CLI to be able to do all things that previously required authd. This means no GUI right now, mostly. The focus is on network stability and then speed. As we optimise for speed, the team will also focus on authd again, perhaps removing it (our goal) in favour of a simpler approach. So test as much as possible with the CLI for now. It reduces user error, reduces the potential bug surface, and allows total focus on the prize - a fully autonomous decentralised network. -
We have discovered a bug with rewards not being paid out and therefore decided to disable rewards so as not to block the testnet release. We will not be fixing rewards or transfer based issues as it seems the DBC work will transform those processes to be simpler and more privacy-aware.
-
In the build up to todayâs release we have discovered a bug we believe is in our IGD implementation. This bug means that trying to join the testnet with your node seems to more often than not fail with an IGD error, unless you have a public facing IP address, or have set up port forwarding on your router and launched your node to reflect this.
If you donât have a public facing IP address, which the majority of us would not, but you feel that you are advanced enough to set up port forwarding on your router, you can give this a try. We canât provide instructions for all the different variations of routers out there, but we will put instructions below on how to launch a node when you are port forwarding.
-
Itâs been noticed after v6 launch that large file uploads can hang as transfers get out of sync. A potential fix for this is known and already planned, as per here.
Letâs try it out!
Please read and follow the instructions below carefully.
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Installing/Updating to Latest
To avoid any pollution from previous testnet iteration settings and data, we ask that everyone removes their $HOME/.safe/
directory, then installs the CLI, authd and node again.
$ rm -rf $HOME/.safe
Note that Windows users may be required to install specific software before being able to install the CLI using the command below - see full instructions for Windows here.
Now download and install the verified latest CLI binary via our install script with the following command:
$ curl -so- https://sn-api.s3.amazonaws.com/install.sh | bash
This script downloads the correct CLI binary for your OS (Windows, macOS or Linux), installs it in the correct directory, creating that directory if it doesnât exist, and adds it to your system PATH.
You may need to restart your terminal window at this point for any changes to your system PATH to take effect. You can now confirm whether the CLI is installed and set up correctly:
$ safe -V
sn_cli 0.29.2
Next, you should install the latest node with the command below.
$ safe node install
Before proceeding weâll just make sure we kill any old sn_node
processes which have been left running:
$ safe node killall
You should now be equipped with the latest CLI and node.
Note that you can find full installation instructions in our user guide:
Joining the Testnet
As with previous public testnets, we are hosting some Elders and Adults on Digital Ocean to kick off the Network. These act as hardcoded contacts which bootstrap you to the network, therefore you will need a network configuration file to inform your CLI which network to bootstrap to. We store and update these connection details on S3 for you to easily point your configuration to.
If you have followed the instruction above to remove your $HOME/.safe
folder before installing everything again, you should have no existing network configurations - you can confirm this with the $ safe networks
command.
You can add a profile for the latest testnet which points to our S3 location, this is done using $ safe networks add
like so:
$ safe networks add fleming-testnet https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config
Network 'fleming-testnet' was added to the list. Connection information is located at 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config'
Now you need to ensure that you are set to use this newly added fleming-testnet
configuration, we can use $ safe networks switch fleming-testnet
for this:
$ safe networks switch fleming-testnet
Switching to 'fleming-testnet' network...
Fetching 'fleming-testnet' network connection information from 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config' ...
Successfully switched to 'fleming-testnet' network in your system!
If you need write access to the 'fleming-testnet' network, you'll need to restart authd (safe auth restart), unlock a Safe and re-authorise the CLI again
(Ignore the ââŚyouâll need to restart authdâŚâ advice in there)
Now to get write access to the network, you can use the following command via the CLI before proceeding to uploading files or transfer testcoins with CLI:
$ safe keys create --test-coins --for-cli
New SafeKey created: "safe://hyryyyy8ogt4yxmtgqd5yj9kuim911yuchbz77mhcrsdqh6dc5noypi9nqa"
Preloaded with 1000.111 testcoins
Key pair generated:
Public Key = f0347407ae2670f604fd53aaff29026ce06fdeaf8c2586ee786cd8a006d7e276
Secret Key = 7c2839cfda20f3c0f7872372d20a28080c7713688f16d9199c1ec02a7b0d185b
Setting new SafeKey to be used by CLI...
We now have our CLI and node components up-to-date, the latest hardcoded contact details to bootstrap to the public testnet, and weâve generated some test Safe Network Tokens that we can use later to upload data.
To Add Your Node To The Testnet
Some differences in this section today due to the IGD bug described at the beginning of this post. We encourage all users wanting to try connecting a node to try the usual method of using $ safe node join
as follows:
$ safe node join
Creating '/Users/maidsafe/.safe/node/local-node' folder
Storing nodes' generated data at /Users/maidsafe/.safe/node/local-node
Starting a node to join a Safe network...
Launching with node executable from: /Users/maidsafe/.safe/node/sn_node
Node started with hardcoded contact: 161.35.36.185:12000
Launching node...
Node logs are being stored at: /Users/maidsafe/.safe/node/local-node/sn_node_rCURRENT.log
(Note that log files are rotated, and subsequent files will be named sn_node_r[NNNNN].log, with values starting at 00000 and up to 99999.)
BUT weâve found during in-house testing that in the vast majority of cases this will fail with an IGD error, when it shouldnât. Weâre working to resolve this now.
The alternative/workaround is to set up port forwarding on your router, then launch your node with:
$ $HOME/.safe/node/sn_node --public-addr <public ip:port> --local-addr <localnet ip:port> --hard-coded-contacts=[\"46.101.59.28:12000\"]
Weâve had success with this in-house, but itâs not for everyone.
Weâre working hard in the background to resolve the IGD issue which should allow more people to join.
If the above steps are successful for you, your node will now launch and attempt to connect to the public network. Keep in mind that it can only join the testnet if the Network is accepting new nodes at that time (see NOTE below). You can keep an eye on its progress via its logs, which can be found at $HOME/.safe/node/local-node/sn_node_rCURRENT.log
.
If there is no space on the testnet for your node to join, your node will automatically try to rejoin until it is successful, or until you kill the sn_node
process ($ safe node killall
). This is an anti Sybil attack feature which we have put in place to only accept new nodes onto the testnet when resources are required, therefore you may be attempting to join the Network but your logs tell you that the Network is not accepting new nodes at this time:
The network is not accepting nodes right now. Retrying after 3 minutes
This is expected behaviour which will happen each time you try to connect until the testnet detects that resources are running low. Your node will automatically try to rejoin in a loop, you do not need to attempt to join again manually, or via a script. We recommend keeping a watch on your node log file to check if you have been accepted - $ tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log
You can help to speed up the process of the network needing new resources by adding some data yourself - you donât need to run a node to upload data! See the Do I need to run a node to participate? section below.
To Connect to the Testnet as a Client
Before working your way through the CLI commands to perform various actions on the network, following the steps above gives your Safe some test Safe Network Tokens to use. This means that there is no need to farm first to earn rewards before being able to try operations such as uploading to the testnet. You can then proceed with the various CLI commands to perform operations on the network.
You can even connect to the testnet in read only mode using CLI, i.e. once youâve installed CLI you can try fetching content uploaded by other users. For example, try download the following image and open it locally afterwards:
$ safe cat safe://hygoyeyx768nkst7qqjjk8khjm67tr1n4otbctcts5j7dten43zae3ga83y > ~/safe-the-planet.png
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Do I need to run a node to participate?
No, you can join with just the CLI to experiment with data, tokens, etc. You can follow the instructions in the Joining the Testnet
section above, but you donât need to run $ safe node join
- at this point, just go straight to using the CLI as per the User Guide.
Further Information
Where are my node logs?
When you launch your node you should see the location of your log file printed on screen - this will be $HOME/.safe/node/local-node/sn_node_rCURRENT.log
. You can tail
your logs with a command such as $ tail -f $HOME/.safe/node/local-node/sn_node_rCURRENT.log
Note that as a result of these log changes, the log file that you will be used to tail
ing in previous testnet iterations has now changed name to sn_node_rCURRENT.log
. You will also notice that the log files now rotate when they hit a capped size limit, i.e. the contents of sn_node_rCURRENT.log
are copied over to sn_node_r00000.log
when it fills and the current logs are clean again, next it will copy to sn_node_r00001.log
, and so on. Searching for historic logs may therefore mean they are not in the sn_node_rCURRENT.log
file.
Where are my rewards?
As per the Known Issues
section above, in the build up to todayâs release, we discovered a bug with rewards not being paid out and therefore decided to disable rewards so as not to block the testnet release. We will not be fixing rewards or transfer based issues as it seems the DBC work will transform those processes to be simpler and more privacy-aware.
____
/ \ \
/ /\ \ \
/ / /\ \ \
/ / /__\_\ \
/ / /________\
\/___________/