Fleming Testnet v5 Release
Fleming Testnet v5 is LIVE!
UPDATE 1 - some Elders nodes on Fleming testnet 5 on first launch experienced an OOM issue due to a bug - this resulted in the network breaking down.
We are investigating the cause.
We have now relaunched the testnet, this time with memory profiling enabled on the MaidSafe hosted nodes. This should hopefully allow us to pinpoint why the OOM issue has occurred.
We need your help to re-test it and see if we can reproduce this issue.
UPDATE 2 - We have successfully reproduced the OOM issue.
We will be studying the results of memory profiling done on the nodes running on Digital Ocean to help identify the source of this issue. Once we have done some analysis of them, we will know whether we can release a quick testnet v5.1 iteration, or hold off for v6. We will keep you informed. In the meantime please consider testnet v5 OFFLINE.
Thank you for your help!
We are excited to announce that Fleming testnet v5 is now LIVE. We are dedicating this one to Gabriel, aka @bochaco, an absolute foundation stone to so much that we do here at MaidSafe. Now one of the longest serving members of the team, he’s the guy we rely on for guidance and expert broad knowledge of the Safe Network components. His fingerprints and influence can be found all over our crates, but none more so than the CLI, that vital component that we’ve all grown to depend on to use to bring everything else together.
Gabriel’s contribution to community discussions and support is second to none. So often, particularly due to timezones, the majority if not all of the rest of the team have signed off for the night only to come in the next morning to see that @bochaco has gone above and beyond to hold the fort.
Gab, the team have spoken and we feel your dedication and Engineering needs recognised. We know you like to quietly work away doing your thing, but you cannot escape this one chap! We are a team who don’t promote good Engineers out of the job into management, and to that end and in that vein we would all like to congratulate you on your new title Senior Engineer. It’s a promotion, but it’s more of a recognition and one that is probably the most well earned in our company’s history. (
on you 
)
v5 Changelog
With the community’s help, we identified several issues with previous testnet iterations, and have been working hard to resolve as many as possible.
Anti Entropy has to skip this particular iteration - we’ve been bogged down in some bugs over the last few days while trying to integrate it into the rest of the code, so we’ve decided to keep it out and allow you to test the various other fixes and improvements we’ve made. We’ll continue working on it in the meantime. 
The fixes and improvements for v5 are:
1. Fix Data Loss Bugs
Throughout testnet iterations 1-4 there’s been recurring reports of some data that was uploaded to the testnets no longer being reachable. After testnet v4 we gathered as much information as we could about these reports and made it a priority to resolve any known data loss bugs.
We made several changes in sn_routing, with PRs 2460, 2470 and 2474 all merged - these all focus on issues we identified where the list of Adults in a section was incorrect or out of date.
We also merged a fix in sn_node here where there was a bug with the way the data was being replicated across nodes.
We hope that this eradicates the data loss issues we have been seeing up until now, internal testnets we have been running seem to indicate it does, but we won’t know for sure until a larger scale testnet involving the community…over to you!
2. Self-Organisation of Blob chunks by Adults
Until now, Elders were responsible for deciding where Blob chunks are stored. They calculated the closest 4 Adults when a PUT request arrived and sent the chunk to those Adults, keeping a track of this mapping as BlobMetadata that was stored at the Elders along with the other data types. Subsequent data requests used this metadata to query the respective Adults to perform the required data operations. On Elder churn and/or section split this metadata was also synced among the new Elders.
This testnet aims to test out phase one of a new approach towards the handling of chunks, purely based on the XOR address space. Through this approach, a chunk will always be held by the closest 4 Adults. When a new Adult joins / leaves the network the Adults themselves (with some help from the Elders) will reorganise the chunks in such a way that the new Adult will hold all chunks that it is responsible for once it has joined the section. In the case of an Adult leaving, a new copy will be given to the next-closest Adult node. This way when there is a data request, it is forwarded to the relevant Adults without the need of metadata at Elders.
In the future, we might move the acquisition of required chunks by an Adult to be part of the resource proof so that any new Adults will already have the data it’s responsible for when it joins the Network.
While testing this feature during iterations of internal testnets, we initially found that the Elders went out of memory while handling messages related to re-organisation of chunks. After some fixes, we could not reproduce this during internal testing, but we look forward to seeing if it happens again in a much larger scale, courtesy of our dedicated community 
3. Performance Improvements
Until now it’s not been our intention to look at performance improvements with the testnets, our primary focus has been to fix bugs first before then tuning a working product. We’ve had a couple of weeks now since v4 was released to give the team some time to work through the issues found and features being added, but some of the team were free during this time and were able implement some code improvements and efficiencies that had become obvious to them. We believe these will result in improved performance uploading and downloading from the testnet.
The improvements were focused on sn_client and sn_node. The general intention of them was to cut down on several unnecessary messages that were increasing network traffic and slowing things down, and to simplify current processes. Some examples from sn_client are PRs 1447, 1468, 1473, 1477 and 1504.
Examples of improvements in sn_node are PRs 1467, 1474, 1480, 1496 and 1497.
These were considered quick and obvious improvements, low hanging fruit if you like. There are many more to be made as we move on with the testnet iterations and major bugs are resolved, allowing us to focus more and more on other aspects, such as performance.
4. Change to XORURL Addresses.
We’ve made several improvements to our self_encryption crate in recent weeks and we will be using these for the first time in this version of the Fleming testnet.
It is important that we highlight this as changes to self_encryption can have the knock on effect of changing XORURLs. This is the case with these updates, so if you were used to uploading a certain file and it having the same XORURL in each of the previous testnet iterations until now, it will be different this time.
So anyone who has scripts, or uses copied commands with XORULs in them, keep in mind that the XORURLs will need to be updated this time round.
5. Register Data Type and API.
We’ve progressed with the implementation of the new Register (CRDT) data type and its client side public API. We have the first implementation now in sn_api.
This new Register API is not yet being used by any logic/flow other than for a new client side Multimap abstraction we’ve also been working on. This new Multimap is not a new data type but just a client side abstraction we are exposing now in the sn_api API that will be available for any application that may need it. The rest of the sn_api abstraction we have planned, such as the NRS and Wallet containers, will also make use of this new Multimap API and abstraction to store the NRS/Wallet data respectively.
We will soon be working on not only migrating the current implementation/serialisation of NRS and Wallet containers to use the Multimap serialisation, and therefore the Register data type as the native storage on the network, but also on having commands that allow users to interact with Registers and Multimap content directly from the Safe CLI.
6. Section Authority Provider
With this PR, we have implemented SectionAuthorityProvider in Fleming testnet v5. This replaces many separate instances of Elder lists or Section Info types, moving them into one type. This is a struct signed by the section and includes the section Prefix, Elders keys (both ed25519 and BLS key shares) and section key.
This is an important aspect of the network and will really help clear up Anti Entropy. An Elder is not an Elder and has no authority unless he is an Elder agreed by the section at a specific section key. Therefore the SectionAuthorityProvider tells us with cryptographic certainty whether an Elder is, in fact, part of the group at a section key. Without this, the code was confusing and did not recognise that an Elder is only relevant when tied to a section key, causing bugs and confusion. We now have a section signed set that shows where the node is acting as an elder and what keys they should use.
What happens to previous testnet iterations?
All previous Fleming testnet iterations have now been taken offline, with all data removed.
This is to allow us to concentrate on any issues that arise from the new testnet iteration, and avoid any community confusion over which testnet they are connected to, or any errors as a result of contamination from previous configurations and data. We ask that everyone who attempts to use this latest testnet iteration removes their $HOME/.safe folder before trying to interact with this testnet, i.e. $ rm -rf $HOME/.safe/ to ensure there is no contamination.
All of the bootstrap nodes for previous iterations that we had on Digital Ocean have been destroyed, with new, clean D.O. nodes created for this testnet iteration - please follow the full list of instructions below to remove all previous testnet settings and data from your machine and to get you started with the latest testnet iteration.
Are We Working on Other Fixes?
Yes. Multiple other fixes and improvements are already in progress, some even from before we released testnet v1.
We also anticipate that the feedback from this testnet iteration will direct us where else to aim resources as a priority.
Where can I report any issues found?
If you come across any issues in your testing, start by checking the Known Issues section of this post (see below) to see if it has already been acknowledged . If it has not yet been added, you can post your issue in the comments of this post. You can also report issues in the Online Spreadsheet for Testnet Results and Issues - see the section below.
We will monitor and investigate reported issues as soon as we can.
Online Spreadsheet for Testnet Results and Issues
If you will be uploading data to the testnet and/or providing safes (nodes), please consider posting your data and results at:
SN Testnet Review
(Massive thank you to @VaCrunch for creating this spreadsheet 
)
This is a new version of this spreadsheet, with the versions used for previous iterations now locked.
For those posting:
• No need to Sign In. The white cells are for your data entry.
• One row for each of your devices.
• Scroll down until you get to the first empty row.
• You do not need to use your Forum name for your ID, but please use the same ID for all your devices.
• Supporting/analysis tabs are at bottom of screen: Error_Msgs, Summary, Thumbnails, Charts, Matrix, Top 10, Map, Resources, Lists, and Comments.
• Use the View/Zoom feature at top to reduce the visible size of the spreadsheet to match your screen. 75% will be the best fit for most people.
• If you choose not to record the name of your country please select “Unspecified” at the bottom of the list.
• If you change the number of nodes (safes) you are running, just edit the existing figure, do not add another line for the same device.
Known Issues
Any reproducible issues that we, or the community, come across will be added to this section:
-
Although we’ve started making some optimisations, we still expect to see performance issues, particularly around
$ safe files put ....operations with larger files/folders. There has been very little optimisation of write times, nor any general analysis of speed so far, with the majority of our efforts concentrated on getting features in place and fully functional before we look at it from a performance perspective. -
We have removed
authdfrom the stack for normal users right now, instead focussing onSafe CLI. To that end, we have updated the CLI to be able to do all things that previously required authd. This means no GUI right now, mostly. The focus is on network stability and then speed. As we optimise for speed, the team will also focus on authd again, perhaps removing it (our goal) in favour of a simpler approach. So test as much as possible with the CLI for now. It reduces user error, reduces the potential bug surface, and allows total focus on the prize - a fully autonomous decentralised network.
Let’s try it out!
Please read and follow the instructions below carefully.
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Installing/Updating to Latest
To avoid any pollution from previous testnet iterations settings and data, we ask that everyone removes their $HOME/.safe/ directory, then installs the CLI, authd and node again.
$ rm -rf $HOME/.safe
Note that Windows users may be required to install specific software before being able to install the CLI using the command below - see full instructions for Windows here.
Now download and install the verified latest CLI binary via our install script with the following command:
$ curl -so- https://sn-api.s3.amazonaws.com/install.sh | bash
This script downloads the correct CLI binary for your OS (Windows, macOS or Linux), installs it in the correct directory, creating that directory if it doesn’t exist, and adds it to your system PATH.
You may need to restart your terminal window at this point for any changes to your system PATH to take effect. You can now confirm whether the CLI is installed and set up correctly:
$ safe -V
sn_cli 0.26.0
Next, you should install the latest node with the command below.
$ safe node install
Before proceeding we’ll just make sure we kill any old sn_node or sn_authd processes running on Linux or macOS:
$ safe node killall
$ pkill sn_authd
Or on Windows:
$ safe node killall
$ taskkill /IM "sn_authd" /F
Note - there’s no longer a requirement to install authd, so the steps for this are removed.
You should now be equipped with the latest CLI and node.
Note that you can find full installation instructions in our user guide:
Joining the Testnet
As with previous public testnets, we are hosting some elders and adults on Digital Ocean to kick off the Network. These act as hardcoded contacts which bootstrap you to the network, therefore you will need a network configuration file to inform your CLI which network to bootstrap to. We store and update these connection details on S3 for you to easily point your configuration to.
If you have followed the instruction above to remove your .safe folder before installing everything again, you should have no existing network configurations - you can confirm this with the $ safe networks command.
You can add a profile for the latest testnet which points to our S3 location, this is done using $ safe networks add like so:
$ safe networks add fleming-testnet https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config
Network 'fleming-testnet' was added to the list. Connection information is located at 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config'
Now you need to ensure that you are set to use this fleming-testnet configuration that we have added, we can use $ safe networks switch fleming-testnet for this:
$ safe networks switch fleming-testnet
Switching to 'fleming-testnet' network...
Fetching 'fleming-testnet' network connection information from 'https://sn-node.s3.eu-west-2.amazonaws.com/config/node_connection_info.config' ...
Successfully switched to 'fleming-testnet' network in your system!
If you need write access to the 'fleming-testnet' network, you'll need to restart authd (safe auth restart), unlock a Safe and re-authorise the CLI again
Now to get write access to the network, you can use the following command via the CLI before proceeding to uploading files or transfer testcoins with CLI:
$ safe keys create --test-coins --for-cli
We now have our CLI and node components up-to-date, and we have the latest hardcoded contact details to bootstrap to the public testnet, so everything is in place and we’re ready to launch our node and add it to the Network. This is achieved using $ safe node join as follows:
$ safe node join
Creating '/Users/maidsafe/.safe/node/local-node' folder
Storing nodes' generated data at /Users/maidsafe/.safe/node/local-node
Starting a node to join a Safe network...
Launching with node executable from: /Users/maidsafe/.safe/node/sn_node
Node started with hardcoded contact: 161.35.36.185:12000
Launching node...
Node logs are being stored at: /Users/maidsafe/.safe/node/local-node/sn_node.log
Your node will now launch and attempt to connect to the public network. Keep in mind that it can only join the testnet if the Network is accepting new nodes at that time (see NOTE below). You can keep an eye on its progress via its logs, which can be found at $HOME/.safe/node/local-node/sn_node.log.
If there is no space on the testnet for your node to join, as of Fleming testnet v2, your node will automatically try to rejoin until it is successful, or until you kill the sn_node process.
NOTE - please keep in mind that we have an anti Sybil attack feature in place now to only accept new nodes onto the testnet when resources are required, therefore you may be attempting to join the Network but your logs tell you that the Network is not accepting new nodes at this time:
The network is not accepting nodes right now. Retrying after 3 minutes
This is expected behaviour which will happen each time you try to connect until the testnet detects that resources are running low. As of Fleming testnet v2 your node will automatically try to rejoin in a loop, you do not need to attempt to join again manually, or via a script. We recommend keeping a watch on your node log file to check if you have been accepted - $ tail -f $HOME/.safe/node/local-node/sn_node.log
You can help to speed up the process of the network needing new resources by adding some data yourself - you don’t need to run a node to upload data! See the Do I need to run a node to participate? section below.
Before working your way through the CLI commands to perform various actions on the network, following the steps above gives your Safe some test Safe Network Tokens to use. This means that there is no need to farm first to earn rewards before being able to try operations such as uploading to the testnet. You can then proceed with the various CLI commands to perform operations on the network.
You can even connect to the testnet in read only mode using CLI, i.e. once you’ve installed CLI you can try fetching content uploaded by other users. For example, try download the following image and open it locally afterwards:
$ safe cat safe://hygoyeyx768nkst7qqjjk8khjm67tr1n4otbctcts5j7dten43zae3ga83y > ~/safe-the-planet.png
IMPORTANT - please be aware that this is a testnet and therefore any data added to it will be wiped once we are finished testing. We do not recommend uploading any important or sensitive information as IT WILL be lost.
Do I need to run a node to participate?
No, you can join with just the CLI and authd to experiment with data, tokens, etc. You can follow the instructions in the Joining the Testnet section above, but you don’t need to run $ safe node join - at this point, just go straight to using the CLI as per the User Guide.
Further Information
Where are my node logs?
When you launch your node you should see the location of your log file printed on screen - this will be $HOME/.safe/node/local-node/sn_node.log. You can tail your logs with a command such as $ tail -f $HOME/.safe/node/local-node/sn_node.log
Where are my rewards?
If you’ve started a node you will have keys generated for you, and stored in two files next to the logs of your node. You can check the balance of the rewards public key with the following CLI commands:
$ safe keys balance --sk $(cat $HOME/.safe/node/local-node/reward_secret_key)
The keys can then be used as per the instructions of the CLI User Guide for transfers, payments and more.
____
/ \ \
/ /\ \ \
/ / /\ \ \
/ / /__\_\ \
/ / /________\
\/___________/
off to the team. The concurrency of all the bits flying about is amazing. Ae/Dbc/register/multimap/nrs update/sn_fs and so much more. It’s all happening so quickly now and we are actually getting time to look at small changes across the board instead of total rework. Feels great.
