There are no secrets in open source. As some of you spotted, the old config file for connecting is no more, replaced by PrefixMap. So what’s going on? @roland is here to explain more.
Another thing that’s headed for the exit is the unmaintained and buggy sledDB. But you also knew that already. @yogesh explains what is replacing it, and why we are avoiding the database option for now at least.
General progress
The whole team is engaged in simplifying the code by stripping out unneeded multithreading. We’ll refer you to @joshuef’s post here to explain why. The good news is, removing this cruft has already been highly positive in terms of performance.
@bzee has developed a new tool that lets us inspect a chunk and see where it’s stored on the network and with which adult. We hope to get this out to the community soon for testing.
Also on observability @joshuef has been digging into heaptrack, which profiles memory use. Memory is generally much better now but there are still occasional spikes.
@bochaco and @chriso are ploughing ahead with DBC integration, and we hope to have the next iteration ready for community testing very soon - thanks a lot for your help with this so far guys 
And @heather_burns is in the media again, this time talking about the UK’s Draconian (and now thankfully delayed or possibly even abandoned) Online Safety Bill.
PrefixMap
We have replaced the old config file with PrefixMap, partly as a simplification of the codebase, partly to optimise the connections and messaging. Config contained IPs and ports of nodes to connect to, plus the genesis key of the network. The PrefixMap contains all of the above plus the section authority provider (SAP), which provides a list of all the current elders in a section and the current section key. Moreover, since the genesis key can be provably linked to the section key in the SAP, the client can verify it is in the right place, in the right network and everything is up-to-date (via anti-entropy).
This also allows the client to home in on the nearest elders straight away, so we can avoid hammering all the elders in a section for every request.
For new nodes joining the network, the PrefixMap provides an overview of the network topology which it can update as it needs to, again, skipping a few interim steps.
@roland explains the background.
The config file, which contained the network’s genesis key and some bootstrap IP addresses was the way we connected to any public/private testnet. But at the same time, we were also maintaining the PrefixMap, a structure which contained the latest SAPs of the sections we are aware of. Since the SAPs contain the IPs of all the elders, we can use this to bootstrap to the network instead.
This is advantageous over the config file because we are not hammering the fixed set of nodes used to bootstrap to the network. Instead we are able to query the PrefixMap to get the closest SAPs to a given XOR-name and use these to bootstrap into the network. This balances the load while bootstrapping.
We also plan shortly to move the SectionChain into the PrefixMap which will resolve some issues in the client, which currently has to go through the anti-entropy (AE) update process to ensure it’s up-to-date, which can add up to a huge amount of AE messages. This is a work in progress.
How does this change the Comnets?
These changes should not affect the running of local and community testnets (Comnets), although you will notice a few small differences.
- The only new process is that we will be using the PrefixMap to connect to local/public testnets instead of the
node_connection_info.configfile. - We use the Genesis Key of the network to name the PrefixMap files, hence they might look something like this
PublicKey(0123..4567)
Connecting to the Comnet:
- Make sure you do not have the old config file
~/.safe/cli/config.jsonand old prefix maps inside~/.safe/prefix_maps/. You can clear out the entire.safedir usingrm -r ~/.safe/ - You can add the Comnet to your networks list by providing the URL/path of its PrefixMap, as provided by whoever sets up the Comnet (e.g. curl https://).
Use the following CLI command to do so,networks add my_comnet "<path/url>" - You can then switch to the added network using
networks switch my_comnet
Running local baby-fleming testnet:
- Make sure you don’t have the old files, as stated above.
- From the CLI, install the latest version of
sn_nodeusing the commandnode install - Use
killall sn_nodeto stop any running instances ofsn_node - Run the testnet using
node run-baby-fleming - You can now use
networkscommand to view all the networks that are available to you and usenetworks switch network_nameto switch between them
Note: The CLI uses the ~/.safe/prefix_maps/ directory to update the list of networks and select the current network it’s connected to, hence renaming files inside this directory can lead to undesirable results.
Ditching sled
The final piece in the sled removal process, storing registers, is now almost in place and the buggy and unmaintained sledDB will shortly be fully replaced with our own in-house disk storage. As we said last week, this is a keep-it-simple implementation with no bells and whistles that performs on par with the other alternatives, while freeing us from an external dependency at this stage. There’s nothing to prevent us looking at other databases down the line when we need one.
We’re confident that this change will remove one of the common causes of testnet failure. But before we get too excited, it’s probably not the only one, and we’re looking into Membership to see if perhaps chunks are sometimes being sent to the wrong adults.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
