Time for a quick review of where we’re at with the various moving parts that make up the Safe machine. As usual, it’s a mixture of rapid acceleration in some areas, with others waiting for progress elsewhere before they can get moving.
Overall, we’re excited by the simplification possibilities offered by the stable set idea, very happy about the establishment of the Foundation, and pleased with progress on consensus. It’s also great to have community testnets back up and running, and to have folks leaping into action with their own GUIs 
.
General progress:
We’re always looking at ways to get the client to do more work and @oetyng is investigating how it can keep the elders in sync with what’s happening in the section (along with AE, gossip etc).
@anselme has been refactoring DBCs and is integrating his changes into the main code, and @Chriso is finishing off the monitoring stack implementation.
@roland has refactored the relocation process into having the relocated node drive the process.
Mostafa and @davidrusu are getting stuck into the stable set, working on a handover protocol that does not require DKG. Mostafa is also looking into the binary compilation process to try to speed it up.
Also on the stable set, @joshuef is confident we have a path to tiered storage with a few extra changes.
Qp2p is still undergoing surgery with @bochaco testing for anomalies around reading and writing streams, while @bzee has moved onto analysing logs in relation to delays on the network when nodes are stressed.
Finally, @JimCollinson has been working on various legal bits and bobs for the board, and also considering launch strategy.
Progress report
Stable set
Treating the elders and the oldest adults as more trustworthy than younger nodes, not requiring them to relocate for node ageing, and introducing tiered storage. Could well remove the need for DKG and relocation, which are both difficult.
Progress: Design stage, testing assumptions, still experimental.
Todo: Complete tests, if successful, work on implementation.
Consensus
Many processes can be handled by AE and eventual consistency (CRDTs) but in some cases, some local ordering is likely required as forks could make double spend possible while we wait for them to resolve. https://github.com/maidsafe/sn_consensus
Progress: Multi Values Byzantine Agreement (MVBA) algorithm has been developed and tested.
To do: Apply to membership algorithms. Trials in live environment.
Quicp2p
Qp2p is a networking library based on QUIC that enables peers on a P2P network to communicate securely. https://github.com/maidsafe/qp2p
Progress : qp2p is being refactored with the aim of bringing most of its functionality into sn_node where it will be easier to work with. Quic is a black box.
Todo: This work is ongoing.
Messages and memory spikes
We were seeing a flurry of messages and surge in memory use particularly when a new node was contacting elders and asking to join. https://github.com/maidsafe/safe_network/tree/main/sn_comms
Progress: We’re mostly not seeing memory spikes any more. Messages are a central part of the way the network works so we’re constantly refining things there. We’re working on better backpressure leveraging Quinn’s congestion control more reliably.
Todo: Squashing bugs when we see them, continued simplification.
Monitoring and observability
We have moved from the ELK setup we were using previously to OpenSearch (a more modern open source fork of ElasticSearch), OpenTelemetry and a Data Prepper, because we can host those services on ECS rather than using EC2 instances. https://github.com/maidsafe/terraform-testnet-infra
Progress: This is pretty much there now, although we’re still looking at the best configuration.
Todo: Finish up, then open it to the community, where feasible.
Data handling
Data handling, redistributing chunks when an adult goes offline, and gracefully handling splits.
Progress: Again, this is a core part of the functionality and work is ongoing, see stable sets above.
Todo: Establish tiered data. Groundwork has been laid, we’ll need to test things out atop of covering relocation and fine tune.
Handover
If an elder notices that the current elders are not the seven oldest nodes, then it sparks a vote on promoting the oldest adult(s) and demoting the youngest elder(s) to make way. Handover is the algorithm that controls this process.
Progress: Completed to plan, although it is being refined with gossip, consensus (see below).
Todo: Introduce gossip to speed communication between elders, and look at consensus to ensure processes terminate.
Membership
This is what allows elders to keep control over the adults in their section.
Progress: Membership is integrated into sn_node but remains the hardest area to fix in ensuring each elder converges on the same view. It’s a coordination problem.
Todo: Remove the dual view of membership we have between SectionPeers and Membership. Work has started here to reduce this down to SectionPeers and is progressing well.
Gossip
Gossip is a type of p2p protocol where a node periodically broadcasts knowledge to another node, often selected at random, and that node passes the message to another node, and so on until it becomes ‘common knowledge’.
Progress: We are introducing gossip at various places where the processes can sometimes get stuck, such as DKG rounds, membership and network splits. In the case of the latter, gossip also allows us to reconstruct the network after a catastrophic failure.
Todo: Look for further opportunities where gossip may help us.
DKG
Distributed key generation is used when elders need to make decisions about membership.
Progress: DKG has been refactored but still occasionally suffers timeouts.
Todo: Looking to reduce the need for DKG as in decision making.
Payment flow
When the client pays for an upload it gets a receipt and the elders and adults in the sections that store the data receive some part of that payment. Payment flow is about how that distribution works.
Progress: Basic design is in place.
Todo: Detailed design to cover security, UX, economic incentives. Awaiting other developments.
Foundation
The Safe Network Foundation is a not-for-profit in Switzerland to facilitate, support, fund and promote research and development, as well as enabling the ongoing delivery, maintenance and adoption of the Network and its technologies.
Progress: The Safe Network Foundation has been established!
Todo: Put it to work!
Safe Network Token Distribution
After the initial distribution of tokens at the inception of the network (10.6% to MaidSafeCoin holders, 5% to shareholders, 14.4% to network royalties pool), the remaining 70% of the total maximum supply will be created and securely distributed as a consequence of people uploading data over the long term. https://forum.autonomi.community/t/updated-rfc-0061-safe-network-token-distribution/37883
Progress: Initial proposal drafted and submitted to FINMA for approval. The mechanics of which node operators will be paid during a particular transaction are still being worked out.
Todo: Design and test node payments.
DBCs
Digital bearer certificates are the vehicle for transacting tokens on the Safe Network.
Progress: DBCs have been refactored removing a superfluous element supposed to obfuscate transactions but with dubious capability to add to the security, and improving performance. This work is nearing completion.
Todo: Improve APIs and the UX around this. Work the DBC APIs into the payment flows and testing there.
Node refactor
sn_node has been refactored to remove as many instances of multithreading as possible in order to simplify the code and remove unwanted locks. Elders now store data so they can estimate storage levels without polling adults, and we’ve been looking at having more smaller nodes in a section, which has been successfully trialled in testnets.
Progress: Done
Todo: Keep monitoring.
Section chain refactor
Section chain, the secure linked list that connects section keys all the way back to Genesis is being refactored as a Merkle DAG. It’s now called SectionTree.
Progress: Done.
Todo: Nothing.
Fault Detection (previously dysfunction)
Regularly testing that nodes are online and taking action if they are not.
Progress: Refactored with its own test suite, awaiting progress in other areas.
Todo: Improve testing with live networks and ensure things are working as expected. Add more fault detection routes.
Name Resolution System and domain names
NRS maps are multimaps (registers under the hood) that contain the mapping of all the subnames to URLs for a specific topname. Each topname has its own multimap.
Progress: No movement on this.
Todo: Changes to NRS ordering to be implemented.
Client / UX
sn_client is the set of APIs and the command line interface (CLI) that allows devices to connect to and interact with the network. https://github.com/maidsafe/safe_network/tree/main/sn_client
Progress : sn_client has been refactored - fixing messaging which shows users whether they are connected or not, especially during bootstrapping.
Todo: Continuous improvement of the APIs and CLI as we move forward.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
and I very much like the [content warning for Willie] bite size summary format with Progress and Todo sections. Looks fairly easy to maintain/update week by week too.
