“Say hello, wave goodbye” warbled Soft Cell’s Marc Almond back in the 80s, and how prescient he was. This week we’re delighted to welcome Mo as the newest member of the MaidSafe engineering team. Mo is a computer scientist and a software developer. He achieved his Ph.D. in Computer Science at the University of Potsdam in 2020 and is interested in researching, programming, and security testing for distributed services and standard software systems.
At the same time, we’re sad to wave goodbye to @yogesh, who has been on the team for a number of years now and has really left his mark, applying his forensic problem-solving capabilities and eye for detail across all aspects of the network. Ever the professional, Yogesh’s most recent work was to make sure the ELK server was running just the way he wanted and completing all the documentation. All the best Yogesh and enjoy your new challenge.
General progress
@joshuef has reworked some of the CmdCtrl module to remove the last remaining lock outside of the Node instance itself. This has had a nice knock on in terms of code simplicity, which is always a win.
@qi_ma is now looking into a possible memory leak on a node joining as well as investigating why some messages are handled very slowly at nodes.
@roland has pretty much finished refactoring the section chain as a Merkle tree and is now updating the API to make it as ‘drop in compatible’ as possible.
And @anselme has fixed a threshold issue in sn_sdkg, the synchronous DKG crate. We now have DKG working pretty well, including with section splits. Integration of this continues!
@bochaco has spotted some issues with the register storage implementation, which meant that concurrent writes to the same data may lose some operations, and in fixing that, we’ve also spotted another issue that may have been plaguing main, whereby similar xorname registers might have been overwriting each other too. A fix is now in place for this.
Progress report
Messages and memory spikes
We were seeing a flurry of messages and surge in memory use particularly when a new node was contacting elders and asking to join.
Progress: Much improved. There are still a few situations that lead to a leak and those are debugged as we come across them. They are much less frequent now and much easier to track with the ELK dashboard.
Todo: Squashing bugs when we see spikes.
ELK dashboard
ELK (Elasticsearch, Logstash and Kibana) is a stack of open source tools for aggregating and visualising log files. Later Beats, which collects metrics from CPU and elsewhere, was added, but BELK sounds stupid so the name ELK remains.
Progress: Done. We can now monitor testnets and see what each individual node is doing. More here https://github.com/maidsafe/sn_testnet_tool/tree/main/scripts/ELK
Todo: Figure out if/how we can securely open this up for all to see.
Data handling
Getting section membership sorted was necessary for proper data handling, redistributing chunks when an adult goes offline, and gracefully handling splits.
Progress: This is done and looks much more solid.
Todo: Awaiting the completion of Membership so it can be completed. More tests are being added to shore up the underlying storage layers.
Handover
If an elder notices that the current elders are not the seven oldest nodes, then it sparks a vote on promoting the oldest adult(s) and demoting the youngest elder(s) to make way. Handover is the algorithm that controls this process.
Progress: Completed, although it is being refined with gossip – below.
Todo: Look at adding gossip as a fallback.
Membership
This is what allows elders to keep control over the adults in their section.
Progress: Membership is now integrated into sn_node and uses consensus to force the elders to agree which new nodes will be able to join the section.
Todo: Get adults to have the same view of members as elders. Many of the decisions that adults have to make, e.g. what data they are responsible for, is based on the current membership state, but currently they don’t have the same view as the elders.
Gossip
Gossip is a type of p2p protocol where a node periodically broadcasts knowledge to another node, often selected at random, and that node passes the message to another node, and so on until it becomes ‘common knowledge’.
Progress: We are introducing gossip at various places where the processes can sometimes get stuck, such as DKG rounds, membership and network splits. In the case of the latter, gossip also allows us to reconstruct the network after a catastrophic failure.
Todo: Begin implementation.
DKG
Distributed key generation is used when elders need to make decisions about membership.
Progress: DKG has been revamped with a cleaner, timer-free implementation and consensus to fix cases where DKG was inconsistently failing and retrying due to the use of timeouts when waiting for messages. Testing is showing positive results.
Todo: Fix a bug in the consensus process that sometimes prevents termination. Look at adding gossip as a fallback communication layer.
Payment flow
When the client pays for an upload it gets a receipt and the elders and adults in the sections that store the data receive some part of that payment. Payment flow is about how that distribution works.
Progress: We are working on the algorithm to manage payment flow. At the moment the thinking is to include all the beneficiaries in the receipt and have them able to claim the funds by having a DBC reissued to themselves at any time. Recipients can potentially be clients, adults and elders.
Todo: Detailed design to cover security, UX, economic incentives. Testing.
Token emissions
After the initial distribution of tokens at the inception of the network, the remaining 70% of the total supply will be created and securely distributed as a consequence of people uploading data over the long term. SNT are not created every upload, but rather when certain conditions are matched. These events must occur with a known probability, but be unpredictable as to when and where they will occur.
Progress: We are currently considering mechanisms that will reward elders, storing adults and clients while making mining by rogue nodes uneconomic.
Todo: Settle on one option and test it.
Node refactor
sn_node has been refactored to remove as many instances of multithreading as possible in order to simplify the code and remove unwanted locks.
Progress: Pretty much done. Code is vastly simpler, performance is improved and only one node lock remains around storage, which is fine. There are a few others around the backpressure feature, but that’s not used by default.
Todo: Keep monitoring performance, edge cases.
Section chain refactor
Section chain, the secure linked list that connects section keys all the way back to Genesis is being refactored as a Merkle DAG. It’s now called SectionsDAG.
Progress: Mostly done.
Todo: Complete the refactor.
Dysfunction
Regularly testing that nodes are online and taking action if they are not.
Progress: Refactored with its own test suite, awaiting progress in other areas.
Todo: Will be continually revisited as other components mature.
DBCs
Digital bearer certificates are the vehicle for transacting tokens on the Safe Network.
Progress: DBC integration and the basic functionality is largely there. Completing the basic functionality is the next step, in particular getting it working with the payment flows, which is far more solid than it was six months ago.
Todo: Improve APIs and the UX around this. Work the DBC APIs into the payment flows and testing there.
Name Resolution System and domain names
NRS maps are multimaps (registers under the hood) that contain the mapping of all the subnames to URLs for a specific topname. Each topname has its own multimap.
Progress: API was refactored to simplify the code. We’ve decided on the topname.subname (google.maps) format as the most intuitive.
Todo: Changes to NRS ordering to be implemented.
Client / UX
sn_client is the set of APIs and the command line interface (CLI) that allows devices to connect to and interact with the network.
Progress : sn_client has been refactored - fixing messaging which shows users whether they are connected or not, especially during bootstrapping.
Todo: Continuous improvement of the APIs and CLI as we move forward.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
adios 
