Summary
Here are some of the main things to highlight since the last dev update:
- We’ve started running multiple internal testnets which bring together all the different components of the network. This is helping us track down issues, which we are working through.
- The authd, CLI and API work to bring them in line with the recent client & node changes relentlessly continues.
- We are experimenting with the metrics of a dynamic
StoreCostvalue, making the cost of writing data to the network take multiple factors into account. - We are homing in on a novel use of a CRDT in a Byzantine setting on a permissionless network - read the Routing section for further details on this exciting development!
Safe Client, Nodes and qp2p
Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan
The authd, CLI and API refactor continues. We now have the basic communication working between authd and CLI, with random keys being generated and passed between the two. The next step here is re-establishing the basic authenticator storage using our standard APIs (previously we had a bunch of specific APIs which added a lot of code complexity). So we’re working towards this now, which is also touching on some API use of various key types and making that more generic (so we can use either ed25519 or BLS keys, for example).
On the integration side, we’ve been pushing more fixes to sn_client and sn_node continuing from last week. With the section start-up more stable with all the updates to routing, we’ve been able to put all the modules together and see them working as one. A wee bug at Actor in sn_transfers was exposed during this internal testing, which on fixing would replace the aggregator we introduced last week as part of the AT2 transfers. Another major change that is coming is in the switch from static write cost to a dynamic StoreCost value. This means that the cost of writing data to the network would now be dynamic which takes multiple factors into account such as supply/demand of storage in the network, number of bytes to be written, etc. Clients would query the network for this amount before writing a piece of data, check for sufficient balance and pay the same. We are experimenting around the metrics for this to get started with a reasonable rate of change for the testnet.
qp2p has been getting its share of love as well this week. The async UPnP API changes have been integrated with routing and sn_node. This helped us identify a bug in the UPnP / echo service which wasn’t working as expected. The two were being done sequentially instead of in parallel and since UPnP was always failing in non-UPnP routers, the whole start-up was always erroring out. We’ve got a fix in the pipeline for this. As mentioned above, we’ve started with some internal testnets which include all the changes this week. This helps us put all the modules together in action and iron out the bugs that could have been missed in the unit / integration tests. If things go well (usual disclaimers apply!), we are on the right path to release a testnet for you guys to tinker with soon. 
We’re continuing a general transition to a messaging system which we saw first in sn_node a while ago. One aim with this is to break up the code in smaller, more cohesive, units of logic that are easier to reason about. Some of the other aims are to improve testing, separation of concerns, and to lay the foundation for a more natural parallel processing of internal tasks, depending less on shared mutable state. In sn_routing similar steps are now being taken, and we are looking further into where and how we can make use of these techniques to our benefit. Additional iterations in sn_node to move ahead some less prioritized parts of this transition will commence after the coming testnet. Hopefully that work will also be helped by the current work in sn_routing to make the interfacing between these layers, as well as the general flow in the system, clearer to newcomers to the code base.
Routing
Continuing with the refactoring work, this week we first got some work to add more unit tests merged to master. This brings back some of the unit functionality checks, and we intend to keep expanding these further. There was also a small API change to add data signing and signature verification public functions merged, as required by the upper-layer. To further improve the routing infrastructure, there is work aiming to remove SharedState and introduce Node, Section and Network modules raised and working its way through our code review process. This involves some object renaming (such as Node to Routing) and breaking complex struct (SharedState into Section and Network), which improves the crate’s readability.
Regarding the CRDT side, we are homing in on a novel use of a CRDT in a Byzantine setting on a permissionless network. This means the strict use and concrete definitions of Authorities. For us, that is Node, Client, Section and Network. To ensure irrefutability and fraud prevention we will digitally sign the Dot<Actor, u64> and the Operation itself (Add(actor)). This allows us to converge under periods of insanely high churn, which is something we hope never happens, but it can.
We also introduced a method akin to AT2 which is called DSB (Deterministically Secure Broadcast) and that may allow us to do something quite different from now and have the Adult running this DSB to get a majority to join the Elders CRDT. That offloads a lot of work from Routing. Additionally, we can see a majority have agreed on a new set of Elders (Replicas) and set off a DKG round to update the SectionChain.
The code is very simple, but the thought process is pretty deep and property-based testing is essential to confirm invariants under a full range of potential inputs. The strength there though is we have CRDT invariants and all we need to do is prove them.
All in all this is beyond exciting and should give us an incredibly solid Routing layer.
A subset of the above means all data can be BFT, plus have NetworkAuthority applied to allow it to republish (a user will pay for network authority).
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
can’t wait
