Summary
Here are some of the main things to highlight since the last dev update:
- Today’s expected testnet release has had to be put on hold as we couldn’t quite get rewards in on time, and disabling what we had in so far was still not cutting it.
- Being able to see network splits occur over recent days, with the latest developments in, has allowed us to find and squash some previously unknown multi-section issues.
- The
qp2pcrate API simplification work to manage the connections and streams internally and take some of the pressure off the API consumer, is now going through peer review. - A BRB PR to fix random packet loss that was interfering with BRB operations has been raised, tested, and is expected to be merged shortly.
- Special thanks to community members @bzee, @mav and @scorch for their efforts over the last week with multiple community PRs merged.
Testnet status - on hold
Our goal at the beginning of the week was to include rewards in a public testnet and open it up to the community today, however there were still several required changes to get it running, and we were not able to finish that for today.
Because of that we stripped out the rewards flow to see if we could get a stable enough network, and made some progress with that, but have been hitting bumps along the way which have slowed us down.
Our current status is that we now have a latest version of a testnet up internally but have not had any real time to test it. We expect that it is likely that there will be further minor issues so would like some time to test that out in-house before making it public with confidence.
We’ve yet to fully discuss and decide whether we will release a testnet with rewards excluded, or whether the issues we are coming across mean that we would be better off continuing on with fully implementing rewards, which we expect will take a few more days (plus testing). The stability of the testnet that we have up now, with rewards disabled, should help us decide but either way we don’t envisage putting up a testnet on a Friday, we would prefer to hold off until next week to allow us to fully support it.
The vast majority of our time this week has been focused on the testnet, but here’s a summary of some of the general work we’ve been doing…
Safe Client, Nodes and qp2p
Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan
First off this week, a PR from @bzee which enables programmatic configuration of nodes has been merged to master today. Nice work! 
We’ve created a new Signing trait in sn_data_types to abstract out sign and verification for our various flows in nodes/transfers, and have been testing and bug-fixing some flows centered around the integration of routing’s new infrastructure queries. This has moved the bootstrapping operations into routing itself, as opposed to within nodes. Previously, we were limited to querying only elders (and elders of our own section at that) for network bootstrap info. Moving this to sn_routing means we can query any node, and get back the correct contact info… essentially enabling clients to operate against a multi-section network.
Beyond that, we’ve been debugging various issues with section splits now that we’ve been able to see that in action (albeit limited due to aforementioned issues), and @mav’s node-logger web app has proved to be a great wee tool for quickly seeing cross-node flows and narrowing down the log output of our multi-section network into something a bit more human readable.
We are also working on the data catchup flow that is required for newly promoted Elders in a section, who would need to update themselves with all the data that the rest of the Elders maintain. Since this data would be relatively big in size, we would ideally want to be streaming the data so that the node doesn’t get blocked with that task, therefore we’ll also be tinkering with qp2p a bit to enable streaming data directly to nodes on request.
Over the last week we have made some additions to the qp2p crate to manage the connections and streams internally to take some of the pressure off the API consumer. This work is completed and currently going through review in this PR, with the new API has also being integrated with routing with all the tests passing. We were holding back for a public testnet iteration to be released before merging these, but with the delay we may decide to merge beforehand. Testing has shown that this change has made things considerably simpler and this will make the debugging of any further p2p networking issues much simpler.
API and CLI
Several more community PRs were submitted in this past week, which were merged and are already part of a new version of CLI, just published today (v0.18.0).
Thanks to this PR submitted by @mav, the NRS validation is now excluding a bigger set of characters which would never have any reasonable purpose being included in any URL. Having a limit of total NRS URL length of 255 bytes, where each subname cannot be more than 63 bytes length, is now being enforced with this other PR.
The safe auth CLI commands now give the --config argument priority over the use of environment variables, this is also now in place in the new release thanks to PR #700, also submitted by @mav.
On the qjsonrpc side PR #698 submitted by @Scorch implements a basic client/server example application using the qjsonrpc API. A qjsonrpc client sends a ping message to a qjsonrpc server, and the server responds with an ack message. This example application simply and neatly shows how qjsonrpc can be used to build any application using JSON-RPC over QUIC. There has been a lot more activity and exploration made by @Scorch with more ideas around qjsonrpc - for those interested in more details and/or in participating in this effort, take a look at its own dev thread where he has been sharing all of it. 
BRB - Byzantine Reliable Broadcast
We raised a PR in brb_node_qp2p to fix random packet loss that was interfering with BRB operations. This makes the node stable in case any technical community members would like to interact “hands-on” with BRB in a CLI environment. Basic usage instructions for doing so are here (Note that you must build the code yourself, for now at least).
Design discussions for integrating BRB with BLS Distributed Key Generation (DKG) have continued with several options presented. Agreement appears to be coalescing around one of the options. Details on these plans should be forthcoming in future updates as work commences.
Routing
We’ve made a first step into moving all message definitions out of sn_routing into the sn_messaging crate, where not only the message definitions will reside, but also any logic related to the serialisation of them. The initial iteration of this, along with having a unified type of message for clients and nodes connecting to a section, is now in place in the latest versions of sn_routing and sn_client. Our next step in this regard is to finally move all messages types definitions onto sn_messaging crate, allowing us to decouple the actual sn_routing logic from what would become the API of the network.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French ; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
FYI that was about a mega realisation I had today - how staggeringly big a deal Safe Network is going to be for developers (
is brilliant too. As always, thank you MaidSafe team even though you were trying to make me lose the bet with 
, trying to create a GitHub alternative and even though there are lots of economies doing this on Safe Network (see 
.
