Sometimes when working with cutting edge tech, all you can do is wait for developments to happen elsewhere. Those advances can be a frustratingly long time coming, but this week we’re happy to announce a significant breakthrough on the networking front. We’re also making significant progress on DBCs, Tx fees, and the way they are handled by the elders. @oetyng explains more.
General progress
Lots of positives to talk about this week. First, some masterful detective work by @qi_ma 
has unearthed the root cause of a longstanding problem we’ve been experiencing with network startup, which was causing CI failures. Pleased to say that headache is no more.
Now onto something described by @dirvine both as “a game changer” and also, a little mysteriously, as “very relaxing”. So, what is it? Well this week, Protocol Labs, the team behind IPFS and Filecoin, updated their libp2p library. Libp2p has proved pretty successful at multiplexing and hole punching, but until now it has only worked over TCP. That has now changed and the library works with UDP and QUIC, which is what Safe uses to manage connections between nodes. Libp2p is used by Filecoin, Eth and Avalanche and has loads of excellent engineers working for it.
Even better, we get free hole punching, local node discovery, DoS protections and more. It looks like libp2p is now what it set out to be and what we were trying to create with qp2p: a solid library focused on p2p. With the recent inclusion of QUIC, we can see the libp2p team realised a lot of the work they did (stream multiplexing, noise for security, etc.) is covered in QUIC.
Anyway, great news all round. It should mean we have great stability in the networking layer at last and for the networking layer to work as we want it. So hats off to the libp2p folks 
With a little tweaking and a few PRs we should be able to finally replace Quinn and qp2p on Safe. The limitations of these libraries have been keeping David awake at night for many months. Finally, he can relax. 
@bzee has already managed to get a prototype Kademlia network specced out with libp2p and with a little more work that should be operational as a test. @Roland is also digging into the documentation to see if we might be able to leverage this.
@bochaco has added some CLI commands to node RPC to allow things like storage level checks, reward queries and wallet mutations.
And @oetyng has been on payments duty. More about that below.
Update on Payment Network
We have covered a fair few steps towards a payment network.
First of all, as some previously already found when running a local network, we had Clients query Elders for a fee (a hard coded value of 1 nano) and extend their transfer to also contain one DBC each for the Elders. That meant that when sending tokens, every input DBC that you spent would cost 7 nanos paid to the Elders.
But the payment was essentially a burn. Because the Elder could neither see that there was a payment to them, nor access it.
Verifying the fee amount
So, the next step was to introduce blinding of the fee amount, so that the Elder could verify that A. there was a payment there for them, and B. that it was a sufficient amount. Because remember, an outsider cannot see what amount a DBC contains, nor who it is for. We must have a way to tell the Elders these things.
So, Clients send a pair of ciphers - encrypted information - together with their spend request. When the Elder receives this, they can decrypt it and get the following:
Derivation Index
The derivation index is used to derive the public key (essentially the unique identifier) of the new DBC containing the fee payment. The Client derives that ID from the static public key the Elder sends, together with the currently required fee amount on querying the Elder.
The Elder uses this same derivation index to derive the corresponding secret key, with which they can unlock the value in the DBC containing the fee amount to them.
As this derivation index is sent encrypted to the Elder, no-one can tell that there is a payment to this Elder, and there is no way to see how much it was for.
Blinding factor + amount
Additionally, the Client sends the encrypted blinding factor and amount.
With the blinding factor, the Elder can blind the amount and compare it with the amount in the DBC transaction. This is because if the same blinding factor and amount are used, it produces the exact same unintelligible garble. This is why the blinding factor is also encrypted. Only the Client and the Elder will know the amount.
For more details on the above flow, you can read the comment section at top of this file: safe_network/mod.rs at main · maidsafe/safe_network (github.com)
Finally getting the fee
When a supermajority of Elders have signed the spend, they each send a SpentProofShare to data holders in the network. When querying for those shares, the SpentProof can be aggregated from these signatures, and DBCs corresponding to each of the outputs in the transaction can be created. An Elder can thus query the network for these, and then build the DBC which contains the fee payment.
From hard coded to dynamic fee
One nano per fee isn’t going to help much, so we dug up the calculation of required tokens to store some data that we already had in our archives.
The implementation calculates the required tokens for write operations, as a number of tokens to be paid for a certain number of bytes, given the current section prefix, the number of nodes in the section, and percent filled. This can also be used to calculate the fee of a transfer, as we just feed it with a fixed number of bytes.
Given the input parameters and the design of the curve, there is an effect on the price based on the supply and demand, albeit with some inertia. The value calculated is higher when there are fewer nodes in the section, when there is less space available, and earlier in the network. Earlier in the network? you might ask. Yes, it is assumed that a larger network means the token has a greater value since more people are sharing a constant number of tokens. Therefore, the required number of tokens per operation is also lower as the network grows. In other words, the price reflects the deflationary character of SNT.
Another property of the curve is that it is very flat at a low level until there is about 1/3 space remaining, whereby it starts to rapidly steepen. The reason is to have the fee stay as low as possible for as long as possible.
But, this isn’t a replacement for a true market. Even though we have some control over supply/demand it is fairly sluggish, and the fee amount based on that is therefore quite rigid. It’s therefore hard for the network to reflect actual changes in the fiat value of the token, and that can lead to large imbalances - which is never a good thing. Such imbalances could be way too many requests or way too few, because the actual value is not in sync with the value calculated from the network parameters.
That brings us to the option for a Client to prioritise their spend, and what that opens up for us.
Spend priority queue
A priority attached to a spend is a large leap forward towards a true market, where the supply/demand properties of the system are nimble and responsive.
This is done by Elders keeping a priority queue, ordered by the value of the fee paid to them to sign the spend.
The benefits of implementing a priority queue are the following:
- Clients can get a lower price for their storage if demand is low.
- Elders can get a higher reward for their services if demand is high.
- Incentives for node operators to run additional nodes increase when demand increases.
- The network becomes more responsive and can grow faster (attract more to run nodes) when demand increases.
- The continuous operation of nodes is more protected from overload of client spends.
- The load on the network is spread out over time as people defer spends at times of high load, and move them to times of lower load.
- There is potential for Elders to access their rewards faster, as they can gather them and reissue into a larger DBC when fees are low.
- …and more…
The first example uses a few discrete priority steps for a Client, to use from the wallet UI when doing a payment. This removes any need to manually set fee amounts.
The priority is translated into a fee based on the current fees in the queue. A high priority would mean that you pay a similar value to those higher in the queue, and vice versa for low priority. You could also set it to above the highest value, or below the lowest, and that’s where the pressure on price can start, based on supply and demand, and the needs of Clients and node operators to find an equilibrium.
By the choice of priority, the Client essentially decides how fast they want it to be processed. They can get it processed more or less instantly, or set the fee at a lower level to have it processed at a time when demand and fees are lower.
Note that there is still a limit to how low the Client can set the fee. Below that it would be dropped and an error returned to Client.
It should also be possible for a Client to update a pending spend, so it would not be stuck there if fees and demand remain at a higher level.
Summary
So, to round up, the basic idea is that spending a DBC can result in many output DBCs. When we send a request to Elders to spend a DBC, we also attach those outputs.
The Elders then verify the amount, and go ahead with processing the spend.
The observant reader would have noted that the fee paid to an elder is in a DBC, and when the Elder wants to spend this DBC, then it would take a fee to spend it, and if the two are the same… then you have… nothing?
That’s exactly how it is. And so, if we remember that the price algo returns lower values the larger the network is, we can see that there is a sort of lock-in + early bird effect. As the nominal value of a fee goes down, previous fees paid will become more and more “accessible”.
This is also where the priority queue helps. Fees that were received during high loads, could be collected and reissued when fees are lower. Another factor which works to even out the load.
Hope that gives you all a fair bit of insight into what is brewing.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
but noticed a few comments around the forum recently showing the team getting excited which makes me excited. I don’t know how this all works and never will but I know we all need the end product so keep it up guys 
)