On this, our last update of the year, we’d like to wish you all the very best for Xmas and the New Year, have a great break, enjoy yourselves , and thanks for all the testing! We’re really pleased to go out on a high with a stable testnet which so far at least is was behaving exactly as we’d hoped [since this was drafted we’ve seen things fall down!]. No doubt saying that will put a curse on it and send it crashing down [yep], but at the moment we’re looking really good. If time permits we’ll put up another one before the end of the year which will allow joins again.
General progress
Mostafa has now finished integration of the consensus protocols we mentioned a few weeks back in the context of membership and handovers. Just testing remains.
@qi_ma has come up with a canny way to separate data replication into batches to avoid the memory spikes that can crash nodes. This seems to be working nicely in our testing.
@chriso has been beavering away on the release process to make sure the testnets go smoothly and @roland is looking into how we can improve observability and tracing with passing logs to the elastic server.
Meanwhile @bzee continues to battle with qp2p and @bochaco is still poking at some edge cases around messaging and connectivity.
Smaller nodes
One issue the previous testnet identified was a bottleneck in replicating data. With large nodes and lots of data, as/when we gain a node or lose a node, the data must be moved around the network. This means we cannot churn faster than we can replicate data. The larger and therefore fewer the nodes, the more this becomes a problem. Internally, we are setting up tests with smaller nodes, say around 1 GB. A node that’s fifty times smaller should have fifty times fewer replication tasks to perform alleviating the bottleneck, thus the load is better spread. This might mean having larger sections (maybe 200 adults per section), or just more sections.
This coupled with the updates to the data replication flow have been looking positive so far.
What would this mean in practice? Well, operators will likely run several nodes rather than one, which would be positive overall as those nodes will be in different address spaces in a section, and when the network grows (it will grow fast), the likelihood is that eventually each node will live in a different section. This means the data will be replicated in 1GB chunks across the network instead of having 50GB replicated in the same section of the network. So, this approach will be more decentralised as well as better at balancing the replication load. There is also likely to be more of a role for elders storing data (currently they don’t) to help in the early stages of a section when we are still busy adding adults.
We are testing this approach internally now with 1GB nodes and if all goes to plan (looking good so far) we should be able to put a public testnet together in the New Year some time. Until then most of the guys are enjoying a well-earned break, although MaidSafe being MaidSafe, there’s always someone, somewhere, tinkering with a new idea or getting stuck into an optimisation. They’re not a put-your-feet-up kind of a crowd.
Take it easy everyone, spend some quality time doing what you enjoy doing. See you on the other side
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; German ; Spanish ; French; Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!