Update 28 April, 2022

Development Updates
1

This week, with some major hurdles now behind us, including membership implementation, handover consensus alllmost merged and the squashing of the memory spiking bugs that were plaguing the testnets, we thought it would be good to have a look at what we’ve achieved and what is still left to do.

General progress

DBC integration continues to go well. We’ve more client side improvements in place, and the start of some POC node-side DBC storage also on the go. Plus a messaging PR in the works to link all it all up.

The Handover consensus work is mostly there, which provides more consensus backed logic to section splits and elder elections (we’re seeing sporadic issues on CI so digging in there now). But the handover code, coupled with some changes tying DKG to specific generations, seems to have improved section startup quite a bit, which is great.

We’ve continued some ongoing refactoring work, organising the safe_network repo, and we’ve started looking at what “light” nodes look like, and what a node running bare “network system” code looks like. The reason for this was to help debugging by having a mode modular system where we could, for example, turn off data and only run dbc code, or turn it off altogether. Our testing there was encouraging - with data turned off, nodes were topping out at around ~45mb memory, and we’ve happily had ~90 running on one machine with several section splits :tada: .

Progress report

Section membership
This is what allows elders to keep control over the adults in their section.
Progress: Now integrated into the network code
Todo: We’re still looking into distributed key generation (DKG), the mechanism by which elders reach agreement on the new section key, to see if there are perhaps other routes here which could remove the requirement for consensus altogether.

Data handling / handover
Redistributing chunks when an adult goes offline, and gracefully handling splits.
Progress: Mostly done but we’re still seeing some failures due to DKG not working as we’d like
Todo: Fix DKG issues; Caching on elders

Messages and memory spikes
Aside from inter-elder communications, most messaging is handled by anti-entropy (we reply with missing information so a node can update and retry its message if needs be, eliminating the need to cache messages), and qp2p.
Progress: Messaging spikes fixed
Todo: Optimising AE

Liveness testing
Regularly testing that nodes are online, and taking action if they are not. This has been expanded to penalise nodes not only for dropping chunks, but also for dropping connections and being behind in terms of network knowledge, so we should hopefully catch more dysfunctional nodes earlier, and increase the network health overall.
Progress: First pass: Completed!
Todo: Improve dysfunctional handling through proptesting.

Name Resolution System and domain names
NRS maps are multimaps (registers under the hood) that contain the mapping of all the subnames to Urls for a specific topname. Each topname has its own multimap.
Progress: API has been updated
Todo: Still deciding what to store as entries in the NRS map. The current (key - value) mapping is “subname” → Url, but for a better UX might be helpful to store the topname along with the subname as key in the map, like: “subname.topname” → Url. On the backburner for now

Monitoring / visualisation
ELK stack allows us to monitor and visualise the performance of all nodes in a testnet.
Progress: Completed. Closed testing underway (and already showing benefits, we caught this bug because we could clearly see one node spinning its wheels.
Todo: Shore up the deployment of ELK nodes and surrounding security so we can open this up to the community.

DBCs
DBCs are the mechanism by which transactions are made on the network.
Progress: Now being integrated into the network
Todo: DBC messaging; implement ordered DAG for auditing purposes; testing on testnet/live network. Implement data payment flows.

Wallet
The wallet is the user interface for carrying out transactions.
Progress: First and basic implementation of wallet reissue API and CLI commands completed
Todo: More work needed to get it production-ready; test with DBCs on the network

Payment to farmers
Adults and elders will be paid for their services by the network. There are some designs on the table, but they await the implementation of DBCs.
Progress: Basic design ideas
Todo: Implementation and testing when other elements allow

Users paying for storage
Similarly, we have some designs in place but await DBCs. Current thinking is for clients to use DBCs to pay for storage upfront, with the elders quoting a price to sign the names of those chunks, making them valid network data. Mutable data will need to be treated differently.
Progress: Design ideas
Todo: Implementation, testing, UX testing

Front end / UX
We have revealed some wireframes and designs for authentication / UX over the last few months, but obviously, those are the icing on the cake, and the cake’s not baked yet.
Progress: Basic look and feel, user authentication, lost credentials strategies, and other designs are done
Todo: Backburner for the moment; implementation when other elements ready, user testing

Governance
Foundation is being set up in Switzerland as a governing body separate from MaidSafe.
Progress: Going smoothly so far; authorities have requested documentation
Todo: Finish refreshed white papers; further discussions over details

Documentation
White papers covering the project from different angles, technical documentation.
Progress: New white papers on Token Distribution, Network Design, Token Design and Project Governance in draft form
Todo: Finish white papers; NRS docs, DBCs, etc

Licensing changes
License the core network under GPL3 with non-safe network crates being licensed under MIT/BDS so as not to limit client apps that can be built on it.
Progress: done, though automation could be in place
Todo: automate licensing management

eMaid
Community effort to optionally allow conversion of Omni Maid to ERC20 Maid.
Progress: Completed!

Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

65 Likes
4

Umm… First?

And sounds like really great progress!

26 Likes
5

Exciting to see the update formatted in this way! The context of what’s dije, in progress and to-do is very helpful. :smile:

32 Likes
6

Great update :fire: thank you.

Is that Davids dog gracing the cover?

ps. @19eddyjohn75 I think you meant to post in what’s up today :wink:

22 Likes
7

There seemed to be a problem around routers too?.. I don’t know if that became a complex issue for later on or is fixable but many testnets seemed only accessible to some?.. looking forward to having time again to play but for now very encouraged with these updates.

Thank you for all the hard work :+1:

20 Likes
8

Excellent update - thank you to all involved. It’s good to see just where we are on each area of work. And its even better to see progress across the board. I’m looking forward to the upcoming testnets and playgrounds.

Not unless he takes the dug abroad, cos thats not Ayrshire (nor anywhere else in Scotland) in that photo. I suspect its beautiful Bulgaria? @dimitar will know. Wherever it is, I’d like to visit it. Is there any fish in there?

15 Likes
9

Nice summary. Keep up the good work. :nerd_face:

10 Likes
10

Thanks so much to the entire Maidsafe team for all of your hard work! :racehorse:

10 Likes
11

Can you say more? In another update or comment, it was mentioned that the team was also looking at potentially getting rid of DKG altogether. Is this what is meant by “see if there are perhaps other routes”?

As far as I know, since the elders need to collaborate and share a common secret key yet have their own respective public keys, DKG will be required. The question is what sort of DKG.

Which aspects of the current DKG implementation are troublesome? If memory serves, it is a multiphasic DKG with complaint rounds, which can be inefficient. Are the issues being experienced related to that? Could something like aggregatable DGK using gossip (https://eprint.iacr.org/2021/005.pdf) help? Or are the issues elsewhere?

9 Likes
12

If we get the handover PR nailed down with these sporadic issues, DKG is already muuuch less troublesome.

That’s a fair summary. It’s not wiildly bad with the race conditions removed (with the handover work in), but it may just be another bottleneck we can get rid of. That’s kind of where the research is looking. Is there something viable/worthwhile… how could we perhaps do consensus without a shared section key (just counting eldest node votes eg). Still very early days though!

13 Likes
13

Nice progress
There is just not mentioned official support of eMAID to SNT swap so might be next week.

eMAID support official statement
eMAID support official statement754×456 61.1 KB

3 Likes
14

This is one of the Seven Rila Lakes. Even if there are fish, it is forbidden to fish - Bear Grill ate a frog there and he was driven out of the park.

Privacy. Security. Freedom

7 Likes
15

Thanks for answering, @joshuef.

But DKG would still be required to establish section keys for section-signed data, DBC mint/spentbook signatures, etc. There’s an aspect of how often DKG would happen (which might bring about its efficiency aspects) and that relates to how often elders change (1>1) or sections split (1>2). So I still don’t see how you can do without DKG given that a secret needs to be shared between elders.

Put differently, the DKG question is separate from the consensus question for handover. DKG (or something like it to establish shared keys) is still required whether it involves consensus or not.

Am I missing something?

6 Likes
16

Im not going then - I always like to (try to) haul a fish out of a loch and grill it. With a beer.
Which is weird cos most folk use a fire of some sort.

3 Likes
17

My bad, I forgot to mention this. We will do it next week.

13 Likes
18

One approach as per the old data chains is to not have a single section key, but a set of all the existing keys.

There are some advantages to that

  • Nodes keep their name between churns (i.e. nodes don’t need to change their name when another node churns, which is nice)
  • clients/nodes half way through getting sigs don’t have to restart their BRB round

Disadvantages

  • much larger history / chain (although we can aggregate BLS keys into a single sig)

We feel; we have BKG under control now, but still considering the options of no dkg but perhaps still, use aggregatable sigs/keys

12 Likes
19

Much much larger? So large that there is a significant performance hit? Cos the advantages seem compelling.

7 Likes
20

Potentially 7X larger, but still working on that part as it’s likely this can also be reduced to a single key plus sig. Then it matches the size of the section chain but requires the current set is calculated from previous state changes. If we get anywhere we can test it easily enough IMO

13 Likes
21

That’s great.

With generation numbers as a tag, nodes also shouldn’t need to change their names between churn.

I like this. Some time back on this forum, I actually mentioned the DKG employed by DFinity as something that would confer this same advantage (in 1>1 handover situations) because the public key doesn’t change even though the secret key does change.

Which ever way this is sliced, I still don’t see how a DKG would not be employed. Even with unchanging public keys, shared secret keys need to change. That requires some form of DKG.

In any event, you might find aggregatable DKG or DFinity’s DKG with persistent shared public key interesting. The persistent pk DKG would only be useful in same section (1>1) handover by the way. It would not be useful in a section split situation.

6 Likes