MaidSafe Dev Update - November 29, 2016

Development Updates
1

The 3 failings tests mentioned last week have been resolved (all the tests in routing are now passing). We’re about to test a routing example application on a network of DigitalOcean droplets to make sure that the network is able to maintain its structure during churn (see the Routing & Vault section below for more details).

In parallel, we are continuing to work on implementing the 3 RFCs related to the new authentication flow + granular permission control.

SAFE Authenticator

We have detailed the authorisation process in the New Auth Flow RFC. Ben is lending some help by working on the Rust code (from safe_core) so that the bindings required for the UI can be provided sooner. Shankar and Krishna are working on the SAFE Authenticator plugin and an example application just to get the development progressing without the actual integration with safe_core.

SAFE Core

We are continuing to implement MutableData and the new paradigm of Authenticator-based interaction in safe_core and routing.

We have completed the provision of RPCs in the core module and mock-routing based on all the requests that the network allows the clients to make. The client creation functionality is in place with new MutableData and mandatory user and config root directories. We are now coding low-level (FFI) access to manipulation of MutableData, provision of standard user directories and MaidManager simulation in mock-vault for app key authentication.

Routing & Vault

The failing tests mentioned in last week’s update were resolved and all the tests in the routing crate pass now! We are preparing for the next round of testing – running a routing example application on a network of DigitalOcean droplets in order to verify that the network is able to maintain its structure during churn (turning droplets off and on again), specifically that it always restores the routing invariant.

With Disjoint Groups, the nodes are organised in groups defined by prefixes, e.g. the group with prefix 10 contains all nodes whose names in binary start with 10, and the requirement is that two nodes are connected if and only if the prefixes of their groups differ in at most one bit.

Andreas is currently writing a script to check that invariant from the droplets’ logs.

We fully expect to find a few more bugs in these tests. And also, there are still a few edge cases that we have yet to write new tests for: Fraser is currently working out how to deal with simultaneous and conflicting group merge and split events, and writing a test for this scenario. (For example, the group with prefix 10 may have lost a node and need to merge with another group, 11, into a new group 1, to keep its minimum size, but at the same time group 11 may have gained a node and want to split into 110 and 111.) And in the same vein, Bart is looking into adapting Byzantine-proof (i.e. resilient against malicious nodes) consensus algorithms to help groups agree on the order in which events are processed.

The next stage will be testing actual vault binaries, but first the safe_vault crate’s mechanisms, and its unit and integration tests, need to be adapted, too. Fortunately that might turn out to be easier than expected: Diggory is currently investigating whether we can reuse almost all of the existing algorithms there! The benefits of the new “group” notion — a simple scheme to securely relay messages, better control over the number of joining nodes per group, evaluating other nodes, … — are all on the routing level and we may be able to just keep using the old “8 closest nodes”-type groups in safe_vault (and still use the secure message routing mechanism, without safe_vault even knowing about it).

And finally, Qi is currently detailing the concrete implementation plan for challenging new nodes to prove that they have the resources (bandwidth, storage, CPU) to contribute to the network. This will be the first part of the Node Ageing RFC.

SAFE Browser

@joshuef is done with implementing all the tasks he had planned in his SAFE Browser proposal. Krishna’s team has reviewed the code. We think it’s fine, we just have a few questions for @joshuef and then we intend to conclude the SAFE Browser RFP by sending him the project payment.

Dev Outreach

Ben attended DAppHack – a hackathon and unconference about decentralised systems. During the hackathon, he ported the noBackend Invoice Example App to SAFE and published it on the Test 11 network. See this topic for more details.

RFCs

@happybeing submitted a pull request for a new RFC called “File Sharing By SAFE Data URI”. It’s a feature request to add the ability to access files using a web URI (without going via a SAFE DNS public ID and service). As @happybeing mentioned, the RFC is a joint effort between him and @lightyear.

It looks like a great RFC! But we’re not able to move it forward at the moment. We want to get some fundamental stuff in the RFC process done (e.g. assigning each RFC to a shepherd – the shepherd is a trusted developer who is familiar with the process, who will help to move the RFC forward and ensure that the right people see and review it). The current process is lacking in selecting shepherds and so we’re looking to get the process in place to assign shepherds to all the community RFCs soon.

55 Likes
2

Thanks for all the hard work and dedication!

17 Likes
4

Congrats on the progress!

We have such a nice community.

11 Likes
5

I’m hyperventilating it’s so close! By so close I mean being able to strike down the doubt of the nay Sayers by showing an efficient/quick and stable network from home! For those of us that have been around awhile, we had heard of what the networks capabilities would be and saw many details that made us fall in love :heart_eyes: but now we are seeing every essential piece put into place. I couldn’t be more grateful for such a competent and dedicated team! Bravo

19 Likes
6

I bet they’ve all put on a substantial amount of grey hair in the last few years… it has been a Herculean effort from a spectator perspective. Most people and organisations would have crumbled years ago. I guess a cause worth fighting for is a good motivator.

The world owes you a big debt of gratitude for trying, if you pull it off we’ll owe you in the most profound way… emancipation! Maidsafe, “the breaker of chains” :grin:

18 Likes
7

Great update, lot of hard work paying off. Seems like Disjoint Groups will get some extra self-tests?

I’ve read this one several times but still don’t get it :yum:. The Disjoint Groups means we have variable groups isn’t it? If they get too small they merge, if they get to big they split into new groups. I guess Vaults need to follow this pattern as well? So how could the 8 closest nodes still be active?

7 Likes
8

Thank you for bringing us forward! One beautiful step at a time :slight_smile:

8 Likes
9

Thanks Maidsafe team,

For your unbelievable dedication to get this job done and rigorous testing.

I had to do it :stuck_out_tongue:

1000×1200 123 KB

6 Likes
10

I’ll need someone to please wake me up!
Am I dreaming or this is all real? Am I really seeing the new internet being built from scratch and participating in discussions with its creators and its community? I’m honestly so confused with all this happening! :sunglasses:

This is all so inspiring! Thanks for sharing these updates with us!

16 Likes
11

@rand_om slaps @bochaco with a trout.

There you go my dear :innocent:

10 Likes
12

No trout, but anyway: Monty Python, The Fish Slapping Dance - YouTube :wink:

5 Likes
13

Great to read the update! Seeing all the pieces coming together slowly. Amazing job to the whole team. There more I read about governments all over the world imposing their agendas onto people the more obvious it is how important Maid and bitcoin really is. You guys are carving out a better future for generations to come. Taking back the power, one line of code at a time. Love being part of this. Keep it up, lets keep the fight going.

2 Likes
14

Truly amazing work as always guys, the revolution is near!

2 Likes
15

Guessing that if the SafeCoin CEP gets finished and the home vaults come out before the authenticator, then we may see a jump from Alpha 1 straight to Beta? Decentralized and SafeCoin seemed to be the requirements for BetaNet

But if authenticator comes out first then we may see the name “Alpha 2” be utilized

Then again, maybe Troon HQ has scrapped the alpha 1,2,3 etc plan from earlier due to things changing / evolving too fast. Never a bad thing!

Great work! Lots of excitement.
Mobile SAFE apps!

6 Likes
16

By “group” we just mean a bunch of nodes doing stuff together. Currently we always want groups to be fully connected, to avoid additional difficulties: So any fully connected set of nodes can do stuff as a group.

One of the things they’ll have to do (which is being implemented in the routing crate) so we can securely relay messages is: agree on who are their neighbours—specifically the neighbouring groups’ members—, and cryptographically sign them, so the recipient of a message has an uninterrupted chain of group signature lists proving the authenticity of the sender without having to trust any single node. For that, it is useful if the groups are disjoint and every node is just a member in one such group: it reduces the number of member lists I have to sign and clearly defines who else has to sign those lists. It gives me a clear picture for that: I am only in group A, and the neighbouring groups are only groups B, C and D, so all I have to do is sign the lists of members of B, C and D (and, crucially, everyone in A sees and signs the same picture!), and exchange those signatures with the other members of A.

A completely separate thing that groups do (which is implemented in safe_vault) is managing data: A bunch of nodes store the same piece of data, make sure that they update it the same way if e.g. Post requests come in, and give that data to any new members of the group. For that, it’s perfectly fine to just consider the 8 closest nodes to the data’s name the “group”. So while each node would be a member of exactly one routing-level group (i.e. those are disjoint from each other), it would be a member of potentially several data groups.

Fortunately, the new routing table implementation still ensures that the 8-closest-nodes-type groups are fully connected, so they can continue to be used, too. In fact, each such group is automatically a subgroup of the node’s routing-level group.

Another way to view these group types is this: in a perfectly balanced network with 8 * 2n nodes, e.g. 32 nodes, eight of whose names in binary start with 00, 01, 10 and 11 respectively, the two group notions are exactly the same! Every data group will coincide with one of the four groups belonging to one of the four prefixes 00, 01, 10 and 11, namely the prefix of the data chunk’s name.
If I add a node to that, I will still have exactly four routing-level groups. But one of them now has 9 nodes and several overlapping 8-closest-node type subgroups.

So the notions are only different in less balanced networks or while the network is growing from 8 * 2n to 8 * 2n + 1 nodes.

15 Likes
17

@AndreasF So a vaults close nodes, are comprised of 8 close nodes. Each of which come from a separate disjoint group? The biggest advantage to using the existing vault algos is avoiding a rewrite that strictly uses the group in YOUR routing table? Just trying to get this clear in my head :smile: are there any disadvantages? Would save som time and resources which is great!

1 Like
18

Another advantage is that each data chunk will always be spread across exactly 8 nodes (unless currently being replicated). There’s no reason to replicate one chunk 15 times and another one 8 times, just because their routing-level groups currently have that size.

No, they always come from the same routing-level group: Every vault-level group is completely contained in some routing-level group. (And if that routing-level group has exactly 8 nodes, the two kinds of groups are even identical.)

4 Likes
19

Maybe a stupid question: but that means that each of the 8 nodes has a real copy of that data chunk, or have some nodes only a hash of that chunck as check?

1 Like
20

They’d each have a data chunk, yes.
(In practice, they’ll get the hash first and then take a few minutes to download the actual chunks, of course.)

5 Likes
21

It’s really pleasing to see how quickly the newest Devs are able to contribute.

The Network seems to have evolved so much since the crowd sale and yes it’s taken longer than most of us expected…but when it’s born, it feels like it will be fully formed.

16 Likes