MaidSafe Dev Update - November 22, 2016

Last week, we released the fourth dev tutorial (Markdown editor).

Now we are focussing on getting the required changes for SAFE Authenticator in place. Earlier today, we moved the 3 RFCs related to the new authentication flow + granular permission control from proposed to active (because we are now starting to work on implementing them). The 3 RFCs are still up for debate and changes will be accepted if they are agreed upon. Our proof of concepts tests are almost done and we’ve started specifying the required tasks and breaking them down into several subtasks.

In parallel, the Routing team is making good progress on the Disjoints Group implementation. Most of the tests in routing are now passing (we’re down to 3 failing tests). Once the tests in safe_vault pass, too, we will move to in-house network tests with actual vault binaries and start implementing the Node Ageing RFC (which we need to implement in order to produce vault binaries that users can run from home).

SAFE Authenticator

We have done some proof of concepts for the SAFE Authenticator plugin (for SAFE Browser) and now we are actually starting the work on the plugin. One of the proof of concepts was to test URL scheme registration on multiple platforms (OS X, Linux, Windows).

SAFE Core

We have started the implementation of MutableData (and the new paradigm of Authenticator-based interaction) in routing + safe_core.

MutableData provides fine-grained access control and more narrow-scoped data operations. This data type is an evolved version of AppendableData. It combines StructuredData and AppendableData features and improves the network efficiency, saves bandwidth, and provides a standardised approach for apps to fetch and mutate data in the network.

Routing & Vault

We finally got the routing churn test to pass consistently with disjoint groups: It simulates a network with joining and leaving nodes, and after each change verifies that the new network invariant is satisfied, i.e. every node has made a connection to every node that it needs in its routing table according to RFC 37, and in particular that the routing tables of all members of a group are identical (except for the implementation detail that a node doesn’t put itself in its routing table). When a group is big enough it now splits up into two and notifies its neighbouring groups, and if a group loses a node and drops below the minimum size, it merges with at least one other group and also notifies all connected groups, to allow them to update their routing tables. The passing churn test confirms that we have now covered all the edge cases and have all the message flows in place for the network to adapt its structure.

We’re down to 3 failing tests out of 102, and those all point to a particular problem with groups failing to send a message while they’re in the process of merging. We’re already working on the fix for that.

After sorting those out, we will move on to testing a routing example application on droplets, while some of us will iron out any remaining test failures in safe_vault. When those tests pass, too, we will test vault binaries on droplets again, and start implementing the Node Ageing RFC.

We are also gathering ideas for improving and unifying some of the synchronisation mechanisms that we have implemented separately and not always in the same fashion in routing and safe_vault, and that would benefit from a better theoretical foundation and a clear, agreed upon order for handling events. In particular, we are discussing adapting the Tangaroa or Chandr-Toueg consensus algorithms for our use cases.

And finally, we have extended the routing simulation to take into account good (non-attacker) nodes joining and leaving, for computing estimates of an attacker’s chances with different parameters (group size, quorum, etc.). With Node Ageing, short-lived good nodes are an advantage for the attacker, as there will be less old and trusted nodes to compete with, so the parameters will need to be chosen in a way to provide a high level of security even in the face of a lot of churn.

Dev Outreach

On November 30, Nick is going to be speaking at the IoT + Data Decentralisation meetup in Edinburgh.

Over 3.6 billion records were stolen between 2014 and 2015, which clearly indicates that the way data is stored today requires a new approach. Not only is data that is centrally stored vulnerable to attack, but storing data in this way is expensive, because the greater the protection, the higher the operating cost. This is why companies and individuals end up spending billions every year on the infrastructure that serves our applications and that processes our data. A new breed of startups are using a different approach to the way data is stored. By tapping the unused potential of billions of connected devices it is now possible to move our data management away from large companies and their data centres. This talk will discuss how harnessing the power of the crowd will impact almost every aspect of our future daily lives including the IoT, and our relationship with the Internet in general.

In addition to the 2 events previously mentioned – Bitcoin Wednesday Amsterdam #42 (December 7) and Kick-off of the SAFE Developer Meetup in Amsterdam (December 9), Ben will give another talk at the Front-Enders Meetup in Amsterdam on December 6. He will explain how to build decentralised privacy-first web apps.

Ben will also be speaking at BOB Conf Berlin on February 24, 2017. His talk will explore the privacy-first data structures provided by the SAFE Network.

The Boston and San Francisco meetup groups have been officially renamed, too:

It’s also worth mentioning that Nikita (who works on the SAFE Core team) gave a Rust talk a few days ago. He said that he got some questions about MaidSafe’s experience on using Rust :slight_smile:

On December 10, he’s giving another presentation about Rust in another city. He’s also trying to schedule another talk for a quite large conference (1500+ participants, all developers and IT professionals) in Russia on April 1, 2017. He’s considering doing a technical presentation focused solely on the SAFE Network in the context of distributed networks.

Recruitment

We’ve signed up with an outsourcing company based in India to scale the frontend team. They’re based local to some team members including team leader Krishna. We’ve already received CVs from them and are in the process of filtering suitable resources to interview and hopefully bring onboard soon. It should certainly help us with the upcoming work in the frontend side with SAFE Authenticator and mobile support.

57 Likes

Great news!

Wooot, that’s quite an achievement. :+1:

Great update again. Most work not visible to most of us as most is done under the hood. But it’s very important to get all this stuff right.

23 Likes

It’s so great to finally get good word on disjoint groups. I think the more detail the better for the more technical but I’m happy enough to hear that it’s narrowed down to three bugs and then hopefully a swift implementation of node ageing! Nodes from home very soon folks, very soon! Great work can’t wait to see how everything goes over the next couple to few weeks

16 Likes

Always good to come home to read these on Tuesdays,

Thanks Maidsafe team for your hardwork and keeping that torch burning.

Sorry that I’m just a cheerleader (I just feel super frustated that everybody is using AI for all sort of @#$$%$%, but can’t use it to suck up all data on github and spit out apps).:stuck_out_tongue:

For now this is the only way that I can support you guys …

9 Likes

Great update! A lot of things moving in the right direction I get the feeling. Im always excited to read the updates. Thanks for the hard work and and dedication!

9 Likes

Great work on the progress of testing. As a developer myself, I can appreciate the importance of rigurous testing. Can’t wait to be able to run vaults from home.

Best wishes,
Gabriel Stratan

7 Likes

Thanks so much for all of the hard work! Keep the good news coming! :slight_smile:

2 Likes

Let me guess, Merry Christmas release? :o I can’t wait.

4 Likes

Could be I suppose haha. Although I’d prefer it to be even slightly after the new year if not further before Christmas. Otherwise I’ll have to cancel all holiday plans because nothing gets in the way of me maidsafing :neutral_face:

7 Likes

@dirvine this has been so drawn-out that i think it would be a good moment to give your personal estimation of the future. While release might be too far away to give an estimate, could you at least tell what is your view on when testing continues with vaults running in the wild? Will it happen this year?

1 Like

@dirvine has stated many times that, in effect, it will happen when it happens, there will be releases when the well-documented goals have been reached.
We all want a beta release ASAP. Hassling the main man to take time out to answer your specific question is not going to make it happen one second faster.
This project is already a model for the rest of the FOSS community in terms of open, honest and frank communication with the external devs and supporters.
What more do you want?

7 Likes

I know I’m not David Irvine but I think we might get a test net of vaults at home with the launcher (not yet the authenticator) before christmas. But there is a chance they want to give us the whole shebang (authenticator+Vault w/async safe core+routing w/disjoint groups) minus data chains, all at once. That would very much be likely to fall into the beginning of 2017 which is close enough to be patient for. This is all just a best guess on my part.

3 Likes

Turn your alarm clock to fall 2017 and come back then.

Looks like I’ll probably be wrong: I'm excited at the 100% chance of SAFE's full 2016 launch!

But I didn’t know they were so low on funds, at the time. So yeah, replace 2016 with 2017 :stuck_out_tongue: .

Dam, I really liked that post and thought 2016 was certain too.

Gotta say it does feel like a little defeat :frowning: but oh well what can we do I guess…

Hopefully with their bigger team things happen faster now.

Huge breakthroughs on the horizon:

  • authenticator / mobile sites & apps
  • distributed vaults that truly will work and don’t have to be reworked a bunch because stability that node age, Dchains etc give
  • SafeCoin wallet CEP

So while it is true that even now we still feel like we’re just here waiting, at least we’re waiting on the final parts, because these points above are really what makes SAFE truly finished, and ready for it’s launch (promise land).

P.S. watched Snowden movie just now

3 Likes

You the man Will, keep positive. @Artiscience could totally be correct about fall 2017 of an ‘official’ release candidate but at least we can be more certain it will be SAFE and won’t fall prey to some hack that could have been prevented by proper testing :slight_smile: I’m chill

5 Likes

What we need is a horde of devs to show up in troon and start coding asap.

3 Likes

Release candidate 1 with data chains?

1 Like

Last week I asked which RFCs would be part of the upcoming Vaults. I got “Disjoint groups” and “Node Ageing” as an answer. “Disjoint Groups” is in the routing layer and David said somewhere that this would’ve taken a full year to implement in C++. So great they’re working in Rust, but still a lot of work. It looks that with the new devs they are getting to the bottom of each and every aspect of routing as they have some extra hands. They created a whole module called “routing_sims” for checks and double checks. You need all nodes to reach consensus on splits and merges of groups and more. That’s quite a task :sunglasses:.

“Node Ageing” is probably easier to implement but not 1 or 2 weeks work is my guess. Well, you can implement it but you need to do tests as well and checks and rechecks and more. “Resource Proof” needs to get implemented as well as part of this. So that’s even more testing with slow nodes, fast nodes etc. And in “Disjoint Groups” you only want several nodes with a particular “age” so you need to implement algo’s to relocate certain nodes without your group becoming too small.

Overall that’s a lot of work. I guess we’ll get Vaults somewhere January or February. And that would be an amazing job by the devs.

8 Likes

Estimate on when data chains may come and if they will be part of the initial release?

1 Like