MaidSafe Dev Update - March 22, 2018

Here are some of the main things to highlight this week:

  • We had the first of what will hopefully turn into a regular catch-up with SAFE meetup organisers today. Lots of valuable feedback and ideas were shared on the call about how best to build and engage communities of supporters which we’ll share very soon.
  • The front-end team has reordered their priorities for the next month in order to prepare for the dev conference in April. They would like to get the dev website ready for the conference. They are therefore pausing the UI/UX work for Peruse to focus on getting the dev website ready.
  • The SAFE Client Libs team developed a new FFI function, test_simulate_network_disconnect, that exposes a previously hidden function that simulates network disconnects. This should help the front-end and app developers to test apps in conditions where the network connection is unstable.
  • The Routing team is continuing with the work of figuring out an approach to deduce a consensus order of observed network events from a Directed Acyclic Graph (DAG) which was composed from the gossiped messages. They have some test code in a private repository that they have working and will progress this to an open solution. We just published a blog post regarding this, you can read it here.
  • The Crust team finalized and merged the chat example.

Marketing

New SAFE Developer Website

As you’ll know from past updates, we’re currently redeveloping the MaidSafe website (https://maidsafe.net/). During that work, we’ve taken a broader look at our online presence and decided that we should build an additional, standalone Developer website. This work is now underway and we’re looking at a release of this website in the very near future. More to follow on this in due course.

SAFE Meetup Organisers Call

We had the first of what will hopefully turn into a regular catch-up with SAFE meetup organisers today. Lots of valuable feedback and ideas were shared on the call about how best to build and engage communities of supporters which we’ll share very soon. Thanks to everyone who took the time out of their day to dial in - and apologies to those who couldn’t attend due to various time zone clashes. As ever, if anyone has any views on meetups or community building when it comes to the SAFE Network, please just contact @dugcampbell or email outreach@maidsafe.net.

Activity on r/safenetwork

As pointed out by @riddim earlier this week, it’s worth noting that there seems to be increasing levels of activity on the subreddit. Thank you to everyone who has been engaging there and helping to spread the word.

Out and About

@ustulation will be speaking at the Rust Edinburgh Meetup on Monday night where he’ll be talking about how MaidSafe uses Rust for peer-to-peer networking on the SAFE Network.

SAFE Authenticator & API

We have reordered our priorities for the next month in order to prepare for the dev conference in April. We would like to get the dev website ready for the conference. We are therefore pausing the UI/UX work for Peruse to focus on getting the dev website ready. Though the UI/UX aspect is paused, the technical aspect of the work in Peruse will continue to be carried out in the meantime. The work is in line with the features that we were planning to have as part of the UI/UX update. Once Jim (our new UX Designer) is also part of the team, we will continue working on the UI/UX designs for Peruse.

@lionel.faber has been trying to get a grip on the safe_app_java codebase. He is proving to pick things fast and we are hoping to see more progress there in the near future.

This week we’ve been testing the latest Peruse changes, and have tracked down and tackled some awkward Electron bugs that were preventing the new background processes from loading properly in the packaged apps. We’re now moving on to ensuring that e2e tests are working in the new setup, before taking a longer look at the DOM API updates we’ve been planning.

As briefly mentioned in previous dev updates, we are exploring ways to enhance our DOM API by removing the handles that are currently needed for each object created through it. We are considering simply exposing the safe_app_nodejs API within the DOM API, with a few exceptions for the functions related to the authorisation requests. These few exceptions are due to our wanting to prevent web apps from needing to send/receive auth requests/responses through the system URI mechanisms and instead using the internal IPC mechanisms as we do today, avoiding all complications we may face with different platforms. This will not only simplify the internal design and implementation of the browser but also make the DOM API much simpler and coherent with the safe_app_nodejs API, allowing app developers to have almost the same codebase if they want to release an app as a desktop and web app.

SAFE Client Libs

This week we developed a new FFI function, test_simulate_network_disconnect, that exposes a previously hidden function that simulates network disconnects. The PR was merged earlier today. This should help the front-end and app developers to test apps in conditions where the network connection is unstable.

We’re continuing to work on the C# and C bindings improvements which turned out to be a little bit more complicated than expected because the library that we used before to generate C headers, moz-cheddar, got deprecated and is no longer maintained. Now we’re integrating the C language support into our safe_bindgen framework which is already based on moz-cheddar and is now tailored for SAFE Client Libs. As a result, we should get fully supported headers for C, with examples and automatic tests. The progress can be tracked in this JIRA task.

Routing & Crust

We are continuing with the work of figuring out an approach to deduce a consensus order of observed network events from a Directed Acyclic Graph (DAG) which was composed from the gossiped messages. We do have a solution that is not available as open source right now and we cannot use that exact algorithm. We have some test code in a private repository that we have working and will progress this to an open solution. We just published a blog post regarding this, you can read it here. So on to the good parts, this ordering consensus algorithm for intra section membership changes is nice and modular as well as mathematically proven. This means less code, less testnets (many testnets were to find all the rules regarding order, which have grown in complexity (although works)). We cannot say this is definitely the last unknown as we have not detailed network restarts after complete collapse or network managed upgrades amongst others, however this is a large step and one that allows us to much more clearly see the remainder of the features for a full-featured network, or in other words Beta. Much of the work (and there is an awful lot) is currently hidden here, but will be exposed very soon. We ask for a little patience with us in Routing for the next few days and possibly weeks. If this pans out then it will be very much worth it and you may find us able, at last, to give reasonable estimates of remaining development work. This does not mean detailed roadmap with timelines, but we will have a very clear route and make it clear which tests we wish to do, most of which will be in client APIs, language bindings and platforms.

In Crust, we finalized and merged the chat example. It is quite flexible and allows us to examine misc aspects of Crust and, as a result, we found 2 new bugs that still have to be investigated and fixed (2589, 2588). Also, we’ve been working on slightly changing the Service::connect() API - Crust will do connection information exchange by itself, all you’ll have to do is provide a relay channel to connect(). But it’s worth noting that the compatibility API will remain the same. netsim is paying its dividends: it helped us spot bugs in both the Crust and p2p crates which got fixed and covered by netsim based tests. Also, we started working on another tool called nat-probe. It is meant to help us test the existence and type of a NAT: is it a full-cone NAT, does it support IGD, etc. Hopefully, this tool will save us tons of hours of debugging in the future.

82 Likes

1st woooo! Ill read it now…

EDIT:

Excellent update!!!

mathematically proven. This means less code, less testnets (many testnets were to find all the rules regarding order which have grown in complexity (although works). We cannot say this is definitely the last unknown as we have not detailed network restarts after complete collapse or network managed upgrades amongst others, however this is a large step and one that allows us to much more clearly see the remainder of the features for a full featured network, or in other words beta. Much of the work (and there is an awful lot) is currently hidden here, but will be exposed very soon. We ask for a little patience with us in Routing for the next few days and possibly weeks. If this pans out then it will be very much worth it and you may find us able, at last, to give reasonable estimates of remaining development work.

Edit:

I will be buying many coins next pay day.

33 Likes

Sounds like an excellent idea to me, and hopefully will generate a lot of community growth for the dev forum as well. Is this going to tie into the SAFE Academy too?

Very exciting! Would also be great if content coming out of this could be captured for further distribution at upcoming meetups. :slight_smile:

20 Likes

Great update and glad to hear alpha 3 being talked about. It sounds like some big steps are being made!

16 Likes

For this new message ordering algorithm, is there a patent encumberance as well as a copyright issue?

Can the tech be licensed?

4 Likes

Is code hidden atm to be documented and patented? The more parents Maidsafe has under its belt with GPL3 the better imo. That would be good news for shareholders!

Great update, always blown away by the innovation and progress. So much grit :fist:

Edit: answer is in the blog post :yum:

13 Likes

Documented yes, patented, probably not. We have felt that the routing and network layers cannot be patented as they are way to useful for many projects. As our patents are defensive though we do use these to promote innovation by allowing projects to use them without charge and with the GPL. In other words this stuff in routing and crust is way to valuable to hold to ourselves. Its our duty to make this all available and as always if any other team can work better/faster we will only help them.

If it helps the planet we do it, that includes defensive patents and much more so open sourcing everything possible and to us possible means everything :smiley:

36 Likes

We wont need to license it and would not be able to, but, there are already moves to provide at least one open source alternative, currently we are doing this in private, but not for long :wink:

24 Likes

@dirvine Just curious how that is done. Does the library need to be rewritten by another developer, or slightly redesigned or optimized?? Very exciting news

5 Likes

Excellent, this is very exciting.

This started me wondering: does MaidSafe partner with any other organizations for research? I’m thinking specifically about IOHK at the moment, since they seem to have huge amounts of resources and very good academic relationships already in place. Although they’re mostly focused on blockchain tech, it seems like they should be interested in the revolutionary techs that MaidSafe is developing.

7 Likes

Ah there are a few ways, in this case we have a few choices and several algorithms to use. Using any of the licensed/patented components is a no no, so any impl has to take at least one different step. It is a complex matter, but not one that worries us at all. The nice thing here is we can use any solution for this small part and remove a ton of our conditional code and move the network on very much faster. So then we can relax about the huge complexity in routing and make it much simpler.

The fact we have sharding sorted, secure message transfer and node age means we have a ton of tools. Then given the self authentication, DNS, self encryption and all the client lib components etc. this small but very important thing allows us to just push the rest of the code/network to beta very much faster and with much more comfort.

Saying that these days are very long and night has fallen into day as well, but not so much with pressure in house, much more with excitement at a massive pressure releif valve going off. Although not a stupidly simple task to integrate this, it is one that is very interesting and you can hear the guys excitement going through the options.

So hard but very interesting days right now in routing. About time to, its been a very long slog down there :wink:

43 Likes

Yes, when we have time, mostly Scottish Universities so far, but there is a lot happening in our India office and I expect a ton more research there as well. The partnerships take so much time and stop engineers working so its a balance, we just cannot afford to many time sinks right now. I expect that to improve when we get a chance though. Its much harder than it seems to get fruitful partnerships, everyone wants the marketing / PR buzz but real work and results is much more intense.

24 Likes

Splendid news! Very exciting, more and more progress! Great work!

12 Likes

Great update, mostly thankyou for following your own path despite all the pressure, it will payoff immeasurably in the end :slight_smile:

The entire Blockchain movement are in for a little bit of a shock hey :smile:

24 Likes

that sounds like a very reasonable plan!

i like that part a lot! (though i’m not a c developer… =)

9 Likes

Will it now be possible to send image files etc?

As discussed here

8 Likes

i was about to highlight the same part now! :smiley: how awesome is that?! messaging on safe!

no additional put costs - just an encrypted direct messaging app that runs through safe :open_mouth:
seems like crappyApps Inc. needs to become more active again soon! xD

How about a radio station on Safe?! (that was more like a general idea … don’t think there’ll be a crappySounds soon - though the name really would be awesome xD )

11 Likes

I think you’ll need this Rid crappyTunes™

3 Likes

Can’t wait for the new website. It will be relevant for the external image of the project.

Think the idea of @Sotros25 about the academy is great! One of our goals imo is to educate people before they use the tools

6 Likes

Very encouraging update. Keep up the good work.

I’m surprised no one has asked about the identity of the closed source, ordered DAG that is mathematically proven. I’m now going to break that good discipline and ask: is it hashgraph (ordered and mathematically proven)? Spectre (mathematically proven but not fully ordered) and Phantom (ordered and mathematically proven) are also candidates, but not sure if there’s available code for these.

6 Likes