MaidSafe Dev Update - May 10, 2018

Here are some of the main things to highlight this week:

  • We originally planned to make all our core libraries (such as Routing, Self Encryption, etc.) GPLv3 only. However, we have now decided to make some exceptions: for instance, our connections networking library Crust will be available under MIT and BSD licences. Many other utility libraries are also now going to be made available under this dual license.
  • Ravinder (@ravinderjangra) has joined us at the Chennai Office. He has good experience with C# and he is an active contributor to Xamarin.Forms. We are looking forward to working with him.
  • The bindgen branch has been merged upstream. This hits a big milestone in regard to the language bindings support.

Marketing

As many will be aware, we are developing a SAFE Academy website as a resource to help people learn more about the technology behind the SAFE Network. The first version will initially be a stripped-back version, with the rest of the content being built out over time. After working on this project intermittently over the past few months, we now have an internal review of the first version of the SAFE Academy scheduled for the end of next week. All being well, we should then be clearer on a projected launch date for the Academy.

As mentioned at DevCon, we have amended our licensing strategy to remove our commercial licence and apply an MIT and BSD license to our APIs and bindings. As some of you will have seen, during this last week, Nikita and other members of the team (Marcin and Srini) have been making these changes in GitHub. We originally planned to make all our core libraries (such as Routing, Self Encryption…etc…) GPLv3 only. However, we have now decided to make some exceptions: for instance, our connections networking library Crust will be available under MIT and BSD licences. We have decided to do this for this library because it’s a highly useful library that will help a number of other decentralised projects. By applying permissive open source licenses, it will enable developers who fork Crust to have complete flexibility in licensing their derivative work as they see fit. This should in turn lead to more developers using Crust. We believe this move strikes the right balance between ensuring that we lock the fundamental components of the network open for all, whilst also giving some highly useful tools back to the Rust and decentralised communities. We are very much looking forward to seeing what they create with it. To support these efforts, we will start to make more developers aware of Crust’s features and benefits in the coming weeks which can only be good for the SAFE Network and the community in general.

In addition to Crust, we are also switching the following utility libraries to the dual MIT/BSD license: config_file_handler, fake_clock, maidsafe_utilities, lru_time_cache, accumulator, resource_proof, get_if_addrs, and secure_serialisation.

SAFE Browser, Authenticator & API

As part of our tasks of making necessary changes to upgrade the safe_app library in our safe_app_nodejs package, we were working on some changes that affect the MutableData entries API. The forEach function exposed to iterate through MD entries has been replaced by a new listEntries function which simply returns the list of entries as an array of JSON objects. This was a pending change to have a coherent API as the other functions to iterate through MD keys and values were also removed in previous releases of the safe_app lib and safe_app_nodejs package.

We have additionally exposed a new function to simulate a network disconnection event which will allow developers to test the logic their apps are expected to follow upon such an event. Testing this type of logic will now be possible even when using mock routing by invoking this function.

Along with these changes in the API, we are reviewing and trying to make some minor changes to a couple of functions to align them to the terminology and names used by the underlying native library, as well as trying to make sure the safe_app_nodejs API is coherent with the other language bindings we are working on. We will provide a full list and details of the changes in our changelog when we release this new version so developers can easily adjust their code if necessary.

One of the features we are now also working on is to allow an application to be able to set the metadata for a file created using the NFS emulation, i.e. the metadata that is stored in the NFS container (MD). This is already supported by the safe_app library so we are just trying to expose the functionality in our API.

Ravinder (@ravinderjangra) has joined us at the Chennai Office. He has good experience with C# and he is an active contributor to Xamarin.Forms. We are looking forward to working with him.

The initial integration of the Java APIs with the bindgen code is going well. The test cases are being updated to use the Authenticator API for end-to-end testing. The next step is to test the Java API on all three (Windows, macOS, Linux) desktop platforms. Once the tests are confirmed to pass, the packaging of the JAR for catering the different combinations will be generated. We are also planning to work on a simple example application to test the usage of the packaged JAR files.

SAFE Client Libs

As we’ve changed the license for SAFE App from GPLv3 to BSD/MIT, we are continuing to work on providing a full API for app development contained in a single library. As was explained in the previous update, the primary reason is to ensure the completeness of the Rust SAFE App API: it should be the only dependency required to be included in the Cargo.toml file to communicate with the SAFE Network from Rust apps. The pull request is raised and being reviewed and the progress can be tracked in this JIRA task.

The bindgen branch has been merged upstream. This hits a big milestone in regard to the language bindings support. And now that the front-end team confirmed that they haven’t found any major issues with the generated bindings, we are moving to the next step: automating the deployment for mobile platforms. This task will entail extending our continuous integration pipeline and including builds for iOS and macOS platforms in addition to the existing ones.

Routing

Following up from last week’s dev update, the Routing team is still working on tackling Eventual Consistency.

TRIVIAL

In a comment on last week’s dev update, @Antifragile linked a recent paper: “Byzantine Agreement, Made Trivial” from Silvio Micali, one of the authors of Algorand.
This paper describes a way to obtain Synchronous Binary Byzantine agreement without the need for a Common Coin. They replace it with a “Concrete Coin” which is a function that will behave like a Common Coin probabilistically often (~ 2/3 of the times).
They then split the decision process into rounds composed of three steps so that imperfect Concrete Coin is enough for reaching binary consensus.

The paper in itself presents, as advertised a rather trivial algorithm; partly because creating a Concrete Coin in a synchronous setting is indeed very easy, and partly because the three steps algorithm needed to converge with a weaker coin is also quite straightforward in a synchronous setting where we can simply wait to hear every node’s voice at each step.

For our asynchronous usecase, it is slightly less trivial, but we think it is possible and actually less difficult than getting a fully asynchronous Distributed Key Generation scheme to work for our decentralized use case. On a side note, we would like to thank @mav for his contributions to the DKG discussions.

We have spent some of the last few days devising an asynchronous Concrete Coin that we call: “gradient leadership based Concrete Coin”. We think that it exhibits all the properties needed for a Concrete Coin, but we removed all the synchrony assumptions from the process of obtaining it. We are in the process of formulating our suggestion precisely in our RFC, with associated mathematical proofs.

We are now tackling the second main challenge of mapping the ideas from “Byzantine Agreement, Made Trivial” to asynchronous consensus: integrating the Concrete Coin in the main protocol in place of a Common Coin.
We only started focusing our energy on this recently but are already seeing some promising leads.

While all this progress is being made on the algorithmic front, we are also continuing with our efforts of producing an RFC to explain the complete idea as clearly as possible, along with visual illustrations that should help convey our points more easily.

If and when we find a fully proven solution for the second challenge explained above, we will focus all our efforts into finalising the wording for the RFC and share it with the community.

We refer to this new adaptation of BOA as: Total Record of Incidents Via Interconnected Agreement Lattice (TRIVIAL) as an acknowledgement to the paper: “Byzantine Agreement, Made Trivial” that gave us the idea of trying to solve the problem without the need for a Common Coin.

Crust

This week we mostly worked on the netsim and tokio-utp crates. netsim is currently able to only parse and construct TCP packets. We extended it a little to recognize flags (control bits) and TCP options, e.g. selective acks. Also, IPv6 is a first class citizen in netsim and now it is finally supported. uTP is another tough cookie and we are still working on graceful connection shutdown issues. In the meantime, we managed to setup CI for the tokio-utp crate which lead us to discovering new bugs. In some cases, our uTP implementation sends Reset packets back, e.g. when the connection limit is reached and we receive a new Syn packet, or when some random packet with unrecognized ID comes, etc. It turns out the Reset packet is not always correctly sent out. Although, it’s worth to mention that the fix is on the way :slight_smile: While doing various fixes to our uTP implementation, we raised some issues on the specification: BEP 29: packets with unrecognized IDs · Issue #83 · bittorrent/bittorrent.org · GitHub, BEP 29: graceful connection termination · Issue #84 · bittorrent/bittorrent.org · GitHub. Hopefully, the rest of the community will be able to make a use of our discoveries too.

71 Likes

First!!! Muahahaha time to read!!! :smiling_imp: first time being first by the way

12 Likes

second, finally made it to the top 3

5 Likes

How dare you!

Next week, me laughs and you says “how dare you!”

(I swear.)

6 Likes

3rd again… lol :joy: (no! @anon28980808 you’ll not steal my 3rd position)
Now read
Edit: Academy will be one of the decisive points for marketing. Can’t wait for its release! Great job devs!

7 Likes

Consolation prize.

Next week, I’ll let you come second bro. Second :star_struck:

Thanks for the update Maidsafe devs

Welcome @ravinderjangra

:clap: @Antifragile & @mav :clap:

:stuck_out_tongue:

10 Likes

yes it should really

2 Likes

@Antifragile you’re a hero! xD (and the team + @mav ofc too!) together this project moves forward =) awesome!

i’m very curious about the outcome of this =)

welcome to writing history @ravinderjangra !

ps: … side note … do you realize the paper Antifragile mentioned was published just a month ago? O.o … so … earlier it coulndn’t have been mentioned/helped moving forward … the world is preparing for safe :innocent:

19 Likes

This whole bit is brilliant. Props to @Antifragile @mav and @maidsafe for progressing this idea. I can’t wait to read the RFC!!!

Btw welcome to @pierrechevalier83! Will be great to see mobile progress even further with you on board.

18 Likes

Those java bindings sound super close now. Great stuff!

Edit: good news on licensing too. I like the focus on being as open as possible, while still protecting the integrity of the network.

12 Likes

So Safecoin will be a “concrete coin” instead of a common coin because it is a better implementation? Did I interpret that right?

Many thanks to all of you for your hard work! And welcome to the new members!

Also, thanks to @whiteoutmashups for all the work he is doing getting us on lots of exchanges!

2 Likes

No, these “common coin” and “concrete coin” are implementation details of the process of reaching consensus on the order of network events. It’s unrelated to safecoin. :wink:

20 Likes

Cant help beeing worried that durring devcon david said that the research is done now time for coding while each week from there there is a new massive issue which then takes few weeks/months to research again. And again there is no clue how long it may take to solve. Last week there was a hint on alfa 3 but now i think we can postpone this at least on month again right?

1 Like

At DevCon David said there’s still one incomplete area of research that is progressing well, and once that is finalised, they will be on to more standard development that’s far more predictable.

No time frame was given for solving this last big unpredictable piece of research, but it’s great to get front row seats to the avenues being explored and new approaches that could be keys to unlocking a great solution more quickly.

While there is still uncertainty around it, it’s very exciting to see rapid progress toward what will be a brilliant break through for the network’s development!

Keep up the incredible work MaidSafe team & community :wink:

22 Likes

Top update! Full 12 points.

what-can-the-eurovision-2015-teach-you-about-presenting-20-638

14 Likes

I just changed this part to:

(since it can be substituted by a Concrete Coin which is more easily obtained)

thanks to @pierrechevalier83 for clarifying this part with me :smiley:

17 Likes

i’ll admit I’m not THAT familiar with the underlying Crust libraries but I know enough about them to be concerned to hear that they’re being liscenced under MIT and BSD, which allow for comercial use. What happens when a corporation decides to start using this tech and makes a copywritten version of the SAFE network using the Crust libraries? Isn’t this like giving out the core blockchain tech without the corasponding decentralizing mining tech to counterweight it? (To use a bitcoin analogy, one which the banking cartel has been chomping at the bit to aquire.)

I’d be interested to know what led to this change of heart and why it’s thought to be safe? In short what are the actual ramifications and potential consequences of Crust becoming a for profit tool? How bad could it get (and seriously do some lateral thinking on that because you know bad can get bad when corporations are involved, especially over the long term).

3 Likes

Interesting question, looking forward to the answer @dirvine

Ok, lets delay launch for a few minutes while I give my opinion on this :wink: since it was asked for. Crust is a utility library, not part of our inventions innovations or core technology. It is a utility that is extremely beneficial to any project in the rust ecosystem, just like our fork of sodiumoxide, rust_sodium (used by Mozilla amongst others). These utilities are handy tools, crust is a large library though and I was shocked when I stated MaidSafe that such a thing never existed, it should be a given. Andrew and Povilas are doing amazing thigs there and are creating other utilities, such as netsim etc. they will also be MIT/BSD and I hope widely used.

Now some corp can take it and do private work and create some closed source proprietary project, I care not a jot, but I do care these utilities are very widely used, tested and improved so we at MaidSafe can get on with the innovation :smiley: It’s that simple really.

21 Likes