Here are some of the main things to highlight this week:
- The individual SAFE DevCon videos have now been published on our YouTube channel. You can check these out on the SAFE DevCon 2018 playlist.
- Community member @cooperka posted an article that is going down very well on Medium titled MaidSafe’s SAFE Network in 90 Seconds.
- @dugcampbell was a guest on @fergish’s SAFE Crossroads podcast where he discussed the highlights of SAFE DevCon 2018.
- We’re going to design and develop a new SAFE Network website in-house.
- The front-end team is working on adding support for multipart range requests in the Peruse browser.
- As explained during DevCon 2018, we are relicensing the APIs and language bindings to MIT and BSD licenses. Also, we’re removing the commercial license on our core libraries (they are now GPLv3 only), and getting rid of our Contributor Agreement.
- With help from @lionel.faber and the mobile dev team, the SAFE Client Libs team were able to fix the known bugs in the Java and C# bindings. Next week, they’ll be covering more test cases and if there are no more known bugs, we’ll announce the public availability of the Java and C# languages support.
- The Routing team is currently following two parallel promising leads to solve the issue of eventual consistency in the network in a robust way, this is in addition to our existing approach which is being integrated into Routing by one of the teams. This is exciting as more and more timers are being replaced as the algorithms in Routing become much more asynchronous and natural. We are hoping that we can remove swathes of complex code very soon and that will be a great step and one that reduces the requirement for so many testnets as we previously mentioned.
Marketing
SAFE DevCon 2018 Videos
After a little post-production, the individual SAFE DevCon videos have now been published on our YouTube channel. You can check these out on the SAFE DevCon 2018 playlist.
Monthly Metrics
It was fantastic to see that so many people watched the SAFE DevCon 2018 live stream last week. We saw a total of 31,857 minutes watched during the course of the live stream broadcast. A big thank you goes to everyone that supported the success of the day via social channels - and in particular to our top three community supporters on Twitter during the day (Twitter handles: @safepress, @SotRos25, @JAMSplayer) who you should all go and follow.
We’re now into a new month so it’s time to share the marketing metrics for April 2018:
Articles, talks, etc.
Community member @cooperka posted an article that is going down very well on Medium titled MaidSafe’s SAFE Network in 90 Seconds. If anyone writes content, please just get in touch if you’d like us to include it on the SAFE Network publication on Medium.
@dugcampbell was also a guest on @fergish’s SAFE Crossroads podcast where he discussed the highlights of SAFE DevCon 2018.
Team Photo
Over the years, people occasionally ask for photos of the team. Despite challenging (and unusual) conditions in Scotland last weekend (i.e. sun), we took advantage of the global team weekend to grab a quick photo of most of the team pre-DevCon.
Update on the Website
The final update this week is in many ways the most significant and relates to the work that we’ve been involved in with the redevelopment of the https://maidsafe.net/ website. For the past few months, the Marketing team have been working with a web development agency that was selected after a lengthy proposal process on this project. However, unfortunately, this has not gone according to plan. Without dwelling on details, despite a number of iterations, the designs still have not met the standard that we required to be able to progress to development and we have now made a significant decision.
In short, we’re going to design and develop the new website in-house. To do this, we’re going to be utilising the skills and experience of our newly expanded design/UX/UI team. One of the key reasons why we’ve gone down this route is because we now have the internal talent and feel that it makes sense to use it while, as you would expect, our team have a solid understanding of the technology, our vision and the community culture. As an additional benefit, going through this work will help these new team members to bootstrap, this whole process is required in any event in order for us to collectively focus on building out a whole range of areas - very broadly around branding, messages, assets and general design. It has become clear that in working with an external agency, the results are sub-optimal as they simply delay - or in a worst case, misrepresent the way that we need to move forward.
More detail will follow over the next few updates but in essence, we’re excited because it gives us the chance to repurpose this into much more of a SAFE Network (as opposed to MaidSafe) specific site. But, in short, working collaboratively in-house on this area should benefit many areas of the project more widely.
SAFE Browser, Authenticator & API
During the last few days, we were reviewing the feedback we have been receiving for our new DevHub site, especially around the experience the community is having with the tutorials, and making the corresponding adjustments to keep enhancing them. Many thanks to all the people providing feedback and especially to @JPL and @southside for all the info and help in this regard.
This week we resumed our tasks related to some PRs we’ve been working on before DevCon, and we are now reviewing, testing and finalising them, to then be able to merge them into our codebase. This effort is mainly around our safe_app_nodejs package. We are also starting some tasks to upgrade the safe_client_libs in safe_app_nodejs to get some enhancements that have been made during the last few weeks.
One of the features we are currently working on is the support of multipart range requests in our Peruse browser. At the moment it’s already supporting the Range header when fetching a file, but we are now working on also supporting multiple ranges in the same request as specified by the Range Requests RFC (a simple description about Range Requests can be found here). We do this by supporting some byte range options to be passed to our WebFetch function in the safe_app_nodejs package, which is used by the Peruse browser to retrieve multiple ranges and construct a valid and compliant response.
As explained during DevCon 2018, we are relicensing the APIs and language bindings to MIT and BSD licenses and we are currently in the process of updating these (e.g. docs/licence: add MIT and BSD licenses and update README.md by safesurfer · Pull Request #224 · maidsafe-archive/safe_app_nodejs · GitHub).
We have put some of the tasks we are working on Peruse on hold until @joshuef is back from a break. We plan to then release a new package of Peruse that incorporates the new Web API, which is almost in its totality the same API as the one exposed by the safe_app_nodejs package (with a few exceptions that we will explain).
Ashwin and Lionel are starting after their university exams. We will be starting to make progress on the mobile development from next week. The linter errors in the C# bindings are currently being resolved and we are planning to freeze the code by early next week. This will be followed by another round of testing before we get it merged upstream.
SAFE Client Libs
As was announced at DevCon, we’re removing the commercial license on our core libraries (they are now GPLv3 only) and also getting rid of our Contributor Agreement. In essence, this streamlines and simplifies our licensing and reaffirms our commitment to open source. So far, we have already switched most of the core libraries, and SAFE Client Libs will be treated separately: while SAFE Authenticator and SAFE Core will remain GPL-licensed libraries, we’ll specifically make SAFE App an exception as it will be switched to a dual license: MIT / 3-clause BSD. The language bindings we provide will also undergo this change and will be licensed under the MIT/BSD terms. With this change, users of our API will have the complete freedom in deciding how they want to license their apps.
With this change, we’re also making sure that safe_app is the only required dependency for Rust apps. From the developer’s point of view, this means we will re-export public functions from safe_core so that you won’t be required to rely on it to develop apps and thus having to make your app GPLv3-licensed. In addition to that, we’re also ensuring that apps can be developed in Rust without using FFI. In the coming weeks, we’ll improve the documentation and provide tutorials for Rust developers on the DevHub website, demonstrating how to build apps in Rust.
In the background, we’ve been working on a lot of improvements to the binding generation framework: as the C headers generator library, moz-cheddar, got deprecated and is not supported anymore, we have revised it and made it into a header generator that is more suitable to our specific needs. Because moz-cheddar didn’t account for dependencies ordering (which is important in C headers), we had many issues reported by the community and reflected in this JIRA task. We have solved this by making the bindgen aware of types we export and import in each header file. With this information, we could build the dependency graph and sort the includes in the header files properly with the topological sorting algorithm commonly used to resolve dependencies in the correct order. With these changes in place, we are now able to produce complete functioning headers for C and C++. This is not the end of the story for languages support though, as we’ll continue to improve our testing and continuous integration scripts to make it simple for the end-users to download all the required libraries and bindings.
With help from @lionel.faber and the mobile dev team, we were able to fix the known bugs in the Java and C# bindings. Next week, we’ll be covering more test cases and if there are no more known bugs, we’ll announce the public availability of the Java and C# languages support. Besides bug fixing, there was another major improvement to our Java libs. The way we handled it before, with the requirement to have JNI wrapper functions calling into native Rust functions and converting the types between Java and Rust, we compiled these JNI functions as an integral part of safe_app, requiring to have a separate SAFE Client Libs build for Java. Now we found a better way of doing this: we compile all JNI functions into a wrapper library, dynamically linking to safe_app or safe_authenticator, simplifying it from the user’s perspective and also simplifying our build process and the CI pipeline.
This week we also raised a fix for a revocation bug raised by @jlpell. The bug was that a SymmetricDecipherFailure (decryption failure) for an app during revocation would prevent revocation for this and all other apps. First, we changed the revocation logic by moving an app to the back of the revocation queue if it fails to be revoked, so that every app will get a “second chance” for revocation when its turn comes again in the queue. If it cannot be revoked the second time, we return an error. The exception is when the aforementioned SymmetricDecipherFailure occurs, in which case we can never revoke the app – so we immediately remove it from the queue and return an error message. We have done extensive internal reviewing and testing for the new revocation logic so that we could confidently introduce this change.
Routing
As mentioned by @viv and @ustulation during DevCon, the Routing team is currently looking at solving the problem of eventual consistency in the network in a robust way.
Eventual consistency means that every node in the network must eventually agree on the network’s structure. This task is made more difficult as nodes may observe different events (other nodes joining/leaving a section) in a different order due to network latency. Nodes may go offline and stop communicating. Some nodes may be malicious and actively trying to prevent other nodes from reaching consensus.
After considering many avenues, we are following two parallel promising leads.
An attempt at full ordering: BOA (Block Ordering Algorithm)
Our most developed idea considers an asynchronous consensus protocol resilient to Byzantine failures. To be of practical use, the algorithm we devise must be scalable and must not rely on any central point of authority. Such an algorithm would allow all nodes in a section to agree on an exact order for each network event. This would mean that sections’ splits and merges would be trivial to handle.
We made a leap towards solving the problem in the week leading to DevCon, when we realised that we could combine the well known concept of a gossip protocol with an efficient, asynchronous binary byzantine protocol to reach full consensus with optimal complexity of O(N*log(N)) communications between an event happening and its order being agreed upon by all nodes.
While writing an RFC, finalising the details of this approach, we discovered an issue that must be investigated. One of the components of the binary byzantine protocol we are using is a common coin. In simple words, a common coin can be flipped concurrently by multiple parties. All parties flipping the coin will obtain the same head or tail result, but no dishonest party will be able to guess the outcome of the coin flip before the first honest party flips the coin. Many papers, including the one we are basing our work on use Cachin’s common coin implementation for this purpose.
However, this implementation requires an initial setup phase involving a trusted party that will distribute some secrets to all participants.
Requiring trust once at network genesis would be acceptable for us, however we are concerned that with dynamic section membership, this protocol may require trust at each node addition/removal to/from a section.
This of course would prevent the network from being decentralized, which means that it is not an acceptable solution for us in its current state.
We feel that finding a common coin protocol that respects the values of our network may still be possible. We effectively reduced the original problem to a much more contrived one, which is exactly what constitutes progress when designing algorithms.
If we find a solution to this common coin pickle, we will present an RFC with our full approach. If we realise such a common coin is impossible, we will keep looking for an asynchronous binary byzantine protocol, stronger from all the research we have done while exploring this particular path.
Could partial ordering be enough? ACME (Agreement by Continuous Merging of Events)
We are also looking into a different approach to eventual consistency that doesn’t require full ordering of all events. The idea also relies on the use of the gossip protocol to communicate individual nodes’ points of view over the network. However, instead of aiming for a full order of all events, we would accept different nodes to view the network in different states at any point in time. We would then have a way to reconcile (merge) different points of views, in such a way that the network would eventually be consistent.
We think this idea is elegant and promising as it would make for a very fluid behaviour of the network, with all points of views interweaving and creating the big picture.
To “continuously merge events”, we would be combining two main graph representations of the network: a graph created by recording the events that were gossiped between nodes, and another “git log”-like graph that would be used for recording temporary divergences of opinions between nodes and resolve them.
We have some concerns about how to handle splits and merges using this protocol, and we haven’t yet found a definite mathematical proof that the network would indeed be eventually consistent, so there is definitely more work needed here before possibly writing up an RFC about the topic.
Crust
During the past 3 weeks, we started testing the Crust dev branch using Vaults. Since the Routing team is advancing fast, we’re trying to make the latest Crust changes usable as soon as possible. Our tests revealed some bugs in the tokio-utp crate, some required changes to our encryption process, etc.
We finally fixed the infinite uTP Reset packet loop (#2589). The problem was that the uTP protocol doesn’t specify how one should handle packets with unrecognized connection ID. In such cases, we used to send Reset packet back to the sender. It turns out this is not a good idea; if the incoming packet is Reset - that resulted in sending Resets back and forth. There’s another uTP bug we’re currently working on: it seems like in some cases uTP connection graceful shutdown is not working properly. Which unfortunately is also not specified. Hopefully, we’ll manage to contribute to protocol specification when we’re done.
We’re also planning to change the interface related to encryption in Crust. We will have a separate trait that defines encryption methods. Hence, Crust users will be able to do encryption on their own. Which is nice when you don’t want to pass your private keys to Crust, etc.
In addition, we fixed netsim issues on Ubuntu and Debian, it should be working fine now. Besides full-cone NAT we also implemented port-restricted and symmetric NAT simulation. Unfortunately, we are still unable to run netsim on Travis CI. We spent quite some time investigating what’s the cause, but our research was fruitless for now. We’re planning to run netsim based tests on Jenkins driven machines, but if you’re interested in using netsim, keep in mind that Travis CI is currently not an option. We’ll keep you up to date with our findings though.
. Man I can’t wait to see eventual consistency get sorted. I’m glad there are so many team members working on it and also in parallel. We’ll be cooking once that’s all said and done. Get creative team and keep pushing! You are the best bunch of minds in this space so we know you can do it!
