Summary
Here are some of the main things to highlight this week:
- MaidSafeCoin is now on Blockfolio’s Signal service, which provides us the ability to send push notifications to share important news about the SAFE Network.
- We are experimenting with providing a short video summary of the weekly Dev Update. Here’s a trial video made using last week’s Dev Update. Please let us know if you have any feedback on this format.
- We released a new Medium post (‘The Successful Conclusion of the Crust Tests’) and @ustulation chatted through the Crust Test results as a guest on @fergish’s SAFE Crossroads Podcast this week.
- A SAFE Network: London meetup is scheduled for Wednesday 28th November and we’re delighted to announce the launch of another new meetup: SAFE Network: Brighton will hold its inaugural get-together on Wednesday December 5th.
- The proposal for having XORname based URLs (XOR-URLs) supported in our APIs has been formalised with an RFC which is already available for review and debate. We’d like to invite all of you to participate in the discussions happening in the PR itself.
- SAFE Client Libs 0.9.0 will be released later today. Numerous improvements and fixes have gone into this release. See the SAFE Client Libs section below for a list of highlights.
Marketing
Big news this week: we’ve finally got onto Blockfolio’s Signal service! For those who don’t know about this, Blockfolio is a cryptocurrency portfolio tracking app. But it’s not just any app. It’s the largest in the crypto space, 5 or 6 times larger than its nearest competitor. Other than the reach, the really interesting thing about it is the fact that it provides the ability for push notifications - in other words, if there’s a major piece of news about the SAFE Network, we can push it directly to the thousands of mobile phones around the world who are already watching MaidSafeCoin. We know that many in the community were keen for us to go down this route and we’re delighted to be up and running - just download the app and click to watch MAID to experience it for yourselves.
Next up - something else we’ve been keen to experiment with for ages. Many (many!) people have asked for a quick video summary of the Dev Update each week to give another (perhaps less-technical) perspective on the weekly evolution of the Network. So we’ve just recorded a quick rough ‘n’ ready video using last week’s Dev Update to trial the format and you can now take a look at the video here. We’ve not gone for perfection - just a quick take on what we feel might be achievable in terms of resources and production quality each week so please let us know - is this useful (and how can we improve)?
We’ve just released a new Medium post (‘The Successful Conclusion of the Crust Tests’) and @ustulation chatted through the Crust Test results as a guest on @fergish’s SAFE Crossroads Podcast this week. It was also great to see our piece on the evolution of the Internet get picked up by the Computing.co.uk website.
Final pieces of miscellaneous news: Trezor have announced firmware support for Omni (and therefore MAID) in December 2018. Perhaps someone within the community is a wallet developer who might be interested in looking into Trezor Connect? In terms of being out and about, @dugcampbell was at the ScotChain 2018 Conference in Edinburgh last week and we have a couple of meetups coming up in the very near future. A SAFE Network: London meetup is scheduled for Wednesday 28th November and we’re delighted to announce the launch of another new meetup: SAFE Network: Brighton will hold its inaugural get-together on Wednesday December 5th (organised by @No_2 and @cryptonomy_louise), when @nbaksalyar will be flying in to give an introduction to Crust and chatting about the recent Crust Test results. As always, it would be great to see the meetups supported: so please visit both groups and sign up as a member, even if you can’t sign up to attend any of the actual events. Every additional Meetup member makes it more likely that someone else will join the group so you can all have an impact on this front.
Recruitment
This week we only have a small recruitment update and that is to welcome @Lindsey to the team as Executive Assistant. She will be working closely with @Viv and the rest of the development teams. Please give her a big forum welcome.
Our other new starts join in December and we will introduce you to them when they start - you don’t have too long to wait… 
The Network Engineer role is still an ongoing project and we will update you when this progresses some more.
SAFE API & Apps
@ravinderjangra and @AshwinKumar have been working to improve the mobile authenticator features. The previous version of the authenticator used to handle only Auth requests. Now, it is being updated to handle Container requests, Share MutableData requests and Unregistered Access requests. Also the strength checker as in the desktop authenticator is being integrated and tested along with a few bug fixes such as auto reconnect and refresh of the registered application list. The authorisation popup is being enhanced to show more information. We hope these quick enhancements will be helpful for the mobile developers who will be trying out the APIs.
During internal testing of the safe-getting-started-android application, we observed that a number of local references were being held in the JNI layer. This eventually led to a local reference table overflow error. Another important aspect we wished to cover in the example application was the handling of the network disconnection event. On adding this feature, we realized that the disconnection event was fired only once. We were able to reproduce these errors in safe_app_java via unit tests through which the Client Libs team were able to debug and resolve them. These fixes will be part of the v0.9.0 release of SAFE Client Libs. We are also updating the packages and sample application to work with these fixes and the latest API changes.
The experimental APIs that we’ve been working on for RDF and WebIDs, exposed from the safe-app-nodejs package, are now being moved behind a flag (--enable-experimental-apis). This is to make it more explicit to the end user, and developers, if an application is making use of any API which is in an experimental phase. As we explained a couple of times, these APIs are made available to allow everyone to experiment with them but they shouldn’t rely on them since they can change, or even be removed, at any time in the future. The PR for the first experimental function, the fetch function, has been already merged and we are now reviewing the PR corresponding to the experimental RDF emulation layer.
As well as ongoing bugfixing in SAFE Browser, this last week has seen a refactor of the application’s network connection logic in preparation for enabling/disabling the experimental APIs in the browser. This refactoring has the knock-on effect of cleaning up some of the older browser application code, helping reduce the number of ‘Apps’ in the browser from two to one (instead of having one anonymous one for ‘Fetch’ and another authed one for ‘save/read’ functionality, there is now only one that gets upgraded/downgraded based on if the user is logged in or not).
Some of you were already participating in the early discussions on our Dev Forum about the proposal for having XORname based URLs (XOR-URLs) supported in our APIs. This week this proposal has been formalised with an RFC which is already available for review and debate. We’d like to invite all of you to participate in the discussions happening in the PR itself. Please feel free to comment and/or ask questions in either the DevForum thread or as a comment in the PR itself.
A very nice and interesting discussion is still ongoing about the RFC for the use of RDF in our Public Name system. We would like to thank those who are actively participating, as well as encourage those who aren’t to do so; it’s very helpful to have the community getting involved in this deep dive of technical discussions.
SAFE Client Libs
Today we’ll be releasing SAFE Client Libs 0.9.0. Numerous improvements and fixes have gone into this release. To highlight a few:
- Java bindings are stable now, with a lot of fixes applied to make them work across different platforms (including Android);
- We switched to base32 encoding for IPC (and made the encoding format future-proof by using Multicodec);
- A severe data corruption bug related to the mock version of the library was fixed;
There are also lots of small documentation improvements that compound to a more polished library overall. There are some breaking changes to the existing FFI API, but they do not break the JavaScript, Web, or Java/.NET APIs. Consequently, you will not need to change any code in your apps. You can find the full list of changes in the individual libraries changelogs.
With this release we stabilised the SAFE Client Libs at its current state and we are ready to undertake the exciting journey of incorporating changes that might be required for better support of SOLID and RDF at the API level of the network. It is yet to be defined how we would proceed with that, but in cooperation with the front-end team we’ll start thinking about our requirements and implementation details soon.
Routing
Following our first steps in the integration of PARSEC with Routing last week, we have focused our efforts on making this integration easier from the PARSEC side of things.
This work has involved three main aspects: More intensive testing of dynamic membership, one expansion to the way we allow client code to customise interesting_events and performance.
Extensive testing of dynamic membership
From the testing side, we now have extended the basic example to support dynamic membership. One can now add and remove peers while running the basic example. With this addition, the example has become a useful tool to reproduce a wide range of dynamic membership scenarios.
We also started working on creating a more extensive integration test. This work already allowed us to identify and fix a bug where the peer’s state wouldn’t be updated early enough.
Customisable interesting_events
A while ago, we introduced the possibility to customise what defines an interesting_event in PARSEC. This means that instead of only reaching consensus on stable blocks of votes, the client may simply require consensus on each vote and assemble them into stable blocks at its own discretion.
This week, we have worked on two aspects of this feature:
Changes that affect the topology of the network should always require stable blocks as this is necessary for the validity of Parsec’s invariants in a dynamic membership enabled network.
We improved the logic with a customisable interesting_event function so that it works well in the presence of forks. Forks allow some gossip events to “unsee” events that their parents saw. We made it so that if an event is interesting, its descendants will always see it as an interesting event in the presence of forks. This fixes a conceptual flaw with our previous approach.
Performance
And now for the most glamorous part of this week’s update: we started having a look at performance.
We improved our criterion benchmarks by creating three reproducible cases: “minimal”, “static” and “dynamic”. We gathered performance profiles using perf and visualised them using flamegraph.
A flamegraph is a representation of the time spent in each function in the code. It helps identify places where the performance can be improved. Here is an example with the graph we used before applying any fixes to get a feel of where the bottlenecks were.
This allowed us to identify three main bottlenecks.
The first one was that we reprocessed events too often. This is a performance aspect we had consciously ignored when implementing dynamic membership, but now the data showed that it was indeed worth looking into.
The second one is about the payload hashes being calculated too often. There is no reason in theory to calculate the same hash over and over again as it is a CPU intensive operation and it can be cached.
Finally, we currently store the events that form the gossip graph in a BTreeMap. It’s a data structure in which access is slower than some alternatives like HashMap or Vec, depending on the use cases. We can see that accessing items in the BTreeMap of events takes a lot of PARSEC’s execution time at the moment.
Armed with these pieces of knowledge, we started implementing performance improvements. Currently, only the first fix made it to master, but it already reduced the time of our Routing integration example from 80-90 seconds down to 9-20 seconds.
We are addressing the second and the third fix at the moment. We will keep looking at the performance profile until there is no obvious “low hanging fruit”, and then we will leave any remaining performance work to the end of this milestone as we will want to take into account the performance impact of malice detection in the data we measure so we can spend our optimisation efforts optimally 
Crust
Work continues on the current milestone of getting Crust ready for Alpha 3. This week we spent time solely on socket-collection as it will save some work in both the P2P and Crust libraries. We updated the library dependencies, enabled Travis CI builds and fixed tests on the Windows and MacOS systems (see this PR). We are planning changes that will touch most of the code base. Hence having the integration tests reduces the chance of regression. So the very first step we took after CI was enabled was more tests. Once we had some tests in place we started refactoring duplicate code more aggressively. And finally the crypto integration into socket-collection has begun.
Ideal… really well done. I wouldn’t expect to engage with that format but plays well to Doug’s strength as clear communicator… 8mins is perfect length too.
