MaidSafe Dev Update 12th April 2016
You will need to forgive the less eloquent update this week, but the team is reduced with holidays and @ross is spending time with his family on a well deserved break this week.
This has been a pretty terrific week for us in many ways. Crust and Routing now appear very stable in tests. We have a droplet deployer script to start up networks quickly and churn these (kill and start nodes) in a programmatic manner. These tests allow an end-to-end test of the modules. Great news crust and routing now are at a point of feature freeze for the MVP. There are some small changes, but only to satisfy other libs, the core of these libraries are now considered stable and MVP ready (as a bonus, with Utp and home networking).
This allows us to relocate resources (not all though, these libraries will evolve) to upper layers and this really is great news. Crust is low level and interoperates with the Operating System and network (I/O) which is very difficult to get right across operating systems and networks. Having this solid is a very large hurdle. Previously this module was a black hole where things could fail in upper libraries with the constant question “is that a crust error?”. Now we have passed that.
Then routing, the “workhorse” of the network became feature complete and tested to an acceptable level for MVP (Minimum Viable Product). This again was a library that was a real developer sink in terms of resources and again had to guarantee its promises were in fact being maintained. It now does! (we satisfy the kademlia invariant with a recursive version of that protocol and managed connections, this is pretty huge).
These are like an avalanche really as we move downhill to launch. Each of these layers allows a relatively huge resource base to be applied to the upper layers. Many of the resources we had in the lower modules were very highly skilled and can now deploy to upper layers, it is a huge relief).
Now we have vaults to confirm and that is happening now with 7-8 developers on a hangout every day to check/double check and confirm vaults do as they promise. Not only do they work, but make sure they are easily extensible. This is important there to allow features which will only exist in vaults and clients to be implemented with significantly greater velocity than previous. It is a very exciting time.
To this and and with the release in clear sight we have retasked the installers project to confirm all is well there. It is!
In addition we are pulling together deploy capabilities for the community as we finalise this library. So expect Docker containers as well as easy deploy to cloud providers to quickly run networks of several hundreds or more nodes quickly and in a controlled manner. We should give the ability to create a vast network very quickly and allow those interested in farming to try and test any ideas they have in a measured and automatic way. We feel this will be very helpful to all community members.
The question we never answer is “when”, but now we are so close we feel comfortable saying we may not even make the next update before vaults are considered ready. We are that close, although the days between updates do feel like minutes. So in terms of where we are, we are comfortable that this last bit of work (confirming churn in vaults) is not insignificant in any way, but completely under our control and very well resourced, it should be over very soon. Please don’t slaughter us with this one, we are trying to convey our feelings and desires, not promises.
This allows the network to release in MVP form, but more important for many is “what can we do with it?”
So the client libraries are the point where the capabilities become obvious to app developers and on to end users of such a network. This part is very important.
Krishna, Spandan and Shankar have all been pushing forward RFC’s and have had great feedback and debates with community members this week. Only the very start, but will allow us to introduce more capabilities to the developer community as efficiently as possible. Expect bigger faster changes here as resources currently in vaults get going in this area as well. If we can apply a decent amount of resources there to encourage more debate then these parts will accelerate through to the people that need them, end users, via the app devs.
We are now looking much closer at the API’s we produce and are very aware of recent developer contributions and projects being developed, funded and resourced. It is our intention to do everything we can to push these projects and help where we can. Expect to see some mechanism to list and showcase these projects very soon, either by us or with us fully supporting some of the efforts there.
At last the development process feels like it can move again and really focus on delivery with code and algorithms we are in control of. This is now the part of the project where we will see progress happening quickly and much more in line with the community. We are appearing from the depths of some tough deep code that in many cases is outside of our control into a place where we are in control.
To this end we have also re-organise the teams in house to better facilitate growth of MaidSafe as a company. This organisation chart is included here to allow everyone to see the team grow and organise.
I should say the roadmap is still not published, it’s become a bit of an issue it should not have. We have time to look and sort this out now and we will. I must apologize here as we let this turn into a mini project requiring a roadmap for the roadmap 
. We will fix this before the next update… No excuse, we screwed up there and will fix it now.
So in summary we are in a great place now and very much more in control. We can begin to allocate resource to tasks with a much more direct impact on you the people that matter. That feels great!
Crust - uTP / API : Vinicius / Andrew
This week saw the conclusion of a working crust. VinĂcius managed to fix the last blocker for uTP. So good news is that the MVP will not be restricted to Tcp alone, but also Utp. This will allow full NAT traversal testing. In short the MVP should now be able to be run from people’s homes.
Andrew is working hard on a redesign and cleanup of crust internals. Mainly making the library more modular in readiness for a move to a more resource efficient mechanism. We anticipate this modularisation effort to complete by the end of next week. It is not a blocker for MVP, but a very good intermediate step to move the library forward.
Routing - Vault Integration : Andreas / Adam
After some more testing and a few final API changes requested by Vaults, we created a stable branch for the version 0.15 of Routing which will be used for the MVP, while development can go on on the master branch.
We also updated and expanded the documentation, and added diagrams of the message flows:
Bootstrap: https://cacoo.com/diagrams/cqX6QPN90ZuKXZ0n-F56A2.png
Get close group: https://cacoo.com/diagrams/PTBt1OgHVcdu0PKt-F56A2.png
New node: https://cacoo.com/diagrams/5VCFe286q4yfQ6Pm-F56A2.png
Tunnel: https://cacoo.com/diagrams/ALXyW2ugR92IxgWd-F56A2.png
Vaults : Brian / Fraser / Qi
The major achievement of last week was having mock_crust in place for the integration test. This allows us to simulated more stressful and flexible scenario than the previous ci_test (using vault executable setup a local network).
The class and sequence diagrams now have also been included to help the flows to be easier to understand:
Personas: https://cacoo.com/diagrams/wl6of3FUFriB0FWO-0BD19.png
ImmutableData Put: https://cacoo.com/diagrams/SCHrwEhLRB86EGe1-EF9A0.png
ImmutableData Get: https://cacoo.com/diagrams/ndcPMKC3WapABSaA-EF9A0.png
Reviewing of the put/get flow based on the above diagrams has been carried out, and some refactor work has been done and confirmed.
A new feature, network full, has been added to reject client’s put request when the network as a whole reaching a storage threshold.
Current work is ensuring churn handling, which is the last step to MVP.
Client - Launcher : Krishna / Spandan / Shankar
The RFC for exposing low level apis was raised last week. @cretz has been really helpful with his suggestions in the RFC discussion. We have been patching the RFC as the discussed points get addressed. We hope to address the concerns raised in the RFC discussion in the next couple of days.
We are also working on a RFC that will detail the standards that the launcher APIs need to incorporate. At present we are trying to get smaller POCs in place to support this RFC. In this RFC we are trying to address the inconsistency in the errors being sent, using http headers where ever we could, avoiding base64 conversions, etc.
Once both the RFCs are accepted, we can start planning the tasks for these two RFCs and get it rolling.
With the standards incorporated and the low level apis exposed, the next version of the launcher should be more dev friendly and stable.
UI Design : Scott / Shankar
Scott is also on holiday this week. He spent last week on progressing the roadmap and producing a video to describe the current API and capabilities from an end user perspective. The video is currently published privately waiting on some detail and links to be put in place then will be made public. This should happen on Monday.
I have not included the individual sheet this week as so many are off, but this will re-appear next week.
