MaidSafe Dev Update - October 12, 2017

Today we finally moved out of Troon to the nearby town of Ayr. The move has consumed a lot of the operations teams time over the past few weeks, but fortunately we have been able to minimise the disruption to the engineering team. As promised last week, here are some pictures of the exterior. Over the coming weeks/months, we will be customising the interior and making it more our own.

It has also been a busy week in development, we just released an initial version of the safe_app CSharp Library. The first version has been published on NuGet and the safe_mobile repo has been updated, so the SafeMessages app uses this package now. This makes it simple for people creating mobile apps as all they need is a dependency to this package and the authenticator and they should be good to start building for mobile right now.

It’s worth noting that the safe_app CSharp library lists specific TODO points that are our current highest priorities as this is by no means a feature complete package for Alpha 2 integration.

SAFE Authenticator & API

The openUri API in safe_app_nodejs had a cross-platform issue and we were relying on Electron’s shell.openExternal API during the development of Alpha 2 applications. We have resolved this issue and tested across all platforms.

Another issue that we were working on during the last few days was related to the URI scheme registration. Up until now, it was needed to package the browser every time we had to test the authenticator integration since the URI scheme wasn’t properly registered when running the browser in dev-mode, i.e. with yarn start. This was time-consuming and solving it was a bit more challenging than expected as we encountered different types of issues in each of the platforms. This should make it much easier for devs to test changes in the browser and plugins as the packaging step can be skipped during the development process.

@hunter has been refactoring the email app to simplify the code mainly by the use of async/await functions, and the PRs reviews are ongoing along with internal testing to ensure the app is as stable as before the refactoring. We are also adding more inline documentation to the example apps, to make it easier for devs to understand the design.

@shankar has resumed his work this week and has ported the Web Hosting Manager app to Electron Forge and SASS in order to use consistent tools across the apps. We are refactoring the Web Hosting Manager app in an analogous way to the refactoring made in the email app.

@joshuef is looking into options to scale the safe_browser to improve efficiency. This is still in the early stages and we will share a detailed plan once we have one in place.

@srini created a set of standalone Node.js example apps which make use of mock routing and the loginForTest testing function to demonstrate the @maidsafe/safe-node-app API usage. The loginForTest function creates a client app without sending authorisation requests to the Authenticator and allows running these samples as standalone targets, which should be helpful to learn about the safe_app Node.js API. Details of how to run these example apps can be found in this README file.
Web examples were being requested by community members; @srini has started working on a simple web example, a Disqus-like comments plugin which can be integrated into any web page.

@Rachit has joined the team this week. He is getting bootstrapped to the SAFE Network, he will be reviving the Java safe_app API soon. JNI integration using SWIG is being proposed for speed and ease of integration. More details in the next section :wink:

SAFE Client Libs

Automatic generation of language bindings has been the focus this week. We’re considering using SWIG for this purpose: it is a tool that is commonly used to automatically generate interfaces for a wide variety of languages ranging from Java to Python, based on C headers. Currently, our primary targets are the common mobile development languages, Java and C# via SWIG (Obj-C/Swift will likely not even require a SWIG layer), but in the future we’re aiming at having more languages in our toolset, including community-supported ports.

We are still considering several other options and weighing the trade-offs, as we are balancing between providing the best possible user-facing API and saving on development time. One of the options is to use a SWIG feature called type maps, using which we can get the most idiomatic, automatically generated Java API, but at the same time it has a disadvantage of being tightly coupled with JNI, the Java binding interface - in practice this means that we can’t reuse the same SWIG interface for other languages and we need a lot of time to delve into language implementations to generate the best possible idiomatic APIs. Another option is to base APIs on a SWIG feature called directors, which provide the easiest automated way to generate bindings, and it is a recommended approach. But we are seeing some issues with it: the resulting APIs might not feel “native” to a particular language, and we want to get easy-to-use well-documented interfaces. That’s why we are also considering a third option, to code the bindings manually, which is a kind of a last resort. In general, our research progresses well, and the current progress and proof-of-concepts can be tracked in this repository.

Continuing our work on updating the API, we replaced all network-events callbacks in safe_authenticator and safe_app with a simpler callback, o_disconnect_notifier_cb, which signals that a disconnect occurred when called. This is a consequence of a previous simplification in the callbacks, when we removed the automatic reconnection (making the front-end responsible for this), effectively making the notification about the Connected state unnecessary. After long internal discussions, we ultimately decided to discontinue the use of value-type structs (namely, PermissionSet) in our function signatures. Value-type structs would hinder our options in generating the best possible SWIG bindings - e.g. see the “5.4 Other Practicalities” section in the documentation. Additionally, using pointers for all struct types is a more accepted C/C++ convention.

Routing, Vault & Crust

Routing efforts this week were still focused on secure message passing and resolving possible issues with it. We identified certain limitations and did some simulations to estimate their severity. Node Ageing and other features certainly influence such discussions quite a bit compared to just the code in the master branch. Apart from that, we are about to start discussions related to the implementation for the first part of Data Chains, which we are calling the “Section Graph”.

We are continuing to port Crust over to use Rust’s Tokio and futures libraries. The implementation side of things is almost done, though the code still needs to be reviewed, tested and battle-hardened. For those interested, the work-in-progress implementation can be found here. Once this refactor is finished we will be able to integrate our Tokio-based uTP library and have a Crust which runs over both TCP and UDP.

80 Likes

Again first!!!..

Sounds like a great improvement!

Nice! My next safe site will have a comment section :grimacing:

Aaaaaaaah here comes the unbelievable stuff too - so cool!

11 Likes

Damn it :laughing::laughing:

Section Graph…wanna know more :thinking:

12 Likes

Congrats to the new location.

12 Likes

Data Chains part 1. It was called option B (which I hated with a passion). Anyway it is the routing layer section of data chains. It’s also by far the hardest part. We have spent ages debating details and simulated a lot of different strategies for this. Good to have choice, but we are really going for a measure twice cut once approach here.

A lot of hard work but probably about the most thought out part of the network so far. Well defo more so than recent implementations we have done in a frenzy. I do hope this one will be smoother.

To make it harder though some crust related things will happen in tandem, Utp support, better NAT traversal and much better async handling.

in parallel we will continue making sure the client side apps API is as expected, but we do know some things like multisig will happen probably after alpha3/4. Just in case there is a surprise in impl for that, but I don’t think so.

33 Likes

That instant response made my morning. Cheers :beers:

13 Likes

So why did you rule out option A in the end. Was it officially a dead end after simulations and so on ?
Did it provide not enough flexibility ?

6 Likes

Thanks so much again for all of the hard work. Is “Ayr” just pronounced “air” is it pronounced “ire”?

3 Likes

Oh man 19 weeks of debates about absolute knowledge in an eventually consistent network was frustrating to say the least. It had a pretty major attack vector as does any leader based consensus approach and there is no way round that we could see. It was closer to traditional thinking and is just did not fit the way MaidSafe was ever designed really. The debates almost finished me off to be honest. It was a tough tough time for sure.

The simulations for section graph have shown no such weakness or error. We spoke about it in a few dev updates as well mind you. Felt to me like arguing against a server based approach, so many great arguments for a server and admin that can be trusted as well. Drains the life blood from you but it’s an important part of the process to debate all the way through.

Bottom line, any system that tries to have all peers in complete agreement at all times before acting on anything will fail 100%, its unnatural and cannot be found anywhere in nature. But humans love that stuff man, we soak it up and believe we can control so much, but in fact the world works very differently from that.

46 Likes

Yip Ayr is pronounced air. As Burns described it “The toon o’h honest men and bonnie lassies”

15 Likes

I’ll tell mom your company has moved to “air”. I’m sure the staff has joked about this already. The name is probably a source of mirth for all the locals.

7 Likes

image
This building has MaidSafe written all over it: Functional and practical with an air of mystery.

22 Likes

It will soon :smiley: We will be branding it up with SAFE network and MaidSafe branding.

29 Likes

Do you own or rent the building? It looks brand-new.

1 Like

Oh we just rent it. There are 4 more there, so we can expand if needed.

14 Likes

Maybe you should place a reservation?

3 Likes

Up to 5 of these same block-shaped buildings, maybe also side by side. Are you sure this is a good idea … :wink:

5 Likes

Welcome on board. Watch out for the person with the highest nodeage while bootstrapping :wink:

Thanks for the very detailed update again. Lot of stuff happening.

Will be part of alpha 3 I guess? More easy hole punching etc.?

15 Likes

Thanks for fighting through the pain and suffering that I am sure of is going on behind the scenes to crack these difficult problems. I can relate through some inventions I have thought on, even if they where not rocket science like this one. All support! A navy SEAL once said and maybe it´s also true, “If it dont hurts then it is probably not worth doing” :slight_smile:

8 Likes

Great news - more fantastic progress. Glad to hear the move’s gone well. Onwards and upwards :smile:

8 Likes