SAFE Network Dev Update - February 27, 2020

Development Updates
1

Summary

Here are some of the main things to highlight this week:

17a
17a830×430 220 KB

General

We have contracted an expert in conflict-free replicated data types to review the SAFE Network and also help to prototype a byzantine fault-tolerant CRDT solution. This could allow us to efficiently handle highly concurrent data as well as offline working and much more. This also, and perhaps more importantly, will provide further formalisation of our data types with concrete guarantees on correctness moving forward. Much of our data is very close to a fit for CRDT, but not formally so. This exercise will, at minimum provide us with much better clarity on the correctness of data mutations. This work is scheduled to begin March 5, and is expected to take about 8 weeks. More details will be forthcoming as that work proceeds.

Vaults Phase 2

Project plan

From the observations in internal testnets, we’ve been making a lot of improvements in Routing, PARSEC and SAFE Client Libs which is bringing us closer to the end of our baby Fleming milestone. With the improvements in gossip (more details in the Routing section) we are seeing stabilised CPU and memory usage by the vaults and we’re having passing tests too. We made some improvements in Client libraries to improve speed by skipping the part where we wait for all vaults to respond to GET requests.

The first implementation of the network launch tool (to be able to launch a local Baby Fleming network) was also finished, and successfully tested in all (Linux/macOS/Windows) platforms. This tool covers just the very basic requirements for the upcoming release and it’s already integrated in the CLI, which means there won’t be a need to release it separately, we can make use of it with specific CLI commands (see below for details).

SAFE API

Project plan

Last week we finalised an enhancement made to the CLI files sync command which adds support for detecting modified files even when their size didn’t change. This was made possible thanks to the ability to calculate the XOR-URL of local files without needing to upload them to the network, enabling us to do this in a very efficient way. By simply comparing the XOR-URLs of local files with the ones already linked from the target FilesContainer we can detect which files were modified and therefore re-upload them in a sync up operation.

We were able to finalise and merge the PR to have vault related commands in the CLI, and we also added detailed instructions to our CLI User Guide for how to use them to install a safe_vault binary, and for launching a local Baby Fleming network, making use of our network launch tool. If you take a look at the User Guide you will already be able to understand how simple it will be to run a local Baby Fleming via the CLI once it’s ready and released.

Since we are in the process of having all PRs and crates ready for our Baby Fleming release, the current state of the master branch of the safe-api repo is not compatible with vault v0.20.1, either running it locally, nor with the shared vault. Therefore building CLI/authd from master branch cannot be used with a single-vault network, and it will be usable only with the Baby Fleming network.

We’ve also started to make the necessary changes in safe-api to be able to support content streaming by allowing to fetch files providing a range of bytes. This will then need to be exposed by the desktop SAFE Browser so any webpage can stream videos or any other type of content in the standard way, i.e. through the HTTP Range header.

Labelled Data, Indexing and Token Authorisation

RFC, Tokens RFC, Project Plan

This week we’ve polished off the vast majority of the token implementation :tada:. With a final PR under review for the vault, enabling proper verification of Tokens and Caveats in vaults, checking these tokens against stored app credentials, which enables us to detect and reject non-current (previously issued) tokens for a given app.

As we get these final PRs merged to our main token branch, we’ve started planning the next steps for data indexing. This is the process whereby all data a user PUTs to the network will be recorded in their own personal indexes so that it can be found later (and it’s this process which we’ll later be using for managing data via labels too).

Some refactors are being made in the Client Libraries to separate the FFI-specific code from the core logic. This clears the path for the new changes coming in SAFE Client Libs which will speed up development of new features, improvements and refactors.

SAFE Network App

We’ve been hard at work on the UX of file and data permissions, and we thought you’d like a whistle-stop tour of the progress to date. @jimcollinson has put together a video on that very subject:

Let us know what you think on how it’s all shaping up!

SAFE Browser (desktop)

Project Plan

After some classic Windows-CI pain this last week, we’ve decided to take the plunge and convert our Continuous Integration (CI) system to GitHub Actions. This has – somewhat shockingly for CI work – been quite a pleasant affair. With GitHub Actions proving to be powerful, flexible and fast. We’ve finally got the CI system signing all our release products here. (Previously Windows had to be done manually). As this was working nicely we decided to move over tests and automate more of the release process too. All of which seem to be working faster than it was on Travis CI.

Now we’ll be automatically generating alpha/beta releases from just a tag (no human intervention necessary at all!). Which should take a few things off of Stephen’s desk.

The only marginal downer we’ve had in the process was getting macOS end-to-end tests to go on GitHub Actions. This has proved a pain as it appears some user interaction is needed to accept some permissions, and there’s no obvious way around this :expressionless: (if anyone has GHA E2E experience and has a solution for this, let us know!). In the meantime, we’ll keep running macOS e2e tests on Travis CI.

SAFE Browser (mobile)

Project Plan

Today we released version 0.3.1 of the SAFE Mobile Browser :tada:

This patch release comes 2 weeks after our v0.3.0 release supporting the Perpetual Web and includes several fixes and improvements based on the feedback we received from that release. For example, we have fixed an authenticate loop on iOS, as well as iOS system-level dark theme overloading issue, and added support logic for audio streaming on Android.

For a full list of changes in this release see the changelog here or have a look at the Done column of our Project Board here where you will see the issues and PRs with the v0.3.1 label.

For full instructions on installing the Browser on your mobile device, and on how to connect it to a vault, please see the previous release post on the forum here.

SAFE Authenticator (mobile)

Project Plan

To accompany the new Mobile Browser, today we also released version 0.2.1 of the SAFE Mobile Authenticator :tada:

As with the Browser, this Authenticator release also contains fixes and improvements primarily from feedback received in the last couple of weeks. This includes a reworked Add a vault page and a generic error message on login details not matching.

Again, for a full list of changes in this release see the changelog here or have a look at the Done column of our Project Board here where you will see the issues and PRs with the v0.2.1 label.

Remember to have a read of the previous release post on the forum here for full details on the Authenticator and the Browser, and how to install and use them.

As always we encourage as much feedback as you can manage to help us fine-tune these apps.

Routing and quic-p2p

Routing Project Plan

This week we finally finished the work on improving gossip (as mentioned in the previous two dev updates - gossip is the mechanism by which nodes exchange messages to update their internal state in order to reach consensus in a fully decentralized manner). After exploring a couple of dead ends we eventually found an approach that removes the artificial delay between gossip messages while also avoiding flooding the network with too many messages. This resulted in a pretty nice performance improvement in our internal “Baby Fleming” testnets. Additionally, we exposed a couple of parameters which allow us to tweak the network further, possibly allowing further performance gains after more testing.

We are continuing to work on quic-p2p improvements which are shaping up nicely. We discovered some areas for improvement while carefully looking into the bootstrapping logic: in several conditions, there could have been a case where some unneeded connections lingered on and consumed resources, and the echo service logic introduced some needless latencies (i.e., the initial connection to the SAFE Network took more time than necessary). We are reworking these modules now and with all these changes quic-p2p should become even more efficient and lean.

Useful Links

Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

73 Likes
3

primero first

27 Likes
4

What a great set of updates, could you give us more information on who the expert is? Would love to hear about their background.

22 Likes
5

:slight_smile: Thanks for sharing weekly updates.

13 Likes
6

Well done guys seems like a lot of pieces are coming together nicely now :+1:t2:

15 Likes
7

Great work as always! :star_struck:

13 Likes
8

Keep up the great work! Excellent to see the new releases and Baby Fleming sounds closer than ever!

16 Likes
9

Nice one, thanks for all the hard work.

Now I know that the mysterious GHA tag means GitHubActions…
So we need to wait for safe_vault-0.21.0 until we can run our own Baby Flemings at home? Have I got that right?

Anyway off to watch @JimCollinson 's video now

Thanks again, team :slight_smile:

18 Likes
10

Have not agreed to name him so chose not to in the update. If you are aware of rust-crdt then you won’t be too surprised :wink:

15 Likes
11

Correct @Southside there will need to be a new vault release to support Baby Fleming from home

13 Likes
12

Will baby fleming have a test wallet and coin?

16 Likes
13

Yes, but not yet farming.

22 Likes
14

YAY! thats super awesome

17 Likes
15

Thanks so much for all of your hard work. This is great progress.

16 Likes
16

nice I can’t wait til you push that baby out :stuck_out_tongue:

20 Likes
17

Cool video again @JimCollinson
Can’t wait to see more of the Safeapp

14 Likes
18

Great update!! Exciting to see stratengic investments being made like the

Also, got a notification on the UXD video earlier today and shared that. The video provides an excellent opportunity and vehicle to broadcast the work MaidSafe is doing :smile:

21 Likes
19

Quietly feels like huge steps are being made forwards at all levels of the project, seems like everyone’s got their heads down and working even harder than usual at the moment, so thanks all for that!

Nice video @JimCollinson, looks like an epic piece of work. The mind boggles at trying to incorporate all the power of a user’s account into one app, but in a weird way I think it’ll actually end up being more intuitive to the end user than the current mish mash of native and web apps to work out and learn.

Also, just to be clear, the mind boggles at all the work going on here! Or at least, it does when it even begins to understand…

20 Likes
21

Really great update guys, impressive stuff :slight_smile:

Haven’t read the CLI User Guide for a while, but that one is really good now. Must read for everyone (also non-technicals) to get an understanding of the network functionalities.

There is still one part where I always get lost, and that’s the safekey part. Because of ‘key’ I always thinks this is the key that gives me access to something, but it doesn’t. And then there is secret key, public key, wallet…
I understand the choice for these names from a technical background, but for an average user it is confusing I think.

To make it easier to understand I would suggest to rename safekey to safelocker.
So we have a safelocker that contains coins, something else valuable, or is just empty. To open this locker we need a secretkey that we own ourselves. Now everybody understands that a safekey(=locker) will never hold the secretkey…
To know the balance you have to open up the locker, so you need your secretkey.
The public key is just the address of the locker. So lockeraddress?
Also a wallet is much easier to understand, it is in fact just an old fashioned keychain with keys that can access your lockers.

Well, at least I find it much easier to understand it this way, maybe it helps others.

15 Likes
22

Thanks very much for the video @JimCollinson—I’ve been getting a lot from these. Also it’s always great to see some of the friendly faces behind MaidSafe.

And of course awesome to see this week’s updates!

:ant:

18 Likes