SAFE Network Dev Update - March 19, 2020

Development Updates
1

Summary

Here are some of the main things to highlight this week:

  • We released Baby Fleming iteration 2 :tada: This update consists of a new version of the safe_vault binary (v0.22.0) which removes the PARSEC step for client requests.
  • We released SAFE CLI v0.10.0 which includes (among other improvements) a new files tree command.
  • We released a new RC MaidSafe.SafeApp NuGet package that supports doing range requests and interacting with the Baby Fleming network.

5a
5a830Ɨ430 215 KB

Vaults Phase 2

Project plan

Today we are releasing the second iteration of Baby Fleming, aka Vaults Phase 2a :tada:

As detailed in last weekā€™s dev update, the performance issues in the first iteration of Baby Fleming were primarily due to all client requests that involved the spending of safecoin being sent through PARSEC to reach consensus before processing the request further. Iteration 2, which we have released today, contains the removal of the PARSEC step from the client request-processing flow.

We will of course still need to reach consensus in this flow and that is where BLS - Distributed Key Generation will come in to ease the load on PARSEC. More on this in the BLS - Distributed Key Generation section below. For now though, it makes sense for us to remove this performance restriction to allow you (and us) to test Baby Fleming.

How do I update to Baby Fleming iteration 2?
This update consists of a new version of the safe_vault binary, v0.22.0.
Using the CLI, you can install this new version with the safe vault install command. See the CLI readme for full instructions on installing the vault, as well as full instructions on running your own section of vaults on your platform.

Known issue
You may experience intermittent failures (AccessDenied or NetDataError - Failed to PUT Sequenced Append Only Data logged here) when uploading data. We are aware of this and we are investigating.

When will MaidSafe be hosting a Shared Section?
Our intention is to host a Shared Section as soon as testing shows the sections are steady, with the major bugs resolved.
Note that when we create a Shared Section we will be taking down the current Shared Vault - data will not be transferred from the Shared Vault to the Shared Section.

SAFE API

Project plan

Weā€™ve released a new version of our safe-api (v0.10.0), as well as a new safe-cli (v0.10.0) which includes all the features weā€™ve implemented since the Baby Fleming release. Weā€™ve updated the CLI User Guide with instructions for the new files tree command.

This new version of safe-api is the first version which exposes its API as async, which is an important step towards fully supporting asynchronous calls from these APIs. As mentioned in previous dev updates, internally they are still not asynchronous, and thatā€™ll be our next step in this regard.

The mobile libs build for safe-ffi were disabled for the last couple of weeks. The issue was related to the mobile support for some of the external Rust crates. This week, we resolved the build failure for safe-ffi in a separate branch (baby-fleming-mobile). Once we have the stable versions of the external crates working with our own libraries properly we will merge the code with the master branch. These updated libs will help us support the release of the mobile apps for the single-section network.

Weā€™ve finalised a basic set of benchmarks to be running against a Baby Fleming network. These tests are simply PUTting data onto the network (files of size 0.5, 1, 2 and 4 megabyte). The tests will run these X times (looking for a sane sample size hopefully at least ~50 times), though this will depend on the progress to improve data PUTs in general. For those interested in running this (or contributing more tests), you can check out the draft PR which adds a -t or --test argument to everyoneā€™s favourite safe vault run-baby-fleming -t to automatically start the network, re/start the authenticator and create a test account for the CLI. (Note, this needs both the safe command and authd to be installed in their standard locations).

SAFE Network App

This week weā€™ve been working on some small improvements to the SAFE Network App, updating our Node.js dependency to get ready for a Baby Fleming compatible release, and straightening out the E2E testing.

The Node.js update has highlighted some potential bugs there, so weā€™re looking into that. But once we have that sorted, weā€™ll be able to get this out to play nicely with :baby:

SAFE Authenticator / SAFE Browser (mobile)

Authenticator board, Browser board

This week we updated the Authenticator app to support the Baby Fleming network and tested the app on both platforms (Android, iOS).

We also released a new RC MaidSafe.SafeApp NuGet package today, to enable us to update the Mobile Browser to support Baby Fleming. We are currently testing the Mobile Browser app internally and will be releasing new updated versions of both mobile apps once we are happy with them.

SAFE App C#

Project plan

Last week the Rust fetch API was updated to support fetching a range of bytes from a file, so this week we updated the fetch and get_immutable_data C# APIs to also support range requests. This will allow devs to fetch the content between the given start and end index. We also released a new RC MaidSafe.SafeApp NuGet package. This new package provides support for the single-section / Baby Fleming network.

Routing and quic-p2p

Routing Project Plan

This week we merged the improving message delivery in the presence of lagging nodes work PR (PR 2068). Other major progress achieved includes merging the work of simplifying the state machine: making Elder and Adult into Approved (PR 2071), as well as BoostrappingPeer and JoiningPeer into one (PR 2072). These simplify the state machine a lot, and we will eventually be looking to remove the state machine altogether. Another improvement this week was the merging of the logging identifier PR (PR 2073). This reduces our code and ensures that all logging in the Routing crate identifies the node the log entry was for.

We are also starting to think about conducting some more quic-p2p tests to have better data about its scalability and resource requirements. We expect it to handle hundreds (or even thousands!) of connections at once and itā€™s important to know the limits and plan for them accordingly, to make sure the network systems will not be overloaded. We are planning to start these tests after the new release which will include the IGD support and the latest QUIC protocol implementation improvements from Quinn, the underlying networking library we use.

BLS - Distributed Key Generation

By depending on PARSEC, the spine of the network, for all types of consensus we might just stress the network from perspectives that we donā€™t usually see. For example, data in the SAFE Network can be fast-changing and querying/fetching such data with consensus from PARSEC would take a while to be retrieved. This duration might just be enough for the data to get updated to a newer version from the fetched one. Therefore, to minimise loads on PARSEC and to achieve quick consensus for operations like querying of shared data, multisig wallets, etc., we face the requirement to sideload some of the consensus onto another mechanism. In here comes BLS-Distributed Key Generation, a cryptographic threshold signature scheme, that allows entities to attain consensus on some object swiftly via the nature of its six phases and forcing 100% participation from the corresponding nodes. You can read more about its workings and details here and here. A PoC implementation is in the works by the team that would be tested on shared data by clients initially, although the bigger picture is to use it for cases like the ones mentioned above and would be a generic crate that will be used for attaining consensus without completely relying on PARSEC at all times.

Useful Links

Feel free to send us translations of this Dev Update and weā€™ll list them here:

As an open source project, weā€™re always looking for feedback, comments and community contributions - so donā€™t be shy, join in and letā€™s create the SAFE Network together :tada:

79 Likes
2

FIRST! (20 char)

EDIT: Great to see solid progress continuing. I hope everyone on the team stays safe and keeps up the great work!

26 Likes
3

Getting exciting!

22 Likes
4

Thanks for the great update Maidsafe devs.

Hopefully I can get this baby running on windowsā€¦

It would help a lot if community members did a step by step how to installā€¦ :sweat_smile:

Usually I can replicate something if I see it, not if I read it especially with cli :upside_down_face:

Iā€™m supercharged that this :baby: is now in iteration mode :test_tube: :baby: :stuck_out_tongue:

24 Likes
5

Amazing work in the face of a pandemic, stay safe team, the world needs you!

29 Likes
6

So is CRDT cancelled in favor of BLS or is this still needed?

4 Likes
8

Sounds like the baby is growing well! Thanks for all your efforts, team! It is thoroughly appreciated and offers a glimmer of optimism in a sea of negativity in the wider world.

25 Likes
9

Look at that speed!! Great to hear things are steadily moving forward, especially with all thatā€™s happening globally right now. Excited to continue seeing this come together :smile:

25 Likes
10

Baby is taking steps forward!! :baby:

Iā€™m liking the differencesā€¦ thatā€™s a nice abc walk through at https://github.com/maidsafe/safe-api/blob/master/safe-cli/README.md#description

It just worked wellā€¦ couldnā€™t find anything bug like; so, just a few queries and impressions from me:

Without looking closely, Iā€™m expecting that FilesContainer is mine, in the sense that if I delete a file from that, such a file might still exist on the network but is not in my FilesContainer.

Does user have more than one FilesContainer?

If not why does

$ safe files sync

not just default to that.

Noting that

$ safe files put

does not behave like

$ safe files sync

and two puts, suggests the same exact feedback ā€œFilesContainer createdā€ etc

A second of two puts, is notionally overwriting but not ā€œcreatingā€ againā€¦ seems dumber than needs be, for a situation in which the difference will cost the userā€¦ and naive user might appreciate that itā€™s not exploiting them at every opportunityā€¦ and I canā€™t think of a reason to want an overwrite as if it didnā€™t exist already. BUT this is assuming that the idea of having two instances in one FilesContainer, is not a deliberate feature - but perhaps it is??

One stop command to close down, would be niceā€¦ knowing to kill vaults and safe auth stop, is more?.. safe network shutdown as a catchall?

Alsoā€¦ doesnā€™t Parsec make a difference - itā€™s so quick nowā€¦ no resources used, not even time! :smiley:

Look forward to seeing a browser and a joined up next step. :+1: :+1:

22 Likes
11

Thanks so much to the entire team for their hard work. Remember ā€œAll work and no play makes Jack a dull boy.ā€

18 Likes
12

Another great updateā€”many thanks as always!

Hope everyoneā€™s staying SAFE ā€¦

16 Likes
13

Looking goodā€¦
210Mb file put to baby-fleming in just over a minute

willie@gagarin:~/projects/maidsafe/safe-api$ dd if=/dev/urandom of=/tmp/rand.dat bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 1.01096 s, 207 MB/s
willie@gagarin:~/projects/maidsafe/safe-api$ time safe files put /tmp/rand.dat 
FilesContainer created at: "safe://hnyynys1tjwkthbg5zzy6boypgonpj7asax6nh4aq1snydy7xtgzsuy8rebnc"
+  /tmp/rand.dat  safe://hbkyyoncf46ioprio41py8o4thxqmkewdy95pxt3nztimdeqgkt44qmos8 

real	1m6.726s
user	0m33.736s
sys	0m6.873s

Iā€™ll keep on keeping on until I break it

Ubuntu 18.04 Intel(R) Coreā„¢ i5-2500K CPU @ 3.30GHz 16GB RAM 16GB swap

27 Likes
14

Aw naw - Iā€™ve dropped baby-fleming on its poor wee heid!!!

It seems like trying to put a 1.7Gb file was a bit much :slight_smile:

Thanks to the fine efforts of the devs filing an issue will be dead easy cos it generated a crash report.
https://github.com/maidsafe/safe-api/issues/464

Next step will be to build from joshuef:MoreE2E and try out the proposed -t flag - if I have understood that correctly.

17 Likes
15

So is CRDT cancelled in favor of BLS or is this still needed?

Nope, CRDT will still be needed :slight_smile:
CRDT is this, kind of a, ā€˜always onā€™ feature that would get data synced among nodes in the network, slowly but surely.
Whereas BLS would be triggered whenever/wherever we require consensus quickly for that certain data/action.
Hope it makes sense :slight_smile:

25 Likes
16

Wonderful stuff ā€“ I mumble through my mask. :mask:

13 Likes
17

This is correct, and when you do a files put you are creating a new container each time, so yes, you can have as many containers as you wish, Iā€™m getting the impression you probably imagined thatā€™s your own container with all your files, but thatā€™s not exactly that, weā€™ll eventually have the labels applied to data and youā€™ll be able to find all the containers and data you own, right now we lack of it. So thatā€™s why a files sync need you to specify which container is the target for the operation, and that will simply sync up updating your container on the network but maintaining its history as well.

17 Likes
18

ahā€¦ I had the impression the FilesContainer name was identicalā€¦ I added one new file and uploaded recursive for the same folder. So, I wonder how that FilesContainer name is determined.

3 Likes
19

Each container you create with files put has a new random name/address, however the files will be linked to the same ImmutableData xorurl if they have the same content, e.g.:

$ safe files put ../testdata/
FilesContainer created at: "safe://hnyynyzyccr9oq7i9gpw811urajsb5j3rz4gukw4emfs7js1un6kgwj7rsbnc"
+  ../testdata/another.md   safe://hbhynydjthmxbrth4tkgrjki8mcihd5frigrz4yunw4rsjh31qhnaombg9 
+  ../testdata/noextension  safe://hbyyyydj99a5hm68szc3ehnt4swbo7caziod9w53b5j87efsykjqyxoo4k 
+  ../testdata/test.md      safe://hbhynydjpor31taroabod6fugmbk5h9i4t7zz1edzu9oohrour9jogank8 

$ safe files put ../testdata/
FilesContainer created at: "safe://hnyynyih8j1oo1fjtnmfjoi59748ryox3ai8kxjygm1e9qirsyi1wr37krbnc"
+  ../testdata/another.md   safe://hbhynydjthmxbrth4tkgrjki8mcihd5frigrz4yunw4rsjh31qhnaombg9 
+  ../testdata/noextension  safe://hbyyyydj99a5hm68szc3ehnt4swbo7caziod9w53b5j87efsykjqyxoo4k 
+  ../testdata/test.md      safe://hbhynydjpor31taroabod6fugmbk5h9i4t7zz1edzu9oohrour9jogank8
11 Likes
20

That clears up some confusion for me - thank you

5 Likes
21

That prefix suffix takes some getting use to

hny .. bnc

:+1:

3 Likes