Safe Network Dev Update - October 8, 2020

[maidsafe]

Summary

Here are some of the main things to highlight since the last dev update:

• We made the sn_fs GitHub repository public and encourage the more technical members of the community to try mounting a filesystem.
• The sn_api repo is now building against our updated sn_client.
• Node and Client integration work took another leap forward with, amongst other things, a PR introducing DebitProofAggregator to complete the work for AT2 Transfers client-side, meaning the code for a secure and successful money transfer would be considered done when merged!
• There was some great refactoring progress in sn_routing this week, with the prime benefit being that we are now able to quickly and easily bulk up our test suite.

CRDT

sn_fs

This week the team kicked the tyres of the sn_fs PoC a fair bit on Linux (Ubuntu), and a couple of brave souls even tried it out on macOS. One minor issue was identified and fixed on Linux. Unfortunately on macOS, several issues were reported that are not yet fixed, so we do not yet advise using it on that platform, although it does build and run, and some basic functionality did work. The code cannot be built for Windows at all yet.

Nevertheless, this week we are pleased to make the sn_fs GitHub repository public so that technical members of the community can build it and try it out and/or review the code. Build and mount instructions can be found in the readme. Please report any issues in its GitHub issues page with all the relevant details.

A reminder: sn_fs is a proof-of-concept to demonstrate that crdt_tree is viable for building a local caching filesystem. As such, there is presently no networking and no integration with the Safe Network itself and everything is subject to change.

Safe Client, Nodes and qp2p

Safe Network Transfers Project Plan
Safe Client Project Plan
Safe Network Node Project Plan

This week we’ve been updating the sn_api crates (api, cli and authd) with the massive changes we’ve seen from sn_client. A lot has changed there since it was safe-client-libs, including messaging and data type naming as well our transfer system. It’s been quite the batch of changes but the sn_api repo is now building against our updated sn_client and we’re ensuring tests are all in a good place for when we have that client fully functional against the nodes.

On the integration side of things, Node and Client modules have seen multiple fixes going in this week. With PR#245 in sn_data_types merged, a handful of bugs were fixed that affected the Node regarding IDs, keys and message signing. These changes were reflected in Node PR#1128 and Client PR#1271. Once this is merged, Clients will be seeing a bit more action with the introduction of DebitProofAggregator, which completes the work for AT2 Transfers client-side. This is the last piece in the Transfers puzzle that is required to accumulate and aggregate SignatureShares from section Elders to produce a DebitAgreementProof for a secure and successful money transfer.

Looking ahead, once the above PRs are merged we should see more than half of the e2e tests going green right off the bat. With a few minor fixes and cleanups we will increase the stability of these two crux modules to work in sync. Next steps after fixing the remaining test suite issues is to expand the farming integration tests, complement the whole set with chaos testing, and integrate these updates with the qp2p IGD updates that were mentioned last week (which re-enables the “Nodes from home” concept)…then freeze the code for a testnet!

Routing

Project Plan

This week we continued refactoring routing in order to make it simpler, easier to test, and easier to reason about. First we landed a PR that removed the somehow confusing FullId and PublicId types. We now use more industry-recognised PublicKey, SecretKey and Keypair instead. This was followed up by another PR to clean things up a bit more. We then landed another PR which refactors the internal logic using a command-driven paradigm and which allows us to re-structure the code into a couple of small, self-contained and loosely coupled components. Most importantly, it makes the code more easily testable. We already wrote a bunch of new unit tests using this approach and they were a breeze. This is quickly closing the test-coverage gap created by removing the old mock-network based tests which were too complex and unwieldy.

On the CRDT front, we continued with testing the PoC code we have for section membership data. We were creating different tests and fixing issues we discovered along the way. We are now looking at and trying to understand the effects of some more complex scenarios when it comes to section churns, and how we can deal with them, using CRDT logic. Thus this is where we’ll be focusing in the next few days by creating some tests which can reproduce these scenarios and finding a solution for them.

In parallel, we are starting to look at how the new CRDT-based membership data can be integrated into the routing codebase and its business logic, which implies finalising the internal APIs of our PoC code.

Useful Links

• Safe Network Website
• Safe Network Primer
• Network Fundamentals
• Roadmap
• Glossary

---

Feel free to reply below with links to translations of this dev update and moderators will add them here:

Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

[Southside]

first !!

[Southside]

@maidsafe

Nevertheless, this week we are pleased to make the sn_fs GitHub repository public so that technical members of the community can build it and try it out and/or review the code.

Haud me back

[Secretariat415]

Thanks so much to the entire Maidsafe team for all of your hard work!

[KafkaLee]

I think we are getting close to… something!

[TylerAbeoJordan]

What!? It’s Christmas already?

Thanks for the presents Maid team!

[davidpbrown]

@maidsafe

the sn_fs PoC

a toy to play with

For those who want to try, it’s not requiring much technical skill…

• Rust from Install Rust - Rust Programming Language with a single line and a logout/login
• for Linux libfuse is installed but libfuse-dev likely will need install.

1 readme could do with use of ./ for commands

./sn_fs /media/safe/test

First impression - it works.

2 command hangs

It’s odd perhaps that the mount command hangs without just mounting and exiting back to command prompt . So, I don’t know if

./sn_fs /media/safe/test &
is a good idea with ampersand the usual way to put command to background or if some fix is needed for once successfully mounted exit.

So, the unmount just works as expected but unfortunate another termial required.
fusermount -u /media/safe/test

3 the executable flag is lost?

So the only buggy looking issue I wonder is that executable permissions are wrong?

Creating a new text document works but oddly trying to open it has the usual challenge for an executable - which is an issue that can arise from transferring from certain kinds of other file system, like FAT32. I would have expected ext3/4 behaviour and a default to executable flag off.

So, is this behaving like a FAT32 filesystem?.. would be better to know that the intent is like ext3 or similar, as perhaps FAT will throw alsorts of issues??

Notably the executable flag is not showing on the file… and toggling, it does stick and is then remembered but has no effect… which is odd… the executable flag on or off in the file properties, does not affect that when opening the file to edit, it looks like an executable… so, this is a problem. Default should be not executable.

Mounting unmounting a few times shows no issue.

Talk of trees and freeze!.. winter is coming… all I want for xmas… is toys!!

[danda]

@davidpbrown

It’s odd perhaps that the mount command hangs without just mounting and exiting back to command prompt

afaik, that behavior is common to fuse based filesystems. At least others I’ve tried do the same thing. I think it should be ok to background it, but haven’t tried myself.

I believe it is possible to run the mount command itself with necessary options to specify fuse + sn_fs, and then it would probably exit back to command-prompt as you expect, while keeping sn_fs running in the background. I don’t have time just now to investigate that though. If anyone here finds it, please post recipe, and I can add it to the Readme.

@davidpbrown

So the only buggy looking issue I wonder is that executable permissions are wrong?

I’ve not seen this issue before. For example I just tried:

$ echo "echo 'hello Safe Network'" > sayhello.sh
$ ls -l sayhello.sh 
-rw-r--r-- 1 danda danda 26 Oct  8 10:36 sayhello.sh
$ chmod +x sayhello.sh 
$ ls -l sayhello.sh 
-rwxr-xr-x 1 danda danda 26 Oct  8 10:36 sayhello.sh
$ ./sayhello.sh 
hello Safe Network

Can you please file an issue on github and specify:

1. which OS you are using (Linux version?)
2. which version of fuse is installed.
3. output of umask command.
4. Your own steps to reproduce the problem.
5. What happens when you run the commands I listed above.

thx!

[davidpbrown]

Curiously, perhaps encouragingly, your commandline method works fine… it’s the noob user route that is odd.

[davidpbrown]

First!.. bug logged as First! Executable issue when creating file through GUI · Issue #9 · maidsafe/sn_fs · GitHub

[19eddyjohn75]

thx for the update Maidsafe devs.

Will try to play in the weekend (not that I’m technical, but I can try)

Cheers and keep up the good work

[Sotros25]

@maidsafe

meaning the code for a secure and successful money transfer would be considered done when merged!

Amazing!

@maidsafe

this week we are pleased to make the sn_fs GitHub repository public so that technical members of the community can build it and try it out and/or review the code.

It’s also exciting to see the increasing avenues for technical community members to get involved. This can serve as a lightening rod to draw attention from devs that aren’t yet involved in SAFE—a key network effect builder.

Exciting update! The ship sails steadily forward.

[Audity]

I saw “refactor” as part of the accomplishments, and immediately sent a Like. After such a huge refactor weeks ago, any further ones are songs to my ears and a feast for my eyes.

[davidpbrown]

Just to note for anyone else who plays with this, that odd behaviour follows also a simple copy from an environment where the file is not executable.

Thinking… could it be that it is a null instead of a zero or sum such… the exec is not set and therefore behaviour is like a transfer from FAT etc where there is no flag?

[mav]

@maidsafe

the sn_api repo is now building against our updated sn_client

I tried building sn_api today but got a lot of errors because it still uses safe_nd in places (recently renamed sn_data_types)

eg in sn_api/src/api/common.rs

https://github.com/maidsafe/sn_api/blob/5f8d61cf41eb547c13b65d2030beefd235b75820/sn_api/src/api/common.rs#L18

Am I missing something? Is sn_api expected to build yet? Or is this only using branches private to maidsafe?

[joshuef]

@mav

Am I missing something? Is sn_api expected to build yet? Or is this only using branches private to maidsafe?

The main repo isn’t expected to build yet, nope. Work is still in progress, but not private, just not PRd yet. My working branch for your interest.

I have a couple more tweaks I want in (get a tx_id in some useful form and get the tests updated for this; and generally tidy up). I should note, you still won’t be able to do much the api until sn_client is fully straightened out, but we’re getting there!

And then once that’s in we’ll start work on updating the authenticator functionality to use the generic APIs. Right now there’s no auth functionality, though authd will run and will receive requests, it’ll die as soon as you attempt to accept anything.

[dask]

Core currency functionality PR up. Buildable (with tweaks) and testable FS. Test suite growing with 50% of e2e passing. This is real progress. Yet another awesome and focused update.

[jlpell]

@maidsafe

sn_fs is a proof-of-concept to demonstrate that crdt_tree

Nice work danda!

@danda

afaik, that behavior is common to fuse based filesystems. At least others I’ve tried do the same thing. I think it should be ok to background it, but haven’t tried myself.

I believe it is possible to run the mount command itself with necessary options to specify fuse + sn_fs, and then it would probably exit back to command-prompt

@davidpbrown yes this is typical.

[davidpbrown]

@jlpell Can you suggest what that mount command is then?..

[jlpell]

@davidpbrown

Can you suggest what that mount command is then?..

Fuse is notorious for this kind of behavior. It depends on your system, kernel, selinux. It may be related to the xattrs support. Based on past experience it helps if you explicitly state the fs type like danda recommended.