MaidSafe Dev Update :safe: 18th August 2015

Development Updates
1

Hiya,

@BenMS and @Peter_Jankuliak had to extend their relentless effort over this weekend to successfully complete the refactor of the routing library.

For this week, the focus shifts to the upper layers (safe_client, safe_vault) to integrate with the actual Routing crate and testing of the same.
@ross couldn’t hold himself back and wait for the upper layers. He has started to run his own tests against the routing crate within the LAN and will be extending his test to the droplets in the upcoming days. As @brian_s advised in last weeks post, the Rust-utp integration with Crust is now complete and testing these new features will keep @ross very busy this week.

Just to summarize where we are after the rust-3 sprint:

  • Unified structured data implementation (multiple modules)
  • Name Service integration (dns crate)
  • UPnP integration in Crust
  • Memory map & compression in Self-Encryption enabling larger file handling
  • Examples (Firefox Add-on and Desktop application)

A diverse set of functionality has been added to the network, giving it a clear direction and shape towards a stable, scalable and secure network. Though this is just the initial implementation, it would definitely benefit even further with the sprints to follow and backed by the technical debt sprints.

As for the client libraries, we needed to implement RFC’s (Unified structured data, Name service) as part of the sprint objective and with the sprint extension we also decided to set up a UI example showcasing the dns crate functionality. This certainly presented us (@ustulation and myself) with a few hurdles (setting the mime/type for a file as part of the file metadata itself) and has also given us a few ideas for the sprints to come. Porting the rust libraries to other languages has been a challenging task to say the least :slight_smile: . There were several hurdles to overcome and there are still a few to be addressed to improve the process here (Using the SWIG port for nodejs and other platforms). However, with our current time limitation and considering this is our first crack at porting rust libraries, @ustulation has made the task look easy :D. We’ve currently exposed a handful of low level API’s through the FFI bindings (a mechanism which enables a program written in one programming language to call routines or make use of services written in another) which caters to the requirement of the dns crate’s example.

The Desktop application and the Firefox add-on are now close to completion. I’d guess elaborating a little on the progress with the examples so far would be helpful for the curious app devs here :). The overall idea was to create a desktop application using FFI and the Firefox Add-on using js-ctypes.

We decided to create the desktop application using nw-js formerly called node-webkit. The first week was pretty much the UI and behaviour integration for the application. All went smoothly and we were able to cross compile to all three major desktop platforms (OSX, Linux and Windows).

The Firefox addon was also developed to intercept the safe: protocol scheme and serve the content from the SAFE network. This add-on was extended from the previous proof of concept which we did few months back. The dev tools for Add-on used by Firefox changed from what we had used previously. Now, we use jpm for the Add-on development. The scope of the add-on for this sprint is limited to intercepting the safe: scheme and serving the content from the network.

We also did a quick analysis on chrome’s feasibility to support custom protocols. But unfortunately they don’t seem to support custom protocol handling through their SDK. Moreover, the Chrome SDK restricts serving the content from the plugin after intercepting the request. We looked at TOR’s implementation and found that the requests from Chrome are redirected to a proxy server via the add-on, which definitely is not an ideal approach for us. We decided to park Chrome add-on for now as we wouldn’t have been able to deliver a suitable solution within the sprint deadline, however, the Firefox implementation is very much a success.

Last week we started to integrate the FFI bindings with the desktop application and a new set of issues started to spring up. We had to follow a different build approach for building the native npm modules using nw-gyp. Even after building the native modules with nw-gyp, we were not able to use it in Windows after packaging the application. Building the native modules wasn’t straightforward either. After a long struggle we decided to move to electron formerly atom-shell over the weekend and the application is now ported to electron successfully. Electron seems to provide better support for native module integration as well.

While everyone is busy integrating routing to the upper layers, I would be fixing the desktop application and the Firefox add-on for this week.

Progress with installers has been good with both the apt repository and yum repository for linux being setup for the Vault binary. Once the vault library has integrated with the refactored routing API’s we’ll be using the same installers to set the network up on droplets. Installers for other platforms have also been coming along well and more info on this should be made available through this week as we get to testing the vault binary. Work in the installers also identified a few edge cases in terms of cache file storage location that various modules were relying on. These have now been finalised and is currently getting worked on being implemented and tested by the guys in the crust module.

Here is the weekly dev transcript

35 Likes
2

@Krishna_Kumar,

After the discussion following the last update I revised my daily strategy to:

  • do a pull on crust, if there are any changes, continue
  • as root do: curl -sSf https://static.rust-lang.org/rustup.sh | sh -s – --channel=nightly
  • cd crust; cargo [clean | update | build]
  • if the build is successful do: cargo test

Just now I got:

crust
Compiling crust v0.2.5 (file:///home/phil/src/rust/maidsafe/crust)
Running target/debug/crust-5a55922afd6d30b1

running 20 tests
test bootstrap_handler::test::serialisation … ok
test bootstrap_handler::test::prune … FAILED
test bootstrap_handler::test::duplicates … FAILED
test connection_manager::test::bootstrap_off_list_connects … ok
test config_utils::test::read_config_file_test … FAILED
test bootstrap_handler::test::oldest … FAILED
test connection_manager::test::network … ignored
test getifaddrs::test::test_filter_loopback … ok
test getifaddrs::test::test_getifaddrs … ok
test bootstrap_handler::test::max_contacts … FAILED
test tcp_connections::test::test_small_stream … ok
thread ‘test_beacon receiver’ panicked at ‘index out of bounds: the len is 0 but the index is 0’, …/src/libcollections/vec.rs:1047
test test::check_rust_unit_testing_is_not_parallel … FAILED
test transport::test::test_ord … ok
test tcp_connections::test::test_multiple_nodes_small_stream … ok
test connection_manager::test::bootstrap_off_list_connects_multiple … ok
thread ‘test_avoid_beacon seek_peers 2’ panicked at ‘index out of bounds: the len is 0 but the index is 0’, …/src/libcollections/vec.rs:1047
test connection_manager::test::connection_manager … ok
test connection_manager::test::connection_manager_start … ok

  • and the testing has hung there . . is that sensible? Is that what other people are getting?

Thanks,
Phil.

1 Like
3

Looks like you forgot:

export RUST_TEST_THREADS=1
5 Likes
4

@happybeing,

Aargh . . quite correct - it is actually still in my script but after major messing around it was now below the “exit” line . .

The result has been like this for a while now:

crust
Compiling crust v0.2.5 (file:///home/phil/src/rust/maidsafe/crust)
Running target/debug/crust-5a55922afd6d30b1

running 20 tests
test beacon::test::test_avoid_beacon … thread ‘test_avoid_beacon seek_peers 2’ panicked at ‘index out of bounds: the len is 0 but the index is 0’, …/src/libcollections/vec.rs:1047

  • that doesn’t look right either?

Thanks,
Phil.

5

What version of rust are you on?
I just tested with

$ rustc --version
rustc 1.4.0-nightly (e35fd7481 2015-08-17)

and got warnings but it did compile and pass tests.

1 Like
6

@Ross,

Me too:

rustc 1.4.0-nightly (e35fd7481 2015-08-17)

Well before I did “export RUST_TEST_THREADS=1” I was getting a lot passed and failed messages (see above) but now I don’t get anything . .

P.

7

This is great Phil, it does help, if you check the badges though and if they are green it means it did work in osx/windows and linux. The versions may differ (there is a libsodium bug preventing 32 bit success jut now, but it’s patched so we await a crates.io update there to switch 32 bit back on in CI).

If I see a difference locally I generally check CI, if all is well there then I know it’s me (or suspect it is) so update multirust and cargo clean cargo build etc. Hope that helps.

PS If you had some weird crash due to threading perhaps there is a long running beacon process on your machine, worth doing a pgrep crust or similar

3 Likes
8

@dirvine,

I don’t know what that means - I run the test stuff in a new xterm - everything is black and white . .

multirust ? - doing “locate multirust” on my system finds nothing with that name . .

doesn’t find anything now . .

Thanks,
P.

9

On the README on github for each project. I think ross has a dashboard with them all in one place.

You need to install multirust (a very simple way to switch rust versions and update it to). Worth your while.

That’s good, not sure why a hang would happen.

Another FYI, we have switched on env_logger in crust/routing so far (rest will follow). So when you start any of the examples with RUST_LOG=debug ./<some example> then you will get info and debug output which is helpful. More info on that tool here http://rust-lang.github.io/log/env_logger/

1 Like
10

@dirvine,

Ah! cool - will check it out.

Another “Ah!” - OK, will also check that out.

Good to know - will test that out too.

Thanks!
P.

1 Like
11

All the badges here - The Windows badges are currently being updated as we move from Jenkins CI for Windows to Appveyor.

Oh and @dirvine is correct multirust really rocks when moving between nightly / beta and stable.

4 Likes
12

I’m loss here. It’s some time I try to build on Ubuntu but I can’t because of "use of unstable feature ‘udp_extras’ even if I have rust 1.4.0-nightly. You talk like there is nothing else to do and you are able to compile it.

And on windows I’m trying to make libsodium working.

What I miss?
How did you managed to compile routing?

13

Are you on windows?
If 32 bit there is an issue you cannot get around and will need to wait for sodiumoxide to update crates.io (which I hope will be very soon)

14

No it’s 64-bit. And I try to use libsodium 64-bit. But on Ubuntu I was able to install it easily. At least if I can build routing in Ubuntu that a start.

15

I am assuming Windows 64 bit?

If so check the appveyor script on github and follow exactly that and then you are doing what our automatic CI machines do so you should be OK then.

1 Like
16

@Ross,

Hmm . . I had a look at it but it reminds too much of Ruby’s RVM which used to cause problems for me because it introduced another layer of potential errors - I think I will stick to trying to solve one problem at a time - in this case: getting nightly crust to satisfactorily finish its tests.

I have started from scratch again but am still getting the same problem as above when using:

RUST_LOG=debug cargo test

- it hangs, I have to break out of it, and I can’t seen any extra error log files created?

P.

17

I would not do that!, for sure then you are introducing a layer of complexity. Really multirust is simple and easy and then you run examples with the log defined and you will have nice output. If you want to use cargo to do that then you need to pass through params and will require extra -- etc. Perhaps don’t use log or mutlirust if it’s going to complicate your setup there.

18

@dirvine,

Doesn’t that mean though, if I install multirust, that I will have to work out how to run each of the 20 crust tests manually? - instead of them happening as a batch from cargo?

Thanks,
P.

19

No, it just automatically changes your rust version and only when you ask it to.

1 Like
20

@dirvine,

OK, I don’t understand, you are going to have to be explicit:

  • Building the nightly crust and running “cargo test” is now not producing any output for me - but it is apparently working for others (ie green badge)
  • So you are saying I have a problem that I can’t get better information on without installing multirust?
  • Why is that true if multirust is only good for changing rust versions?
  • Isn’t that “RUST_LOG=debug” option available in the nightly version of rust?
  • Are you just saying that I can’t use that option in combination with “cargo test”?
  • If I install multirust, I will still be using exactly the same version of rust and crust source, so the only thing that changes is how I start the testing
  • If I don’t use “cargo test” what do I use exactly?

Thanks.
P.