I’m almost a week late here ! Not because of Nick’s effort to timely transcribe Monday’s dev update, but as always I’m the bottleneck on putting it out. Apologies guys !
It’s been a crazily productive week so far. People watching JIRA and the MaidSafe · GitHub will have noticed an unfamiliar silence. This is all because of the current sprint. Viv has been full-time scheduling every developer with very specific tasks, and the online progress of that is now on the github readme to-do lists (the Rust repos). We are now in a great position to extend from what was there in the C++ implementation.
A brief glance over the past few months shows that with testnet1 we had a C++ implementation of roughly 560.000 lines of C++ code. With routing_v2 we fundamentally restructured the logic to be both more clear and secure (importantly introducing the sentinel into routing) and we have about 73.000 lines of C++ implementation that is much faster and more stable. Importantly the essential goal for the SAFE network, is to extract a clear protocol from the accumulated knowledge; an accessible codebase is the only way forward. That is what we achieved in testnet2.
And that brings us to today: preparing for testnet3. In testnet3 we plan to introduce new designs to the project, which includes safecoin. In order to do that we need to ramp up our ability to fast-prototype and safely build new technology on top of the MaidSafe stack. This is what the past four weeks of sprint have been about: evaluating the merits of Rust for the future of MaidSafe. The official results are not yet in, but it is looking extremely promising. We now have a Rust implementation (say 90% there) of slightly over 7000 lines of Rust. Going from 560k to 7k lines of code over the past 4 months, is quite a feature and a crucial one to get all of you involved into the core-development. We are aggressively breaking down the barriers to enable an active community; after all the SAFE network belongs to all of us!
So let me see if I got this right, the code base went from 560k lines of code to 73k lines of code written in C++. Then as side project the code base was written in Rust only needing roughly 7k lines of code.
If the Dev team keeps this up my grandmother is going to be able to understand the code base!
That’s the aim! For your grandmother to have her pull request merged into the code base
Especially the modular approach inherent to Rust forces logical units to be independent. Contracts on the API allow for the project as a whole to move forward. Compared the monolithic code base we had in testnet1, this is real breath of fresh air. So yes, tell your grandmother to fork the repos
Lets hope so. Now there are very few barriers to entry for this system and that makes us less valuable and all you folks more valuable. A fantastic move. As a sample just look at GitHub - dirvine/self_encryption: file self encryptor and click Documentation link. Nothing to stop anyone just becoming a core dev. Here are the steps (assuming you have rust installed)
Fork the project on github (click fork at top right hand corner)
You have everything you need for self encryption
now do cargo build and it’s built with all dependencies etc. [Edit removed info about sodium lib, it’s not required for self_encryption]
now cargo test that’s it tested cargo bench that’s it benchmarked [ EDIT as @happybeing spotted this will require a rust nightly version as opposed to the beta channel]
then edit the docs even vim src/lib.rs
Edit some of the documentation (mind this is API docs) git commit -m"Edit to change XXX"
then git push to push to your repository
Once done select pull request and add comments and click go. We get the PR and if good we merge, your now a core dev. Easy enough so then start to get more comfy and look at improving algorithms, tackle a todo list item or whatever. Just feel it’s yours and your important cause you are
The days of looking at mountains of massively complex (although very well done) code are over folks. This is simpler, faster and much easier to engage with.
I am keen to see how you feel if you get a chance to play and perhaps do a pull request. We would love to meet you in that place and you would be very welcome.
Be aware the pace is breakneck don’t be surprised something already moved on minutes after you fork, that’s OK though, don’t be put of. Pace will settle as soon as testnet3 is live and at this pace you will all see very clearly how close it is. Not only that but get an idea of how fast we can now move so expect a lot of apps in short time and from lots of sources. This is not a wee bit simpler, it’s a step change.
or an even simpler trick: there is a hidden file .travis.yml in every root folder. It’s a little file that tells the continuous integration test computers how to set up and run all the tests: it has the explicit commands you can copy paste into a terminal (without thinking) and gets you up and running (edit: on linux/unix! ).
sudo apt-get install -qq libcurl4-openssl-dev libelf-dev libdw-dev wget https://github.com/jedisct1/libsodium/releases/download/1.0.0/libsodium-1.0.0.tar.gz tar xvfz libsodium-1.0.0.tar.gz cd libsodium-1.0.0 && ./configure --prefix=/usr && make && sudo make install && cd ..
Great work folks! Will dive in it some more. But which software is needed to compile the client once it’s in Testnet 3? Is that a lot of work? I can compile a little program using Visual Studio 2013. But do we need more for RUST etc?
the compiler for rust code on any platform is rustc, but it has a build system called cargo so running simple commands like cargo build in your directory will pull all dependent code, compile it and link it.
This is the amazing thing about an open project like this. We get to see these incredible developments happening right in front of our eyes, week by week. Normal companies do this in a box, and I think that there are many people who have come to expect that the only things the outside world gets to see are final finished (closed-source) products.
This is a whole different ball-game with a group of people who are willing to put their money where their mouth is.
I did all the above and it seemed to work without any libsodium, except for the following:
the tests all showed “0” (zero tests run, passed etc.)
the bench command errored:
Compiling self_encryption v0.0.8 (file:///home/mrh/MaidSafe_rust/self_encryption)
benches/lib.rs:22:1: 22:18 error: unstable feature
benches/lib.rs:22 #![feature(test)]
^~~~~~~~~~~~~~~~~
note: this feature may not be used in the beta release channel
error: aborting due to previous error
Build failed, waiting for other jobs to finish...
Could not compile `self_encryption`.
So I installed sodium but no difference. Is this the current state or a local problem?
Yeh that might have just been a typo mark, self_encryption doesnt have a dependency to libsodium. Routing does. Readme for routing gives the instructions to get libsodium.
You don’t need Visual Studio for Rust projects(can build as mentioned by ben with cargo build in the command line). There are a few early releases for plugins to support the language in VS IDE, One of the “reliable” editors for Win currently is Sublime Text 3 with Rust-Plugin & RustAutoComplete-Plugin. Eclipse also has decent support and similar to Sublime Text 3 is cross platform. Atom also is pretty good.
Yes sorry I should have said bench is a feature as it’s called and not stabilised for Beta channel, to run those you need the nighly release of rust (not a bad thing really). If on linux/osx then curl -s https://static.rust-lang.org/rustup.sh | sudo sh -s -- --channel=nightly will get you there.
We are tracking these as they improve. Beta release of rust disallowed any unstable features, but it’s OK in nightly. These features are being cut over to the beta channel, but may take up to 4 weeks. Our evidence so far is they get this done very fast.
Anyway this should help. BTW was this on your arm device ?
It is to do with the rust compiler itself. Since last week the rust compiler entered beta-stage. At this milestone the beta-rust compiler no longer allows unstable features.
So a quick note on rust: #[feature(test)] is an attribute that says, you explicitly want to allow the use of an unstable feature in your code called [test][1].
We use it to do benchmark tests. It will in a later beta release become a stable feature, but currently beta is locked and focusses on debugging all present stable features.
You only compile and run the benchmark tests when you execute cargo bench;
on cargo build or cargo build --release you only compile the actual code
on cargo test you will also compile the unit tests and run them
In self encryption we only use stable features, and it should build fine with the beta version of the rust compiler. If you want to run unstable features (for cargo bench, or in routing) you need to use the nightly build of the rust compiler.
curl -s https://static.rust-lang.org/rustup.sh | sudo sh -s -- --channel=nightly and run cargo clean and cargo update before running cargo test
Oh man, from my very knees covered in blood and sweat (just like ice cold in Alice at times ) I can tell you the current plan is a review and implementation review next week (keyboards down) for a 10 day sprint that will probably see the beginnings of testnet3, but we are doing all this really as fast as we can to get in front of the curve. When we are in front we will give all the dates and times with 100% accuracy as we will only make release dates after the code and tests are complete. Please don’t take timescales from us right now and let us go at this pace, it will get us there much faster if we keep this pace up, but it is pretty focussed.
This will be a rolling iterative release. The intent is to match up all the TODO items on the libraries with the roadmap to automatically check them off and turn them into a link to download that version.
This burst of work has been ferocious to allow us and from a personal perspective, me to get in front of the game and state with clarity what is happening, but to have a ton e of evidence folk can click on. The enormity of the project was preventing clear messages under so much pressure, this will come to and end now, but we will need a few mins break (just not this month).
The other issue you may see is many of the libs are under my name ? This is something I want to encourage. MaidSafe will fork these and create releases from their fork, but where they get developed need not be a maidsafe repository. So if you did a module we could use (say for instance a stand alone sentinel) we will fork it and add it and leave to you manage it in your repo. If we like it then it’s in if we disagree we are on our own forks all of that is cool.
So now I would love to see the San Francisco Montreal Melbourn London etc. Pods all with their own repo’s and building stuff for core as we move on, even doing PR to us for fixes/docs/more tests/examples/benchmarks etc. Then were moving swiftly and sharing quickly which will make all the difference.
For now I would say just look at the libs and the todo’s done in a couple of week s and compare with what’s required for 0.1 versions and you will see when we are in 0.1 then network live again and then we will add further version updates for testnet3/beta etc. as we move on. I see testnet3->beta as feature freeze and just adding efficiency and security analysis etc.
