Time for a wee celebration and for us to take a breath and more importantly to let the community know, this works, our faith is well founded. The hurdles we faced recently were not insignificant, the remaining work cannot be underestimated, however, we can now stand up and stop arguing if this can work and show this does work. We can have autonomous networks, self authentication, self encrypting data, secure mutation of data in a secured private environment…etc… We can at last show anyone who is interested. Our quiet steady progress has all been validated. We will continue quietly and steadily, but now a little bit quicker with a larger team and less need to get involved in the chat about the impossible network. This milestone should make everyone feel a little happy inside and knowing we have the staff, the resources and obviously the grit and determination to take this to completion. All in all, it’s a good day 
We also can’t thank you enough for all the feedback, support, testing and gathering log files, this is truly a community effort.
Test 12c is now offline. As @dirvine mentioned here, we identified a bug in Crust which caused nodes to partially connect (drop connections to only a few other nodes). This was a good thing in fact as this is extremely hard to model and is a great test to look at potential DOS attacks on some sections and work out how to handle them. While such attacks are very difficult in the real world this bug modelled it perfectly. It had also gone unnoticed for a while now. We’ve seen the same issue from Crust also occur in the Alpha 1 and Test 11 networks. This bug has now been fixed (see this PR) and we also got Routing updated slightly to handle similar issues from Crust, so if we do see partially connected nodes we now have several methods to handle them. As we move to data chains though this kind of bug is highly detectable and we can allow it up to a point and then agree to kill nodes that cannot maintain enough connections to group members. We also are analysing the possibility of even more dramatic network outages (>90%) and what can be done there. Full network restart is part of data republish, but handling these extreme cases requires an exceptionally good design (not all systems can handle such extreme cases).
So, now that we have a stable core network in place, it makes sense to talk about the future and what we have coming up. We have really been defensive during the recent months as we had a large job and we did get into a very high pressure state in house with immediate issues to be solved (of course none had obvious solutions). We can change this a little now and define, just a little bit clearer, our near term desires.
Our current thinking is that once we have confirmed the stability of the soon to be launched Test 15 (some may have expected 12d), this will become Alpha 2. The Alpha 2 network will supersede all existing networks. We’re planning to deprecate both Alpha 1 and Test 11 towards the end of this week. These networks have helped us test the client APIs extensively and have encountered a few issues over the last few months they’ve been running. Some from our bugs like the Crust one we noticed this week and some from network outages on these networks which weren’t suited to handle churn well. As you know we have taken down TEST-12c today and are doing soak tests on our fixes over the weekend. If this all goes well over the weekend, we would expect TEST-15 very soon afterwards. There is one more small fix we may apply tomorrow, but testing on that continues for now.
List of features for the upcoming alpha networks:
Alpha 3
- Mutable Data and authenticator pattern (with the new SAFE API)
- SAFE Browser JavaScript APIs and Node.js SDK along with example apps, tutorials and documentation
- Vaults from home (continued, more people should be able to run more nodes and on more platforms)
This will be a huge testnet for the community. We will work extremely hard with many Engineers on our forum daily to get feedback on the APIs, help with apps and more. This will be the Alpha we settle the client-side APIs so we expect huge debates and help from all the usual suspects and beyond. As a company many of our Engineers are just too busy to get too involved, but this changes in Alpha 3 as this is the real engage moment. We will have the ability to quicky alter an API, test it out and let others play with it. This will be critical to the future of all of the network and the networks developers. It should be an exciting period.
We anticipate having several mutable data testnets confirming that the implementation and documentation is good enough.
Beyond Alpha 3
- Data republish (ability to upgrade the network without deleting all the data) will likely be the focus.
- A security audit of the network
- Test safecoin
- Real-time network upgrades
- Network validated upgrades
In between alpha networks, there will be many testnets. Each alpha will focus on a few major components. Major features will be added one by one in testnets.
Once all those things are in place, there may be other alpha networks, or we may move to beta. With the team constantly expanding, there are a lot of reasons to believe we can be more managed in this rollout.
A few other things are happening in parallel (e.g. in Crust, we are making changes to allow more computers to run vaults from home, to increase the amount of people on the network as well as security and efficiency).
Also, you can expect to see an increase in activities for documentation in the next weeks and months. We will also look again at a simpler clearer roadmap. Not a mini project this time, but a helpful and friendly simple roadmap.
MaidSafe Asia
As mentioned in an earlier blog post, the Maidsafe Asia agreement has been signed off and the Jakarta conference was the first direct output of that partnership. We are currently working on a strategy with them to identify the most effective way forward, including; where to target, how we position our offering, what materials we need, which languages we support (linked to where we target), what skills we need…etc… The initial focus is all centered around working with SAFE Network developers.
Thus far, we have the guys at MaidSafe Asia who are highly motivated, industrious, and have a huge amount of energy, and as can be seen from the conference intro video, are prepared to put their money where their mouth is. This is an unfortunate rarity in this world and we feel fortunate to be working with them. As strategies are agreed and plans put in place we will share these with you moving forward.
In addition to MaidSafe Asia, we are also seeing other aspects of the business developing. This year we have seen a number of commercial enquiries and possible partnership opportunities develop. These have come from a small and diverse range of companies from all around the globe. We anticipate that the opportunities will only increase as we build the team and roll out the technology, and we have decided to hire a Business Development Exec to take these forward. You will likely see the job spec being added to our Careers page in the coming days. We are primarily looking for someone with knowledge of system architectures and a strong commercial background, possibly from a very technical sales engineering type role.
Community Engagement Program
We haven’t been able to give time to the Community Engagement Program, because of the pressure of getting vaults from home and other work we’ve had to do recently. There is still considerable pressure until we get safecoin running, but we have slightly more bandwidth now that we have a stable core network.
So, we will focus on it once more and to get things started again it is probably best that we solicit your advice once more in telling us what you would like to see. These ideas could be things that are of strategic importance to the network, such as a decentralised exchange, or an app that you feel we be highly useful. Once we have discussed some ideas we could poll the community and gauge interest, before we start requesting proposals. Click here for a refresher of the CEP.
Riot.im chat
We’ve noticed that #safenetwork:matrix.org (a chatroom hosted on Riot.im) has been getting more active recently. It’s a good place to discuss the SAFE Network and ask for help! See this topic for more info.
SAFE Authenticator & API
We updated the build scripts for the browser and the example application. Now it, should be easy to build and test the applications. The dependency downloader will download the native libraries based on the platform and it is not needed to build the libraries locally on the app developer’s machine. The authenticator and the example application have been tested on Windows, OSX and Linux to ensure the IPC process is working fine.
Shankar and Gabriel are working on exposing DOM APIs for dynamic web applications (just like safe-js). Meanwhile, Ben is working on porting the email example application to use the new safe_app APIs (with mutable data).
SAFE Client Libs & Crust
The latent bug in Crust should be fixed in this commit. This was like a slow timebomb for all connections severing them randomly and after long gaps. Also, we added a new config option to address the peculiar behavior of some routers that we found out. So if you think you have successfully forwarded a port and you should be externally reachable (no firewalls etc.) and if bootstrapping still complains otherwise for you, you can try enabling that option to see if it works.
We have also started looking into various error-recovery mechanisms in safe_client_libs. This will be a big topic and we are currently investigating what transactions can be rolled back or which ones should have the successful units left as is and skipped next time when it’s reattempted (due to failure in one of the intermediate steps the previous time). We need to factor in the behavior of our network, the way our data is expected to work, bandwidth concerns etc. so it’s interesting but also non-trivial.
Routing & Vault
This week we have spent a lot of effort monitoring Test 12c and also analysing a bug we noticed. This was a Crust issue, but did show a possible part of Routing that could be stronger. Several PRs waiting in the wings were also validated and merged. We also planned several short-term code changes in respect to simplifying several parts of Routing once more. These are the multiple IDs reducing to single IDs per node and also making IDs immutable. This affects relocation but makes way again for data chains type paradigm and node ageing will also be simpler to implement.
In design (which is kept mainly private as it’s extremely deep, fast and very much Engineers arguing bluntly and logically) there have been several advances in the longer term approaches for Routing. Again we are focussing on simplification as well as additional functionality, such as node ageing and data republish. This is all a parallel task that when complete will lead to implementation with most of the code already written and tested (at least locally). We have taken this parallel approach to prevent any delays similar to what happened with disjoint sections, where we charged ahead with huge changes right into the current working code base and created a huge delay.
Routing will continue like this and keep out of the way of the client-side work in releasing authenticator and the new SDK and APIs that this brings. Just as Crust is moving forward with many new features and keeping out of our way.
.
Really impressed and very nice to see the optimism!
Hope to buy you all a round some time soon