Vaults Phase 2
The past week we’ve been testing and finalising the next iteration of the vaults from home testnet. After a few rounds of testing and bug fixing we are happy to announce the next iteration of the vaults from home testnet that has been released today 
.
Head over to the release post to play your part, all are welcome.
In parallel, we have been working on some of the next steps, which are primarily focussed on multiple sections. We have also been refactoring our test suite with multiple improvements which include using real Routing for the tests on our continuous integration platforms.
SAFE API
Today we released new versions of safe-api, safe-cli, safe-authd, safe-ffi and jsonrpc-quic, all to support today’s new vaults from home testnet iteration
See the changelog for highlights of the new features and bug fixes which are in the new releases, and of course see our detailed CLI User Guide for step by step instructions on how to use all new and existing functionality.
We have begun to explore possible design improvements for the SAFE File API.
In particular, a recent research paper (and video see 35:30) caught our attention that could dramatically improve performance, help simplify our API, and solve collaborative editing challenges. The paper is titled A highly-available move operation for replicated trees and distributed filesystems and it details a formally proven CRDT Tree data type that is suitable for filesystem usage and, being eventually consistent, solves a longstanding move operation problem that affects even Dropbox and Google Drive when dealing with concurrent updates. It is also suitable for working offline.
The core design idea is to add Tree as a SAFE Data type, and implement a FileTree as a specialisation, just as FileContainer has been a specialisation of Sequence. The important difference is that each Directory and File is stored individually in the Tree, instead of all serialised together as a flat JSON map in a single Sequence entry. This makes updates and retrievals much more efficient for any non-trivial directory structure.
Going further, we would like to incorporate elements of Safe.NetworkDrive for fast local access and offline usage, as well as native file system integration via FUSE.
To be clear, the idea is that SAFE apps could access the File APIs directly if they wish, but non SAFE apps could still interact with SAFE files via a local mount. This would mean that tools such as rsync could be used easily, file explorers, etc.
For the moment, this remains brainstorming and high level design. There’s a lot to solve. A first challenge would be to implement the CRDT-Tree algorithm in Rust. It presently exists only in the form of Isabelle/HOL formal logic. If any community member is proficient with Isabelle/HOL and could help translate it to Rust, or even pseudo-code, that could help speed things up.
In parallel, we have been trying to expose the Authenticator FFI APIs from the safe-ffi crate. This will unify our APIs on all platforms. During the last couple of weeks, we were facing an issue with FFI APIs when trying to connect with the real network while the mock APIs were working fine. This week we fixed this issue and we can now continue with the final round of testing and review.
SAFE App C#
As we mentioned in the SAFE API section, resolving the connection issues in the safe-ffi has made it possible to test the new Authenticator APIs in the C# APIs. This means in the coming weeks we will be releasing a new version of the NuGet package which will contain all the new APIs, including the Sequence Data APIs and Authenticator APIs. Once this package is released it will be easy to update the Authenticator and SAFE Mobile Browser apps to support the latest Vault.
These recent changes have already proven very helpful and for the very first time we can easily set up our CI to run the tests against a real network . Until now we would have to run these tests manually before a new package release, while on CI only mock tests were able to run.
SAFE Browser / SAFE Network App
The Browser and the SAFE Network App have been somewhat neglected of late with the focus being on the various testnets and underlying AT2 work. But we’re at a good place now with the latest vaults and underlying SAFE API changes so we’re working on a new release to be compatible with the latest Vault and the test network.
This won’t be arriving today, but we’re building and testing things locally, so these should be with us in the not too distant future.
CRDT
This last week we continued with our research around Access Control on our CRDT data types. We are mainly looking at two different approaches at the moment, a first one which allows operations to be applied even if they depend on an older version of the Policy, and a second approach where such operations are not accepted, or even undone if a new Policy makes it invalid.
Both approaches have their own complexities and implications which we are still trying to fully understand, not only from an implementation point of view, but also from a user experience point of view.
In both cases we need to make sure we keep track of this dependency (causality) information between data and Policy operations, and that’s why we started a small refactoring in our current Sequence CRDT implementation to make sure it can accommodate the upcoming changes, regardless the approach is taken.
Transfers
SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan
This week has seen our network messaging refactor settle down somewhat, after an additional iteration. With the major changes now into safe-nd and applied in vaults, we’ve been bringing safe-client-libs back inline and making some improvements to message handling there now that we’ve increased specificity of the network messaging. Additionally, this has allowed for a more or less complete flow of farming rewards to be implemented in vaults. We are currently proceeding with cleaning up some internal messaging patterns there.
In SCL we’ve set things up to allow proper handling of network events (such as validation of an AT2 payment request), the full query flow (which all our messages were going through previously, even if that wasn’t necessary), and finally put in place the basics of receiving errors from the network. Actually handling such errors will likely be more of an app-level decision for developers to take.
Now the focus for SCL + vaults will be testing testing testing these interactions.
Routing
As part of the work to improve the code quality, we started to split out some Routing code blocks into separate crates. The first crate we split out this week is safe-network-signature-accumulator, which is dedicated for generic BLS signature aggregation for the SAFE Network. There will be more code blocks split out when this becomes stable, to reduce the complexity of the Routing logic. Tidy up like this is something we feel is essential for launch as each small module can be thoroughly tested and documented with greater clarity than when mixed with other code.
As mentioned within last week’s dev update, the first task towards removing the parsec usage, ensure mutations to SharedState are approved by the section, was merged this week. This enables us to go further with the task of overhauling the node promotion. However, due to Adam’s parental leave (huge congratulations to him ), the work is replaced by PR 2160, which was merged today. We are now a step closer to implementing message-based voting, which replaces the current parsec-based voting.
Useful Links
Feel free to send us translations of this dev update and we’ll list them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together!
me too.
