Summary
Here are some of the main things to highlight since the last dev update:
- Check out the awesome SAFE Network Wallet Bot developed by @DeusNexus here - Telegram has now been connected to the SAFE CLI!
- There has been a huge refactoring and simplification effort in the SAFE Client Libs over the past few weeks, with circa 18,000 lines of code gone, and a simpler, clearer and cleaner codebase in place.
- Work to expose a Routing
asyncAPI continues smoothly with the same code simplifications we have seen in other crates over recent months.
CRDT
safe-fs
Work has continued on the crdt-tree implementation. Some refactoring was done to move each data structure into its own file. Also, some additional examples/tests were added that deal with log truncation via causally stable threshold. Finally, some minor optimisations and cleanup were done. The code still needs to be commented and basic unit tests added for each struct, but is nearing completion.
In related news, we received a Ph.D. dissertation authored by a Doctoral candidate who has written a distributed filesystem which implements the crdt-tree algorithm. Even better, the filesystem is implemented in Rust as a FUSE filesystem. We couldnāt ask for a better fit with our efforts.
As such, some time this week was devoted to reading the paper carefully, taking notes, and evaluating vs our own design decisions, as well as trying out the code and reviewing it. We have identified 2-3 ideas that we may be able to incorporate into our design.
We do not presently have permission to share the paper or code with the community, but will do so if that changes.
SAFE Client Libs and Vault
SAFE Client Libs Project plan
SAFE Vault Project plan
SAFE Client Libs and Vault integration
Refactor (with a machete)
This last week in SCL weāve been working on finalising some larger refactors and simplifications to our codebase on the back of the recent quic-p2p async push.
So on the one hand quic-p2p has enabled us to remove a lot of complexity at the client layer, removing internal event loops and extraneous structs to get towards a much simpler and leaner client.
Weāve also been re-evaluating the role of the SAFE Client Libs more generally. For a while it has been a big library, with code for a core client, applications, authenticators and FFI integrations. But as weāve been diving around the code more, some things have been looking like they needed an overhaul.
At the highest level, the main thing we have done is strip back what a client is. Previously it was a number of things (app, authenticatorā¦ āclientā), now itās just a āclientā. A client can connect to the network using a random set of keys (and so not write data), or you can pass a secret key to your client, and if thereās enough money, youāll be able to do write operations.
This drastically simplifies things. āAppsā just become āclientsā (more on how thatāll work below), using secret keys in the same fashion as any other client. We no longer need the āauthenticatorā layer now that we have no apps. Which again, means less code both in SCL and in vaults (where an appās permissions would have been checked prior).
So with these (rather sweeping) changes, we have been able to reduce the SCL codebase by something like 18,000 lines of code in the last few weeks. Getting a codebase that is much simpler, clearer, and cleaner. Easier to document, and easier to review as well.
New āappsā and āauthenticationā
As noted, weāre dropping the āappā distinction in the codebase, but not in general. What weāre looking at is this: Instead of relying on another layer of network code to check applications (with all the complexity and specific network requests that brings), in the coming time weāll be reworking our Authenticator app, to be just that. An application which works using standard network APIs. And this application will manage any āappsā in much the same way as previous. The main distinction being that each app will have its own keypairs, which the Authenticator will know and control.
This opens up an increased amount of flexibility in controlling app spending and permissions, and itās these permissions which will now solely rely on the Permission structs at the data. Which means weāll still be able to fully control what/where an app can write, as well as increasing the control over spending.
Itās been a large refactor. But the benefits are already abundantly clear. With 18,000 lines of technical debt gone, weāre free to move much faster in SAFE Client Libs. We are now continuing with finalising integration with updated Vaults and quic-p2p to get back at working towards testnets.
Vaults async and tests
Integration of async routing has begun also in Vaults, in parallel to the efforts in routing.
The minor (multi-threaded) test framework we mentioned last week is more or less completed in this PR, has already been put to use and is working well.
Alongside the above, we are continuously testing and fixing any issues found in the wake of recent messaging refactors, and the new transfers and farming features.
Routing
The work to remove SendNeighbourInfo votes, as mentioned in the last dev update, has now been merged with all failing tests resolved, along with a PR to (fix incorrect section key assigned to non-elders).
Meanwhile, the implementation of the relocation refactoring work (Remove relocate_queue and make relocation independent on the order of churn events) is progressing well. We also started to use the testing framework from Vaults to verify Routingās behaviour under real network usage.
The effort related to exposing a Routing async API continues smoothly, and as in previous refactorings made in other crates we are already seeing code being simplified as we introduce async functions in the codebase. So far we were able to expose an async stream which is what the vault will be using to receive each of the events the routing layer produces, and right now we are working on getting vaults integrated with these changes and successfully bootstrapping with other vaults using the new APIs.
Standardisation
We are pushing our standardisation process a bit further forward now. We will switch bincode to MsgPack for serialisation (as @mav pointed out in a post a while back, bincode is Rust only). We will standardise on base32z for human readable encoding.
This builds on the move to QUIC from Crust which gave us a standardised secure transport layer.
We will also have a push in-house to document the message API that exists in Vaults. This represents the most fundamental API and so using only industry recognised formats and types in this area will allow folks to build vault clients in any language, or indeed from any device capable of sending and receiving messages.
More importantly, this should allow us all to see how everything works at a single level, i.e. a vault receives message X, it then does Y to X and sends a message Z to another vault. Or a client sends message A to 7 vaults (its elders) who take A and agree B, then send message C to a remote section to ādo Dā, and so on.
Lots of fast moving parts right now, but we want to be ready for some quick testnets and to have everyone much more informed about all the moving parts in the network.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
As an open source project, weāre always looking for feedback, comments and community contributions - so donāt be shy, join in and letās create the SAFE Network together!
