Summary
Here are some of the main things to highlight since the last dev update:
- Thank you to all who contributed to last week’s vaults from home with IGD testnet!
- We’ve been examining the results of the testnet and are now working on areas identified for improvement.
- There is a community-run testnet for those who wish to get involved - see forum post here.
- The safe-cli symlink PR is now going through testing and review.
- Sequence CRDT progress continues at pace with associated PRs merged in safe-nd and safe-vault.
- A GitHub issue has been created listing all the steps we think are needed in order to remove parsec from routing.
Vaults Phase 2
We’d like to thank everyone for their participation in last week’s testnet. Even though the testnet was limited to users with IGD supported routers, we had a good response with nodes joining the section 59 times. There was a good amount of client participation as well with 2214 immutable data chunks stored in 107 files containers. An important part of the test was to demonstrate data recovery when nodes leave the network and this showed good results. The network executed 4193 successful data recovery operations.
The testnet helped us understand some real-life network scenarios as well. The implementation had already considered some of these scenarios and the testnet helped us identify some areas for improvement that we have already started working on. An interesting example is ahead of time responses. The network needs only quorum number of Elders to approve a request and process it. Once processed the response is sent back to all the Elders for confirmation. While some of these elders will be aware of the request others might not know about it. In such cases, the response is cached until the Elder eventually receives the request. We identified that sometimes the response was sent back twice. The first one was processed and the second one was cached waiting for a request that would never arrive again.
While the double-response is the culprit here, we also identified that caching the responses might not be the best choice for such scenarios. The Elder can verify if the response has a request that was validated by the section and process it accordingly. This would improve both time and memory efficiency. We are also reducing the Elders’ responsibility of taking care of the chunk duplication. Instead, the new holder of the chunk will fetch and store a new copy of the chunk which again removes the need for signature accumulation for data recovery related messages.
We will be including these features (and more) in the testnets to come. In the meantime, you are welcome to join the community-run testnet. Kudos to @tfa, @Traktion, @neo and everyone else who has been involved. Visit this post to be a part of the action.
SAFE API
Symlink support is wrapping up this week. Work progressed with test cases, documentation, and fixing some path related issues on Windows that turned up in tests. A large pull request was raised on Wednesday and awaits final testing and review before it can be merged.
CRDT
Good progress has been made in the past week in terms of making the Sequence CRDT ready for a first testnet iteration. We’ve finalised and already merged a PR in the safe-nd crate where this new data type is defined and where the core CRDT logic resides. The AppendOnlyData was also removed from it as the Sequence data type is superseding it.
We were also able to finalise the PR for the safe-vault crate, which was approved and merged today. This means we now have the code ready in the vault to accept requests for storing and mutating Sequence CRDT content. Some of the last things we’ve worked on were to also have the permissions and owner of the Sequence be CRDTs, and the creation of some basic integration tests towards a first iteration and/or testnet. The changes needed to remove the support of AppendOnlyData in safe-vault are now ready for review in a separate PR.
We are now trying to finalise the PRs required to support this new Sequence data type on the client side, which means having the safe-client-libs and safe-api crates expose APIs to be able to send requests to store and mutate Sequence content. The PRs are already functional, there is only a need for some final clean up and the addition of some automated tests before a first testnet using this CRDT data type becomes ready.
Transfers
SAFE Transfers Project plan
SAFE Client Libs Project plan
SAFE Vault Project plan
We’ve been working hard on transfer actor integration into SAFE Client Libs and Vaults. The former now has our mock tests passing with some placeholder APIs, and we’re now working on integrating with our updated vault code. This has needed the creation of some new flows and messages. We now have some SimulatedPayout request types for mock to load accounts with test safecoins as if they had received a farming payout. This is a small but helpful tweak, which gets us closer to how the network will eventually work.
The AT2 implementation itself necessitates some tweaks to our flow for various other operations. Most notably, CreateLoginPacketFor requests (which allow an account to upload login data for someone else, and preload it with some money) will have to effectively wrap two pre-validated transactions: one for the cost of a put, and another for the transfer of money to the new account. Fortunately, we have the basis for the transaction validation in place so this should not be any great shakes to incorporate into the client flow. (And what’s neat here is that we’re effectively setting up transfer bundling – albeit in a small way – while still keeping the onus for co-ordination of this work on the client).
Routing
The shared state signatures PR was finally raised and is currently being reviewed. The DKG replacement PR is also almost ready apart from some kinks that need to be ironed out first. We created a GitHub issue listing all the steps we think are needed in order to remove parsec from routing. It seems like a lot of work, but we believe most of it is quite straightforward. The whole effort can be summarised roughly like this:
- We will implement parsec-less voting based on messages and BLS signatures. It will give us consensus / authority but not total order. We believe that is enough for our needs. The resulting mechanism will be much simpler and more performant than parsec.
- We convert all the data structures that are used to store section-approved information into CRDTs (Conflict-free Replicated DataTypes), which means that all mutations to these structures will be commutative, associative and idempotent. In simpler terms, that means resilient to message reordering and duplication. This will give us eventual consistency for very little cost compared to parsec.
- Additionally, we only apply a mutation if it was approved through the above-mentioned voting process. This gives us resilience to various kinds of failures and attacks because any mutation must be approved by the quorum of the section elders.
Useful Links
Feel free to send us translations of this dev update and we’ll list them here:
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together 
[ EDIT, this really changes from local RAM use to network use, more traffic, but for small devices this is cool as a tiny device can push loads on the wire, but not necessarily handle huge RAM requirements, apart from that it’s also more natural and correct]
)
