With the HeapNet2 testnet having just left the launchpad, it’s time to review the previous one, MemDebugNet .
MemDebugNet served us extremely well, before inevitably spiralling into Weimar Republic territory due to the highest StoreCost in a close group being the one that gets paid. We are changing how that works by discarding the higher fees and only paying the majority, which should cut out wild price increases. And while it isn’t yet available for HeapNet2, it should follow soon enough.
Other things we learned included that uploading files in batches where the batch size is less than the total number of chunks can fail.
We had a problem with our streaming self-encryption/decryption leading to downloads retrieving prior attempts’ chunks and yielding larger than expected (and corrupt) files. Thanks to @19eddyjohn75 for alerting us to that one. It’s fixed now.
The cash_notes directory can get very large. Saving CashNotes is just a temporary measure for debugging purposes, so this won’t be an issue long term. Its size is also dependent on batch_size. See @anselme’s explanation for why that is.
Work continues to understand and address memory usage and clean-up issues during file transfers and decryption. We’re testing some of these on the new testnet.
As always, a thousand thanks to everyone who’s helping out. Every squashed bug is one step closer to launch. 
General progress
@Chriso has been working on a node manager binary, including an install command, and looking at extending the service-manager-rs crate to suit our needs. This will allow nodes to be updated on the fly, like clients can be now. This is working on Linux, but needs more work to support macOS and Windows.
Meanwhile, @bochaco continues to work on having nodes publish and subscribe to predefined topics such as storage payment notifications over GossipSub. The notification message contains the payment’s recipient key and the transfer details, and the owner of the corresponding public key can listen and receive these messages. Initially this will be used for rewards payments and for paying Foundation nodes. We will be trialling this shortly.
@joshuef has been turning his attention to the issues turned up in the last MemDebugNet testnet, including PRs 811 and 812 aimed at reducing memory allocation and simplifying connection handling by removing identify cloning/collecting and unroutable peers. He’s also been leading testing of large (multi-GB) file uploads/downloads where we’ve been seeing some flakiness, looking into hangs which seem to be related to StoreCost, and implementing some wallet cleanup in the event of network tx not being registered to disk. He also fixed an unfeasibly large timeout issue that’s a likely cause of the hangs spotted by @happybeing.
@Roland has been digging into the finer points of Terraform and Ansible to help Chris with the sn-testnet-deploy tool. He’s also refining the logging process to capture test client logs to individual files instead of stdout, and working on data location tests during churns to find out where chunks really end up.
Based on heaptrack analysis of resource usage, @Bzee is looking into memory allocation for multi-address conversion. Could that be one of the causes of the slow memory leak?
@Qi_ma opened two PRs (814, 815) to clean up temporary chunks after uploading and compare downloaded files twice to ensure integrity. He has also fixed error handling issues in the GetClosestPeers function which, when an error occurred returned the wrong peer, and a memcheck CI failure which turned out to be related to StoreCost not being paid properly.
And @Anselme has been digging into wallets. He found that wallets keep spent CashNotes, allowing reuse, and so is working on locking them during transfer and deleting spent ones after. Tested locally on Linux, this makes concurrent uploads much faster as previous concurrent CashNote uses would create errors and thus retries. This should also allow folk to run various client commands in various terminals at the same time without incident.
Useful Links
Feel free to reply below with links to translations of this dev update and moderators will add them here:
Russian ; 
German ; 
Spanish ; 
French; 
Bulgarian
As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!
is so 
the devs really nailed this
)
