Update 7th November, 2024

News this week, for those that missed the Discord Stage session on Tuesday, is the imminent launch of the WASM web app, which should now really be called the Vault. This will allow users to interact with the network without needing a CLI, so obviously that’s a big step forward in making it more accessible to “normal” people (whatever they are).

The latest release tackles some CPU/memory concerns that we’ve observed and which is why we’ve been going softly-softly with the uploaders. These include some changes to range based searching to reduce resource usage.

In addition, pressing L on the Launchpad who show an individual node’s logs, and the upgrade screen has an estimated upgrade time. Launchpad now runs on multiple threads to allow operations to be performed while nodes are being started and upgraded. More instructions here.

As mentioned, there’ll be more of these fixes and features incoming, including some with breaking changes. We’re aiming for a code freeze on 10th December. It’s going to be a busy old five weeks, but we’re quietly confident of being ready by then.

We’ve also launched a new reward system for node operators, allocating 1.5 million test tokens over six weeks, with points earned based on data hosted. You can get going with this now by downloading the latest Launchpad (v0.4.4)

Thanks to @loziniak for his PR on unifying register names.

General progress

@chriso worked on the latest release, which includes data pruning and key management upgrades, among other things.

And out back, @shu has been analysing metrics to inform our choice of instance types and capacities for upcoming networks, and wrote up some documentation for prune and cleaning Influx DB as and when required as part of our monitoring process.

Meanwhile, @anselme has been running a review of all the performance and data security features and needs, so that, with the basics in place, we know what to focus on over the coming weeks. @Qi_ma has also been heavily involved in this area too, as well as putting in several bug fixes and analyses.

@bzee has been profiling CPU by joining with a port-forwarded host on his PC. Most of the CPU usage seems to come from either QUIC handling (connections) or XorName::from_content. He’s also succeeded in getting web workers to work with self encryption, correcting an error that was blocking the UI of the WASM Vault.

And @dirvine has improved his Python module for interfacing with the network, with recursive data maps, functor based decrypt (for direct network reads), and more.

On testing, Ermine worked on vault integration tests, fixing some issues with Windows CI, and raising a PR that adds functional test cases to the Autonomi Vault CLI utility.

@roland has been looking into the behaviour of the ExternalAddressCandidate event including setting up tests with websockets. Roland is also looking at edge cases in payment flows to make sure this vital aspect is working smoothly.

@mick.vandijke Implemented private file/archives upload through the WebApp, including showing the quote prices and paying using MetaMask. He also implemented listing all files from a user’s vault, and opened a PR to check a wallet’s balance before uploading a file. This is to avoid the situation where an uploader runs out of tokens during a file upload, having already paid for a bunch of chunks.

And @mazzi has been working on improvements to the Launchpad including individual node log view, and adding and removing nodes individually. There are also fixes allowing Ctrl+V on rewards address popup, and to eliminate a bug reported by Windows users.

49 Likes

Wowser :flushed: that’s impressive if it’s what I think it mean’s :clap:t2:

17 Likes

Thanks so much to the entire Autonomi team for all of your hard work! :muscle: :muscle: :muscle:

And good luck to everyone with your Beta Rewards! :four_leaf_clover:

17 Likes

Nice work everyone! Latest version seems to use far less CPU than the last one. Happy days!

21 Likes

What do you mean by this? Oo

Since we don’t even have an api right now I don’t think it can refer to something including the api layer…?

12 Likes

Guessing it means no more breaking changes after that point

11 Likes

So 4mb+32gb is settled now and has proven to be perfect I guess :slightly_smiling_face:

11 Likes

I’d love to see a detailed breakdown as to how these figures were judged to be optimum and the internal tests carried out.

12 Likes

Indeed - don’t think that decision can be based on the public testnets (but really would be interesting to know what it is based on)

6 Likes

Just think of cheapest in fees and still able to work somewhat
In my opinion.

Maybe setting window size in QUIC flow control will solve some of QUIC CPU hogging. Each data block check would not be on 4MB but 128KB each time. Removes some of the hogging hey devs and splits it up more over time

12 Likes

Thank God you solved the ctrl+V problem and we’re finally getting our vaults (or an early version at least). At least we’ll be able to upload our cat pictures now lol.

2 Likes

I did notice very high download as the node starts with what seems a lot more than the 4MB x 1000 records. Also I’d assume the 1000 records are not at 4MB each. So why it needed that many GB to be downloaded is beyond me.

I have limited the port out of the router to 300Mbps download and was seeing around 150Mbps download speed for a couple of minutes then periods of none and periods of 150Mbps for another couple of minutes.

That is a lot of data coming down. I suspect by the large number of nodes already shunned by that new node (> 5) that there were many records in error too many times from those nodes it shunned. Again a potential problem the QUIC window size could resolve by not requiring a whole 4MB to be checked and retried at once. Also this might solve some of the cpu load mentioned with QUIC by spreading out the data block checking (hash of data block) over many tests instead of one big one.

13 Likes

Any hints about what to grep for?

Im trying to play BegBlag with @traktion’s sn_httpd and getting many Handshake errors.

But after a good 10 mins it worked :notes: :musical_note:

Thats with 15 nodes running with Formicaio .
Memory use is consistent across all nodes ~160 with not quite regular spikes >200MB


All 15 nodes show much the same graph.

4 Likes

Thanks for the update. Looking forward to the app, and things kicking up a gear once stability improves.

Though it’s early days, the CPU usage on my VPS’ seems to be down massively with the same number of nodes running. I need to be very slow on starting nodes to prevent the CPU going above 50%, but once they’re up, the usage is way less than the previous network, so it looks like a big improvement.

Well done Autonomi Team, and keep up the amazing work!

10 Likes

We are getting close to “Making the World a Materially Better Place to live”. :slight_smile:

It’s interesting to note on Coindesk one of the lead articles discusses just that,

“how does the blockchain community get back to its original ethos, or reason for being…”

Questioning the real purpose of blockchain and what it represents

" perhaps the blockchain is not about money, the post suggests but is about time…"

Anyway some food for thought…,

6 Likes

Thx 4 the update Maidsafe devs

@loziniak :clap: :clap: :clap:

Latest update was just 2 CLI “super”, also that nodes auto start. :+1:

Can’t wait to play with the Wasm web app, big, big step forward :heart_eyes:

keep coding/hacking/testing super ants

7 Likes

And people say “time is money” :smiley:

7 Likes

Btw, I’ve just pushed another version with some tidy up and with most of the logging silenced (sn_networking is very noisy).

4 Likes

I just wanted to highlight this.

There was lots of vitriol, conspiracy, tantrums, after this setting was given a second run. Perhaps the team deserve a bit more respect next time, folks? :face_with_monocle:

5 Likes