Please follow these instructions to upgrade your nodes to the newest version to ensure the best performance and stability for everyone!
For Node Launchpad Users:
Open Node Launchpad v0.5.2 if you are not already on that version
Press O to access the Options screen.
Then, press Ctrl + U, and hit Enter. This will upgrade your nodes. Upgrading can take several minutes for each node. Please donāt close the app or stop your nodes during this process.
Your nodes will now stop
Press Ctrl + S to start your nodes again
For CLI Tool Users:
If youāre using the CLI tool, please update and upgrade. Run the update first: antup update
Then run the upgrade: antctl upgrade --interval 60000
For ALL Users:
Please start your nodes gradually ā especially if you plan on running multiple nodes.
Be conservative with CPU allocation to maintain stability
Changelog:
Memory and CPU metrics use more precise measurements
Apply a timeout for EVM transactions. This fixes an issue where some uploads would freeze indefinitely.
The ant CLI was not selecting its network consistently from the environment variable.
The purpose of this hotfix was really just to release two small fixes to the client:
The main one being to stop an issue we had with uploads becoming frozen.
Even though thereās a new version for the node, there are no changes. Each crate/binary gets a āsafetyā bump for each release, to guarantee a unique version number. This is the easiest thing to do with an automated process.
So mainly, this is just a release for us to stop our uploads freezing.
Before TGE, we might see some hotfixes coming for other, bigger issues.
The changelog section of these upgrade announcements is helpful. It would be even more helpful if the enumerated changes were prepended with [module], or some other method, to help node operators know if they need to take nodes offline to go through an upgrade process. It might also help network stability by not having potentially many nodes churn unnecessarily.
Logging to directory: "/home/ubuntu/.local/share/autonomi/client/logs/log_2025-01-06_23-55-33"
š Connected to the Network Uploading data to network...
Uploading file: "ubuntu-24.04.1-desktop-amd64.iso"
Error:
0: Failed to upload file
1: Failed to upload file
2: Error occurred during payment.
3: Cost error: NotEnoughNodeQuotes(3a8868(00111010).., 1, 5)
4: Not enough node quotes for 3a8868(00111010).., got: 1 and need at least 5
Location:
ant-cli/src/commands/file.rs:58
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Right now running nodes is work, from one day to another 30% of all nodes might have stopped, there seem to be no upper limit to how much resource usage a node can have. It is very difficult to find which have stopped so all nodes needs to be reset. To get as many as possible to run nodes it has to be as close to fire up and forget as possible. There has to be an easy way to find which nodes have been stopped through the metrics and then easier command to stop/remove nodes. There needs to be a good phone app that can tell if nodes go down and an easy restart/reset button, maybe auto restart when resource usage goes down. Is the dev team aware of this and will the team work with these things after launch? I am not sure at the moment who to talk to about this.
I am not sure if I can help here, but I can point out some things.
Nodes will spike in cpu b/w and memory, so lets say resources
They will go quiet, quiescent
They will get to max ā¦ that may mean nearly all available resources
if you have maxed out. machine with lots of nodes you will risk some will die
Nodes donāt have an upper limit on some resources except disk space. Limiting b/w or cpu automatically is a very non trivial thing to do automatically
So given that and not knowing how many nodes you are running on what machine at what capacity, I think we can say the above.
Then we have to think, what can we do?? Hereās more thoughts
nodes should not crash, they may get throttled, but not crash, if they crash we have issues (I think you are saying they do)
Nodes may become unconnected and totally shunned
So if we have folk trying to run as many nodes as possible we will have issues for sure and those running nodes in this way will become service providers. So they will need to have very high levels of knowledge on their hardware and b/w resources, including routers and bridging protocols etc. (bgp)
Now what I hope for is many users running some nodes, but not even know they are doing so. They will be using apps and those apps will run nodes as part of the app. So the apps will earn money via nodes for the apps to use! (i.e. an infinite dropbox for free with perpetual data, no wallet visible).
@bzee is working on the API and you will see how devs can integrate nodes with apps in this way. That IMO will b e a very large part of the eco system. Itās a vital thing IMO for us to get mainstream and not having to teach folk digital currency of any type (native or etc20 or ā¦)
For node runner farm types we will have to do much better at finding out those sweet spots you are looking at, for sure though.
When i start a node with the node command (not antctl) I still see a spike of about 1/2 to 1.5 seconds. (monitor updates every 1/2 second and is there for 2 updates)
This is a 24 core 48 thread machine hitting 80% total CPU usage when normally below 10%
The RPC port is specified but not used. I use /metrics but that is not happening while a node starts.
Yeah, there are CPU issues that are independent of the node managerās use of RPC. They are to do with relaying and the number of open connections. We are working on that.
I understood that feature to be a temporary ārev-limiterā for the pre-launch phase,
I may be wrong but I never took this as being a long-term behaviour.
I can understand the thinking about running many nodes then you also have to setup some form of tool that controls the nodes, when the registry update thing will be gone that might make things easier. Right now the cpu hangs at 80% for many minutes which might kill the nodes just by running the command to stop nodes. The more nodes you run it seems almost exponential the time for the command to update the registry.
Problem with this size of the network 80-90k nodes is when someone pulls the plug on 10k nodes, others in the network might get trouble as the load increases heavily on some machines.
As there are no upper limit, on linux OS if a pc hit 100% cpu then the whole system freeze, canāt even type or move the mouse. When someone pulls the plug on 10k nodes then it might have a house of cards effect when others in the network get into trouble hitting the 50% cpu limit or as before hit 100% cpu. It would be good as you said if the nodes somehow donāt choke the system. Maybe the 50% cpu load limit will be replaced closer to launch for something better, @Neo had some suggestion on the topic.
Looking forward to good solutions. My initial concern is that I want as many as possible to run a lot of nodes so the network can reach some kind of critical mass as soon as possible, maybe 1-10 million nodes.
Apps that run nodes sounds interesting, will they run when the app runs or all the time. It reminds me of when sites installed crypto miners a few years back, will be interesting to see how it might work and be received by clients.
Not given it much thought, but maybe it would be better to stop nodes that are consistently shunned (through slow CPU, lack of disk, etc), rather than pre-emptively killing nodes for chewing CPU.
Announcement: Antnode Upgrade 0.3.2