they earned this last Sunday or are the earning this since last Sunday? (or daily consistently since last Sunday?)
2k nodes I ran fell below 20 ANT per day this week which turned them into money burning machines 
There was a power outage on Monday, so from start until this last Sunday they earned without a problem, earnings only went down because network grew larger.
Well, I donāt use swap, but it doesnāt hurt to run 2000 nodes just to see if thereās a difference. Anyway, I keep the machines only to support the network - 20k nodes bring me only 12 ant every 12 hoursā¦
Privacy. Security. Freedom
1 Like
I need to do some changes to the hardware, when fire up the nodes again I will measure how many days they earn consistent.
1 per PC. ā¦
1 Like
fyi
500 node machine last 2 payouts - thatās ~9x what you got on avg per node ā¦ maybe thereās a bottleneck involved even though the machine looks healthy ā¦?
3 Likes
Also my 700 node machine had a power cut and I had to restart the nodes.
There was zero improvement in emissions earning over the old nodes that ran since TGE. The emissions just reduced because of the network size.
In fact on other machines I also do not see any measurable changes in earnings when I have had to restart the nodes. I do see the emissions earning vary from one 12 hour period to another up and down.
I am wondering if for some it is more a case of the randomness favouring your new nodes for that initial payment after restart and confirmation bias says ah ha see restarting nodes works.
Or maybe its because I am not maxing out these devices with so many nodes due to lower available upload b/w
Or maybe in this version of the node software there is something in the restarting and in my case it was the randomness did not favour me.
But without rigorous blind testing I think with experiences giving the opposite effects when restarting, it is dangerous in my opinion to consider it factual that restarting all your nodes works in earning you more emissions. To much randomness for one and too many experiences that suggest opposing effects.
2 Likes
yes - for me it varied up and down (even though it was 2k nodes) Ā±30% or so ā¦ no restarts ā¦ which is quite significant but not an issue if you know thatās just how it is ā¦
All on 1 address, but I also have another machine with a different address that is currently starting nodes every 1 minute and I just stopped them on
pgrep -c antnode
1131
Here is the analysis of the machine at the moment:
Brief Analysis of vmstat and iostat Output
CPU (vmstat)
Very high idle time (95%) ā The system is mostly idle.
Minimal user (1%) and system (4%) CPU usage.
No significant I/O wait (wa = 0%), meaning no noticeable disk bottlenecks.
Memory
Large amount of free RAM (~550GB).
No swap usage (swpd = 0).
Cache usage is ~198GB, which helps with disk performance.
Disk (iostat - nvme0n1)
Low read activity (19.24 kB/s, 0.94 r/s).
Moderate write activity (3119.98 kB/s, 63.53 w/s).
Write latency (w_await = 17.71 ms) is slightly high but not critical.
Disk utilization (%util = 1.03%) is very low, meaning the disk is not a bottleneck.
Conclusion
The system is mostly idle with no CPU or disk stress. There is plenty of free memory, and disk activity is low with no I/O wait. The system is in a healthy state with no immediate performance issues. š
Privacy. Security. Freedom
1 Like
and youāre running exposed host and not --home-network (just for completeness of the info) which means breaking relays canāt be the issue on your side - right?
1 Like
And as another example just yesterday my machine had to be restarted and the nodes earned 1/4 with new nodes in the period following all the nodes being started
2 Likes
1 Like
Yes we should be very cautious to make assumptions. Randomness combined with smaller rewards caused be growing network can make us easily think there is something wrong while it might be just normal.
4 Likes
Ask it about the cache usage of almost 200GB, what is that, where is that cache located and does it effect the cpu?
I can imagine having too many nodes (too many for whatever reason) can cause the system to slow down.
Nodes do have many times the context switches in the operating system due to it being a heavy communications program. Then having 5000 nodes means the OS core is doing a lot, a hell of a lot of task switching and this does NOT show up in TOP and other system resources since context switching is not considered part of any operations (processes user or system). I am sure it can be worked out, but you are not going to see it easily.
So in fact 5000 nodes could be using up 10% of your CPU, at least one full core (both threads), or many cores just context switching if a lot of messages come in a flurry of activity
2 Likes
What is the ~200GB Cache?
The cache (cache = 198GB) you see in vmstat refers to disk page cache, which is managed by the Linux kernel. This cache stores frequently accessed disk data in RAM to speed up future reads and reduce disk I/O operations.
Where is This Cache Coming From?
- Linux automatically uses free RAM as a cache for disk reads/writes.
- This cache includes file system metadata, recently accessed files, and buffered write data.
- The large cache suggests that the system has processed a lot of disk activity in the past and is keeping frequently accessed data in memory.
- This is not a memory leakāLinux will free up cache memory when applications need more RAM
Check out the Dev Forum
2 Likes
It will be interesting to see if you earn more consistent running fewer nodes. Something seems problematic running more 5000 nodes or above, as I also seen others with similar problems on Discord.
Maybe @shu can give his thoughts on the situation?
I have a dev friend with setups in the cloud with significantly weaker machines with so many nodes that earn significantly more and on that I based my assumption that it is related to the quality of the internet and loss of connectivity over time - because the last months of the beta were earning well on significantly higher load. But anyway I will test with 1131 nodes and I will tell you the difference. 
Check out the Dev Forum
3 Likes
My Hetzner nodes earned about 50% more than my home nodes, every time, during late Beta.
1 Like
I would like to weigh in here with my experience. Personally I do not think the lack or earnings have nothing to do with connection.
As most people know I run both from home and from the cloud. When it comes to data chunk storage, before TGE my home nodes performed close to equal (though slightly less) than my cloud nodes. After TGE home nodes have dropped off horribly, earning around 10 to 15% of what my cloud nodes are (have been: Iāll get to that) earning. My homes nodes are behind a very strong and capable router, are connected with fiber and have a very stable and high bandwith connection (not close to the limit). Both setups are equally loading the systems and have around the same amount of nodes running per device.
My earlier assumption was that this had something to do with latency, where nodes try to find nodes closest to them and most of the nodes being run at central places like data centers (Hetzner for example). I kinda expected that because of this, home nodes far away from those data centers are eventually going to lack connections as they do not have as many nodes nearby as the cloud servers.
Whatās weird is that recently I slowly started killing my server nodes to restart (using a different reward address), but now earnings have dropped off by around 50% since. So the newer nodes are apparently not earning. Same servers, same settings, just started a week or 2 later.
Iām not sure if this is at all helpful, but I do think it paints a pretty clear picture that server load is not really a thing in general (perhaps for some individuals it still is).
1 Like