So swap is used when 60% RAM is used? Then I need to look at that setting and bump it to 95%. Do you know how to check the setting? System should never use swap until it can feel the flames of hell.
Within the last couple of days I stumbled accross something very odd.
1 of my systems had a meltdown with swap enabled. Both swap and RAM got depleted.
I turned swap off and restarted, it now uses less that 50% RAM.
More experimenting planned today.
2 Likes
Chatgpt told this, I will try and change setting for all servers. I wonder if the swap setting could affect that Ubuntu feels sluggish under moderate load.
To ensure that your system primarily uses RAM up to at least 95% usage before swapping, you would want to set the vm.swappiness value lower, so that the kernel avoids swapping unless absolutely necessary.
Recommended setting:
vm.swappiness = 10or lower: This will make the system very reluctant to swap out memory and will prefer using RAM up to high utilization before considering swapping. At aswappinessof 10, the kernel will only swap when free memory becomes critically low (i.e., when youāre close to 100% RAM usage).
Change RAM/Swap setting:
Edit the following config with;
sudo nano /etc/sysctl.conf
add line
vm.swappiness=10
save and exit
Ctrl + O
Ctrl + X
To apply the change
sudo sysctl -p
To chek the setting
sudo sysctl vm.swappiness
1 Like
Good news, the issue has been identified after a deep dive on this topic yesterday, and a fix is in the works internally (to be vetted out).
Running 870 nodes at home with the problematic area of code removed/certain tweaks (not the final PR / solution just yet) as a POC now only used up 108GB of RAM for me, i.e average of 127MB per antnode.
11 Likes
Can you summarise what was causing it? I am interested
File Descriptors specific to /proc/<pid>/task/<threadid>/stat were being created per antnode from 10s to 100s to 1000s as more antnodes got registered.
Its due to the sysinfo rust crate having to do a refresh on CPU usage (OS scan) that is causing the pain here (periodically), as the file descriptors got built up, and never reduced, which I believe led to the increased memory.
After trimming out that code as a quick confirmation, the FDs dropped down to 14 per antnode and remained there.
10 Likes
nice quick āintegrated logic memory leakā find. Kudos!
Anyone having using tiny form factor devices, such as Lenovo M910Qs?
Decent CPUs, sufficient RAM, small/quiet, but maybe small SSDs. Could upgrade or add external storage.
Seeing these things pretty cheap as refirbs. Maybe pretty eoconomical for ant node duties? 
edit: M710Q uses about 12w on idle, apparently.
If its Windows running, about 3GB RAM is needed vs. running Linux on the same device to run the same number of antnodes on each OS type in a performant manner, on what are essentially small āNUCā like devices coming into the market cheap as refurbished. That and getting enough ISP bandwidth for the node count and you will be good to go. All of them have 1Gbit and you can backdoor manage them via USB from your notebook USB and run them headless to keep it cheap, and even use and space space as backup storage.
1 Like
My only thought here is once again the power draw of small SBCās or āNUCā like devices versus a larger PC. If a āNUCā like device or laptop has a power draw on idle of 30-50 watts then its not going to be worth using them versus a PC drawing 80-100 watts on idle. The PC should be able to run more than double the āNUCā like device.
Do your sums and make sure if a 2nd hand PC might not work out better than 3 NUC like devices
That is why RPis and other SBCs are rather interesting devices to consider since the power draw is low and can run many of them with an old PC power supply saving on power bricks. SBCās often have M2 slots for disk space already so no hats needed like many RPis need
Costs of power for devices dedicated to node running has to be considered and accounted for since that is the recurring cost associated with dedicated node running. This cost over a year can amount a major portion of the initial outlay costs.
EG 100 watts draw in a year is 876 KWHr and if $0.30 per KWHr then its an extra $260 you are paying in outlays
4 Likes
I used 2xDell OptiPlex 7060 USFF / Intel Core i5-8500 / 16GB / 500GB SSD 4 months ago and they held 80 nodes then without a problem. I havenāt put them up on the current network yet, but plan to next week.
It took me a while to figure out how to share the net through ubuntu to another computer with a second network card, but I finally did it.
A second-hand dual-port 1 GB network card is 5 euros in BG and together with a bridge mode is the cheapest option - no expensive routers and complicated configurations.
Check out the Dev Forum
4 Likes
From comparing my 14yo laptop with SSD vs my new medium range laptop, both power and performance seem better. Cooler and faster.
Iām thinking these wee devices should easily have enough CPU for their disk space, which seems like a good starting point. Hard to change a CPU vs storage.
Then Iām thinking they are small and quiet vs bigger boxes. I donāt want a big cabinate full of noisy fans, etc. I have a bit of space near my router that Iād like to fill without too much noise too. I could get a few of these little fellas, without causing issues.
We also have solar, which means power is pretty cheap, at least when not winter. So, less converned with draw cost, but I suspect these small devices are good that side.
I bet the bigger boxes may be cheaper though, as I canāt imagine they are in high demand. At least less so than these wee ones.
Iāll do some more research, but recycling these old devices for cheap ant nodes could be a nice little earner. Especially if it means Iām getting more value out of spare solar.
2 Likes
I have Odyssey SBC (4 core intel) with 2TB SSD drawing <10W and could run 12 or so nodes on the previous versions (30 in the very early beta versions). Have not tried with 3.3 and expect to run more than 25 nodes per SBC.
I 3D printed a 10" rack for my router and SBCs and patch panel. Then printed a 1RU unit to hold the router and a 1 RU unit to hold 2 of these Odysseys. Then itāll sit inside a bookcase out of sight. Hope to get 75 - 100 nodes with 2 of these and duplicate that for the starlink link mounted in the same rack.
I am limited in upload b/w so that is why I am happy with this setup. Already had the odysseys
3 Likes
Iām running 100 nodes steady on an 8 year old laptop (2 cores, 4 threads i7 6500u) drawing ~12w and Iām not CPU limited (~30%) but ram limited (16GB). So yeah I think most SSF or laptops are great for running nodes with limited power draw.
Also, CPU usage should be lower in the next releases with last tuning from @Shu, removal of rpc and removal of logs.
80ARM CORES
160GB RAM
10TB NVMe
Rack-mountable, PoE-powered carrier board for Raspberry Pi Compute Module 4/5 and compatible devices with all the necessary interfaces. With Compute Blade, you can create a high-density, low-power-consuming, plug-and-play blade server for home or data-center use. Individuals and businesses.
4 Likes
The NUCs with Intel CORE running i7 12thgen are fast, and very power efficient, a base set up with 32GB give you a workable desktop in MS and really good in Linux, with 1TB base storage expandable to 2TBā¦ The price on these NUCs, Geekom, BEElink, ASUS, etc from Taiwan are dropping fast, you can spec which type of processor, Intel or AMDā¦ the number of nodes you run will be performant, provided you leave at all times at least 20% disk space free, thatās the oxygen that keeps GC, defrag and wear levelling working fast without disrupting foreground tasks, so your antnodes dont get shunned
Doing the math thatās 700GB useable, after desktop apps, linux, and related storage needed locallyā¦ 700/35 thatās 20 nodes running and you have Daily driver.
If you run these NUCs āheadlessā with just base Alpine Linux say via the USB connect, and 1GE or 2.5GE connect to the Router/Cable modem GE Hub, you can run many more antnodesā¦
Add another x2 512GB NVME SSD Strips in these NUC and you are at 2 TB, but its really 1.4 TB to keep your antnodes performant, some of these NUCs now upgrade RAM up to 96 GB , but most coming off biz. leases (these are the big buyers of NUCs, strapped to monitors) and sold refurbished in the next few years will be 32GB upgradeable to 64GBā¦
One needs a small UPS of course to protect your ISP modem and router and NUC(s) , should your grid go down, you might get lucky and the ISP is self powered and you stay up and keep earning , such ISPs have low powered fibre optic repeaters to the home and your service stays up when that happens, because such modernized ISPs have their own power feeds aerially or undergroundā¦, tied into a dual backup substation configuration to keep bix customers at home up and runningā¦ (rare in many old NA and EU neighborhoods)
2 Likes
Iām not sure that is a huge concern, as long as you live somewhere without frequent cuts. Autonomi nodes seem to bounce back to earning attos pretty quickly after a restart (new release upgrades for example).
1 Like
Hey guys! Still looking for ways to scale up nodes from home. I have an option to buy a secondhand server set, Iām completely unfamiliar with server racks but it feels very clean and scalable. Before purchasing Iām looking for some advice from someone more experienced. Hereās what the set contains:
2x HPE Proliant DL380 Gen9, each server has:
- 2x Intel Xeon CPU E5-2650 v3 @ 2.30 GHz (20 cores)
- 192 GB RAM
A module for M.2 SSD-expansion
A High-performance storage array:
- HPE MSA 2040 ES SAS DC SFF-storage (2U), model K2R84A
- 2x MSA 2040 SAS-controllers
- 13 SAS HDD
- 13 brackets + 11 fronts included
2x Cisco SG500X-24P Layer 3 switches (1U):
- 4x 10 Gigabit Ethernet SFP+
- 24x 10/100/1000 PoE+
- Including rack mounts
Cables:
2x SFP-H10GB-CU1M
4x Molex 1110751004
APC switch and a half height server rack.
I can get this for ā¬2000, which seems like a very sweet deal. Wondering what you guys think though.
What sort of internet connection are you trying to fill with that lot? I hope youāre on good terms with your broadband provider! 
Seems expensive.
I recently got a E5-2699 v4 2.2Ghz (22 cores each dual socket) with 1.5TB ram for $2800 (refurbished) (including 6x10Gbps SFP+ network cards + 12Gbps 12 bay disks sas/sata), compared to your 2 x E5-2650 v3 2.2Ghz (10 cores each dual socket) with 192 GB RAM for $2000.
4 Likes