Sure, I agree there. I took only what was known (a generic multi threaded passmark score), else I have no other means of roughly estimating or extrapolating a suggestion for @Mightyfool . Its a start as oppose to real world benchmarking off antnodes on every hardware, and its impossible to do so prior to procuring the hardware itself.
I am keeping an eye on context switching, playing around with pinning the antnodes to rotate between 2 cores (bitmask) each based on 22 buckets for a 44 core machine etc, and see if the CPU can be lowered some more without causing an increase in shunned or error rates at home.
I would not like to see the memory footprint off an antnode pass or cross boundaries across the NUMA sockets here either.
The 4 nodes is my notebook, I am working on 11th and 12thgen Intel CORE based i7 NUCs, to see what I can run using Alpine , in the base config 32GB RAM and 1 TB (x2 512GB) storage with and without(headless an LXDE or similar Window Mgr. Current ISP Max is a 1Gigabit planā¦, I can expand these NUCs to 64 GBRAM and 2TB NVME SSD Storage maxā¦ , so thatās my challengeā¦ There are alot of these NUCs scheduled to fall off business leases in the next few yearsā¦ SSD wear will be light for many of them. Iāll take your advice and run the numbersā¦ ty.
@Shu These two NUCs will get a special Alpine build with a LKM insmod and phy media format to increase durability, reduce power, and also speed up write.
I am running Ceph FS at home, where I present a mountpoint to the LXC container.
After some more tweaks, here is the latest numbers after it took 6hrs+ to startup 3000 pids:
3000 Nodes at 300GB RAM at 340 Watts power draw within Alpine LXC.
Therefore:
102MB RAM per Antnode.
(340 watt current - 240 watt baseline ) = ~33mW per Antnode
LXC CPU usage at 62% (artificially increased temporarily due to higher density testing)
Do you have an estimate of how much of that baseline is now being used to run nodes.
The baseline has the cpu still running 100% doing work, prob at lower cpu frequency. But now for 62% (just using that figure of yours for an example) of the time the CPU is running nodes. Thus 38% doing unrelated system work. IE approx 38% of the cpu baseline power.
I would now expect the baseline of non-node work to be lower than 240 watts by a not an insignificant amount.
Wonder if you are able to do an estimate. Is the 240 watts baseline the CPU power of total system power, since youād need the baseline cpu power used to get that estimate.
Not sure if I followed completely, but in idle steady state, the machine draws 240 watts with 0 antnodes and nothing else running on it (i.e CPU < 5%).
I was thinking about this the other day and harking back to my writing RTOS systems for microprocessors/mini-computers I realised just using a baseline figure will not be accurate.
The CPU is doing work 100% of the time in baseline. Just mostly not user mode work so the CPU usage is very low since that only is showing the user mode programs total work (user mode and system calls).
Thus when you are using 62% of the cpu running applications (user mode and system mode) the cpu is only doing 38% non-application work.
Thus for the cpu power the baseline is only correct for 38% of the time. Thus if baseline CPU is 100 watts then when running nodes its 38Watts ābaselineā and the other 62watts is part of the running of nodes.
@neo - the system is idle and using 240 watts (in steady state) doing minimal i/o and cpu work in general. Anything that raised the power consumption here is directly related to antnodes, so I am okay with my math (user or kernel space (doesnāt matter) it was all associated with running more applications and in this case only antnodes.
The LXC CPU usage is very close to Host CPU Usage (I am just not showing that chart), as the LXC Usage went up, so did the Host CPU Usage. There is only 1 LXC on the host.
I donāt know where folks are getting this figures from. There is literally two power watt readings for the numbers here, with and without load . And only one contributor to the extra power consumption aka antnode (it doesnāt matter to me if it uses kernel or user or both) to get the work done from the baseline of 240 watts (steady state w/o antnode).
@Shu there is apparently dpdk support for Freebsd, so in theory it will run DPDK open vswitch offload on say a mellanox or naptech dual port card , if you want to located one of your fleets of antnodes in a colo and the colo operator gives you two TOR switch leaf drops for path redundancy from your monster system with say 10Gbit per port using an offload NIC, that would be a real beast of an antnode system setupā¦ The Napatech NIC is a nice fit in 1U rackmount. For the record I know and worked with CMO there Jarood Sikket when we were at FORE systems together bak in the 90s, he is a good guy and the Napatech Danish techies are the best in that NIC offload space.
It might be something the Autonomi foundation would want to fund, a POC with your stuff and the DPDK offload setup running on say mellanox and napatech?
There is no stopping big node operator deployments of Autonomi Antnode fleets in the colo, its going to happen (is happening?)
imo we are better to get ahead of the curve on this one and dictate how that happens to ensure the quality of antnode āfleetā growth performance
so i tried as normal user and i kept getting antctl command not found. however logged in as ārootā ive managed to get 10 nodes up and running now. Is it not a good idea to run it as root?
Note the anm subdir. It may be worth exploring as it JustWorks
It JustWorks in a weird manner but does Work.
And you can always ping @aatonnomicc or myself for hand-holding
The main directory NTracking is useful but not essential now IMHO - go to the scripts in the subdir and confusingly the first actions you need are halfway down the README
. And only one contributor to the extra power consumption aka antnode (it doesnāt matter to me if it uses kernel or user or both) to get the work done from the baseline of 240 watts (steady state w/o antnode).