Yes, they are: if I run ./formicaio start after upgrading (so, killing old ./formicaio start process and unziping new files), it tries to start all running nodes (the nodes are already running, they were started using old version of Formicaio)
That shouldnāt be the case if you killed the old Formicaio app, all nodes are killed when Formicaio is stopped/killedā¦unless you are trying to run more than one Formicaio instance from within the same folder using the same Formicaio DB file, so that will be a problem. But, when you kill Formicaio, all nodes managed by it are killed.
I see, I didnāt know you were running it that way, then I see what can be done. Currently Formicaio assumes nodes are killed when starting and thatās why itās trying to start up those which were running. All make much more sense now, Iāll be trying to add those options. Thanks again, very helpful feedback!.
I can confirm this does not always happen.
Several times I have had to do
pkill -e formicaio && pkill -e antnode
Another nice thing while you are pandering to our every need @bochaco would be an option to choose only Inactive nodes and an option to set the timeout somewhere here
You are not doing anything wrong, I need to allow that since Inactive now has several sub-status then itās broken, Iāll find a solution for next release.
Formicaio V0.4.8 was just published to fix the above issue with node status filtering/matching from CLI (no other fixes/features are part of this version).
Something that I noticed: after 1 or 2 days, CPU usage of Formicaio is always 100% (1 core), wether I have 500 nodes or 2500 (and the fetch rate of metrics set to about 3 minutes). Note that at first, even after all the nodes are launched, CPU usage of Formicaio is low, but after some time (1 to 2 day) it takes 100% CPU (and doesnāt go down)
What do you see in the logs when this happens? anything unusual or just the "Fetching status and metrics from N node/s ..." message every ~3mins ? (I may be publishing another release today with some minor improvement I suspect could be related to this, Iāve actually identified where the issue is but Iām looking for a proprr solution).
Formicaio V0.4.9 was just published to introduce some improvements which I believe could solve the root cause of the issue discussed above (no other fixes/features are part of this version).
It seems to be better, I updated one machine (and although it is still trying to restart all my already running nodes) and cpu usage is now really low.
Iāll see how it goes on a fresh install, but it bodes well
After one day load is still ok most of the time but there are some times of high load (100%cpu, which can last multiple hours), and I usually see these kind of logs when load is high:
Removing oldest metrics from DB for node 516e366b6377 ...
Removed 0 metrics records
I have been following your efforts for awhile, kudos!
Iād like to try Formiciao out now, however my infrastructure setup is incus(LXD) using LXC containers. Whatās the best way to proceed given your use of Docker containers?
That is a periodic task which keeps database size capped by cleaning too old metrics data, it performs this for all nodes. Itās currently not possible to change its frequency and/or DB size limit, but it could be in future releases.
Start your nodes and look in /formicaio/formicaio_data/node_data,
choose one of your node dirs and check formicaio/formicaio_data/node_data/-nodenumber-/logs/antnode.log for errors.
[2025-03-21T12:24:48.373875Z INFO antnode 450] Node is stopping in 1s...
[2025-03-21T12:24:49.374294Z ERROR antnode 459] Node stopped with error: Node terminated due to: "UPnP gateway not found. Enable UPnP on your route
but . . UPnP was already on . . and has been for quite a while . .
UPDATE: I should have been more specific that there was no improvement - the nodes still fail . .