Formicaio

Yes, they are: if I run ./formicaio start after upgrading (so, killing old ./formicaio start process and unziping new files), it tries to start all running nodes (the nodes are already running, they were started using old version of Formicaio)

1 Like

That shouldn’t be the case if you killed the old Formicaio app, all nodes are killed when Formicaio is stopped/killed…unless you are trying to run more than one Formicaio instance from within the same folder using the same Formicaio DB file, so that will be a problem. But, when you kill Formicaio, all nodes managed by it are killed.

1 Like

My nodes are up and running (and earning) and were not killed after stopping Formicaio.
Here is my full process from starting to upgrading:

  • Start:
    • start Formicaio using nohup ./formicaio start &
    • launch nodes using GUI
  • Upgrade:
    • terminate process ./formicaio start using btop (after that, all my nodes are still running, I can see the processes)
    • download and unzip new formicaio version on top of old install (but I’m keeping the formicaio_data and everything that is not on the zip)
    • restart new version: nohup ./formicaio start &

Also, I don’t want my nodes to be killed when upgrading so I’m quite happy that they are not killed when terminating Formicaio

2 Likes

I see, I didn’t know you were running it that way, then I see what can be done. Currently Formicaio assumes nodes are killed when starting and that’s why it’s trying to start up those which were running. All make much more sense now, I’ll be trying to add those options. Thanks again, very helpful feedback!.

2 Likes

I can confirm this does not always happen.
Several times I have had to do


pkill -e formicaio && pkill -e antnode

Another nice thing while you are pandering to our every need @bochaco would be an option to choose only Inactive nodes and an option to set the timeout somewhere here

Thanks for all the work you are putting in on this, you should publish a wallet address for donations on the app as well :slight_smile:

EDIT: Also what am I doing wrong here in Terminal


* > nodes start --status inactive --interval 5

* error: Invalid value for '--status <status>...': invalid type: unit variant, expected newtype variant

#>
4 Likes

You are not doing anything wrong, I need to allow that since Inactive now has several sub-status then it’s broken, I’ll find a solution for next release.

True, I’ll be adding that.

1 Like

Formicaio V0.4.8 was just published to fix the above issue with node status filtering/matching from CLI (no other fixes/features are part of this version).

1 Like

Something that I noticed: after 1 or 2 days, CPU usage of Formicaio is always 100% (1 core), wether I have 500 nodes or 2500 (and the fetch rate of metrics set to about 3 minutes). Note that at first, even after all the nodes are launched, CPU usage of Formicaio is low, but after some time (1 to 2 day) it takes 100% CPU (and doesn’t go down)

2 Likes

What do you see in the logs when this happens? anything unusual or just the "Fetching status and metrics from N node/s ..." message every ~3mins ? (I may be publishing another release today with some minor improvement I suspect could be related to this, I’ve actually identified where the issue is but I’m looking for a proprr solution).

Nothing unusual, only the Fetching status… message.

1 Like

Formicaio V0.4.9 was just published to introduce some improvements which I believe could solve the root cause of the issue discussed above (no other fixes/features are part of this version).

5 Likes

It seems to be better, I updated one machine (and although it is still trying to restart all my already running nodes) and cpu usage is now really low.
I’ll see how it goes on a fresh install, but it bodes well :grinning_face:

1 Like

After one day load is still ok most of the time but there are some times of high load (100%cpu, which can last multiple hours), and I usually see these kind of logs when load is high:

Removing oldest metrics from DB for node 516e366b6377 ...
Removed 0 metrics records
2 Likes

I have been following your efforts for awhile, kudos!

I’d like to try Formiciao out now, however my infrastructure setup is incus(LXD) using LXC containers. What’s the best way to proceed given your use of Docker containers?

2 Likes

You can use the binary version (without docker): GitHub - bochaco/formicaio: Le formiche sono insetti sociali che vivono in colonie e sono note per la loro organizzazione e cooperazione.
That’s what I’m currently using inside my LXCs.

3 Likes

That is a periodic task which keeps database size capped by cleaning too old metrics data, it performs this for all nodes. It’s currently not possible to change its frequency and/or DB size limit, but it could be in future releases.

Back again for my periodic check in - I installed Version: 0.4.7 with 12 nodes:

  • each node is created with 60s between each creation
  • each node works for about 10s
  • each node stops working with a message like:

ā€œStatus: Exited (exit status: 1), 5 seconds agoā€
and never comes back:
ā€œStatus: Exited (exit status: 1), 2 days agoā€

1 Like

Firstly upgrade to v0.4.9

Start your nodes and look in /formicaio/formicaio_data/node_data,
choose one of your node dirs and check formicaio/formicaio_data/node_data/-nodenumber-/logs/antnode.log for errors.

1 Like

Open the logs from the GUI, exit code 1 is usually due to UPnP issues, you will probably have to create nodes with UPnP disabled.

Thanks people . . from the log:

[2025-03-21T12:24:48.373875Z INFO antnode 450] Node is stopping in 1s...
[2025-03-21T12:24:49.374294Z ERROR antnode 459] Node stopped with error: Node terminated due to: "UPnP gateway not found. Enable UPnP on your route

but . . UPnP was already on . . and has been for quite a while . .

UPDATE: I should have been more specific that there was no improvement - the nodes still fail . .

1 Like