I don’t see this update available on Umbrel.
You should see it now or in a few minutes, the build for docker images takes quite some time so they are available later than native binaries.
2 Likes
Isn’t it that now the most problematic resource is network connections count, so router being the weakest point, not RAM or CPU?
Perhaps someone could just maintain a community fork? Simpler code means less work for the team and less bugs/problems.
1 Like
No, I started clean, no docker running, no old nodes running.
But it’s not a Formicaio problem, I asked some help in a more relevant thread before switching to LXC containers.
1 Like
I am now finishing with debugging an issue similar to yours where a node stays in status Inactive (batched), I’ll be pushing a hot fix hopefully today.
Edit: Formicaio v0.4.6 has been just published with a couple of fixes related nodes status tracking when monitoring large number of nodes.
2 Likes
When I try to run Formicaio, this happens:
$ formicaio start
formicaio: command not found
Also, with the frontend this happens:
I’m on Ubuntu 24.04.2 LTS. I did update my Ubuntu before these problems arose. I have used Formicaio successfully natively on this machine before,
Should it be “./formicaio start” unless the command is on the path?
And you should “cd” to the folder where the binary is.
3 Likes
Oh, it was on that folder.
1 Like
I’m wondering if
cp formacaio ~/.local/bin is a GoodThing to put in the ReadMe
Formicaio v0.4.7
Formicaio version 0.4.7 has been released and it’s ready for installation across all supported platforms in native mode.
This release is only to introduce some improvements to node status reporting, specially for nodes which exit or fail to start, by showing as part of the node status info the exit code obtained from the exited process (when available).
I was observing the following which you may find useful:
-
Nodes exiting with
exit status: 1were all due to UPnP issues, checking the log I was able to find the following error:
[2025-03-13T06:04:35.087102Z ERROR antnode 459] Node stopped with error: Node terminated due to: "UPnP gateway not found. Enable UPnP on your router to allow incoming connections or manually port forward." -
Nodes exiting with
exit status: 101were those which tried to open/use an already used port. In this case there is no error message in the logs since such error is not being sent to theantnode.logfile. -
Nodes which were running ok with UPnP for some time, and for some reason the UPnP address/link then expired, were exiting and I could see the following last log messages:
[2025-03-09T03:35:47.937545Z INFO ant_networking::event::swarm 84] UPnP event upnp_event=NewExternalAddr(/ip4/<ip-addr>/udp/<port>/quic-v1)
[2025-03-09T03:35:47.937555Z INFO ant_networking::event::swarm 93] UPnP: New external address: /ip4/<ip-addr>/udp/<port>/quic-v1
[2025-03-09T03:35:48.066007Z INFO ant_networking::event::swarm 518] external address: expired address=/ip4/<ip-addr>/udp/<port>/quic-v1
[2025-03-09T03:35:48.066083Z INFO ant_networking::event::swarm 84] UPnP event upnp_event=ExpiredExternalAddr(/ip4/<ip-addr>/udp/<port>/quic-v1)
[2025-03-09T03:35:48.364907Z ERROR ant_node::node 355] The `NetworkEvent` channel is closed
[2025-03-09T03:35:48.365022Z INFO antnode 450] Node is stopping in 1s...
[2025-03-09T03:35:49.366025Z ERROR antnode 459] Node stopped with error: Node events channel closed!
If this is your first time installing Formicaio, you can easily follow the installation instructions in the README.
Make sure that, after upgrading, you refresh the page of any instance of the Formicaio frontend/GUI you have open on a browser with Ctrl + F5.
2 Likes
I didn’t found support for “–data-dir-path” is it there?
Anyway, very handy software, thanks!
1 Like
If you mean to change where Formicaio stores data, you could use the NODE_MGR_ROOT_DIR env. var. if running in native-mode.
Or simply move the formicaio binary to wherever you want to store all the data and it will create a formicaio_data subfolder in it by default.
Edit: you may wanna also set the DB_PATH env. var. to change where to store the Formicaio database file/s, which could be different from where the nodes data/files are set to be stored with NODE_MGR_ROOT_DIR.
3 Likes
When using the bin version of Formicaio and when updating the version of Formicaio (following the guide on how to upgrade), I have 2 problems:
- When I run ./formicaio start, a batch to start the already running nodes is launched (eventually killing the nodes), it’s not a big problem as I can cancel the batch but it always kill 2/3 nodes in the meantime.
- the ./formicaio start process takes 100% of a core all the time (I greatly decreased the fetch rate of metrics at about one fetch every 3 minutes to no avail, the machines that are running a fresh Formicaio with equivalent number of nodes don’t have this problem)
1 Like
Sometimes I get this error repeating for some node:
Failed to fetch metrics from node 5a7373356141: error sending request for url (http://127.0.0.1:44042/metrics)
Restarting or recycling sometimes helps but usually not.
Edit: from the logs it seems like it is running normally, just don’t get the stats.
Feature suggestion: count how many nodes has been upgraded to the latest version.
2 Likes
not sure how to make sense of the info but its in the metrics from the node
# HELP ant_networking_node_versions Number of nodes running each version.
# TYPE ant_networking_node_versions gauge
ant_networking_node_versions{version="2025.1.2.6"} 30
ant_networking_node_versions{version="2025.1.2.11"} 3
ant_networking_node_versions{version="old"} 8
1 Like
btw, how is the estimated network size calculated? Some average of the numbers got from nodes? I think it should be calculated differently; it should be at least the amount of largest node estimate.
Let’s assume network size is 40M. One node would see half of the network, 20M. Another node sees also half of the network, 20M. What is the formula which would give us 40M with unlimited number of nodes giving estimate of 20M?
Edit: Actually, it depends how a node has estimated the network size.
We should be able to get some estimation by using only 2 nodes list of visible nodes. It is called Lincoln-Peterson estimator or Capture-Recapture.
We need a list of id’s the nodes sees:
n1 = number of nodes node1 sees
n2 = number of nodes node2 sees
m = number of nodes both sees
Estimated network size is N = (n1*n2)/m
But I guess the autonomi node code has something like this already built in.
1 Like
Yes, the idea behind this is that if you have Formicaio set up as a service or simply running it all the time and the host/PC restarts it gets all nodes which were previously running to be started up again automatically. I guess in some case it’s not desirable, I’ll be working on adding some option to the start subcommand, e.g. formicaio start --no-auto-start.
Can you please explain what you mean by “(eventually killing the nodes)” / “it always kill 2/3 nodes” ? Im investigating an issue related to batches which seem to fail to act on 100% of targeted nodes, so perhaps it’s the same issue, can you please explain a bit more what’s is it you see about nodes being killed?
Is this still with ~4k - ~6k nodes? those different machines are using the same and latest Formicaio v0.4.7?
Can you please give it a try to have a GUI opened against the instance with high CPU usage and confirm if that makes any difference? (it’s just I’m suspecting where there could be a bug).
Thanks a lot for reporting all this.
I’ve seen this myself too, which led me to add a feature where these nodes are reported with status Unknown. I was not able yet to confirm if it’s the antnode itself which is not responding to the service after some circumstances, or a bug in Formicaio. I’ll be debugging it more when I get to reproduce it again.
I have a branch with this in place, but I had the same issue with not knowing exactly what the reported numbers meant (i.e. is it reporting % of nodes or num. of nodes for each version…). I’ll have to dig a bit more to understand and then display it, I just didn’t want to display something which could be misleading if presented in the wrong way.
Yes, it’s just the average of all reported estimation of active nodes. It used to do a weighted average using the number of connected peers as the weight, but I then realised that may not be a good way so decided to simply show an plain average. The dilemma I faced was: do I trust every value when there are enormous discrepancies among the values reported by nodes…?. As you say, perhaps using the largest number is the best we can do with the reported values.
3 Likes
From the logs, it seems it tries to start an already running node and since it’s already running it fails and the node is killed:
Spawning new node 777552507863 with cmd: <....>
Node process for 777552507863 spawned with PID: 3052121
Process of node 777552507863 was killed.
Killed process for node 777552507863: exit status: 101
Process with PID 3052121 exited (node id: 777552507863) with status: exit status: 101
No, it’s on machines with ~2K nodes, and yes all my machines are using latest version of Formicaio (fresh or upgraded). If I open the GUI on an upgraded one, it takes more time to load than on the fresh install with same number of nodes (but it still loads)
Would be good to have the choice to not start anything (in the case of an upgrade of Formicaio), and to restart all nodes but maybe with some delay (in case of a reboot). I don’t think starting 2000K nodes with no delay is a good idea
2 Likes
I think you have nodes trying to open ports already in use, double check those which are failing and see if the assigned ports are not being used by another node OR any other app/service/program in the host/PC.
Ok, thanks, I’ll be trying to reproduce and fix it. One last thing, make sure you refresh the GUI with Ctrl+F5 anytime you upgrade, that has shown to be an issue many times as the browser won’t fetch the latest GUI from the backend.
True, I’ll see how it can be improved. Thanks!.
1 Like