some other thing I see now. in the dashboard it says i have 4 tokens, but on arbitrum sepolia scanner it says i have 2? anyone else seeing mismatches in dashboard and ledger?
It’s a known issue, I’ve been seeing it myself too, the node’s metrics reports inexact balance values.
Edit: hopefully next version of Formicaio will fetch the balance directly from the ledger.
@bochaco’s earlier instructions got it sorted out, but I tried running your recommended command anyway.
My Umbrel machine is at home and I’m remote at the moment. Logged in to CLI from remote (via the Settings path to CLI) and entered the command above.
First try got me “permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get “http://%2Fvar%2Frun%2Fdocker.sock/v1.44/containers/local-formicaio-1/json”: dial unix /var/run/docker.sock: connect: permission denied”
So I tried it with sudo and got: “Error response from daemon: No such container: local-formicaio-1”
No idea where to go from there, but I’m not worried. It’s mostly above me anyway. I’m just being patient and running a couple nodes as the network and the Formicaio app get more foolproof.
Anyway, the couple nodes I’m running seem to be running nominally, so the above is provided as feedback.
Please be aware that the Formicaio backend application, as well as the safenode binary running within each user-created node instance, utilizes third-party RPC services to retrieve information related to the Arbitrum L2 ledger.
Specifically, the Formicaio backend application queries the RPC server at https://sepolia-rollup.arbitrum.io/rpc to periodically check the current rewards balances for each node instance based on the configured (ETH) rewards addresses.
Risks:
Privacy: Using third-party RPC services may expose your IP address and other metadata to those services. This means that the service provider can potentially track which addresses you are querying, which could lead to privacy concerns regarding your activity on the Arbitrum network.
Data Exposure: Any data sent to the RPC service may be logged or monitored by the service provider, which could include sensitive information related to your node instances.
It’s recommended to consider these risks when using the application and taking appropriate measures to protect your privacy.
I’m not sure if that will support docker-in-docker that Formicaio currently uses, or alternative any docker engine it can access to, it needs it to launch the nodes instances. Launching the Formicaio backend and UI for sure would be possible to run on that platform as you just need the image repo which is bochaco/formicaio:latest.
I took a deeper look, a bit more educated response is that even Formicaio can be run as is on the flux net using that web portal, some integration work would be required so its backend can connect to the flux net programatically to deploy/launch the nodes the user instantiates, i.e. use the flux net not only for running the Formicaio app itself but also as a replacement to the docker engine it currently uses for the nodes it launches.
I couldn’t find an SDK which would be needed for such an integration, so cannot tell how difficult it would be to implement, but for sure it would also need some integration with a wallet so you can pay the flux net for each node you instantiate from Formicaio to deploy on fluxnet.
I’m running 18 nodes via Formicaio on Umbrel, on my 2012 Macbook hardware. It’s finally operating with great stability, producing steady awards, easily monitorable via Tailscale from anywhere. It’s dead easy and I’m delighted.
Past problems have cropped up when the network updated and thus the need to update Formicaio, and thus upgrade the nodes. I see that you’ve included an automatic node upgrade option, which will be very helpful, and hopefully allow for a smooth change, without having to start from scratch every time.
My question is: Will there be an option for Umbrel to upgrade Formicaio automatically, which will then upgrade the nodes, or will it always have to be done manually?
Not sure, I haven’t investigated that yet, I’ll have to research if Umbrel could/does support it, or if I could implement some hack to do it.
There is another thing which I’m more interested in having, which I don’t think it’s currently supported by Umbrel: upgrade Formicaio without stopping nodes. I’ve recently updated the README with instructions for how to do this, which is possible but only when running with docker-compose on Linux.
With nodes-auto-upgrade this will already happen, the auto-upgrade performs a first check when Formicaio starts up, regardless if Formicaio was upgraded manually or automatically.
In any case, upgrade nodes can also be performed manually without needing to restart or upgrade Formicaio, each node will show this button when new version is available:
This is all great. I know the system will improve–it’s what you do!!
Again, I’m really appreciative that you are making this easier and better. I haven’t decided if I’ll be making further use of Umbrel beyond Formicaio, but as both develop further, I’m pretty sure I want to make a formidable hardware instance for it all as a home server. Great work!!
Also when I’m pushing the number of nodes to the max. A restart of one node can start a restart cascade which maxes out the CPU and causes a restart loop. Maybe time between node restarts could be set or automatically calculated based on the number of nodes running. Just some feedback. Thank you for the amazing work. I love the app.
Was this after manual upgrade? auto-upgrade? I’ve seen a case like this which was due to the node not being able to fetch the bootstrap contacts file and bailing out. Since Formicaio sets them up to automatically restart if the node exits/bails, it must be causing that loop, or perhaps some other issue, not sure tbh. Next version will allow you to remove a node if this happens, so you can at least get rid of it and create a fresh instance.
This must be due to that logic in the nodes themselves to commit suicide when they see they are using >50% CPU for a few minutes, plus what I mentioned above that Formicaio sets them up to always restart the instance if it exits/bails for any reason, and also nodes using more CPU when they start up, all seems to end up in this cascading and loop of restarts. I’ll have to think about a way to prevent it as you suggest, I guess even if that suicidal logic is removed from the nodes it would be desirable to have some smarter way of automatically restarting them as you say…, perhaps analogous to how it’s done during the auto-upgrade process…I’ll have to figure out a solid approach…
Auto restart with delay would be amazing. Launcher with interval would be great. Select the number of nodes you want to start. Enter your address and select launch interval. Not to be greedy with feature requests
