Was this after manual upgrade? auto-upgrade? I’ve seen a case like this which was due to the node not being able to fetch the bootstrap contacts file and bailing out. Since Formicaio sets them up to automatically restart if the node exits/bails, it must be causing that loop, or perhaps some other issue, not sure tbh. Next version will allow you to remove a node if this happens, so you can at least get rid of it and create a fresh instance.
This must be due to that logic in the nodes themselves to commit suicide when they see they are using >50% CPU for a few minutes, plus what I mentioned above that Formicaio sets them up to always restart the instance if it exits/bails for any reason, and also nodes using more CPU when they start up, all seems to end up in this cascading and loop of restarts. I’ll have to think about a way to prevent it as you suggest, I guess even if that suicidal logic is removed from the nodes it would be desirable to have some smarter way of automatically restarting them as you say…, perhaps analogous to how it’s done during the auto-upgrade process…I’ll have to figure out a solid approach…