this and all other from my top-10 IPs are behind the hetzner’s firewalls.
Funny is that from my 59 nodes most nodes have 0 “is considered as faulty.” errors but some nodes have logs with bursts of these faults. Like thousands in a single log.
Oh, this seems to be been fixed by itself, at least for that antnode13. Some other node was still suffering this issue, but looks like they are better in general.
[2025-04-02T08:59:46.785477Z ERROR ant_networking::event::swarm 478] IncomingConnectionError Valid from local_addr:?/ip4/0.0.0.0/udp/32092/quic-v1, send_back_addr /ip4/144.76.4.81/udp/18111/quic-v1 on ConnectionId(12779) with error Transport(Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }))
[2025-04-02T08:59:46.785592Z INFO ant_networking::external_address 372] Removed external addresses due to connection errors on port 32092: [/ip4/109.204.xxx.xx/udp/32092/quic-v1/p2p/12D3KooWPbdwoMPCu4rXrWdtY7PF5LDpfvcBaaRRQsAasmxMRyn1]
[2025-04-02T08:59:46.785625Z INFO ant_networking::external_address 489] All External addresses: [], and listen addresses: [/ip4/127.0.0.1/udp/32092/quic-v1, /ip4/192.168.50.181/udp/32092/quic-v1]
[2025-04-02T08:59:49.801265Z INFO ant_networking::driver 916] Set responsible range to Distance(299465604199726157556491101820833646750420999470443891207051898524172840)(Some(237))
[2025-04-02T09:00:04.802323Z INFO ant_networking::driver 916] Set responsible range to Distance(299465604199726157556491101820833646750420999470443891207051898524172840)(Some(237))
[2025-04-02T09:00:19.802481Z INFO ant_networking::driver 916] Set responsible range to Distance(299465604199726157556491101820833646750420999470443891207051898524172840)(Some(237))
[2025-04-02T09:00:23.578735Z INFO ant_networking::external_address 144] External address confirmed, adding it to swarm: /ip4/109.204.xxx.xx/udp/32092/quic-v1/p2p/12D3KooWPbdwoMPCu4rXrWdtY7PF5LDpfvcBaaRRQsAasmxMRyn1
[2025-04-02T09:00:23.578777Z INFO ant_networking::external_address 489] All External addresses: [/ip4/109.204.xxx.xx/udp/32092/quic-v1/p2p/12D3KooWPbdwoMPCu4rXrWdtY7PF5LDpfvcBaaRRQsAasmxMRyn1], and listen addresses: [/ip4/127.0.0.1/udp/32092/quic-v1, /ip4/192.168.50.181/udp/32092/quic-v1]
Then all it is saying is that it cannot contact that node.
This is normal since most routers cannot do hairpin routing by default. Special rules need to be setup to translate the external address into a locally routed packet as if its external.
I was able to ping that node so it is able to give quotes and expect it to be working fine. Just your other nodes cannot contact it since they are behind the same router
Yes, but why most of the other nodes does work? And why error on incoming connection? I think it means the other node can send to my node, but it cannot receive my node’s answer. It is either blocked on my side or on the other side. And my node can talk to other nodes.
There is more than likely that one of your nodes in the bootstrap node file since it is populated by and used all nodes with the first (maybe) node will be added to the file so the other nodes may try to contact it.
Yes a bit of “if” situation.
It boils down to, if excluding one of your own nodes then I would ignore such errors. Why one or more nodes have attempts to connect to them I do not know but can be random situation that causes it, including being an entry in the bootstrap file.
Basically in the end it is safe to ignore the error and is working properly removing the node. As far as I can tell
Without searching the logs, I can tel you what nodes suffers this. They are nodes nr 1,3,5,6,7,9,10,11,13,14,17,18,19.
And nodes 2,4,8,12,15,16,20 are feeling well, they do not keep “removing external address” and then after a minute or so “External address confirmed”.
Hmm… just a wild guess, maybe the other nodes are trying to connect some earlier node I did have on the same address/port? And since the node is no longer the same, they just cut the connection out → my current node thinks it’s external address is wrong?
Since this has been going on so long, and no one seems to be interested, I’ll reset those nodes and create new ones, this time to completely different port scale.
I do have a bunch of other nodes, started with formicaio, I’ll let them stay as they are.
Edit: Actually that might explain a lot; I did have old formicaio nodes on the same ports when I started the new 20 antctl nodes.
Edit2: Maybe the fact that some node answers in the ipaddr:port keeps also the old node ‘alive’ on the network?
Edit3: maybe the extra dead nodes causes a lot of another problems, ie upload problems, download problems, unfair emission distribution etc…
[2025-04-06T19:19:34.152941Z INFO ant_networking::external_address 372] Removed external addresses due to connection errors on port 2000: [/ip4/109.204.184.55/udp/2000/quic-v1/p2p/12D3KooWFRsq9ps9ZrRA1w5f2tbBrQN1rkAhUtf1MKC2H2bhW9B7]
[2025-04-06T19:19:34.153002Z INFO ant_networking::external_address 489] All External addresses: [], and listen addresses: [/ip4/127.0.0.1/udp/2000/quic-v1, /ip4/192.168.50.181/udp/2000/quic-v1]
[2025-04-06T19:19:34.159903Z ERROR ant_networking::event::swarm 478] IncomingConnectionError Valid from local_addr:?/ip4/0.0.0.0/udp/2000/quic-v1, send_back_addr /ip4/88.99.99.41/udp/41872/quic-v1 on ConnectionId(2584) with error Transport(Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: Connection(ConnectionError(ConnectionClosed(ConnectionClose { error_code: APPLICATION_ERROR, frame_type: None, reason: b"" }))) } }) }))
[2025-04-06T19:19:39.027750Z DEBUG ant_node::node 435] Handling NetworkEvent "NetworkEvent::QueryRequestReceived(GetVersion(NetworkAddress::PeerId(12D3KooWFRsq9ps9ZrRA1w5f2tbBrQN1rkAhUtf1MKC2H2bhW9B7 - bf035c7de7f2f8da130dd9e671c628a386d2e9e2303141c612cfd606c3d565d0)))"
[2025-04-06T19:19:39.027832Z DEBUG ant_node::node 506] Sending response Query(GetVersion peer NetworkAddress::PeerId(12D3KooWFRsq9ps9ZrRA1w5f2tbBrQN1rkAhUtf1MKC2H2bhW9B7 - bf035c7de7f2f8da130dd9e671c628a386d2e9e2303141c612cfd606c3d565d0) has version of "2025.1.2.13")
[2025-04-06T19:19:42.014147Z ERROR ant_networking::event::swarm 478] IncomingConnectionError Valid from local_addr:?/ip4/0.0.0.0/udp/2000/quic-v1, send_back_addr /ip4/94.130.160.30/udp/34304/quic-v1 on ConnectionId(2585) with error Transport(Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }))
[2025-04-06T19:19:44.154445Z WARN ant_networking::event::identify 35] identify: Error { connection_id: ConnectionId(2417), peer_id: PeerId("12D3KooWQj2JbgS8osjGezRiX9fwYNBbNzV6RRLCTvkoowb6tUyn"), error: Timeout }
[2025-04-06T19:19:44.154562Z ERROR ant_networking::event::swarm 478] IncomingConnectionError Valid from local_addr:?/ip4/0.0.0.0/udp/2000/quic-v1, send_back_addr /ip4/94.130.160.30/udp/34304/quic-v1 on ConnectionId(2587) with error Transport(Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: Connection(ConnectionError(ConnectionClosed(ConnectionClose { error_code: APPLICATION_ERROR, frame_type: None, reason: b"" }))) } }) }))
[2025-04-06T19:19:44.687907Z INFO ant_networking::driver 916] Set responsible range to Distance(2279802664230173328494577420313443246229011479839508333060137033925961000)(Some(240))
[2025-04-06T19:19:52.614988Z INFO ant_networking::external_address 144] External address confirmed, adding it to swarm: /ip4/109.204.184.55/udp/2000/quic-v1/p2p/12D3KooWFRsq9ps9ZrRA1w5f2tbBrQN1rkAhUtf1MKC2H2bhW9B7
[2025-04-06T19:19:52.615063Z INFO ant_networking::external_address 489] All External addresses: [/ip4/109.204.184.55/udp/2000/quic-v1/p2p/12D3KooWFRsq9ps9ZrRA1w5f2tbBrQN1rkAhUtf1MKC2H2bhW9B7], and listen addresses: [/ip4/127.0.0.1/udp/2000/quic-v1, /ip4/192.168.50.181/udp/2000/quic-v1]
There is also my current external address, I will reset my router to get a new one.
Edit: Now I have only these 20 antctl nodes, I’ll let them run for maybe a day before resetting ip.
