LOG:Removing all addresses with this port

Support
1

Found on the logs:

[2025-03-25T19:43:13.555983Z ERROR ant_networking::event::swarm 478] IncomingConnectionError Valid from local_addr:?/ip4/0.0.0.0/udp/32031/quic-v1, send_back_addr /ip4/167.235.X.X/udp/19436/quic-v1 on ConnectionId(16203) with error Transport(Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }))
[2025-03-25T19:43:13.556040Z INFO ant_networking::external_address 343] Connection on port 32031 is considered as faulty. Removing all addresses with this port
[2025-03-25T19:43:13.556136Z INFO ant_networking::external_address 373] Removed external addresses due to connection errors on port 32031: [/ip4/MY_PUBLIC_IP/udp/32031/quic-v1/p2p/12D3KooWAjcAa...]

Router logs tell me that this node does have some successful connections, but also some which are not so successful. I have Asus RT-AX58U, currently total 60 nodes. What could be causing this?

1 Like
2

Later:

[2025-03-25T19:47:56.651547Z INFO ant_networking::external_address 144] External address confirmed, adding it to swarm: /ip4/MY_PUBLIC_IP/udp/32031/quic-v1/p2p/12D3KooWAjcAa44..
[2025-03-25T19:47:56.651614Z INFO ant_networking::external_address 490] All External addresses: [/ip4/MY_PUBLIC_IP/udp/32031/quic-v1/p2p/12D3KooWAjcAa44...], and listen addresses: [/ip4/127.0.0.1/udp/32031/quic-v1, /ip4/192.168.50.181/udp/32031/quic-v1]
[2025-03-25T19:47:56.668243Z DEBUG ant_node::node 434] Handling NetworkEvent "NetworkEvent::QueryRequestReceived(GetVersion(NetworkAddress::PeerId(12D3KooWAjcAa44X58ms9vkoGuPhZG8mA5xhbExDoqNn68WRyZ83 - d3bacf235e9ce0fb1e00d4ba8431ad7ad99fb243737d1ae3d1e2ce482645d486)))"
3

And soon after that:

[2025-03-25T19:49:13.666587Z ERROR ant_networking::event::swarm 478] IncomingConnectionError Valid from local_addr:?/ip4/0.0.0.0/udp/32031/quic-v1, send_back_addr /ip4/138.199.X.X/udp/16540/quic-v1 on ConnectionId(16222) with error Transport(Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: Connection(ConnectionError(ConnectionClosed(ConnectionClose { error_code: APPLICATION_ERROR, frame_type: None, reason: b"" }))) } }) }))
[2025-03-25T19:49:13.666659Z INFO ant_networking::external_address 343] Connection on port 32031 is considered as faulty. Removing all addresses with this port
1 Like
4

This repeats in a few minute intervals, ie external address is removed and then again found and confirmed?

1 Like
5

Looks like some nodes do this all the time, some does notā€¦

1 Like
6

Does anyone have similar issues? Just cd to your nodeā€™s log folder and

grep "is considered as faulty." antnode.log
7

Some interesting findings:

grep "IncomingConnectionError" antnode.log

grap some ipā€™s there
traceroute xx.yy.zz.aa
ā†’ a few I searched ended up to this address:

.
.
 6  decix-gw.hetzner.com (80.81.192.164)  25.007 ms  24.345 ms  24.239 ms
 7  core22.fsn1.hetzner.com (213.239.224.9)  29.141 ms core24.fsn1.hetzner.com (213.239.224.81)  29.033 ms core23.fsn1.hetzner.com (213.239.224.65)  29.122 ms
 8  blocked.hetzner.com (88.198.255.174)  31.478 ms  34.116 ms  31.445 ms
8

My assumption is that there are some servers which have nodes which are blocked.. and those nodes try to connect to my nodes (maybe 30%), which end up in error and when this repeats long enough, my nodes think itā€™s public ip is nonfunctional.
So it is possible to make some kind of DoS attack by installing nodes and block incoming connection to them or something like that?

9

Most (all?) of the ā€œIncomingConnectionErrorā€'s are coming from Hetzner Online GmbH / Falkenstein.
What does mean

{ error_code: APPLICATION_ERROR, frame_type: None, reason: b"" }

?

10

just checked the logs for successful quotes for that hetzners ip address 88.198.255.174 and i dont see it in the list of ipā€™s. not sure what it means but im not getting any quotes from it.

11

No, that address is not what I found on the logs. 88.192ā€¦ is what blocks the route to the destination ip.

12

This ip cretes a lot of both incoming and outgoing errors lines to the log: 188.40.64.177

13

This has been going on all night:

~/.local/share/autonomi/node/antnode13/logs$ grep "is considered as faulty." antnode.log
[2025-03-25T22:14:19.207952Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-25T22:14:19.208159Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-25T22:17:01.363559Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-25T22:17:21.185318Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-25T22:17:31.542279Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port

.
.

[2025-03-26T05:18:52.170795Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-26T05:23:02.994286Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-26T05:23:29.964031Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-26T05:23:52.002697Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-26T05:25:39.432604Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-26T05:25:50.808308Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-26T05:27:23.346945Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
14 
~/.local/share/autonomi/node/antnode13/logs$ grep "188.40.64.177" antnode.log | wc -l
158
xxx@yyy:~/.local/share/autonomi/node/antnode13/logs$ 

~/.local/share/autonomi/node/antnode13/logs$ grep "is considered as faulty." antnode.log | wc -l
404

~/.local/share/autonomi/node/antnode13/logs$ grep "IncomingConnectionError" antnode.log | wc -l
414
1 Like
15

All in all, I had 80 formicaio nodes, reduced their amount to 40 and started 20 antctl controlled nodes (did this to see if emissions are down because of too many nodes). Almost all of the formicaio nodes are working ok. About 30% of the antctl nodes suffers from this. I assume itā€™s because they are ā€˜youngā€™ nodes and formicaio nodes are older. Or maybe they are just unlucky to be close to nodes on that failling ip. But time will show.

(btw, with 80 nodes I got ~4 payments / day, ie ~0.2 ANTS)

1 Like
16

Ping @rusty.spork I think someone who knows should examine these logs. I can PM all logs if needed.

2 Likes
17

After a couple of days, this still continuesā€¦

[2025-03-28T07:14:41.525240Z ERROR ant_networking::event::swarm 478] IncomingConnectionError Valid from local_addr:?/ip4/0.0.0.0/udp/32092/quic-v1, send_back_addr /ip4/144.76.4.81/udp/52269/quic-v1 on ConnectionId(51598) with error Transport(Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: Connection(ConnectionError(ConnectionClosed(ConnectionClose { error_code: NO_ERROR, frame_type: None, reason: b"" }))) } }) }))
[2025-03-28T07:14:41.525318Z INFO ant_networking::external_address 343] Connection on port 32092 is considered as faulty. Removing all addresses with this port
[2025-03-28T07:14:51.903576Z INFO ant_networking::driver 903] Set responsible range to Distance(160425741525564883819777036738613464084945058691560768388191645304539966)(Some(236))
[2025-03-28T07:15:01.901824Z INFO ant_node::log_markers 69] IntervalReplicationTriggered
[2025-03-28T07:15:01.926364Z INFO ant_networking::external_address 144] External address confirmed, adding it to swarm: /ip4/MY_PUB_IP/udp/32092/quic-v1/p2p/12D3KooWPbdwoMPCu4rXrWdtY7PF5LDpfvcBaaRRQsAasmxMRyn1
[2025-03-28T07:15:01.926457Z INFO ant_networking::external_address 490] All External addresses: [/ip4/MY_PUB_IP/udp/32092/quic-v1/p2p/12D3KooWPbdwoMPCu4rXrWdtY7PF5LDpfvcBaaRRQsAasmxMRyn1], and listen addresses: [/ip4/127.0.0.1/udp/32092/quic-v1, /ip4/192.168.50.181/udp/32092/quic-v1]
18

ping @vphongph did I brake it :sweat_smile: :laughing:

19

Iā€™ve just checked in the logs for a node Iā€™ve had running for a few weeks and I see messages like this as well:-

antnode.log.20250304T145900:[2025-03-03T18:46:18.955012Z INFO ant_networking::external_address 343] Connection on port 12001 is considered a
s faulty. Removing all addresses with this port

a couple a day on average.

And only for port 12001.

12001 is the port Iā€™ve setup this particular node with!

From the other discussion weā€™re in Nodes from home issues I know that 32092 is the port of your node. So Iā€™m guessing this message is innocuous.

I donā€™t understand why the node would be trying to contact itself.

It occurs to me though that it might be damaging for the node to remove all addresses that use that port though. Does it really mean ā€˜allā€™? What if my node is trying to contact other folkā€™s nodes on port 12001 because that is what they are using? Are those addresses going to be removed as well?

20

I think it is some kind of healthy check, it gets an incoming connection, but fails, so it ends up thinking itā€™s local address/port does not work. Later it gets another incoming connection that does work and then reverts the port back.
For this unlucky node number 13 it happens all the time, usually a few minute interval. For some nodes it happens never.