I’m looking into my node health from home nodes and it’s driving me absolutely insane, hopefully some of you can help me out.
I’ve been monitoring incoming udp traffic in my OPNsense logs. I’m seeing a ton of (blocked) traffic on udp ports below 30.000 (from server IP’s I actually run myself with cloud solutions). This all, while my home nodes are being started on ports exclusively above port 30.000. It appears as if my node is not starting on the port provided.
When starting a node and monitoring outgoing traffic, I also see several request going out from ports other than the port I’m starting the node on.
I have set up firewall nat outbound mode to Hybrid outbound NAT rule generation and added a rule that sets the source (server ip), source port (port range for the home nodes) for tcp and udp traffic and setup static port.
I also have a port forward rule in place, forwarding the ports for the node (both tcp and udp, even though just udp should work I believe) to the node server ip.
Does anyone know what could cause this issue? This is probably the underlying issue to my poor performance from home compared to my cloud solution.
Your nodes talk to other nodes on their port since that is the port they respond to.
I do not change the outbound rules from default. The firewall and router are setup as default to correctly handle these and do not need changing for nodes. You will be for a world of hurt if you get these wrong
This was actually something Shu mentioned I should set up as otherwise the ports could change while running.
Doesn’t the router block any of that outgoing traffic then as those ports are not setup?
Also, why would a cloud server have outgoing traffic to my hone server ip on a port below 30.000 if i do not start nodes on that port? That traffic is just getting blocked right now.
What might be important to add is that when launching with launchpad, i continue to get the error after 5 retries message. The fix as presented on the autonomi website does not work. So there is probably something fishy going on there to.
You absolutely need the outbound rule with OPNsense (as described here: Configuring OPNSense for Autonomi nodes - #6 by d3su but I know you already know that).
You can try removing it, but your nodes won’t be able to communicate anymore (and earnings should be 0).
@Shu I hope you can shine your light on this as you also replied in the topic of d3su.
In that topic you mention you would restrict the outbound NAT traffic rule to the antnode UDP port range (or at least just the UDP protocol). What I’ve noticed is that outgoing traffic is not sticking to the port range assigned to the antnode itself. @neo has mentioned that this is correct behavior since you’re talking to other nodes on their assigned ports. So I’ve now setup the outbound rule to not just my antnode assigned ports, but to all ports, I think this is the correct way to set it up, but hoping to hear your thoughts on this.
Since I set it up this way, I’m seeing a lot more traffic passing, which is a good start I guess. But I’m still seeing massive (and I really mean massive) amount of blocked incoming UDP traffic on ports below 30.000 (the ports that are not assigned to my node and therefor are not port forwarded either).
If I understand correctly, this could potentially be ok and not necessarily a mistake from my side. But I would also like to verify this with you and the community. So If I understand correctly, my outgoing ports can be pretty much anything, depending on the ports other nodes around the globe have set their nodes up. My incoming traffic however should A, always be on my Antnode assigned ports, or B, return traffic from one of my outgoing requests? If the last, I believe my firewall should recognize it as return traffic and allow it. I’m seeing some incoming traffic on ports outside my Antnode assigned range which are being allowed so that could be a clear sign I’m correct in my assumptions.
Could it potentially be that some people have their nodes not setup in a correct way which is making them use the wrong port and getting blocked by my firewall? If so, others should probably see similar behavior in their routers firewall logs.
Please research symmetric nat vs full cone nat vs port restricted cone nat etc.
You need to setup static rules or equivalent for outbound rules on those ports, else the ports will keep changing at least in pfSense. Its not the fault off antnode, but your router’s software.
I suspect OPNSense is similar to pfSense in how its operating (I am not familiar with the GUI).
The ports and connections form a 5 tuple NAT session entry on your router. Its only on the one side, will the IP/Port of your antnodes match, the other side is the dst ip/port or src ip/port of other nodes’ endpoints (depending on direction off traffic). You don’t need to create a rule for ports that are not assigned to your antnode.
I doubt it, if they are misconfigured it will not allow you to send outbound to them as their ports’ will keep changing as and when they initiate outbound connections to your peer. However, they should be able to always reach your nodes due to port forwarding setup by you (if done correctly) (NAT port forward rules).
Note: My quick 1 minute response for now. Its a bit late here without catching up on long running thread. I had mentioned something along the lines above before on prior posts in this forum.
I’ve been testing with this for over a week now, and even though I cannot say for 100% sure I’m not doing something wrong, I’m pretty sure that some things just aren’t working as intended with port forwarding.
My outgoing outbound traffic rules are set up to be static. There is no UDP traffic leaving my router from ports other than the ones I assign to my nodes. A big bunch of the incoming UDP traffic is through the right ports, but the more nodes I spin up, the more traffic outside of the assigned port range starts pouring in. Some statistics:
Currently I’m running 40.000 nodes from home
They earn 8 ANT every 24 hours (5% of the earnings I get with the same amount of nodes from a cloud solution)
around 50 denied inbound UDP traffic OUTSIDE assigned port range every second
I’ve tested with smaller node amounts, inside containers and outside containers. It continues to stay the same. Before TGE, this exact same setup was underperforming my Hetzner nodes by ±20%, which wasn’t great, but nowhere near the 95% I’m at right now. I’m not necessarily asking someone to help me out, but I do feel there is quite an important issue here that should be addressed as home nodes is still the absolute fundament of the network.
That’s good. So your nodes can receive traffic from other nodes on the ports you’ve specified.
We need a proper networking person to comment on this but this part stands out to me:-
Well there should be I believe. Other people’s nodes will only accept traffic on the ports they are listening on so you should have traffic going out from yours through your router to theirs on ports you haven’t set your nodes up to listen on.
Is this the problem:-
Are you saying that you’ve limited outbound traffic from your nodes to the internet to be just the ones you’ve set your nodes to listen on? Your nodes definitely have to be able to send traffic to other people’s nodes on any UDP port because they could be using any port.
So they are responding to quotes from MaidSafe because they come in on the ports you have set to listen on. So they earn (the thing that must not be named!). Presumably the reply goes back on the same port.
EDIT:
Do your nodes have any records stored on them? I’m sure they won’t have been paid for storing any new records as there’s very little of that going on but do they have any records at all?
What numbers do you get if you run this:- ls /home/safe/.local/share/autonomi/node | grep antnode | sort | while read f; do echo ${f} ; ls /home/safe/.local/share/autonomi/node/${f}/record_store | wc -l ; echo ; done
What I’m driving at is that your nodes might not be working properly in terms of storing records. Although with the number of nodes in the network and comparatively small number of records it might not prove anything having 0 records. I have nodes that have been up for the best part of a month with low single digit records and plenty with none.
Which one of those is the source and which is the destination?
Anyway, This is an entry from the NAT table on my router:- 6 SACFs protocol=udp src-address=10.0.0.21:12037 dst-address=65.109.66.240:53300 reply-src-address=65.109.66.240:53300
So it could be I’m barking up the wrong tree here because it clearly says there is a connection with the source of src-address=10.0.0.21:12037 (My computer) and the destination of dst-address=65.109.66.240:53300 (someone elses). So traffic can exit from the router to the internet on one port and arrive at someone else’s computer on another port. You learn a new thing every day!
It’s supposed to be static according to most people here on the forum (and what was successful pre TGE). I was new with this too, but you can have an outgoing static port, communicating to a different port.
No, they have access to the internet and nothing is blocked outgoing. There is a static outbound rule for the ports assigned to the server (as mentioned here above)
Yes, and I even have chunk payments outside of the 12 hour period payment window. I recently reset my servers as they’re only consuming energy when not earning anything of significance.