Started looking at my router for the first time, embarrassing really that this crucial part to running nodes has been largely overlooked on my part at least.
This is my old (business) router that will be replaced today by something significantly more robust but as most folk are running standard consumer grade routers it is worth noting.
I have 53 nodes “running” but are they really if my sessions are maxed out?
Going to drop down to 35 now and see if things clear up.
Please will other folk share what is happening with router resources and how many nodes you have or can run without bogging it up.
I had my port forwarding for both TCP/UDP, I just changed it to TCP only but I don’t understand why I would have so many UDP connections, I thought we use TCP now.
Restarting my nodes and UDP climbing rapidly after only 5 nodes joined
Most of the connections shall be pruned after a while if no more traffic on it for a while, which could be reported as timed out.
During the startup, as the node is keen to discover the entire network, it will contact all accessable nodes, which will result in a spike regarding the UDP connections.
It sure does!! I don’t want to be bound by asic limitations off consumer grade routers .
I use pfSense and it works great (state table limit set to a max off 2 million currently), only limited by the amount of RAM I allocate to the hypervisor etc. And yes, my physical host(s) for the router is on a hypervisor with VM and is diskless, lol. .
Most consumer routers have a limit of 16,384 or less… for the state table.
It was precisely hitting these kind of limits that caused HandshakeTimedOuts on a wide scale for me in many testnets (limitation off ISP equipment) that I needed to bypass the ISP equipment.
In general even in steady state, it occupies a ton of src ip/src port - dest ip/ dest port and it quickly adds up per node if running 10s or 100s of nodes. Its expected due to P2P (I suppose), and a single node widely communicating with distinct and unique peers (the list constantly changing and evolving etc).
Complicated to setup and run but lots of options if they are needed.
I was previously using a Turris Omnia
Also about £250
as my router. It’s an OpenWRT router, switch, Wi-Fi device. I was very happy with it as my router but the interface was crapping out when there were more than about 4000 connections being used. It was still working fine but the GUI would crash and I’d have to go in to the CLI and restart the rpcd service. So that is relegated to just being the Wi-Fi hub. Anything important is connected directly to the MikroTik by wired ethernet.
The MikroTik is over specced for what I need but it’s nice to know that it will have the capacity to cope with whatever I throw at it even if I can get the 1Gb fibre that a company has been promising for my area ‘soon’. With 20 nodes and 9900 connections open it’s sitting at 3% CPU usage and 20% RAM usage of 1GB. The RAM usage isn’t much different when there are no safenodes running.
Also that it has lots of features that I might need if things get more complicated. And if the fibre ever materialises I’d be able to configure it to use the BT line as a failover.
The MikroTik is not doing the actual connection to BT because I need something with a ADSL port for that. I’ve got a DrayTek Vigor 167
setup in Router mode for the connection to BT and the BT set top box connected to that directly with the other port to keep the traffic completely separate. So I’m actually double NATed just at home and have to have port forwarding rules on both routers!
I would recommend the MikroTik RB5009 but it does need some networking knowledge to set it up. I’m certainly learning a lot from it!
But doesn’t the ISP router still have to handle the connections too? Or have you bridged the ISP router to your pfsense box? I am looking at setting up a small opensense router device between my gear and the rest of the house to limit exposure and so am reluctant to bridge my ISP router (if I can even do it)
@Josh My ISP router doesn’t give me any of those stats you are able to get. But in the stats I can see, which is more of a bytes/packets/errors up/down there weren’t any errors to speak of, but these may have been stats for router to their internal network
It looks good on specs for a 1G WAN but for a 5G WAN it will likely be slowing the traffic to under 4G and looks like under 3G from the specs given for routing. You are never routing max size packets all the time.
I only mention this so you are aware if increasing the WAN speed beyond 2G
So am I reading you right in that it can “easily” do multi homed (2 WANS) configuration
I would want at least 2.5Gbps on the switch ports since all my new machines are 2.5G connections or 10G on PC and NAS
Also looked at the specs and for routing the speed is going to be less than [EDIT] 1G 2.5Gbps in typical situations.
I have seeed Odyssey box and bought a 2x2.5Gbps B+M M.2 device to add giving 2x2.5Gbps native and 2x2.5Gbps on the add on board.
Planning on using that for my opensense/pihole/VPN out device with dual homed WAN with intelligent use of the 2. Maybe based on local IP address as primary decider
No, I have removed the ISP’s equipment from the equation and the workflow (no such thing as bridged mode or passthrough or whatever you may want to call it). Prior to the upgrade coming up, their box currently is only used to do 802.1x authentication traffic (passing through their box via my router to the wall), and the rest off the network traffic comes directly to my pfSense from the wall.
After this week’s planned upgrade, I will attempt to make the fiber go into a custom ONT SPF+ XGS-PON device directly to my router, and completely eliminate their equipment from the workflow.
You look at the testing data and go to the 25 ip rules with 512 byte packets. That is a more realistic figure for random traffic being routed through the router. The switch uses a chip that handles full speed switching, but routing involves the CPU and is typically lower than for a commercial high priced router that is much faster.
In normal circumstances for business the router&switches are mostly switching the data and routing lesser amounts and why a lower routing pps is fine. You can in a big business logically group your ports to do this with vlans and l2/l3 managed switches.
For home & SOHO routers a lower routing speed is acceptable to keep costs lowish. Switch function is still on a chip. In fact a better than normal ISP router will typically have better routing speeds than a cheap router for home use since most of the traffic hitting the ISP router is going to be sent to the internet, but unfortunately most ISPs penny pinch.
For Autonomi though routing is everything, all your nodes are required to be routed to the internet and so routing speed is of utmost importance. And as you point out the state tables need to be large which ISP routers don’t have
Although the 5009 and yours have the SPF+ port (10GB capable) where I would connect my portion of the home internet to 10GB/2.5GB switch and go from there.
Precisely, based on the large number of state tables required per node pid, most folks will be likely bottlenecked by their current router or ISP’s router first (i.e. errors and dropped packets and connections will start taking place), even before the bandwidth is peaked. This will lead to all nodes experiencing issues until one can track down the safe maximum one can do against their router for a healthy / stable state for all their nodes.
For those non advanced users, they wont even notice this is happening and think everything is okay, but in reality they are running all of their nodes at sub-optimal conditions and higher chance (probability wise) of being shunned or marked as bad depending on just how much they are saturating their bandwidth with total # of safe node pids that require X # of states on average against their router with the rest of the internet. I suspect rewards will also be less optimal due to going in and out of super healthy state and not.
Its been a tremendously long journey to isolate, identify, and correct the HandshakeTimedOut issues on my end (finally got to the end of it (at least without (private nodes (i.e --home-network))) (previously due to the NAT session tables limitations of ISP’s equipment), and for the advanced users, and serious farmers out there, they will need to be vigilant and ensure this bottleneck isn’t silently happening in their environment.
.
.
