Routers for advanced setups

[Southside]

@peca

Huawei, ZTE - full of China backdoors

Was that ever proven or was it just more of the frenzied thrashing and China-bashing from the sclerotic US developers and their bankers?

[peca]

Yes it was proven, not only in USA, look here for example.

All economically strong countries do it and all big tech companies are in bed with them. IT is full of dark secrets.

[drirmbda]

Here is a datapoint with a Unifi DreamMachine Pro, 4GB RAM, after doubling the conntrac limit using command “echo 131072 > /proc/sys/net/netfileter/nf_conntrack_max” and on a PON with 200 Mbps symmetric bandwidth:

With about 300 safe nodes, using port-forwarding, there are about 115000 entries in the NAT table (per /proc/sys/net/netfileter/nf_conntrack_count.) Not impacting RAM at all, but CPU utilization increased to about 80%.

The instantaneous numbers can swing wildly around these typical values. It appears that the router’s CPU will be the limiting factor.

[neo]

Yes routing is the big load on router/switches. And the big figure they like to present in the sale pitches is the switch capabilities which is all done on a separate chip built to do full rate switching.

MikroTik for instance have their router/switch devices and often people zoom in on the switch capabilities and skip the routing speeds. Often the routing speeds are a lot slower than one wants, and then are the models where they excel

In normal situations you don’t want the routing to be anywhere near the switching speeds because most of the traffic is between ports and internet (or upstream) routing is only a small %age of the overall traffic.

But for nodes it can be the major portion of the traffic is routing. And this is where people need to be careful in they plan to upgrade their router. One of the things to help find a router is if it is meant for replacing a ISPs router, because many of these will have higher routing capacity.

[peca]

@neo

Yes routing is the big load on router/switches.

Not that long ago this was what you needed to be able to route full 1 Gbps, only 3 ports. I know about few places where these machines still do their job (and eat a ton of electricity)

@drirmbda

Not impacting RAM at all, but CPU utilization increased to about 80%.

Try to also double number of buckets, it should help (/proc/sys/net/netfilter/nf_conntrack_buckets).

EDIT: On Mikrotik (and other platforms too) routing is optimized and you can do nothing about it. What takes performance down are firewall and NAT rules. Every rule packet goes through slows it down. You want to keep the number of rules down and think about order of rules. If packet hits rule 2 in 10 rule list it is done in two steps, if the same rule is at the end of the list it takes 10 steps.

[drirmbda]

UDM-Pro runs kernel Linux 4.19.152-ui-alpine. I hoped that changing /sys/module/nf_conntrack/parameters/hashsize would do the job but it did not seem to make a difference. Reported usage of Memory and CPU load remained the same. Neither did setting /proc/sys/net/netfilter/nf_conntrack_buckets.

[Josh]

@neo you may find this useful when your package arrives.

FastTrack is a feature in MikroTik RouterOS that significantly improves router performance by bypassing certain processing steps for eligible network traffic. It is specifically designed for high-throughput scenarios where you need to handle a large amount of traffic efficiently.

Key Functions of FastTrack

1. Bypassing Firewall: FastTrack allows eligible packets to bypass most firewall processing. This reduces the CPU load as the packets do not have to go through the entire set of firewall rules.

2. Connection Tracking: FastTrack works by marking connections as eligible for fast tracking. Once a connection is marked, all subsequent packets belonging to that connection are processed faster.

3. Bypassing Firewall Processing: Just like with TCP, FastTrack allows UDP packets from established and related connections to bypass extensive firewall processing. This reduces CPU load and improves overall performance.

Benefits of FastTrack

1. Reduced CPU Load: By bypassing the firewall processing for eligible connections, FastTrack significantly reduces the CPU load.
2. Increased Throughput: With less CPU involvement in packet processing, the router can handle more traffic, leading to higher overall throughput.
3. Lower Latency: Faster processing of packets can result in lower latency, improving the performance of latency-sensitive applications.

[storage_guy]

@Josh

FastTrack is a feature in MikroTik RouterOS that significantly improves router performance by bypassing certain processing steps for eligible network traffic

Looking at the status on my MikroTik it seems that most of the traffic in and out of the port of the RPi 4 that the safenodes are on shows as ‘FastTrack’. Although it has to be said that most of the traffic on ports for other machines shows as FastTrack as well.

I don’t know how to assess how much it is helping anything.

[peca]

FatsTrack is default on, you only need to switch it off for some functions to work, for example speed limiting with multiple queues or packet prioritization.

[Josh]

Did you add the rules to your firewall?

I don’t fully understand it but before adding the rules I saw connections marked for fasttrack.

However you want to add these two rules.
High up on your list so that they are hit first.

/ip firewall filter add chain=forward action=fasttrack-connection connection-state=established,related
/ip firewall filter add chain=forward action=accept connection-state=established,related

you can then check it with /ip firewall filter print stats

https://wiki.mikrotik.com/wiki/Manual:IP/Fasttrack

[Josh]

@peca

FatsTrack is default on

That explains why I saw fasttrack connections, but why do we need to add rules then?

[peca]

It is probably for different device or different version of RouterOS, Mikrotik documentation is terrible.

My router came with this default FastTrack rule:

/ip firewall filter add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes

[Josh]

@peca

hw-offload=yes

You must have a beefy router there, mine does not have hw-offload I was just salivating over that yesterday :)… ohhh hold on I was wrong. I think I can!

[peca]

@Josh

You must have a beefy router there

Nothing too fancy: MikroTik Routers and Wireless - Products: hAP ax²

I have only 200/200 Mbit connection, but so far it performs nicely. With 100+k connections the CPU load is around 20 %.

[Josh]

I can offload, don’t know how I got that wrong. Seems silly that it is not default on for me, the explanation appears to be not all routers support it so ships as off by default.

However in your case it was default.

[chrisfostertv]

Great info in this thread, I’m running 24 nodes on a HP T640 thin client (2 core/ 4 threads with 6GB useable Ram) running a steady 80% CPU…probably drawing around 15W juice.

Bottleneck is definitely the routers CPU (NetComm NF18MESH), regularly spiking 50-100%…so probably horrific things happening with the packets.

The RB5009UG+S+IN looks good…either that or repurpose a T640 maybe.

Haven’t seen any discussions around NIC settings like MTU; Jumbo Frames, removing bindings etc

[neo]

@chrisfostertv

Haven’t seen any discussions around NIC settings like MTU; Jumbo Frames, removing bindings etc

Doubt changing those settings will make much difference since most people are not maxing out their NIC or even getting close, and quite possible to make no difference.

[Joey_Jo]

so is there a list of specs anywhere that we could use if we were to go “router shopping” ?

[Josh]

Look for a high max sessions (connections), good CPU and a decent amount of RAM.

My advice is get the best that you can afford as is always the case you get what you pay for.

[peca]

Look at Wifi6 routers, not because of wifi, but because they had to use new high performance CPUs to support that, not some 10 years old shit in new packaging like many other routers are.