Routers for advanced setups

Community
667

Great to see the numbers! I thought it would be useful to share mine, at the other end of the hardware spectrum (mini PCs).

Currently, running around 2500 nodes, 6 boxes, with around 400 per box (similar hardware, between 250 and 500 nodes each). Getting around 3 ANT per day, per box (again 1.5 to 4.5 range), so around 20-25 ANTs per day, at price of 25c per ANT = $5/day to $6/day;

Power consumption (10w per box) = 60 watts (about 1.44 kwh/day)

Milliwatt per Node = 60*1000/2500 = 24mW per antnode

We have free solar during the day and dynamic pricing for cheap night (about 18c). So, in most months, power is probably under 10c average.

Power Cost per Day (10c per KwH) = $0.15/day

Each mini PC cost around $120, includes core i5 2.4ghz - 2.8ghz (4 core), 8 GB ram, 128/256 GB m.2. SSD for os/apps/logs, 2 TB HDD for node data. However, that cost was covered by rewards from beta.

I suspect as the network gets busier, the nodes per box will likely drop though. The 2 TB drives should support around 60 nodes at 50% full network.

So, pretty much $5-6 profit per day, given broadband was an existing fixed cost (about $2 per day). Each box takes around 3-4 months to break even, so 200-300% return in year 1. Pretty decent tbh and much better than most investments!

Edit: I should add, a MikroTik router was needed (for a stable LAN), along with wired connections for nodes. That added about Ā£130 iirc, to startup costs.

Edit2: I also provision the nodes using ansible, so adding a new node is pretty quick. Most of the time as verifying that the used hardware is functional (lenovo has bootable app to do thorough checks). From a bare box, to contributing takes under 30m though.

As part of ansible provisioning, it sets up port forwarding and traffic shaping on the mikrotik router too. Additional shaping is configured per box to restrain outbound to prevent large spikes. I also have a custom node runner script that targets system load, ram and disk usage and slowly adjusts node count to fit capacity.

16 Likes
Best Ant (was Safe) Node hardware
668

After multiple tweaks (and OS changes), I finally have a stable connection with a large number of nodes (~15k) filling about 1.2M NAT entries on the router.
But there is still one issue that I canā€™t pinpoint: the more connections, the less bandwidth (especially upload).
Without any nodes, I measure 7.5gbps upload (and download) through iperf3 and I currently measure 2.2gbps with 15k nodes and 550mbps upload traffic from nodes (and about 450mbps download traffic from nodes) for a total of ~2.7gbps upload (though I measure about 5.5gbps total for downloads) . Note that with 7k nodes I had about 5gbps upload.

Router (openwrt on i5 12600h) CPU is almost idle (even when doing the iperf test, I checked all cores through btop), RAM is 900MB/32GB, node traffic is about 250 kpps up and down.

What can be the cause? Can it be my ISP? How can I test?

2 Likes
669

So, after buying some beefy Fortinet router (Fortigate 600E) and a lot more tests, I also had the exact same problem with the same limits, it turns out it was my ISPā€¦
I changed my ISP and it works :tada: (for nowā€¦ I hope they wonā€™t restrict my usage at some point)

2 Likes
670

FortiGate-600E
2 x 10GE SFP+ slots, 10 x GE RJ45 ports (including 1 x MGMT port, 1 X HA port, 8 x switch ports), 8 x GE SFP slots, SPU NP6 and CP9 hardware accelerated#FG-600E
List Price: Ā£11,648.96

Now that is what I call commitmentā€¦

671

Wow, I think I may have come across this brand onceā€¦ but I had totally forgotten about it.

I had to look it up just now. :smiley:.

As always, nice to folks sharing details of some of their hardware specs at their locations!

I prefer a network topology where the switch is just a single component of the overall design, and the failure of such a switch (no matter how dense or simple its offering) doesnā€™t lead to outages for the stakeholders (appliances and servers traversing through that switch). Personal preference is to keep the router appliance/hardware as a router/gateway role only (high availability design), and not be used as an expensive switch/combination, and still ensure HA on the network design exists via multiple switches (hardware), and still be able to segregate and partition network traffic for different use cases.

For instance, I have 9x48 port switches + Uplinks, and redundancy on the switches/wiring with the server hardware itself, so a failure off 1 or more switches (for any reason), doesnā€™t bring down the service running on the servers utilizing the switches (HA) and the router/gateway (HA). At best, if designed properly, and depending on the mode of failure, its a degradation of service, but not an outage in terms of connectivity.

For what its worth, running 1.5M NAT session tables on physical commodity hardware that is probably under $500 dollar value (bought refurbished), but it is in a HA configuration.

1 Like
672

So with your new router equipment and ISP change, whats your current state of the network traffic for 15K antnodes?

Just curious, what your NAT session table count is, and the Upload/Download in Mbps in steady state?

1 Like
673

I had a (very) good deal on it: ~400ā‚¬ refurbished (with no license, but the license is only useful for the advanced security features like deep packet inspection, antivirus etcā€¦ the license price is several thousands a year).

I was using it this way, only as a router with just one WAN and one LAN interface (connected to the switch).
Also, this is a homelab setup, my priority is that it works with minimum hassle (and also minimum power usage), high availability is very low on my list of priorities right now (and in my case, failures have 100x more chances to come from a power outage than anything else even though most of my equipment is backed by an UPS).

Iā€™m using my OPNsense router (minisforum MS-01) -I stopped the fortinet- it has less power usage, itā€™s way more silent and itā€™s beefy enough at the moment (though I have one core at ~68% cpu usage average from interrupts only even with Receive Side Scaling enabled).
With ~15K nodes, I have ~1.8M entries in the NAT table and an average bandwidth of 450Mb/s down and 500Mb/s up.
Running iperf3 on top of that gives my 7.5Gb/s remaining on my 8Gb/s symmetrical internet connection (only when testing in unidirectionnal mode, if I test using bidir mode, I top at about 4.5Gb/s whit current NAT table whereas it was almost 8Gb/s before starting the nodes but I think it might come from some hardware limitations at this point).

So now, Iā€™ll start investigating why my Epyc Milan server can only handle half the number of nodes per core my old Skylake laptops are handling (with the milan cores being 1.5x more powerful than the skylake core from the benchmark I ran)

4 Likes
674

Does anyone by chance have experience with virgin media hub 5 ?

675

Now thats what I call a bargain

Any more going?

676

:laughing:, no, but refurbished units are usually listed at ~2kā‚¬ on ebay if you really want one (but you can build yourself a router almost as powerful -at least for handling a large NAT table- for 1/4 of this price, maybe less).

1 Like
677

I have a ā€œAsus ROG Rapture GT-AX11000ā€ expensive toy. I thought this can handle few hundred nodes easily. But after 700-800 nodes, I am unable to get even 150Mbps through 2.5Gbps ā€œgaming prioritizedā€ port.

I have another cheap router TP-Link Archer C6, is it possible to transfer some load to it if I connect TP-Link router to Asus router in bridge mode? Idk what I am talking about.

678

This is already a rather good figure for a consumer router, I had the same one (still have but configured as AP) and I could get up to ~100K NAT entries (~1000 nodes) before noticing lags in my LAN. You wonā€™t get much more from consumer routers.

No, and you would not be able to launch many more nodes if you could anyway.

If you plan to run less than 8K nodes, you should consider the Mikrotik Rb5009.
For more than 8K, go for OPNSense or OpenWRT on SFF computer with at least 6 cores and 4Ghz.

679

NAT entries mean number of ā€œTotal Connected Peersā€ right?
And how do I reduce internet lag for AX11000? even Google opens slow.
Currently 1339 nodes are UP!

When I do speed test, its 30-40Mbps UP/Down. My connection speed is 150Mbps.

680

Reducing the amount of NAT table entries (lower the amount of nodes). Like people said, consumer grade routers are just not optimized for so many connections.

1 Like
681

A 500$ router cant handle 1000 nodes? :smiling_face_with_tear:

682

Having never configured a router, would this be simple to set up?

683

A consumer grade $500 router cant. Iā€™ve had massive success with running own hardware with open source router software like OPNsense.

A $200 router could run 100.000 nodes without issues. Itā€™s just that the consumer grade routers are such a rip-off and very poorly optimized for so many NAT entries. It does make sense, because lets face it, pretty much no consumer needs so many NAT connections.

Edit: you can take a look at a home node guide I once wrote. Could be outdated, but it does show my setup and helps with setting up a custom made home router with OPNsense: The ultimate home node guide

1 Like
684

No, but you could easily find help (this is a well known router and OS, so there are plenty of resources, also lot of members of this forum are using this particular router) and LLMs (ChatGPT, Geminiā€¦) are really good at helping setting up the routers.

1 Like
685

If you are paying 500$ then a mikroTik RB5009 is a much better choice for running nodes.

Retail routers are not designed for huge number of connections, each node can at times have hundreds to 1000 or so connections. Average will be much lower but still in hundred(s) each. The RB5009 has a one million capable NAT table, and is quite fast for the price.

Retail routers are either just to get average person connected to internet and handle a couple to few people doing things. Gaming routers are for speed not connections.

2 Likes
686

Thatā€™s at least 10M NAT entries, Mikrotik are limited to 1M (even the high-end routers).
Only high-end enterprise routers (Cisco, Juniper, Fortinetā€¦) are able to handle such traffic and these donā€™t start for less than 10k$ (and you need to add licensing costsā€¦)