I’m not sure that is a huge concern, as long as you live somewhere without frequent cuts. Autonomi nodes seem to bounce back to earning attos pretty quickly after a restart (new release upgrades for example).
1 Like
Hey guys! Still looking for ways to scale up nodes from home. I have an option to buy a secondhand server set, I’m completely unfamiliar with server racks but it feels very clean and scalable. Before purchasing I’m looking for some advice from someone more experienced. Here’s what the set contains:
2x HPE Proliant DL380 Gen9, each server has:
- 2x Intel Xeon CPU E5-2650 v3 @ 2.30 GHz (20 cores)
- 192 GB RAM
A module for M.2 SSD-expansion
A High-performance storage array:
- HPE MSA 2040 ES SAS DC SFF-storage (2U), model K2R84A
- 2x MSA 2040 SAS-controllers
- 13 SAS HDD
- 13 brackets + 11 fronts included
2x Cisco SG500X-24P Layer 3 switches (1U):
- 4x 10 Gigabit Ethernet SFP+
- 24x 10/100/1000 PoE+
- Including rack mounts
Cables:
2x SFP-H10GB-CU1M
4x Molex 1110751004
APC switch and a half height server rack.
I can get this for €2000, which seems like a very sweet deal. Wondering what you guys think though.
What sort of internet connection are you trying to fill with that lot? I hope you’re on good terms with your broadband provider! 
Seems expensive.
I recently got a E5-2699 v4 2.2Ghz (22 cores each dual socket) with 1.5TB ram for $2800 (refurbished) (including 6x10Gbps SFP+ network cards + 12Gbps 12 bay disks sas/sata), compared to your 2 x E5-2650 v3 2.2Ghz (10 cores each dual socket) with 192 GB RAM for $2000.
4 Likes
good box for a 3-5Gigabit ISP biz connection, if you can get one in your area.
Already have a 5Gbps so all good.
2 Likes
Just curious, whats the highest number of antnodes folks have been able to run on a single container/VM/physical machine out there in the wild?
Feel free to share your hardware specs off that machine too.
I am attempting to run 2000 to 3000 antnodes just now on my single Intel E5 server… will provide an update if successful.
I suspect even the beefier modern 5nm AMD EYPC CPUs can probably cross 10K nodes on a proper dual socket server? 
.
2 Likes
Once the hotfix is released, likely then 50/8 * 500 = 3125 nodes before CPU threshold limitations.
3 Likes
Which GUI is that?
I have removed the graphical environment and access my machines with this, there is a built-in terminal through the browser:
There is also a usability graph:
Check out the Dev Forum
3 Likes
Managed to hit 2000+ nodes on this host (new record for myself).
Probably can get to 3000 to 3750…but will try that test a bit later, as it took 4+ hours to spin up 2245 antnodes using 270 GB RAM. NAT session tables less than 200K entries on the router.
313 - 240 watts = 73 watts additional usage for 2245 antnodes = ~32mW (nice!)
7 Likes
I have a question that’s only somewhat related to hardware. If there’s a topic that’s more appropriate for asking about “Best Safe Node configuration” please ask a mod to move my question there.
Has anyone observed nodes running with the port forwarding connection type earning more than with the home network connection type?
Most folks aren’t going to run 50/50 split off nodes in two different configurations at a single location, so the data to compare properly will be limited (all else equal).
Autonomi does run --home-network nodes but its 1:10 ratio compared to the remaining public nodes on DO. They are both earning, however, private nodes currently show slightly less payment received count average value. The comparison isn’t valid as the sample size for private nodes is very small here compared to the public nodes.
Having said that, both configuration do earn, though its best to do port forwarded than --home-network. As stated before, --home-network is the last preferred option, port forwarding and UPnP is the preferred method here.
i.e. With --home-network, why be dependent on other nodes more than necessary for your communication/transport requirements, as its yet another layer, where the things could go wrong, resulting in potentially less earnings.
3 Likes
Thanks for the reply! I’m now looking at some different setups. Found this one:
HP Proliant DL380 Gen10
- 2x Intel Xeon Gold 6146 3,2ghz (12c, 24t p/s)
512gb DDR4 RAM (16x 32gb)
Room for 8x 2,5 inch disks. No brackets included
Onboard SATA controller
Power supply Redundant 1600 watt
1x HPE Ethernet 10Gb 2-port 562FLR-SFP+ Adapter + 1x HPE 562 SFP+ Ethernet 10GB Dualport Adapter
For €1250, this comes a lot closer to your 2699 v3 setup. Not sure it more than 512 GB ram would be needed looking at your current numbers. What do you think?
A dual socket Intel Xeon Gold 6146 (24 cores total) is roughly 7% slower in performance compared with a dual socket E5-2699 v4 (44 cores total) (Intel Xeon Gold 6146 vs Xeon E5-2699 v4 [cpubenchmark.net] by PassMark Software).
Having said that, the re-occurring cost due to TDP requirements is slightly lower in my setup, though my CPU is 2 years older (release date).
If I can run 2000 nodes at < 50% CPU, with an average memory requirements of 130MB per antnode, you are right under 256GB RAM. Say at a future date, if and when CPU limit threshold is altered or removed, 512GB of RAM should suffice to make maximum use off your hardware (both on CPU and memory front). Don’t forget with 2000 nodes, you need up to 125TB off storage capacity too, though that limit will likely never be hit (network being full).
However, I went with higher memory just in-case the requirements change off antnode, choosing to maximize the density of RAM sticks in the system, because if you have to increase RAM in future and no spare slots, you will end up throwing away the old RAM sticks (wasted $$$), so I opted for highest memory capacity per slot right out of the gate. Also, I have other home lab projects VMs that require higher memory in general.
1 Like
One is into DC Grade HVAC to keep that EPYC cool in dual socket mode, as such one will need air cargo handler grade noise suppressors to cover one’s ears if the fan systems are average quality. Either that or one employ’s liquid cooling. 
I just did a quick check on RAM per antnode after the config settled, seem’s 130 MB per node is tight for UPnP? Thoughts? Do you know if there is a different antnode RAM consumption usage vs. Port Forwarding?
I am on an LXC Alpine container, however, for 2580 antnodes now, its at 315 GB RAM = 125MB RAM per antnode (port forwarded). This is running with custom modified code base at home with a recompile to reduce file descriptors (soon to be released to community early this week) (PR: Avoid scan entire sys during cpu threshold check no sysinfo upgrade by maqi · Pull Request #2639 · maidsafe/autonomi · GitHub).
However, for just 4 antnodes in your screenshot, the above PR won’t make a big difference in memory usage. I do not think the memory difference will be much different between UPnP vs Port Forwarded, but I am not running UPnP at home so I cannot accurately comment on that.
1 Like
I think for antnodes 44 cores would be much better than 24 cores. With thousands of nodes, there will be significant CPU time spent on task switching and less nodes per CPU core means less time wasted on that.
Benchmarks typically run one thread per core, our use case is far from that so real CPU performance for nodes may be far from universal benchmarks.
I don’t have enough HW to do more tests, but I have AMD 8-cores and one 12-core of the same generation and the 12-core performs better than 1.5x of the 8-core.
1 Like