Launch node memory usage improvement

I just want to take a moment since no one in my surroundings is into this :blush: I just managed to run 5 nodes on my laptop! Feels good to contribute, even in a small way. Feel free to throw confetti :tada::yum:

9 Likes

Problem is that some countries have not impemented or fully impelmented IPv6 yet, will it be possible to run both IPv6/IPv4?

1 Like

From experience people might need to turn on UPnP in the router settings. It would be interesting to know how many % of consumer routers have UPnP on by default.

1 Like

The issue is, even if a large % of nodes would work then we cannot guarantee 100% of clients would. So even with a smaller % of nodes working with ipv4 (which today is not the case as ip4 is a much larger proportion) then with ipv6 the issue is clients.

So then it’s dual address ipv4 and ipv6 nodes and that is where is gets a bit complex. As nodes can be shunned fro being bad then an ip6 node may suffer and not be able to be connected to with ip4 clients and other ip4 only nodes. So it’s a mess really, but one we need to keep watching

3 Likes

Dual stack, aka IPv6 on extra fun mode. Top comment from today in fact.

2 Likes

IPv6 is a mess right know, I won’t switch until it is fully or near fully implemented.

It’s been like this for years now, decades, came out in the 90’s world ip6 day in 2012 was supposed to be widespread adoption and still we are here. So waiting is what the whole world is doing. We really need some really valuable network or similar to push it at some point in time. It’s the networking version of “don’t look up” as we ran out of ipv4 and now have a nightnare centralised only foundation to build real decentralised network upon.

But we will do what we can I am sure.

DNS is the other nightmare we need to kill and work around

6 Likes

I think the politic in EU should force a change, tell companies/organisations that they have a limited time to change or there will be penalties.

2 Likes

It could be orchestrated like the mandate for universal phone chargers/USB. All networking capable devices could be mandated to support IPv4/IPv6 or IPv6, then let companies make the business choice to simply move to IPv6 as it’s more costly to support dual stack.

1 Like

Just to give an update on this, I have 3 near identical mini PCS. Same motherboards, same amount of 8gb RAM, similar m2 SSD for boot/app/log, similar 2tb HDDs, similar CPUs (core i5 6500T, 7500T, 7600T about 300 mhz spread, all 4 core).

The 7600T is a fresh build, live about 24h. I set it to scale to up to 250 nodes and it has been pretty much pinned there for most of that time. It’s also running AntTP, which is worth about 10 nodes CPU time in top.

The 7500T has been on since launch. It is now sitting at 28 nodes, with load being heaviest on the oldest nodes!

Checking the logs of the oldest node, shows it essentially dedicated to relay! It has a 15m system load of 2.03:

Compared to the spritely new box, node one seems to be doing useful storage work. It also has a 15m system load of 1.72.

I suspect if I took the 250 cap off (the number of ports I have forwarding to it), it would do at least 10x what the first box is doing.

How about 6500T? Also 28 nodes with 15m system load of 1.89. Node 1 on that box is also essentially dedicated to relay for home network nodes it seems:

Restarting the services gives temporary rest bite, but the relays find them again pretty quickly - within hours, it’s back to this state again.

I may try powering off the 2 boxes that are getting hammered. Then turn them on later when the relays have found someone else to pick on! :sweat_smile:

1 Like

Fwiw, only the new box earned rewards in the cycle 4 hours ago too. It looks like the other 2 didn’t contribute enough nodes to register.

I’m sure 3.5 million nodes isn’t helping. My node rewards are in the gutter as well.

1 Like

That’s interesting how things change over time.

I had nodes running of a few servers since launch. At first, they had a very low CPU usage, which crept up over time towards around 70% CPU. I thought It’d try a reboot of the system, and now they’re under 20% again.

Not sure if that’s likely related to relay duties? I’ll see if they stay low CPU overnight.

1 Like

I switched off both boxes that are getting hammered for 30m. Turned them back on an I’m relay central again, according to the logs! :sweat_smile:

2 Likes

If it’s this, it took about four days for the traffic to die off. Peer nodes are persistent hehe. I wonder what would happen if you disabled the port forwarding and ran your nodes in home network mode?

Possibly related! Maybe relay node peers are sticky?

I just recreated the port forwards on a different range and started booting the ant nodes again. So far, no relay logs and the load seems more normal (i.e. similar ballpark to the new box).

The logs look like more normal data storage ones too, e.g.

[2025-02-18T22:06:06.653309Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:06:21.653351Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:06:36.653050Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:06:43.658261Z INFO ant_networking::bootstrap 162] It has been 180s since we last added a peer to RT. Slowing down the continuous network discovery process. Old interval: 537s, New interval: 575s
[2025-02-18T22:06:43.664025Z INFO ant_networking::bootstrap 75] Triggering network discovery with 0 candidates. Farthest non full bucket: Some(240)
[2025-02-18T22:06:51.653133Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:07:06.653459Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:07:07.652534Z INFO ant_node::log_markers 69] IntervalReplicationTriggered
[2025-02-18T22:07:21.652882Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:07:36.652812Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:07:51.652741Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:08:06.652747Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:08:21.653247Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:08:36.653224Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:08:39.652421Z INFO ant_node::log_markers 69] IntervalReplicationTriggered
[2025-02-18T22:08:51.652716Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:09:04.656891Z INFO ant_bootstrap::cache_store 153] Creating new CacheStore with config: BootstrapCacheConfig { addr_expiry_duration: 86400s, max_peers: 1500, max_addrs_per_peer: 6, cache_file_path: "/home/autonomi/.local/share/autonomi/bootstrap_cache/bootstrap_cache_1_1.0.json", disable_cache_writing: false, min_cache_save_duration: 300s, max_cache_save_duration: 86400s, cache_save_scaling_factor: 2 }
[2025-02-18T22:09:04.656968Z INFO ant_networking::driver 946] Scaling up the bootstrap cache save interval to 2552s
[2025-02-18T22:09:04.656980Z INFO ant_bootstrap::cache_store 327] Flushing cache to disk, with data containing: 0 peers
[2025-02-18T22:09:04.704936Z INFO ant_bootstrap::cache_store 376] Cache written to disk: "/home/autonomi/.local/share/autonomi/bootstrap_cache/bootstrap_cache_1_1.0.json"
[2025-02-18T22:09:06.652993Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:09:21.653444Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:09:36.652992Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:09:51.653525Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:10:06.653001Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:10:11.652787Z INFO ant_node::log_markers 69] IntervalReplicationTriggered
[2025-02-18T22:10:21.653401Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:10:36.653260Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:10:51.652878Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:11:06.652897Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:11:21.653079Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:11:36.653391Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:11:43.652636Z INFO ant_node::log_markers 69] IntervalReplicationTriggered
[2025-02-18T22:11:51.653508Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:12:06.653220Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:12:21.653619Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:12:36.653235Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:12:51.653329Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:13:06.653279Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:13:15.652272Z INFO ant_node::log_markers 69] IntervalReplicationTriggered
[2025-02-18T22:13:21.653078Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:13:36.653214Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:13:51.653246Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:14:06.652693Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))
[2025-02-18T22:14:21.652905Z INFO ant_networking::driver 894] Set responsible range to Distance(785265362123726368703687778107963784812215065278052870276269422796719900)(Some(238))

So, that seems to have shaken off the zombie hordes after my ant nodes, for now… :sweat_smile:

2 Likes

For those following along, those 28 node boxes had nearly 260 nodes this morning. If you get selected for relay, the impact is substantial (assuming it is all relay related).

2 Likes

Relay behaviour is a serious issue for us and one we are seeing. Expect some changes here very soon

7 Likes

This topic was automatically closed after 60 days. New replies are no longer allowed.