Launch node memory usage improvement

[born2build]

Since launch, nodes appear to use substantially less memory. Is anyone else observing the same?

[storage_guy]

Yes, I’d definitely agree with that. About 40MB on average. I have a few outliers that are using 140MB to 150MB. We’ve got to remember the network is quite in terms of uploads and downloads at the moment though.

[Shu]

With very little chunk records at best per node (near 0), the record cache isn’t maxed out in memory which is likely why you are seeing a lower footprint right now.

Capacity plan accordingly, as and when records do appear, memory will rise (up to a certain amount). In addition, relay circuits and open connections also will add to the memory footprint.

[born2build]

@Shu

relay circuits

Does that mean that home network nodes use more memory, or that port-forwarded nodes that are supporting home network nodes use more memory?

[Shu]

I will leave it up to you assess that, I am not going to suggest ever to use --home-network as its not good for the network. The burden off connections shifts to those who are considered public nodes or nodes that are port forwarded properly.

The team is looking to rely way less on that option or entirely remove the flag (so its all automatic) along with DCUTR enablement.

[born2build]

nod and very much looking forward to those optimizations.

[neo]

Yes for some the automatic gets it wrong and they end up with nodes not talking or just unable to earn since clients cannot reach them

[Traktion]

@born2build

or that port-forwarded nodes that are supporting home network nodes use more memory

This is what I observed previously and suggested rewards should reflect that.

[Profess]

I’ve noticed this myself, but I have another question that I’ve been meaning to ask for a long time - once the nodes are up and running (in a home network), should the number of connected nodes remain the same? In recent tests (i.e. for a few weeks now) and in the current network, after the nodes are started, the position of the “peers” does not change, only the number of “conns” positions changes. Is this the correct behavior?

[Traktion]

I’m finding that my nodes are getting crushed pretty hard by relay duties. This didn’t happen in the final test network, which makes me wonder whether there are more home network nodes on the live network.

Boxes which could run 150 idle nodes, have dwindled to only being able to run 20 nodes at the same target system load.

It seems like the oldest nodes have the most load and when I check the logs, there are lots of attempts to relay:

[2025-02-16T22:36:21.256060Z INFO ant_networking::event::swarm 102] relay server event event=CircuitClosed { src_peer_id: PeerId("12D3KooWA9BS2LL2b5RtU9oVav2rctQDeAdit9Sfk3AE53BvJNJe"), dst_peer_id: PeerId("12D3KooWHbybb5ZPGoFwHjpwJq7T18mVuwBUbESBQcSr23d8ikHB"), error: Some(Custom { kind: ConnectionReset, error: Stopped(0) }) }
[2025-02-16T22:36:21.437457Z INFO ant_networking::event::swarm 102] relay server event event=CircuitReqAccepted { src_peer_id: PeerId("12D3KooWL9hLv26BojYnzXbE4b17Vk38Wcex2Mxu7eTrHinMyYSi"), dst_peer_id: PeerId("12D3KooWA4kVW2ck2wSrFV3BfZyFhc1sPDM15WZ1f41RgbvcSS19") }
[2025-02-16T22:36:21.647299Z INFO ant_networking::event::swarm 102] relay server event event=CircuitClosed { src_peer_id: PeerId("12D3KooWL9hLv26BojYnzXbE4b17Vk38Wcex2Mxu7eTrHinMyYSi"), dst_peer_id: PeerId("12D3KooWA4kVW2ck2wSrFV3BfZyFhc1sPDM15WZ1f41RgbvcSS19"), error: Some(Custom { kind: ConnectionReset, error: Stopped(0) }) }
[2025-02-16T22:36:21.652464Z INFO ant_networking::event::swarm 102] relay server event event=CircuitReqAccepted { src_peer_id: PeerId("12D3KooWJHLUsfHFwGaupaxRjmzxAwEQw4UuSUMAiCT9CHhDW2KG"), dst_peer_id: PeerId("12D3KooWMZnjs4xkCLPPsFyMXQLxkSiY5FzwvhXoherHp3MJJB6U") }
[2025-02-16T22:36:21.854074Z INFO ant_networking::event::swarm 102] relay server event event=CircuitClosed { src_peer_id: PeerId("12D3KooWJHLUsfHFwGaupaxRjmzxAwEQw4UuSUMAiCT9CHhDW2KG"), dst_peer_id: PeerId("12D3KooWMZnjs4xkCLPPsFyMXQLxkSiY5FzwvhXoherHp3MJJB6U"), error: Some(Custom { kind: ConnectionReset, error: Stopped(0) }) }

I restarted a near identical box, with a slightly slower CPU, and it is running 50 nodes still.

However, even the services restarted on the second box show that the oldest nodes are creating the most load on the box. They are also the ones with the most relay log messages.

I can restart the services to give the boxes a breather, but I think this is something that may need investigating further.

Tagging: @Shu @dirvine I can provide logs if needed, but sample above is repeated in similar form frequently.

[neo]

My PC had to reduce from 800 to 700 nodes, not because of resource usage but because more upload traffic is happening. Relay duties may account for some but I suspect its more to do with increased downloads happening since some nodes now have records and high chance they will be targetted for downloads. So few files and so many people testing downloads.

Relays have little B/W needs since they are just relaying connection details and the real B/W is direct between the two nodes.

My SBC (4 core, 8GB, intel) was increased from 100 port-forwarded nodes to 150 port-forwarded nodes some 14 hours ago. Reset first and added 150 nodes (custom script) and from the monitoring around 90 have given out quotes already. This is pretty good considering the network being around 1 million nodes.

The system monitor is showing an increase in CPU usage over 100 nodes but not that much more. with 150 nodes 3 cores are now doing the same work as 4 cores with 100 nodes.

Here is a snapshot of the graph of usage

[born2build]

I have many InboundCircuitEstablished relay client events in logs as well, and that’s with nodes started using --home-network.

[neo]

Considering those are saying your relay connection (incoming to your node) was established then I’d expect that

[tobbetj]

I don’t understand why not more people are encouraged on discord to try and get port forwarding or UPnP going before they test home-network, as devs have told it is good for the network the more people that can run port forwarding or UPnP
@rusty.spork @neo

[Shu]

@tobbetj

I don’t understand why not more people are encouraged on discord to try and get port forwarding or UPnP

Team is working on enabling UPnP as a priority by default before further actions are decided based on the state off the antnode, and its perceived connection to the outside world.

@Traktion

I’m finding that my nodes are getting crushed pretty hard by relay duties. This didn’t happen in the final test network, which makes me wonder whether there are more home network nodes on the live network.

True statement here. We are already aware off this situation. We are also trying to identify if its abnormally higher ratios off --home-network requesting relayers’ assistance than anticipated, or there is a regression in how the relay to relayers’ code base works with respect to active number of connections.

Thank you for bringing this up to our attention (both of you)! Cheers!

[born2build]

I think using UPnP if it’s enabled, and port forwarding isn’t configured, and only then falling back to relay behavior makes sense. It has the benefit of ease of use without the disadvantage of putting extra work on port forwarded nodes, as relay behavior does.

In concert, I believe there’s low hanging fruit available in helping more people to run port forwarded configurations.

The following two issues document trouble I’ve encountered in my goal of running 100% port forwarded nodes. The first issue is a feature I need.

Antctl add --node-interface?

For node machines with multiple NICs, passing --node-ip to antctl add is helpful to direct node traffic over the intended interface/IP address. Once added, however, I’m not aware of a way to update the configured IP address for nodes. Is it possible? If not, it would be great to be able to pass --node-interface to antctl add, with the expected behavior that upon each start of nodes, they obtain the current IP address of the interface.

The second issue is documentation I need.

Optimal firewall rules for port-forwarded nodes

I have port forwarding set up, and now I’d like to lock down my firewall. What’s required to be open for nodes to have optimal network access? I’m asking because it appears, though I could be mistaken, that nodes require more than UDP to the ports that are being forwarded to the nodes. I’m seeing many HandshakeTimeOut errors in node logs, and much blocked traffic in to UDP ports outside of the range of ports that are being forwarded to nodes.

This documentation could be augmented with a simple feature added to antctl/antnode, or a small dedicated tool, that people could run. It would report their connection status, and take the guess work out of knowing if they have port forwarding properly configured or not, and perhaps also if they have UPnP enabled or not.

[Traktion]

Tbf, its harder to set up and support. I can understand why home network is easy to bootstrap folks with.

Ideally, there should be a carrot to earn more if providing direct ports. If home network nodes use less resources and put more load on direct port nodes, they should earn less.

Anyway, it sounds like the team is on the case, so I look forward to trying out the results. Thanks @Shu and team!

[QrJ955nxcap]

While this may be based on personal experience, I observed that my port-forwarded connections earned more attos during the beta phase compared to my home-network nodes.

It is advisable for home routers to have UPnP functionality disabled by default, as leaving it enabled poses security risks. I am concerned that if Autonomi utilizes UPnP for configuring NAT firewalls, it may give the impression that they are recommending users to enable this feature.

[dirvine]

@QrJ955nxcap

I am concerned that if Autonomi utilizes UPnP for configuring NAT firewalls, it may give the impression that they are recommending users to enable this feature.

I get you, however many many devices now do use upnp, it can be argued even port forwarding also is a security risk and so on. So it’s all relative and 100% important to continually check out.

For us we need

• No relays (yes zero relays as they will kill the network)
• Direct connected nodes
• Nodes connect via manual port forward (so we need very clear instruction)
• Nodes via upnp
• Nodes via full cone NAT

These are all good nodes, not 100% of home users running nodes, but 100% of any user running apps (we need to be really clear on this part).

Then we have all these flags gone to hell (as I hate these flags and mental configurations) and make nodes as automatic as possible for folk to run. Apps 100% automatically OK to run from anywhere.

This is also why I keep poking ipv6, sure it’s not great yet, but if we remove dns (who needs dns where we are going??) then it’s better. In any case 100% nodes from home is a journey we have to make, but not wait on IMO.

[born2build]

@dirvine

remove dns

Glad to see this is a goal as it’s a vector for censorship.