Are you sure about this, resources are shared among many and are limited too.
I think we could assume better specs but by how much, especially shared cheaper plans.
Bare metal better perhaps?
Is there anyway to see?
True - but shouldnāt that then translate to smaller peer count maybe?
Idk but I would be very conservative with cloud nodes as it is a bit like driving blind and hoping for the best.
1 Like
i guess i need to check the log files for error rates then too ā¦ damn ā¦ i thought iād get away without it tbh ā¦
1 Like
I really hope the cloud nodes are not needed in order for the network to work. No true decentralization if they are.
1 Like
I canāt see it relying on cloud nodes, you have either near zero cost to run from home, or a once off cost for bigger setups but why pay the cloud in perpetuity to run nodes?
Earn nanos to pay for your cloud nodes in a never ending cycle, thats if you even cover the cost.
To bootstrap the network sure, home nodes long term.
2 Likes
You kid not, crash course in setting up a router from scratch underway.
Really cool to finally understand what happens under the hood, I had no idea the thing comes completely raw, rude awakening!
Itās all good though few hair less on my head but I have managed to remove my ISP router from the equation and got things working.
Unfortunately the priority had to be getting the boring stuff done, DHCP, house online, firewall up, kids placatedā¦ tomorrow comes nodes!!!
1 Like
Trick is to work out exactly what you want to do. Without that it becomes a much bigger problem because you have so many options to work it
And internet search will be your friend initially
1 Like
Yes there is a lot online and Mikrotik has pretty good instructions, still took a bit of cussing but it I am really happy with the new knowledge although I am sure I have barely scratched the surface and will pull some more hair out in the coming days. 
2 Likes
@Shu if i read your posts correctly ā¦ when the NAT Table exhaustion happens nodes donāt manage to connect anymore (peer-count maybe going down (?) ) and in the log files there should be other hints like:
many
external address: new candidate address
or many
error: DialFailure
right?
with high peer count and just a couple of DialFailure errors per day a connection problem should be unlikely?
1 Like
No, the peer count likely wonāt go down, it will keep attempting to connect with new peers and keep rotating with the list. You will see a lot more HandshakeTimedOut messages once the NAT session table is hit.
external address: new candidate address ā if you are seeing this more than once or twice, it means your NAT port forwarding (outbound) is not setup properly assuming you went down the road without the --home-network flag.
DialFailures usually happen in the early stage off spin up off node, and may not be 100% correlated with NAT table exhaustion.
If you run enough nodes and exhaust the limit (verified), you will constantly be bumping against that limit until you reduce your # of nodes to a level that leaves some buffer room against that table upper limits.
Basically nothing in the logs is going to directly infer or tell you your router is maxed out, however, for me, HandshakeTimedOut on a huge scale were the leading contributor to realize āsomething is definitely wrongā.
5 Likes
does this count as huge scale for one node?
ā¦sometimes 10 minutes none ā¦ sometimes 3 within one second ā¦ (vps running 10 nodes ā¦)
/var/log/safenode/safenode1/safenode.log:39509:[2024-05-08T11:24:33.993090Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:39779:[2024-05-08T11:29:43.726502Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:39787:[2024-05-08T11:29:43.902645Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:39792:[2024-05-08T11:29:44.139863Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:39799:[2024-05-08T11:29:44.140091Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:39808:[2024-05-08T11:29:44.261521Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:40280:[2024-05-08T11:38:38.749474Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:40285:[2024-05-08T11:38:38.824499Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:40495:[2024-05-08T11:42:44.794445Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:40505:[2024-05-08T11:42:45.225569Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:40647:[2024-05-08T11:45:24.867523Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:40655:[2024-05-08T11:45:25.153777Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:40829:[2024-05-08T11:48:04.941793Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:40980:[2024-05-08T11:50:44.975932Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:41108:[2024-05-08T11:52:40.295243Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:41115:[2024-05-08T11:52:40.296777Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:41165:[2024-05-08T11:53:25.053362Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:41308:[2024-05-08T11:56:04.803038Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:41316:[2024-05-08T11:56:04.979001Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:41323:[2024-05-08T11:56:05.166787Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:43897:[2024-05-08T11:58:31.042291Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
/var/log/safenode/safenode1/safenode.log:43914:[2024-05-08T11:58:31.505192Z WARN sn_networking::event] Problematic error encountered: Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) }
or is this just to be expected because theres others out there not reachable ā¦?
2 Likes
I had definitely underestimated the complexity of that monster. I ordered it with the idea to have the capsman manage the wifi through the house, setup different vlans on different ssidās for me, my iot gear and for my extremely demanding endcustomers (wife and teenage girls) etc etc. Split the incoming traffic to different vmās on my proxmox box etc etc.
Then once you open that interface for the first time, navigate through the menu on the left, then a submenu opens, then you choose something and a new screen pops up with different tabs and all kinds of abbreviations no one in their right mind has even heard aboutā¦
So yeah in the end I paid a guy on fiverr. Argentinian dude that works at an ISP and knows these things inside out. Highly recommended.
2 Likes
Iām wondering if Iām hitting my limits as well now this is from one machine running 70 nodes started about 12 hours ago
grep HandshakeTimedOut /var/log/safenode/*/* |wc -l
678395
Both seems like a lot.. however, both of you need to find the router settings or stats page associated with NAT table entriesā¦ or NAT session table.. or NAT session statesā¦ (its the only way to confirm it)ā¦ the rest is just pure speculation, though might be accurate to some degree, but the official answer is you need to identify & troubleshoot your routerās metrics, 
.
Note: If the HandshakeTimedOut shows up frequently and you are without the --home-network flag, likely the router is the culprit, however, for me, when on a super high upper limit, I was still getting HandshakeTimedOut for --home-network flag (so that alone isnt a proper indicator), unless you become a public node (w/o --home-network and still experiencing this issue).
4 Likes
Worth noting for all home node runners.
My previous router max sessions was 16384.
Now that I am monitoring these numbers more closely 36 nodes and general household use has my active sessions is at around 11000.
I think it is very important that we all pay close attention to this and stop thinking more is better.
More is very likely the complete opposite of better.
7 Likes
How do I check these numbers @josh?
It is router dependent, you may be able to find it in there somewhere but I couldnāt tell you where.
Google is your friend or share your model if you want and I can try help you. DM if you prefer.
1 Like
thinking the average node-runner should be able to know or recognize this seems a bit of a stretch isnāt it ā¦?
is there no way the node can detect such a connection issue and give the node-manager/user feedback or a hint that something is going very wrong / exclude the node from the network @joshuef
ā¦i mean not even those swings between 100 and 500 peers seem unusual from what iām seeing ā¦
cloud node with 10 nodes on it:
local node - 3 nodes running all alone in my network with --home-networking (i canāt believe anything can be exhausted here ā¦)
3 Likes
What are you tracking there? Peers?