there was an issue discovered by Josh when running nodes (same hw, same internet connection, same everything … just a long ethernet cable)
… somehow I cannot post it because I posted the same content by accident in the wrong topic …
EDIT:
ofc by now everything could be different … not sure what changed within the last weeks and if that is still representative …
I have a feeling that the problem might be with the ISP, server halls like Hetzner have no ISP in the way, from home data still needs to go through the ISP and that might have some impact and limitations.
Maybe @anon26713768 have some thoughts on it.
This is just speculating, but this is what I think might happen. Please correct me if this is cannot happen:
When time passes, there will be more node pairs which will shun each other, but not the complete network.
For good connectivity nodes (ie. cloud servers), this will happen less often.
I am currently getting 11-15 ANT rewards from 2.4k nodes. 5 physical machines + 2 VPS, all running since start with no restarts.
This could be about quality of BGP connectivity and peerings. Hetzner has motivation to have good and reliable peers everywhere. Your average home ISP needs line to Google, Meta and few others and for the rest the cheapest option will do. Not many customers will complain that communication to random addresses on the other side of the world is slow, has packetloss and sometimes doesn’t work at all.
Yes, this is very important and hard to pinpoint. Also HW plays a big role - CPU caches, RAM latency, network card and its driver.
Another thing is OS tuning. Defaults are for general use on average HW, it may be highly ineffective for running thousands nodes on multicpu server. Sometimes changing few kernel parameters can make huge difference.
Btw has somebody tried setting nodes with fixed CPU affinity? In theory it should reduce CPU cycles needed for context switching, especially on CPUs with multiple core groups or multi-CPU systems.
I don’t believe the lenght of the cable was the issue in term of latency. Probably the cable was bad, not bad enough to loose link but maybe packetloss, malformed packets that need to be resend etc.
Just to note that the same setup was working fine on a much heavier load for the last 2 months plus it was working fine the first few days and gradually reducing the token yield to zero, it hasn’t stopped abruptly.
Check out the Dev Forum
Have the nodes, that earn less, actually been shunned? Can you see that from the logs?
Also, when talking about “earning 50% less” (for example), I think that the only way to make that statement comparable to any other number it must be:
50% less in an exact same UTC hours, than the other nodes it is compared to.
Also, I think we would need to see some good statistical analysis about what kind of variance is expected within certain sample sizes.
But, the most important question, has the nodes been shunned or not?
heavier load meaning more nodes or just %cpu/mem/hdd activity at a lower node count …?
because the context switching stuff might be mainly process == node count dependent if that causes some issue …
Just to add some info I have a hetzner running 1k nodes earns about 10 ant a day and home nodes (multiple machines running from 50-200 nodes each) earn between 1-2 ant per day each so pretty equal for me.
Until my mining stopped, I kept the same amount of nodes - 3000 per machine. I then scaled up to 6000. Now on 1 machine I’ve throttled their initial launch to 1100 nodes and will monitor how it plays out.
Check out the Dev Forum
I think they are not completely shunned. If I have understood it correctly, it’s not like some node is shunned or not. It is shunned to some other node/nodes, not necessary to all other nodes.
Since the nodes are spread all over, there are quite much always some node pairs shunned.
But this is speculating, maybe we should try to prove it true or false? Either by searching the logs or by reasoning based on the real behaviour of the nodes?
~100 nodes, 1.5 Ants / day (4 x 0.381 ANT), some variations day by day. One old laptop, 2 addresses.
Not really other than my finding that a group of machines on a very long ethernet cable were earning significantly less.
I observed higher ping times and attributed it to that, Rob suggested packet loss but I couldn’t be bothered hunting the issue down and just moved them closer. Solved the issue
This could easily drop the connection from 1G to 100M. Or even to 10M..
Maybe, still have the cable. Cat 6 but copper clad I was too tight to get solid copper. Should have at that length.
For interests sake I’ll run a speed test on it later today.
In my experience UDP packet loss (between say my cloud VM and home server) can be quite bad and lead to significantly reduced earnings, all while avoiding shunning. Further, UDP packet loss also depends on ISP’s setup, no matter how great your own router is.
Has anyone done packet loss testing? What is typical UDP packet loss between home nodes? 1%, 10%? Things are clearly better in a datacenter than at home.
I considered monitoring packet loss to optimize number of nodes per IP but even just measuring the loss was tricky. I hope that ant nodes will become able to measure the quality of their connectivity to the network using metrics such as autonomi message loss rate.
Yes.
Formicaio gives node-specific reports, how many other nodes have shunned it. I wonder if these low earning nodes have gotten some shuns by other nodes.
I don’t have any nodes right now, but I used to run 6-8 nodes for almost 2 weeks behind a crap(ish) router, and didn’t get any shuns. Oh, but I didn’t get any records either, so of course there is no shuns… 
When the network is empty, there is apparently little chances to know how well the connections work, because you could be out of reach, but not getting shunned.
There would definitely have ben something else, quality of cables, too far for Category being used, faulty cables.
The distance of a network LAN cable is so small compared to the distance to the ISP that 10 meters or 20 meters mades zero difference.
But low quality cables then 10m and 20m will make a big difference due to errors. Thats why those using smaller cables often do better, because of the cables themselves not the lag due to distance. nanoSeconds in extra lag in your home will have no, zero, material effect on the network.
Also the network is not supposed to be phased because a datacentre saves 10 or 20 mS of lag time.
In my home I have quality cat 7 cables even though its only 1Gbps being used for the most part and they are 20 meters from the router, router through a patch panel then 20m cable to another switch. Below the max length for cat 7 and well within the specs for 1Gbps
I can definitely tell you the lag time due to a LAN network cable has zero difference. Its not lag time but quality of cable or it could be faulty, or just toooo long for the specs (equivalent to faulty cable)
Nodes do not consider IP address other than it is needed to get to a peer node. The only address that affects connection decisions is the XOR address of the peer/node
