I wonder what will be the price on that
1 Like
large commercial. Not cheap at all.
And if his other commercial plans are anything to go by you also pay for any data over the undersized included quota.
Understandable since the satellites do have limits on how much overall they can carry and at what level others will be affected in your coverage cell.
100 nodes was able to bring the ISP network to a halt. We agree I canāt run 100 nodes but what if people in the community run 5 nodes here or there totaling 100 nodes we again arrive at shutting down the local ISP.
1 Like
I have run in excess of 2000 nodes from home. This restriction by your ISP is crazy.
1 Like
It seems to be a hardware limit due to overwhelm by small packets. Iām going to talk to them in an hour. If you guys have any questions post them and Iāll report back on what they say. Iād like to get the bottom of this.
The connection works great 100Mbps up down when I do a speed test. If I start 50 or 75 nodes it grinds down to about 1/3 of the total bandwidth. I canāt load web pages. Other customers connected to the transmitter also can load pages.
The connection is provide via radio access point.
Okay so they are refunding me the install and what Iāve been billed. Their system canāt handle constant traffic. 30Mbps of constant traffic ends up deprioritizing other customers even though they have the service as advertised 100Mbps Up Down no limits. LOL
Kind of wish I did not have a hole in my house from this LOL.
1 Like
Caught lying about the service they provide. 
do you have alternatives?
1 Like
Comcast which Iām not a fan of but it will have to do. Iāll get a Mikrotik router so I donāt upset the family. This should allow me to manage traffic properly.
2 Likes
Probably being spread out on many people (links to each person) the limit will be a tad higher than when its coming in on one account. Maybe a handful more, or not any.
@Josh Its a limitation of the wifi supplying the internet. There is no possibility to replace their wifi router since its supplying a number of accounts. It is a hardware issue and not set by the ISP deliberately, but by the last mile hardware they installed. WiFi is always sub par to wire connection.
That is one positive
A decent model will do you well. Mine has been great. I had decent internet with them, but now replacing their router with a proper one, Iāve had a much better experience and thats with running a lot of nodes.
Get a real cheap mikrotik and youāll wish you put the money into a better one. The reason ISP use pretty looking but shitty routers is they buy cheap, and cheap routers are cheap because they donāt have the power to be good. While a cheap mikrotik will be better than a ISP router, it still might not run the number of nodes you want.
2 Likes
Been one of those days when I question why I donāt spend my time fishing instead.
The old issue I had with the connection cutting out at a certain number of nodes has returned.
I am now more certain than ever the issue is at the ISP.
Nothing I do on my side will restore the issue.
It was intermittent before, my connection would slow to a crawl for 5 minutes and then recover.
After 2 weeks of no issues and me thinking that I have it dialed in and under control, one of my IPās slowed to a crawl after much time spent trying to sort it out, it recovered over night by āitselfā.
A few hours ago it happened to a different IP.
I called the ISP and got the stock response it must be a config issue on your personal equipment.
So I set it up on their equipment and got exactly the same result. This one IP is maxing out at 40Mbits when it is meant to be 2Gbit.
Called them again with the issue happening on their equipment.
Shocking surprise, now that they canāt blame me anymore. āThere is no way for them to troubleshoot only one particular IPā call back tomorrow and we will assign you a new block.
So did I fry something on their end?
Was really hoping that they would investigate and I could finally get a answer to what is happening.
7 Likes
Maybe if it wasnāt the ISP then it could be random attack on that one IP. Why I have no clue but maybe if you check the logs for indication of such an attack on that IP
But as you suspect the ISP has over provisioned their services and at certain times there is just not enough bandwidth to handle all the customers
3 Likes
Nah, they either have problem with load balancing somewhere or buggy SW on one of their machines.
SW bugs on machines for big $$$ are more frequent than you would expect and it is pain in the ass to debug, most of the time impossible to duplicate in lab and you donāt want to piss of your customers by playing with production machine where simple reboot means 20 min outage.
You can try traceroute to some internet addresses to see if the path is same for all your IPs, and depending on your ISPās topology it may show you multiple hops in their nerwork so you will see on which hop the latency jumps up.
3 Likes
I have tried everything. Direct attached server to ISP, OPNsense, VyOS on massive server as router, tuning conntrack on router hardware supporting millions of entries. None of this would support more than about 0.5k nodes. And while alive and maintaining sufficient numbers of peers, nodes seemed to be earning much less than expected. I guess due to packet loss in ISPs network.
Itās not really worth it to run a server at home with this kind of performance and itās a waste of a broadband connection because all other traffic suffers also, especially tcp.
Itās probably cheaper to run those nodes on a bunch of VPS instances. Each get their own IP, they are closer to internet backbone, UPS protected, easier to manage, and earn better. Itās still a nuisance to VPS providers, because the massive traffic looks much like DDoS attacks on their infrastructure. Fewer nodes per IP works better.
To make autonomi nodes (and users) look more āreasonableā, and to encourage running nodes from home, the traffic (hundreds of. connections) to storage (2GB) ratio needs to be reduced by increasing storage per node, by a factor of 10, and reducing number of connections by another factor of 10, if possible. What a difficult balancing act.
The idea of increasing the node storage size by up to 10 was one of the improvements to be implemented in the last update. Obviously other fixes took too long and the size upgrade was one too many factors for testing purposes. The 10 times increase was suggested in a few discord events ago, I would expect them to be testing this before beta program is over.
I feel though that the connections per nodes is high because it needs to be for creating a web of nodes in a decentralised network without making the network super slow. Also keeping connections with 100+ nodes protects against the network becoming semi segmented. Just take an extreme thought if each node only connects to 5 others, you could easily end up with a segment that doesnāt know there is more than 5 nodes. By increasing the connected nodes then the probability of segmentation reduces, also more connections reduces the messaging needed to find the nodes responsible for a chunk.
What the optimal number of connected nodes is more of a mathematical problem to worked out. There is a minimal number but that would not account for multiple nodes going offline in a small period of time. So you need more than the minimal. So whatever mathematical model they use would also need to include factors for that.
1 Like
Now we are cooking. I was told i cant remove their iSP supplied hardware. After some excellent support and product advisement from fs.com that pos got removed. initial tests seem to be fine with upwards of 5000 connections. il give it a little while and push it to 250k and see what happens.
Il keep yas posted, if anyone cares.
6 Likes
I find that if you call back often enough you will eventually get someone capable of trying to help.
3rd call today got it escalated to level 2 support who did some backend troubleshooting, confirmed there is a issue with this particular IP, butā¦ we first need to have a tech come out because they canāt understand why.
Frustrating as it is, how can they trust a customer when 90% of their own tech support donāt know what they are doing.
The first one yesterday offered a wifi extender as the solution, after explaining that it wonāt help and why, I was told that she is just trying to be helpful
2 Likes
I had 15 staff ish, in office, technical support, business support, tell me its not possible. My conclusion is everyone is full of shit.
2 Likes
2 Likes
Currently seeing around 100 errors per second on 300 ish nodes. connections seem to be holding up but its slow and a lot of shunned nodes.
also pulling up the status drives to nearly a complete halt. or crashes.
1 Like
Iāve skimmed this topic (I had no need to check here until yesterday, sorry if I missed things in the skim) and only saw you with a UDM. Iām going from 500/25 to now getting 2000/2000 next month. Iām already in the unifi system and wanted to stay in it. Was also looking at the UDM pro max or SE. But itās sounding like youāre still having issues.
This post in particular makes it sound like you have not been able to pinpoint the source of the problem? Or you do actually believe that itās an ISP hardware limitation and the UDM is not the limiting factor?
I plan on getting something that will handle 2-5 gig bidirectional regardless, (whether it will handles the number of nodes I want or not, I have other [personal, not mission critical] things Iād like to move off VPS and bring to my local network) But Iām generally curious your overall impression of the setup - even though itās obvious youāre having trouble with the nodes themselves.