With just 5 nodes running I have noticed earnings start to show up after such a change, (I have oscillated between HOME and UPnP changes to see the difference, there isn’t any) closer to 36 hours than 24 hours, as stated earlier because the network has grown so big and the faucet output has not been increased,
so that makes sense, not seeing earning until 36 hours
given the network has more than doubled in size since the faucet output was set and organic uploads are definitely not keeping pace with the growth of the network because of the Dave (Android and MACOS)mobile only upload client or ‘cli only’ options being the only way to upload,
the latter effectively excludes/reduces the potential number of ‘uploaders’ by not including those of us currently running desktops and being node launchpad operators.
(Linux, WIN10/11 and Mack Desktop operators avoiding the ‘cli circus’.)
Why is this such a hot topic right now, hot enough to have two threads on it?
Folk that run many nodes will manage those nodes with care so I fail to see why it matters. Nobody who goes through the effort to take this seriously will have their nodes run out of space.
Its our form of managed speculation…, sort of ‘Italianesque’, live for today, to heck with tomorrow (because today is reality)…, I don’t think anyone should worry about this, its stimulating network growth, readying for what will be a reasonable tsunami of uploads thereafter (because the big capacity means a low upload price.)
The likely outcome? imo ‘they’ will stay on their toes ‘and hang ten’ to stay ahead of that wave, it’s in their best interests.
Its more a problem for the very large farms which can see 2000 to 5000 nodes fall over all at once and this is not good even on a 250K nodes network. Lots of churning and even a small potential for lost chunks.
As an example I restarted my SBC on the starlink connection and decided to reset the nodes and add 80 with 40 on each of 2 eth addresses. Did that late last night.
Come back in the morning to an unresponsive SBC, no I/O response except seeing the mouse move and the system monitor frozen. I was running 75 nodes just fine, but 80 just was too much and locked up the system. I am assuming that since the system monitor before freezing showed 100% CPU, 90% RAM used and no swap, that there was a cascading situation where some events caused 100% CPU and that just ended up backing up processing from other nodes which then needed even more CPU and so on.
Going to try that again, glutten for punishment, with 35 nodes on each eth address and see if it collapses again
Point being over provisioning under resourced nodes on the system can cause fast collapse and for large farms this can be an issue. And maybe caused by as little as a few nodes requested to send chunks at the same time, or a hiccup in the internet connection.
I predicted this situation and mentioned it to the team the first time it was announced we were going from the 2GB node size to 32GB ave/64GB sized nodes. That was so long ago. I have no problem with not having enough storage to cover 32GB per node, especially in the early network, as long as the people are given the WARNING about risky activities. Let them do it with eyes wide open.
Later on though in the life of the network the average size of nodes will be much higher and this having too many nodes for the storage will not be occurring as much. But by then we also should be having a lot more people running nodes to compensate.
How do you know? It is entirely possible that we will see the opposite if the network is successful. For example, in Bitcoin we have a constant growth of the hash rate. I personally would not be surprised if the resources in the network grow at a faster rate than the usage and the stored data.
Say someone is running 200 nodes locally (feel free to make the calculation for more local nodes) - network needs +15gb - I define suddenly as within half an hour.
15gb * 200 nodes / 30 minutes == 222mbit bandwidth usage for that half an hour
Why on all earth do you think disk space is the bottleneck in such an event?!
(and in parallel we’d see skyrocketing storage prices - I’d spin up a dozen cloud machines in that time span that have plenty of bandwidth and can compensate more than those 200 local nodes…)
I don’t know really, how fast is the replication, when a portion of the network goes offline?
I think in December we saw a collapse of the network, when a large node runner went offline, and the rest of the network couldn’t cope with the data.
But did I understand right, you say it’s more about pipes getting blocked, than not having the space to have that data?
Whichever it mostly is, I think that thin margins contribute to the problem. If a large number of nodes are run by thin margins, they will go out with the sudden addition of data, and that has cumulative effect for all the rest of nodes etc.
Do you have an automated system to do this, or are you monitoring your nodes all the time, or how is this happening?
What do you think is an appropriate margin to have per node?
For sudden events I would look more to undersea cables faults, country goes off line due their “kill switch”. Churn happens causing other nodes to be used for keeping 5 copies
In December churn was super cpu heavy and many systems ended up completely blocked (possibly hundreds of nodes leaving therefore at the same time with all their storage space) which had a cascading effect… Nonetheless the network survived every time (and the majority of my cloud machines made it through without issues - and having learned from the first dips even my big rigs survived the last drop… )
Maybe it’s simply okay there are different kinds of farmers and we don’t need to protect them from themselves…? For home runners it’s most convenient to just install huge disks… For cloud farmers it’s just some clicks/terminal commands to spin up a couple hundred/thousand nodes temporarily when there is the need for it
I’m not at all into protecting any individual node runners. As long as their problems are limited to their setups, it’s all fine. And even in the worst case, they are probably up and running again within 24 hours.
I’m, just worried about the network as a whole. But you seem to think that thin margins are not going to cause the scenarios I’m worrying about? I’d be happy to be convinced that it is indeed the case.
On the other hand, If home runners have just say 30GB free, it’s more profitable to run 10 nodes instead of 1.
I think the needed margin very much depends on the strategy and I’d leave it to the runner to decide …
If a home node runner over commits and runs 10 nodes instead of 1 that won’t risk the network
Larger runners will all have a strategy to protect their earnings (==uptime of nodes) … Cloud machines cost by runtime… Dropping out and not earning when the network fills would mean they’re loosing out when it’s the most profitable period… It’s in their best interests to have a strategy for this and act accordingly
The heterogeneous structure of runners will protect the network…
As a general principle, that sounds true. In our case, how exactly does the variance of margin of disk space lead to robustness? Especially how does it make the network more robust to have a portion of nodes to have thin margins?
I meant that people tend to buy too much disk space… My laptop drives are filled <50% and have been for many years now… Disk space is too little of a cost factor to be cheap on it…
I’m always running out of space on my main, work laptop, and on my phone too. Only a hobby Linux laptop has a lot of space.
But if people in general really have so much excess space, then no problem!
But one question still remains, namely what is to be considered “overprovisioning”? I personally find the current pre-defined 35GB setup in Launchpad to be way too large.
That’s noble, and long-term rational way to go. However, I don’t believe that could be taken as a way people in general behave.
Not yet. But if the network becomes a hit, I expect that it will become a matter of getting an alternate Launchpad, made by unofficial parties. It will let you set the GB per node what you like easily.
My point, behind all of this, is that a successful and robust network should be designed so, that the self-interest of node runners always aling with the general good of the network. And that should apply for long and short term. And if that is not the case, then the behaviour the network needs, should be enforced by other means, like verifying that a certain amount of capacity is there.
