How much disk space per node are you having?

Ant-Node (was Safe-node)
1

The official recommendation is to have 35GB of storage space per node.

However, this is not enforced in any way, and people can actually run as many nodes as their setup allows. I’m curious, how many nodes are folks running, and how much disk space they have allocated?

To be clear, I’m not morally against running more than recommended. Now I’m only running something between 0-12 nodes, and they all have 35GB assigned. With a network as empty as the current one, I would try to run much more, but my router is the bottleneck that prevents me even trying.

So, could you answer the following two questions:

  1. How many nodes you are running?
  2. How much disk space you have per node?
3 Likes
2

I thought we should allocate 64GB per node, not 32GB.

3

Launchpad sets it at 35.

1 Like
4

There was talk in other topics, I believe by Jim Collinson that said each node, theoretically, could use up to 64GB. Whatever it turns out to be, 35 or 64, you should really plan on using only about 75% of available disk space, though, to maximize throughput.

2 Likes
5

JIm said to me last week that 35GB was fine.

6

It’s all kind of confusing. This is what he said on Discord:

JimCollinson — 1/17/25, 3:35 PM

So chunks are not all the same size… they just have a max. But nodes commit to a max number of chunks. So in reality, they never get to 64GB

2 Likes
7

And I think that statement is actually inaccurate… Nodes do have active chunks and inactive chunks and they remove the inactive chunks only if they need storage space for active chunks I think… So the goal probably is to have 50 % filling level (so 50% active records) and currently the record count is multiples of the active records…

I’d assume with 50 % active records the total record count will probably max and this would mean the node would be 64gb in size… Maybe someone from the team might correct me here if that’s not the case @JimCollinson @rusty.spork

8

Unless I’m mistaken, it’s not documented on autonomi.com. In the absense of official docs, I followed the only official guidance I could find, 35GB/node from the launchpad. I’m running several thousand nodes, and all of them have 35GB provisioned. I could receive more rewards playing games, but I’d rather put the health of the network and sustainable ops first.

4 Likes
9

Well this is a double edged sword to use 64GB in calculations for storage capacity.

While technically true, practically if one node is at 50GB then there will be for sure one or some outlier nodes at 64GB (ie totally full) and a failing network in some regards. Not failed but failing although recoverable.

The 32GB arose out of the times prior to ERC20 where on average 1/2 the chunks were tiny holding the chunk store transaction. Thus 16K records at max 4MB would only be 32GB.

That was considered the average size a node could get to.

Now married with the charging algo from long ago continuing with charging near max amount at 60% records stored, and I see no reason why that wasn’t baked into the smart contract too with an adjustment being made to factor in the market price. Basically if the market price of the token remains the same the price will be near maximum at around the 60% mark.

This now means that at 32GB, the desired size of nodes, the price will be about half way to max. A sweet zone. Go much higher and the price jumps and in theory people will be incentivised to add more nodes, or start running nodes.

Thus for better honesty, in this one fact, 32GB (storage) as the size of a node is better figure than 64GB.

It is a risk if people try to run nodes at 32GB and use 95% of their disk. Yes, but hopefully people and the OS will be warning them long before they get to 95%. Maybe launchpad should be keeping an eye on that too.

@Toivo can I suggest you start a new topic for a anonymous poll on this. Maybe ranges like less than 5GB, 5 to less than 10GB, 10 to less than 15GB and so on.

Make sure you tell people to ensure they do not include the 20% free space on disk they must always keep free for the FS and OS to utilise

eg disk size * 0.8 less used space all divided by number of nodes.

EG a 2TB drive with 200GB already used and 100 nodes

space per node == ( (2000 * 0.8) - 200 ) / 100 = 14GB per node allowed for

3 Likes
10

6000+ at location A.

Not fixed, variable, filesystem can grow indefinitely provided I keep stock piling hard drives in advance. Currently, have 200TB of free space allocated to the 6000 nodes (> 32GB per node).

Storage space isn’t the bottleneck for me, its the hosts’ CPUs.

4 Likes
11

I think things like maximum theoretical storage requirement per node should be documented on autonomi.com. It’s really hard to properly provision for imprecise specs. Launchpad implies 35GB, now I’m reading 64GB and 32GB figures.

Network rules must be baked in at the protocol level, but it would help node ops do the right thing from the outset if it’s easy to know what that is.

5 Likes
12

Cannot really control these things at a protocol level as they are different beasts.

At startup it might be possible for a management program to advise, or limit numbers of nodes being started.

If try to do at the protocol level it becomes a hybrid thing of checking things out of its control. Like it needs to know how many other nodes there are running and so on

1 Like
13

The real limitation is the # of records in the record_store, capped at 16,384. With chunks being off different sizes, this can vary on what will be the average utilization off this folder (requirements), as it varies with the network size and data uploaded.

On the extreme end, its 16,384 * 4 MB (max chunk size) = 64GB, though that’s theoretical, the network should never reach that upper limit with continous re-shuffling and re-balancing off the data among all nodes.

6 Likes
14

How would what I suggested at Update 23rd January 2025 - #12 by born2build not discourage oversubscription. (underprovisioning)

15

This is good information and should be documented in a ‘node ops best practices’ page on autonomi.com IMO.

2 Likes
16

Some background on the LP 35GB limit:

Launchpad enforced a 35GB because in the prior testnets, I believe there was a spend record 4KB and a chunk record 4MB max, so for 2 files used in the record_store, the average was about 2MB per file.

Therefore, the suggested allocation size at that time in LP was 2MB * 16,384 = 32GB + 3 GB (log folder) = 35 GB.

Based on our production antnodes’ metrics, average file size in the record store currently is around 3.47 MB, and antnode’s relevant record utilization at .11%.

6 Likes
17

@Toivo I have 10 nodes running at the moment and out of the allocated 350 GB, and memory usage is ~1.5 GB, unfortunately still all nodes have reached 0 Attos.

1 Like
18

Running 350 nodes on 1 machine with 3TB and second PC is around 550 nodes on 6TB. Can’t afford to take the PC’s offline right now, have a 14 TB waiting to get added but that won’t be enough after some time after TGE. After upgrade, 1 PC will run 550 nodes on 14TB and second PC 350-400 on 9TB (3 + 6). Hetzner servers running 450-600 nodes with 16TB each.

2 Likes
19

I personally don’t expect this to happen in the first few years. Yes, the cost of storage may be very low, but the transaction fees are high enough to prevent massively casual use. I personally won’t buy hard drives until my machines reach 50% full, currently with 1500 nodes per machine it’s under 20%.

Check out the Dev Forum

2 Likes
20

My nodes still have not received any Attos, the 10 nodes were storing about 1-1.5 GB at first, then this value dropped to ~0.8 GB, and now it is about 1.2. Is the lack of Attos due to low allocation of stored data, or could there be another reason? After what time should the rewards appear?