[Offline] 50GB Static Testnet

[oetyng]

@Bogard

Why not go further with #2 above and have the network be also unaware of node storage size? If a node can’t store and provide the data that was assigned to it, then it gets booted. This would mean that there may be nodes on the network with 5gb or 10tb of storage and anywhere in between or more.

I think a better way to think about this limit than we have so far (by calling it max_capacity) is to see it as a trigger to expand the section, and ultimately split.

It means it’s a variable that Elders monitor to decide when it’s time to split (via allowing more nodes to join, to hit the 2 * MinSectionSize needed to split).
Having it hardcoded is just a way to decide upfront (together with MinSectionSize) how large the network tree will be relative to data hosted, or in other words approx. GB / section, nodes / TB, or any of the other view angles on the same thing.

So really, it’s not a limit on the node hardware.
Under the hood the node can pack trillions of zettabytes if it likes. We don’t interfere with that.
But with this growth trigger level, we tell nodes that they need to have at least this amount of space available to play, otherwise they are likely to be kicked out (when we find out that it is full, or can’t deliver what it is supposed to hold).

The max_capacity variable that we use in node config, should really be min_capacity.
Then we can have the Elders abstract a GrowthTriggerLevel to be, say 90% of that, for some additional variance margins.

---

I know that the trigger level could theoretically be dynamic (as well as section min size), but it’s far simpler to start like this (as you of course already know).

@Bogard

But it leaves the question of how to regulate node joins, which can be similarly addressed by letting any node onto the network as long as it does some useful pow first

Yes, it’s a good question.
The specifics of when to allow an additional node to join (i.e. beyond replacing a lost node) on the sections’ path to reaching the GrowthTriggerLevel (where we will make sure to grow until split), is yet to be defined I think.

There is currently (I think) one source: relocations (the source section replaces it with a new node, so that gives net growth of the network).
Do we need an additional source? Or can we stick to a single one for simplicity, and just work with the relocation rate to govern the network growth? Or would that be a coupling of needs detrimental to both aims (sybil protection and network growth)?

There are still some things to figure out.

But a join queue, where nodes pre-load what they need to hold to join in a specific sub-range of the section xorspace, could act as a useful PoW and trigger/barrier to allow nodes to join (and have them hit the road running as well). Probably a bit too easy as a sole barrier, so perhaps combined with some randomness and probability proportional to used space…

---

[joshuef]

@danda

I like this idea in principle, but given the high storage variance at present the estimate will be very loose indeed, eg 12G - 18G. otoh, if we could have a nice uniform distribution of chunks (as discussed above), then the estimate might be accurate to within tens of Mb.

It’s also worth noting that the decisions are across all the elders, so you have that variance / elder_count (i think). But yeh, more nodes in general helps us here.

[Bogard]

I like the design and the simplification that’s being aimed for. There’s a small wrinkle that might have to be addressed. It can lead to all kinds of considerations/complications, but I’ll try to keep it simple with the following rough sequence:

• Elders notice that they reached their minimum storage capacity and decide that the section needs more storage

• At least 5 of 7 elders agree to allow more joins until the section splits

• The section is preparing to split and the next seven oldest nodes in the parent section are elected to be new elders in the sibling sections

• All of the original 7 elders as well as the new 7 elders will likely still have their minimum storage at capacity and so will immediately vote in their respective sibling sections for more nodes to join and for those sections to split

• This would continue on in the majority of sections because elders and elders to be (who will be making the decisions) will typically have had their minimum capacity reached and new nodes joining does not change that minimum capacity at the elder making the decisions. It gets more complicated from here on out with potential side effects on network function (e.g., will elders mostly be busy trying to split?), security, etc.

To prevent this issue, there might have to be an increase in the individual nodes’ storage capacity along with the allowance of new node joins. This can be done while maintaining the simplified design. For instance, perhaps with every increase in node age there’s a concomitant and predictable increase in storage capacity. This would still allow for exactly the same properties being aimed for with the current design while avoiding the issue I tried to describe above. The only thing that would change is a small change to the arithmetic used to estimate total storage capacity in the section or to decide whether more nodes are needed by simply taking into account the age of the section’s members.

As nodes get older, their size might become large enough that the churn concern might crop back up, but this could be mitigated by tapering the increase of node capacity or it might not be an issue at all given that few nodes will reach such a size or hopefully by that point well after release other solutions would come along.

[oetyng]

@Bogard

and new nodes joining does not change that minimum capacity at the elder making the decisions

This here is what flips it.

Elders do get their used space lowered with every joined node.
Every new node reduces burdon by the new amount of available space.

So, to illustrate this, using an extreme case without any other join source:
Say there are 100 nodes after split.
Then a massive upload which fill them up, and Elders trigger the fail-safe to allow joins until split. (Assume insignificant amount of data uploaded meanwhile.)

That means the section will double in node count, and with fix size, this will see a halving of used space.
Then they split.

The new sections have 100 nodes each, with each of the nodes at half full.

[Bogard]

It’s possible that’s I’m missing something, but my understanding is that local used space of existing members wouldn’t change with new node joins while the global used space of the section will change. That is, in your example, the existing 100 nodes would still be full, the new 100 nodes will be empty, and the section’s used space will be half full. Unless data is being deleted from existing members with new node joins, which I didn’t think was the case?

Assuming there’s no data deletion, here’s how I see it. Let’s define:
A = Existing members of the section
B = New members of the section

The elders of the parent section are all A. The elders of the two child sections are all A due to node age. The used space local to A nodes does not change with the arrival of B nodes, even though the used space of the section as a whole does change with the arrival of B nodes. So A nodes, who are the decision makers, are still not aware of the change in used space for the section with the joining of B nodes because their own local used space didn’t change, and they make decisions based on their local used space giving them a sense of the global/section used space without consensus. A nodes’ local used space will be decoupled from the section’s used space with the join of the first B node.

But if every agreed new join of B nodes came with a local adjustment to the used space of existing A nodes, then the decision makers’ local used space will remain coupled to the global section used space, which they can calculate as a simple function without the need for consensus (or more precisely piggy backing on the consensus already done for membership).

With node age as the first parameter of the adjustment function, options for the used space parameter include:

• Increasing the percentage of filled local used space that’s considered node fullness for existing A nodes with every split. This could be the easiest to start out with, but its limit would be quickly reached in a very large, long last network with many section splits.
• Increasing the local used space of existing A nodes with every split. This size of the increase could be very small to start off.
• Any other?

This all changes if there’s data deletion from A nodes after equilibrating with B nodes.

[oetyng]

@Bogard

This all changes if there’s data deletion from A nodes after equilibrating with B nodes.

Yep, that is just now being added.

New nodes fill up with data in their range.
Old nodes clean up data they are no longer responsible for, but only when they have reached min_capacity.

So, that’s how we can keep the network growing without ever becoming full.

[Bogard]

Sounds good

[stout77]

Looking forward to seeing the churn happening in distributed testnet!
In the meantime, I hope you guys @maidsafe are enjoying the break, between a line of coding and the other!
Cheers!

[tfa]

@stout77

@dirvine

Think like this, Adults are like disk drives in a computer. Store stuff there and let it be. In a section they are almost append only immutable data stores.

Elders have massive amounts of work to do and possibly too much. They take care (atm) of all mutable data, transfers (safecoin), client connections, Adult penalties and all network agreements/consensus.

I only go by what was said in the past, when the decision was taken to have Elders not to store data. Although, as others, I didn’t necessarily agree, I thought it made sense if the load was to much for Elders.
If the optimisations that have been put in place in the meantime allow to bring back storage to Elders, happy days. But frankly I believe it’s premature; churn has taken down elders in previous testnets way too easly even without that, and that’s even without splits.

I remember my opposition to this change in 2020 and I welcome MaidSafe changing their mind again and intending to bring back storage to elders.

This is what I replied to this post at that time:

@tfa

@dirvine

Elders have massive amounts of work to do

If adults work is lighter, then continuing it shouldn’t be a problem in that case.

@dirvine

and possibly too much.

So a very powerful device is needed to be an elder and I like that because an attack to get control of a section is harder to implement.

Furthermore, I would find unfair that any requirement for elders be less constraining than for adults. It could be storage (capacity and speed), cpu, download, upload, …

I wouldn’t be alone, and the risk is a perceived injustice damaging for the network success if the requirement could be lighter in one of the dimensions (say for example storage capacity).

The only way to ensure this is that elders work is additive to adults work. If the work is different we can’t be sure of that or at least it might be difficult to convince people of that.

[happybeing]

@tfa

So a very powerful device is needed to be an elder and I like that because an attack to get control of a section is harder to implement.

Furthermore, I would find unfair that any requirement for elders be less constraining than for adults. It could be storage (capacity and speed), cpu, download, upload, …

Just commenting on the above. The first point isn’t a given IMO because if you reduce the population of potential elders in that way you don’t just make it more costly to run a potential elder node, you also reduce the time for such a node to become an elder because they have fewer competitors.

The second point seems more sound.