Node size considerations

What about another case for active management: Killing for example 20% of your fullest nodes periodically and launching new ones to replace them?

Motivation: if the clients choose to pay for the cheapest nodes, then the fullest ones don’t have a chance to earn. So why not kill them and launch new ones in hopes to get them in a better place?

Eventually it would lead to a more evenly filling nodes and that might be good thing for the network, right?

How is the network fullness, r in the equation actually gauged? Or are the nodes just trusting that their fullness is representative of the fullness of the network?

:100:

Yup, is already in there in fact.

5 Likes

Don’t see any advantage other than rolling the dice and fluke landing on a XOR address that currently isn’t closest node to many chunks, compared to everyone else, and then fluke chunks coming in where its in the closest node group for those new chunks. Its 1 in 2^x where x is between 30 and 256 (assuming like 2^25-2^30 nodes in network)

The network nodes will on average be around the same size and the fuller nodes are just the differences in random distributions and could be the least full by morning with churning happening around it. Basically a zero gain game.

The only time it might help is in the very rare case that you have a node that is a lot fuller than the rest of your nodes. That could indicate a faulty node or that outlier to the random distribution. And in that case it might make sense, but likely to never occur for most PCs with less then 1% of all nodes.

Network fullness would be gauged by taking your nodes and getting an average of them all and that will be close to network fullness. Its all in the random distribution of chunks that will determine how accurate that is.

The best algo for active management at this stage seems to be killing off faulty nodes and starting new ones to replace them. Otherwise the other active management is monitoring your key parameters of your setup, being CPU usage, Memory Usage, Bandwidth, and Quota used, and managing your nodes accordingly to keep those parameters within certain bounds.

4 Likes

If each machine is running 100s of nodes, isnt there a chance that a chunk gets replicated and exclusively stored on one machine.

Surely there is no closer XOR address than one on the same machine.

Or to put it another way: what does “close” mean here?

The XOR address of each node is random. So for 100 nodes in 500,000 node network you have a 1 in 1000 chance of 2 of your nodes being close nodes to a particular chunk. And at 100 nodes per PC then that is only 5000 PCs in the network. By beta we’d be hoping for 10 times that. Some people will be running more than one machine, especially in late beta/launch since the more technical will be there

Close means closeness of the chunk to nodes in terms of their XOR address

3 Likes

Not a chance in a megabillionzillion. Think infinite improbability drive while you sip your tea. :laughing:

For this to happen you’d have to have a significant proportion of all the nodes in the network, like >30%

2 Likes

thanks v much.
Just to make sure - there is no other metric to define closeness other than XOR address similarity?

None.

The 5 nodes with the closest XOR address to the hash of the chunk (its xor address) are the ones the chunk will be stored at

1 Like

my daughter actually brought me in a cup of tea as i hit refresh. :grinning:

2 Likes

Thanks. I didnt know that. Maybe the term “close” could be abandoned altogether then.

Since the XOR addresses are random, so is “closeness”, yet the term suggests geographical proximity.

It’ll depend on how you are viewing things. For the network the only thing it sees for location is the XOR address. For the network its landscape is the XOR space.

So in some respects it is perfectly appropriate to use close. But i certainly can see for those not so into computer networks, programming, maths will see close as geographical. Whereas for the network it has no understanding of geographical at all. Even IP address has little meaning, its just how packets are routed through the hardware of the internet.

5 Likes

The problem is creating an attractive visual representation (2D or 3D) of very very long line with a few points on it that represent your node address, the XOR addresses of chunks and the chunks that you are “closest” to at any one time.

Been doing a fair bit googling and asked some AIs. I didnt see anything that was anywhere usable in a Dummies Guide to XOR addressing.
And without a decent graphical representation then this is a major communication problem we will have and need to somehow address.
Yes, its easy enough once you get your head round it - just requires some fancy manoeuvres of the head first that are quite unfamiliar to the vast majority - and why should they be familiar? This concept does not raise its head significantly in any other field. We need to cut a lot of slack here and be patient in our explanations to n00bs.

3 Likes

Hmm… I might give it a try… Is the XOR space (and node address space) linear, or are the ends connected making them a circle?

2 Likes

This older Maidsafe article might help:

3 Likes

So here is two sketches. Not attractive yet, but are they conceptually correct? Every chunk is connected to 5 closest nodes. Every node is connected to various number of chunks based on how close it happens to be situated.

If the concept is right, you can give me any suggestions. In the beginning I’m mostly interested in finding out all the information we want to have there. Maybe number of chunks next to every node?

One thing is to find a right balance with the number of chunks and nodes so that the information we want to present is there - but not messy. It gets messy really easy.


Got to go to sleep soon though.

5 Likes

A couple more, this time in 3D. One level is populated randomly with chunks and the other with nodes.



Once I have the parameters right, it’s quite easy to make several different representations. And these can be made like 1000 times more beautiful, but that will take some time.

OK, I’m off to sleep now.

No, one more. Here the blue chunks denote a file.

All kind of comments welcome!

9 Likes

They look great to me. Good job.

2 Likes

The closest is a string and join the two ends to make a circle. Then the position along the string is the XOR address. And one end is 0xFFFFF… and joins to 0x00000… Since 0xFFFFF… + 1 is 0x00000…

XOR address is similar to binary in that it is the same representation and XORing gives binary anyhow. Really from the outside you could just think of it as binary and you’d be pretty right. XOR really defines an operation on 2 binary bits giving a resulting bit. 1.1=0 1.0=1 0.1=1 0.0=0 dot here us xor operation, I don’t see a char with x superimposed on a o

yes see my reply just above

1 Like

nope - a line from zero to scary big number, like a REALLY scary BigNumber at least in my pished brian right now.@neo has answered above and its usually a good idea to at least consider what Rob has said so I will - back later

2 Likes

That would be smart thing to do in this case

Remember that a chunk with hash of 0xFFFF…FFFE and nodes

  • 0xFFFF…FFE0
  • 0xFFFF…FFF0
  • 0xFFFF…FFF8
  • 0xFFFF…FFFF
  • 0x0000…0008

with none in between those nodes would mean those nodes are closest to the the chunk

Don’t ever expect nodes to be that close in real life LOL

4 Likes