Non-persistent vaults

[BenMS]

@anon40790172

CRUST in action! I saw the video by David, talking about CRUST and that it detect’s if a node goes offline in milliseconds. Really great! So while walking around on the streets, you connect in a MESH-way, maybe to some others only 2 or 3 minutes. In that time CRUST will use it to PUT/GET a couple of Chunks, while at the same time it uses connections to others as well. So if one goes down, no problem, it will use the others that are still live? Really magic, and now I really want to see the stuff in action!

As a client you only need to keep minimally one connection to any SAFE node (always better to have redundancy though). Any SAFE node can act on your behalf and transfer your requests. So as you move around through the city, the only thing you need to do is send a tiny message to your client managers, every time informing them where you can be found next; such as switch would be almost instant. This all has to be fleshed out but the possibility is certainly there. And this is without mesh-networking in some sense; it would require ideally SAFE software on routers to act as a SAFE node and have an open WiFi signal where traffic is encrypted from the SAFE client side (and as such unreadable by even that first WiFi router) instead of the ridiculous notion we currently have of encrypting the signal with a router-set password…

You could be paying that router safecoin for the data used, and there you go: worldwide internet access on the go without data roaming charges:) - if life was only that easy LOL

---

real mesh networking would be a different story, where CRUST indeed would have a new protocol that is a mesh-networking protocol. The beauty of CRUST in the SAFE network here is that a destination is named by the overlay network with the 512bit id, regardless of how it is connected. So we currently rely fully on the IP network, but it would be imaginable that you store on the network that you are reachable from IP-to-your-local-meshnet at bridge-points X, Y and Z. This would allow routing to jump from IP to arbitrary meshnets and back. This again is future music, but without a consistent overlay network that SAFE now offers, this is even not really worth considering.

[happybeing]

@BenMS

Those cooling costs evaporate if you are a desktop or a laptop, by a user). So from that we’d expect that an average vault could easily go up to 500GB for desktops and brought down by say 50GB for laptops (both smaller disks and effectively reduced by being offline more).

Thing is it is very hard to get a handle on this from thought experiments, and even testnets. So we can bat this about indefinitely! For example, in your quote, very few desktops or laptops will ever store that volume of chunks because vaults are no longer persistent. Only vaults that are on for a long unbroken period will accumulate a lot of chunks, and at the moment its hard to see who would accept the cost of this, given that its not clear that this behaviour will be rewarded significantly. I think this because +20% of average farming rate is very low, and upping the global rate rewards everyone, so no incentive to be online constantly from that. If the reward is only +20%, I think it will be hard to attract archive vaults. So I’m concerned we won’t have enough archive vaults, and that this will affect data security. Maybe less serious, it will be easier for someone malicious to dominate this aspect of the network, and to use it for a big DDoS now and again. Yes that’s costly, but I’m not sure if we’re making it costly enough

[BenMS]

@wes

I’m curious about what happens to very small (phone) vaults that never gets turned off. My phone rarely ever gets restarted. It also has very small storage. If it becomes an archive, it’s small nature means I’ll get very very few hits… Would this be an incentive to turn the device off more? If I chose not to turn it off, would I be compensated more for a hit? Have I missed something?

It seems like small storage space would have an advantage to restart vaults periodically so they are more likely to have “fresh” data?

It seems like the only advantage to being an archive in 1.0 is being big.

Your phone might not ever be turned off, but it will loose signal occassionally. So we are currently not yet at a stage where we think it is reasonable to run a vault on a phone; in fact phones are the main reason why clients are brought into the game; they don’t route data, perform network tasks or store data. A client simply uses the network services at minimal bandwidth and power cost to the client as possible; the network carries the burden here. You offset that though by providing a vault at your house that earns net safecoin for your phone to consume.

I would say as argued above, the main incentive for increasing your safecoin farming rate is exactly being online as much as possible, where the size of your optimal vault is determined by the average bandwidth and how long your computer stays online on average.

[BenMS]

@happybeing

Only vaults that are on for a long unbroken period will accumulate a lot of chunks, and at the moment its hard to see who would accept the cost of this, given that its not clear that this behaviour will be rewarded significantly.

The network is not helped by oversized archive vaults. It is funny how quickly and repeatedly it is forgotten that farming from the start is designed to be self-sustaining farming: ie you are motivated to farm as much as you consume. Farming was never designed to be a feeding ground for profit margins and ever growing farming companies, feeding off the majority of network users. The benefit for our early adopter community, as described from the start is that the farming rate at the early days of the network is exponentially higher than later on, and as such helps to jump start a good network size; that these safecoin prices are pushed up by increased demand for network resources (not plummeted by an excessive over-offering of big archive vaults).

And importantly, Secure Access For All, also contains Access For All: that implies that the network should be free and fully accessible for all people who just open their laptop to use it, and close it again half an hour later. On those 30 minutes they should have roughly earned what they needed, aiming for a balanced network.

So yes, we will advantage your grandmum who only knows how to turn on her computer, and has never even heard of farming - it just automatically happens in the background without her worrying about it, and she can keep using the network without logging in to a safecoin exchange.

And yes, we will advantage a rural schoolgirl in India that now can get a modern phone and access to the worlds information because her phone pays for itself while it is charging at night.

To just throw two melodramatic stereotypes out there! But we are not joking, the idea never was and never will be to create a small elite of safecoin millionaires for just turning on your computer. If this disappoints anyone of you, then I deeply apologise, but maybe go and invent your own bitcoin.

So yes we keep a margin for profit to stimulate net-contributions, but these should be shared by the many and not the few. If we get this balance wrong then safecoin will simply act as a centralising force sucking the whole network into the controlling hands of a few. So the reality of this world is that the majority of computing devices are not always online, and might not be dedicated to safecoin farming. The network is designed for that ‘quid-pro-quo’, and the non-persistent vaults not only make the network more secure and robust, it also just favors normal behaviour compared to your usage.

As a dedicated farmer your challenge will be in trying to be online as much as possible, to get a net profit margin. But we would be wrong to allow the network to start depending on dedicated farmers. The network needs to rely on the contribution of all nodes.

[happybeing]

@BenMS I think you are misconstruing my question here. Maybe I overestimate the importance of archive vaults to the network, but it is my understanding that the network wants and needs (and will therefore need to reward) vaults that store data that is not accessed often. This hasn’t been detailed, but its the impression I’ve gained from following David’s writings on this area over many months. I’m sure he talked at some point about the “need” to find a way to reward vaults that may accumulate large amounts of data that doesn’t earn much, obviously because this is an important function for the network, else it wouldn’t be a “need.”

Since then we recently shifted to non-persistant vaults. The concern was raised that when they go offline this would cause lots of traffic. The comment was (I think made - or maybe just occurred to me, I’m not sure) that this will be ameliorated by archive vaults and that it would therefore be good to have rarely accessed data stored on these.

So as you can see, my questions are not about my concerns for earning lots from setting up my own archive vaults. They are about how the network will function in this new setup. I’m seeking information about this change and how it alters our thinking about the network. I’m surprised to read you interpreting this the way you did, rather than respond on the technical implications, which is what I was talking about. Incentives for all needed participants need to work, or the network won’t have the resources it needs.

[dirvine]

@happybeing

Since then we recently shifted to non-persistant vaults. The concern was raised that when they go offline this would cause lots of traffic. The comment was (I think made - or maybe just occurred to me, I’m not sure) that this will be ameliorated by archive vaults and that it would therefore be good to have rarely accessed data stored on these.

This is the case, pseudo [harsh term by me] persistent vaults (old way) still have this issue I feel. The new way says right up front, sort this issue and do it quickly (not in weeks but months). Archive nodes will be very important and will have a different profile, they may be distributed (so large chunks across vaults). The network will balance what is in archive and is not. This will require a similar algo to safecoin network rate balance (as equilibrium is the target).

It’s a tiny issue/concern what the algorithm looks like, it is a balancing thing and has to be simple. It will tie back to the Farming rate as well so have a double input to balance the reward.

This is a simple thing really but requires we get it right (like safecoin algorithms). IT will look different to existing vaults and we are already considering Archive persons as a type of NodeManager persona form the language of the network. Hope that helps

[happybeing]

Ok, so early days archive vaults are not really catered for, but will be in time for them being needed (hopefully) :-), and this will require special treatment (incentives), which makes sense. I guess this was always the plan but didn’t appear in the new discussion of non-persistent vaults. Thanks

[Seneca]

What confuses me concerning vault size and centralization is that it’s possible to create multiple vaults on the same machine right? So the reason for the sigmoid function in the farming rate is decentralization on the network level, in XOR space. I don’t really see how it decentralizes in the real world. If person A has one vault of X GB on one machine and person B has twenty vaults of X GB on one machine, then - all other things being equal - person B will earn twenty times more SafeCoin for serving twenty times the amount of resources than person A .

But I personally don’t consider this an issue. Honestly, I think we’re overly afraid of centralization because of what happened to Bitcoin. Yet Bitcoin centralization is an extreme case. SAFE will neither have ASICs (I’m pretty sure farming requires Turing-completeness) nor mining pools. In Bitcoin, ASIC’s hardware costs are about half of GPU mining hardware costs, and more importantly, ASIC mining is almost 20 times more power efficient than GPU mining. In SAFE, people with dedicated hardware will be maybe 20%, maybe 50% as efficient? Maybe 100%, though that seems unlikely, considering dedicated cooling costs. The centralizing effects of that are incomparable to the more than 2000% difference in Bitcoin.

[Caleb_Allen]

@Blindsite2k

I don’t know of an ISP around here that doesn’t cap internet download usage

Damn, that’s horrible. I guess I’ve been taking Google Fiber for granted.

Capping internet is horrible, it would completely change the way I used the web.

[DavidMtl]

That’s right but since it only takes 1 disconnect to reset it all, investing a big amount of money for disk space becomes more of a gamble since it takes a while to fill up whenever you get disconnected.

[BenMS]

@happybeing

The concern was raised that when they go offline this would cause lots of traffic

This is maybe where I lost some of you guys from the start:

CASE a) persistant vaults
network keeps 4 live copies at all time; up to 16 dead copies on offline vaults
one 1TB vault goes offline → the network needs to verify that one million 1MB chunks are still redundantly stored;
that same 1TB vault comes back online 5 minutes later because Windows forced a reboot → the network needs to report that those 1 million chunks are available again, while for a fraction of them, redundancy copying is already ongoing anyway, simply creating too many copies.

CASE b) non-persistant vaults
network keeps 4 live copies at all time; can’t be ars*d what happened to those offline vaults
one 100GB vault goes offline (a big one by this discussion) → network needs to verify that 100.000 1MB chunks are still redundantly stored, and for some fraction start copying.
that same 100GB vault comes back online 5 minutes later after Windows reboot → the network says ‘hi! oh and thanks for helping out. here are some chunks.’

Question: which one causes less data traffic and less network overhead?

Not only that, but it also does not affect your incentives/profits as a dedicated farmer. What it does do is force you to be a reliable node if you want to host (and get paid for hosting) 1TB of data. The reasoning is really simple. As a farmer you have a silent contract to the network: you provide net services to the network, the network rewards you. If you reboot/drop connection, that causes a burden on the network (read above), so why should the other nodes pick up your slack? Instead with non-persistant vaults, there is only an added pressure on you to be reliable to the network. The more reliable you are, the more safecoin you can farm. You will not get paid for showing up with a big disk, sorry.

There are some really interesting points here:

• the coupling of supply and demand is much tighter; this in turn pushes the price of safecoin towards the actual usage, making it free for all by earning as you spend.
• occassionally you always have to reboot your machine, especially if you are a home user. For a persistant vault you’d have to worry about protecting your private keys and backing up your stored chunks. At this point we’d have an automated turk-network, not an autonomous network. The network needs to manage every single key, and even as a farmer you should just be able to turn off your machine without losing a moment sleep.

[BenMS]

@happybeing

I think you are misconstruing my question here.

No, @happybeing, I know you’re a true hearted supporter of the SAFE project, so I just use the opportunity to speak to the many, and let off some steam

[BenMS]

@Seneca

What confuses me concerning vault size and centralization is that it’s possible to create multiple vaults on the same machine right? So the reason for the sigmoid function in the farming rate is decentralization on the network level, in XOR space. I don’t really see how it decentralizes in the real world. If person A has one vault of X GB on one machine and person B has twenty vaults of X GB on one machine, then - all other things being equal - person B will earn twenty times more SafeCoin for serving twenty times the amount of resources than person A .

all things equal and 20 vaults on the same machine, those 20 vaults of person B need to share that one CPU and same bandwidth. What I’ve increasingly realised is and what has always been mentioned is ‘the resources provided’, that includes disk space, but also CPU and bandwidth. So if these are out of proportion, maybe the realisation I have is just that to provide abundant storage (because disk space is cheap) we have to actively work on archive nodes, more then they already exist as PMID nodes.

Im am pretty convinced though that the model for the PMID node is essentially that of the archive node.

[jreighley]

Southern CA doesn’t have reliable power…

It is what it it is… But often the “keep the power on” is outside of the users control… Even if the user stocks up with UPS’s - the upstream routers may or may not have them…

If you have an outage at an ISP level (like Comcast) or a backbone router, it seems like a pretty ugly storm could ensue… But with persistent vaults wouldn’t you get the benefit of extra copies on machines known not to be effected by certain known centralized points of failure?

It seems like a semi-persistant system would be better all around. Unreliable connections ought to be non-persistant, but those that are 95% or more ‘on’ ought to be considered pretty valuable…

[janitor]

@Caleb_Allen

I guess I’ve been taking Google Fiber for granted.

Do you think Google Fiber will remain cap-free forever?

[anon40790172]

@jreighley

It is what it it is… But often the “keep the power on” is outside of the users control… Even if the user stocks up with UPS’s - the upstream routers may or may not have them…

Buy these Tesla Power units and put them on your wall. It’s good for my stocks But seriously, power outage sucks. But how often do they happen?

[anon40790172]

@janitor

Do you think Google Fiber will remain cap-free forever?

Yes they will. They’re always talking about net-neutrality, it would give them a really bad name if they would cap things. And by the way, networks are becoming faster and faster, and all these videos are not becoming bigger and bigger. So for Netflix to deliver 4K to over half of it’s users will take some years (people need to buy 4K televisions). In these years, Amazon has dropped prices for data again and again. Netflix uses AWS so they can deliver way more at lower prices. The great difference between Google, Amazon and the old cable-companies is that the last ones are very lazy making easy money. A company like Amazon is disruptive. They make things faster and charge you less. Google did the same with email. I was used to have 5mb. storage for mail and than Google provided me Gigs.

Here in Holland I have the same provider for years now. They disconnected some users some years ago because they were downloading from usenet 24/7 at full speed. But for the rest, no one got any caps for years now. That’s almost 15 years since I got cable at my parents home without any caps at all. That’s gonna be the future.

[janitor]

@anon40790172

They’re always talking about net-neutrality, it would give them a really bad name if they would cap things.

Losing money on building infra for anonymous users (that is, SafeNet farmers and users) doesn’t seem like a way to get rich (not quick, but ever).
Either everyone will get crappy service, or the T&C’s will have to change.

[Traktion]

Lofty and admiral goals, but if the econics don’t work, no one will get to use safe net.

Big players can use economies of scale to reduce costs. While this may centralised storage, it also makes it cheap.

Ultimately, the network needs to both be cheap and highly distributed. There is a delicate balance to be struck. Idealism will be beaten by reality time and time again.

I am sure this is all being considered, but the economics are critical for the success of the network.

[Traktion]

Look at what makes the best data server and we can observe that ASICs can’t compete. If they could, we would have server rooms full of them.

Ofc, it begs the question over whether massively distributed, commodity hardware and connectivity can outweigh data centres filled with specialised hardware. Moreover, it makes me wonder how this dynamic will change as mesh networks become more popular.