Decentralized Cloud Storage

Beginners
1

Hi all,

I am a newcomer trying to understand how would a storage solution in the Safe network compare to standard cloud storage. Here go a few questions. Forgive my ignorance. There is a sea of information out there and it is hard to dig all this info.

  1. Tokenomics. From what I read, Safe network provides immutable permanent storage. How is that achieved economically? I mean even if a fee was paid for the initial storage. How can a farmer store something permanently without regular payouts like it is common in centralized cloud storage?
  2. Immutability. To what degree is data immutable? (E.g., how many shards, etc)
  3. Timeline. How far are we from being able to use any form of Safe Network for commercial storage? I.e., reliable enough to market it to companies. Months? Years?
  4. Is Safe Network blockchain-less? One of the problems with other immutable storage solutions like Sia is that the end user need to sync a blockchain.
  5. Storage capacity. If demand for storage increases, do nodes also increase organically?

Thank you very much!

8 Likes
2

SAFE is a lot more than a cloud storage system, but would replace a cloud storage system.

The network is based on people supplying their spare resources which equates to a near zero cost to the farmer to supply resources. The farming rewards will cover the insignificant cost plus provide a small profit.

  1. The storage fee will be set by the network and the algo will take into account that it is permanent. For the last number of decades storage units have increased in size 10 times every 5 years and costs have come down per unit.
    So what I store today will be a fraction of the total storage in 5 years then what it is today.

  2. The files above 3KB are split into at least 3 chunks and each chunk is a maximum of 1MB. So a 3KB-3MB will be 3 chunks and above that 1 chunk per MB

  3. Timeline is standard for a large project which is breaking new ground. Last week was released a test network which provides a section of what the full network would be. The next stage is to increase the number of sections, and testing many aspects. The next stage after that is to incorporate more, if not all, features for a fuller test. This of course can change as requirements dictate. Do not expect a live network this year, unless the test networks show up no bugs (do not expect that).

  4. The network is blockchain-less. There is certain data that is kept for the operation of consensus and network operation. But this is not a blockchain and is often discarded after it has served its purpose. Some have tried to call keeping of operational data a block chain because it is stored in blocks in internal storage of the node, but no its not blockchain technology

  5. Yes the network grows organically. By adding sections to the global network as more nodes are added. The farming algo will increase rewards as spare storage reduces thus encouraging more people to add their spare resources to the network, and as this is done the network splits off new sections so that no section becomes too large.

Finally some people feel that data centre resources will be used, but really that will only happen when the rewards can cover the costs that are involved in data centres (wages, air/con, security, electricity, etc). Now this may become viable in some cases but the home user with spare resources and virtually zero incremental costs to running a node will be king

14 Likes
3

For the last number of decades storage units have increased in size 10 times every 5 years and costs have come down per unit.

You do realize that there are physical limits to that, right?

Storage will not continue to drop in price eternally. For that matter your “last number of decades” rate of “10 times every 5 years” (annual rate of about 1.6) is way below the old “Kryder’s law” rate of “doubling ever 13 months” (annual rate of about 1.9). That’s mostly because recent rates of improvement have slowed as we near those physical limits.

From 2010 to 2020, cost per gigabyte of generally available HDDs went from about $0.10 to charitably $0.02… whereas your model would have had it fall to 0.001. Even ignoring your claim that costs per unit has reduced, you’d off by a factor of 20. It’s only going to get worse from here on out.

Nobody knows exactly where the bottom is, but in the limiting case, if you could somehow maintain your “10 times per 5 years” rate (which again is not being maintained now and hasn’t been for a very long time), apply that rate to the USB stick on the desk in front of me, then assume that you could do volumetric rather than areal storage and scaling, you would end up having to store a bit on every single atom in about 60 years. I guess you could eke out another 5 or 10 years by storing a bit on every single electron. Either one is, of course, completely unachievable, but they’re also unarguably absolute limits.

The “all storage is eternal” thing is the absolute worst mistake in the SAFE network concept.

2 Likes
4

It will be interesting to come back to this point after launch. There a some factors you may not be aware of such as tertiary storage (archive nodes) and via data chains network republishable data. So not all storage has to be eternal and held in primary storage space. However data being eternal is a core principle we have. It’s the magic of how you achieve that principle that involves innovative approaches.

12 Likes
5

@jbash I’d like to like that last post of yours - and bar the last line, I still do like it a lot.

2 Likes
6

And neither you, not me, not our society, not our world, not our sun and not our galaxy are eternal.

And about the price, I will be satisfied if it continues to drop over the next few decades in a similar way, which is very likely.

2 Likes
7

reminds me of The Last Question (Asimov)

[Edit: Aspects of] What @jbash brings up is fair but not priority right now. Once the technical work is done and the network in beta, economics concerns can be revisited.

2 Likes
8

Your 60 years limit is meaningless and senseless and I suspect designed to belittle a prediction for only 20 years in the future. And in doing so has completely missed the point of the message (HINT: point is not if a prediction is perfectly true but that storage is out striping unique data increase and will be for life of SAFE)

See below for the explanation

I used my experience from the sixes to now. Sorry it does not match someones law that is now wrong.

And I’d have hoped people are sensible and realise this will not happen forever.

BUT you have not considered new technologies either. SSD is on the beginning of the “S” curve of growth and rotating magnetic media appears to be starting the ending of the “S” curve. New optic storage is just starting too. DNA storage is being developed.

So you have missed so much in an attempt to prove me wrong and attempt to show SAFE cannot work for the next 20 to 40 years, which is an approx time frame before everything is migrated to a new system.

Realistically we are nowhere near a “bottoming out” and we are still in a massive growth phase of storage that started some 5 to 6 decades ago.

Not quite true, and yes perhaps the per unit size in last 5 years has not quite kept up, but the increase in number of units produced per year makes up for any shortfall. ALSO there have been periods where the figure drops off for a year of two (EG the RLL limitation) until a new technology is implemented then we see a spike that makes up for it.

1987 Large disk drive for a personal type computer was 5MB, 32 years later, 2019, (~2x10^6 predicted increase) we should have 10TB drives. I bought a 16TB drive last year. So not bad prediction would you think. Now SSDs are taking up the slack of the slowdown in rotating media. If you want to compare server and mainframe storage then it too is similar, but since this is for the home user we will stick to that.

The last 5 years has seen a drop off in rotating magnetic media unit size increase (3TB to 20TB) but there is a massive increase in SSD media and SSD is slightly different because rather than the increase being in unit size it is as much in the explosion of the number of units produced with a unit size increase.

The increase in available storage produced per year has increased and the technology coming up supports the statement that storage capacity increasing at a large rate for my lifetime and even without knowing your age, for your lifetime.

Do you have any clue how many atoms are in a USB stick. Approx 10^23 to 10^24 atoms.

But seriously you had to go out to 60 years to try and disprove this. Guess its difficult to disprove my point that for the life cycle of SAFE even if 60 years will see a increasing storage capacity.

To to show that you were trolling, people today are backing up their data multiple times whether you have SAFE or not the data is going to be stored. At least with SAFE there is only one copy of each unique dataset rather than potentially up to a billion copies of win7 and win 10 being backed up by millions to billions of people on a regular basis

And guess what just this year a paper came out describing how they now can control a nucleus with an electrical field. The first stages of storing data at the sub-atomic level. In 60 years we will be storing data inside of atoms, maybe 64 bits even sometime in the future, who knows in 60 years

Your 60 years limit is meaningless and senseless and I suspect designed to belittle a prediction for only 20 years in the future. And in doing so has completely missed the point of the message

9 Likes
9

20 years is a completely inadequate design lifetime for something that’s trying to be the basis of “new Internet”. It’s “senseless”, to use your word.

Out here in the real Internet, we are still using IPv4 (in active use for over 35 years). We may just now be managing to really move to the “new” IPv6 (whose specification reached essentially final form over 20 years ago). We’re able to do that mostly because IPv6 doesn’t make any breaking changes in the services offered by the network layer… which would not be true for relaxing SAFE’s “eternity” assumption.

At layers above, we have TCP (35 years again, with some backward compatible tweaks up to maybe 25 years ago), and HTTP (in use in its present form for over 20 years; almost 30 years if you count HTTP 1.0). DNS has been in operation for about 30 years as well. None of them are going away soon. If they do go away, it will again be because other protocols offer compatible drop in replacements that don’t violate application assumptions. Immutability is an example of such an assumption.

Fundamental protocols live a long time… if they succeed. It’s amazingly shortsighted to think in terms of 20 years or even 40 years. Everything does not just magically get “migrated to a new system”.

For example, the existing Internet applications are not all going to migrate to SAFE in the next 10 years, if they ever do. IP, probably even IPv4, will absolutely have a 60 year widespread operational life span. Maybe longer. If SAFE gets where it supposedly wants to be, it will have a similar life span.

Also, MaidSafe was apparently founded in 2006. 14 years is a very long time to spend designing a protocol suite if you only expect it to have a 20 year working life.

Anyway, the whole point of the “a bit on every atom” reductio ad absurdium is that it is not achievable, ever, period, with any technology, magnetic, SSD, DNA, nanorods, fairy dust, or otherwise. The point is to show that even if you’re allowed absolutely any new technology that could ever be built, and even if you’re allowed technology that could never be built, you will still stall out in the reasonably foreseeable future. The particular number of years doesn’t matter much.

60 years happens to be what I came up with when I started with “a bit on every atom”, decided to apply that to every atom in a solid volume, and calculated how long it would take to get there at your stated rate. But I chose those assumptions because I didn’t want to leave any room at all for argument.

I could equally well have shared your reliance on the last 50 years or so, and taken the view that since every single storage technology so far has been fundamentally planar, you shouldn’t expect to stack layers more than say 1000 deep.

And the figure I was using (3x10^24 atoms in a 3ml volume) would allow for a “USB stick” made out of metallic hydrogen or something exotic like that. That’s why it’s higher than your number, and, yes, thanks, I do know how many atoms are in a USB stick. I could just as reasonably have assumed a less dense silicon lattice spacing.

Silicon spacing on 1000 planes of 3 square centimeters would have given you roughly 5.5x10^18 atoms to play with. Which would mean that you’d have run out of space in that USB stick in 30 years. Except, of course, that you still wouldn’t actually be able to achieve that anything close to that density, because it would leave no space for the stuff that manipulates the atoms and moves the data.

These hypotheticals depend on a whole bunch of ass-pulled numbers, and I got to 60 years by choosing to ass-pull numbers that gave you every possible benefit of the doubt. The only thing I refused to do was to assume that some kind of completely unexplained (and frankly implausible) technology will show up and solve everything.

In reality you will not have 60 years, because you will not be storing a bit on every atom in a solid volume. You will not have 40 years. You probably will not even have 30 years. You may not have 20 years. And you will deifinitely see costs go way up well before you hit the final wall.

1987 Large disk drive for a personal type computer was 5MB, 32 years later, 2019, (~2x10^6 predicted increase) we should have 10TB drives. I bought a 16TB drive last year.

I looked at it again, and the curve has been smoother than I thought. But the rate of improvement is still slowing down, especially on cost as opposed to pure drive size.

As for new technologies, since you seem to reject the technology-independent argument…

SSD is not at the beginning of its curve. SSD is just plain old VLSI integrated circuits (with die stacking). VLSI has most of its growth behind it. Flash is not going to get much denser before charges start tunneling out of cells. You’re not going to be able to stack chips much higher before they fry, or the interconnects become impossible, or they just plain fail mechanically. And there is no obvious better replacement.

DNA storage is, frankly, a gimmick. If you could get that to work, you could get other, better things to work. But those better things still wouldn’t get you to a bit on every atom, and neither, obviously, would DNA.

As for “controlling nuclei”, I can guess what paper you’re talking about: the “Nuclear Electric Resonance” one, right? If you think that’s going to lead to storing different bits on every atom, you really have no clue what it’s about. You might be able to put multiple bits on an atom… with millions of non-storing atoms around it to manipulate it and maintain its state. You won’t be able to put different bits on the next atom beside it, and the one beside that, and the one beside that, and read them back separately.

“Control a nucleus” in that headline means “line up the spin in the desired direction, while also lining up the spins of every atom for an enormous distance around it in the exact same direction”. It does not mean “separately set, maintain and read back separate distinguishable per-atom states, reliably, over any number of atoms”.

When you say “In 60 years we will be storing data inside of atoms, maybe 64 bits even sometime in the future, who knows in 60 years”, I can only read that to say that storage is going to keep getting denser and cheaper because you really, really want it to… not because you know any real way for that to happen. It’s magical thinking.

You have no idea how to actually do it, and no reason to believe it can be done, beyond a completely invalid extrapolation of an exponential curve. Exponentials always break down in physical systems, usually sooner rather than later.

The two concrete ideas you mentioned (DNA storage and the weird NMR business) show that you don’t begin understand the limitations of the work you’re relying on.

At least with SAFE there is only one copy of each unique dataset rather than potentially up to a billion copies of win7 and win 10 being backed up by millions to billions of people on a regular basis

People have their own copies of data because they need to process those data. Your local disk has copies of stuff that you need to access quickly. You can’t just keep stuff you use constantly out on a widely distributed network. Not if you want to actually get anything done.

3 Likes
10

Seriously. You picked the one time I said 20 years and left off the times I said 20 to 40 years and the times when i said the next network takes over. You know like morphing into SAFER (next iteration after SAFE)

You want to prove storage will fail, good on ya, go for it

Bottom line storage is increasing fast and no signs it is stopping for decades.

And in any case not using SAFE means there is need for EVEN MORE storage to cover the needs.

7 Likes
11

Can I just mention that private data can be deleted. It is public data which cannot be by design, as it may be depended upon by other public data.

If the network becomes too full because public data storage is outstripping the supply of storage media, then the price to store will increase to reflect this. It will make people more discerning over what they wish to store.

With public data, we also have de-duplication, which we be at a global scale by the time available media becomes an issue.

We must also consider supply side dynamics. Are people going to be willing or able to create the sheer quantity of data to exhaust all available storage? Even if they have the will, will they have the econcomic capability to pay for it? What will it even be? 4k, 24/7 video logging of their every movement, by every person on the planet, shared for everyone else?

SAFENetwork needs people to store new stuff, to keep data moving, to remain robust and secure. Much like money in our economies, which needs to maintain velocity. Should this scare us? No - it is a reflection of our personal interactions, our enterprising nature, our desire to learn, share and evolve. If one thing is certain, it is that change will continue.

11 Likes
12

Current hdd grain size for storing bits is approximately 200nm. However, finest resolution of sem imaging today is approximately 0.5 nm. Resolving power based on Nyquist consideration let’s us presume 1 nm.
Assume 50% fill factor gives us a max data storage and retrieval density of 5Tb/cm^2 using current available tech. Subsurface ion implant and imaging techniques could increase this by 10x. Better resolving power down to .125 nm gets us another 8x. Multi atom deposition allows for non-binary encoding that would improve this another 8x. This rough ‘back of the envelope’ calc shows a plausible 3.2 Petabits/cm^2 using no crazy exotic tech. Archive vaults could see a single ‘LTO-200’ type cartridge storing 327.5 Exabit. Good Economic policies and mass production needed for cost reduction and miniaturization to 3.5 inch hdd form factor. Plenty of more innovation and improvements to be had. There is no real ‘need’ to delete anything, although I do agree that it is prudent to be thrifty and not wasteful of resources. There is a challenge in that 90% of data is read once and then never used/read again.

6 Likes
13

And this is where SAFE’s other features come into play.

It costs to store data so people will use care with what they store anyhow. As spare space reduces then more people are encouraged to become farmers due to rising cost to store until enough new farmers join.

De-Duplication will reduce a lot of waste, that without SAFE will require even more storage devices to be produced with them rarely ever being full. Even today if you only want to use 120GB total for your PC storage unit then you need a minimum of 500GB and even now that is a more expensive option than going for a 1TB device. SAFE benefits because the 380GB or 880GB could be used for the SAFE network thus making better use of what storage is being sold.

3 Likes
14

One of the things I was searching for when I first found SAFE was a good way for automated file deduplication. I’ll admit that about 35% (or more) of my storage is wasted on duplicates and backups of backups that are poorly organized.

5 Likes
15

I wrote a script decade or two ago to CRC32 every file then compared file size and crc32 and made a listing of the dups.

2 Likes
16

Funny, that’s exactly what I’ve been doing recently.