Ceph might be the only open source source solution that can scale that high but there are commerical ones. Scality is an example.
1 Like
Nice, I’m going to read up on that one. Looks promising at first sight !
CEPH is a one size fits all ‘catch all’ solution for obj/file/blob/block. the reality is based on our own testing and testing of others it is average in file/obj sense for 4K random 70%read/30%write data flows compared to NetApp which is arguably the most wide spread ‘filer’ NAS solution out there for obj/file stores. Keep in mind block storage is typically 8X to 10X faster than obj/file on average, and there are many different types of block storage which blow CEPH out of the water in terms or write IOPS,
Reads come for free as they are generally always fast,
n.b- one just needs to make sure with FLASH systems you always maintain 30% freespace/unassigned space on FLASH so the GC/Defrag/Wear levelling tasks have space to work ‘out of band’ so those processes, when the drive, when it gets close to practical maximum capacity full of data, they do not impact your read and write speed maximums.
On ceph, if you fio it with single threaded I/O with a qdepth of 1, you can’t get more than 1600 iops (if you are very lucky). Databases will never fly, and it’s not the best tool for that job. In fact, if you can name a storage system that is slower on a single i/o than ceph I’m happy to hear it 
But it scales almost infinitely in terms of performance. On a reasonably sized ceph cluster you can keep adding clients until your storage clusters nics/cpu’s are swamped and due to the distributed architecture each client/thread will still enjoy those 1600 iops. And even if you swap your cpu or nic and hit the limit, you could just add a couple more servers to spread the load. Which is way simpler than adding storage to an already maxed out netapp for example. And it doesn’t come with fan in/fan out ratio’s, itn limits, max sessions and all other kinds of nasty limitations that you run into if you connect too many hosts to an iscsi or fibrechannel target.
Though in reality, 99% of servers (and even databases) don’t require that much I/O to begin with.
Most enterprise vm’s are slow dormant application servers, webservers, load balancers, firewalls and such that don’t do much after they have booted.
But if your primary storage consumption is high I/O databases, ceph is the worst possible tool for the job.
Interesting
Autonomi though is not providing 30 mS access or even 100mS access to single records. Can those systems tolerate 100-500mS access for single records, or will parallel access that autonomi can do be able to compensate for it.
1 Like