At this time it seems that it will not make much difference to nodes.
1 Like
we know backblaze, that is my day job co knows them well face to face in CA. bblaze started strictly in the tape backup space as a BackupaaS player in CA, grew from there…, they have some testing capability for sure, that said, they were way late to the flash party…,
drive durability is not just about reading smart drive data , the current CEO guy at Hammerspace was early into the flash space, just after us, late 2006, he was the Fusionio dude that got swallowed by SANdisk, they got Whiptail to use their code for v2 of their box, which didn’t work and that is why Cisco shut down Whiptail , Flynn and SANdisk got swallowed by WDC, see the pattern here?
the FIO test tool is under control of the FB/meta domain, best to rely on that one tool, Jens Axboe theauthor of FIO knows what he is doing and the little test community surrounding him are diligent…
imo Its all about the flash controller, that is running it in RAM with FTL in memory spreading the cost of same and system hdw compression across multiple cheap drives , which will be a lot cheaper than running a DAS array of flash drives or a small NAS of flash drives where each 50% more expensive drive included chips and logic to run both FTL and compression per drive.
ie- leveraging system hdw compression (Intel) to go really fast is a good thing , yeah it costs system CPU clock cycles and threads, and RAM but its way cheaper (like at least 50% cheaper) spreading the FTL and compression costs across 4 cheap dumb durable drives than buying expensive drives with FTL and compression built in.
Its important the ‘in memory’ controller manages the flash GC, defrag and wear levelling in RT opportunistically (fine grained always working, so perf of write and read i/o is minimally impacted, a lot of the drives out there running hdw controllers use a batch mode which is pretty disruptive to say your desktop Apps working smoothly (vid. conference will burp the most when you don’t want it doing that
) during peak write traffic times), as well as use any unassigned capacity (don’t let it shrink any lower than 30% of total capacity otherwise your write speed will decline rapidly as the controller spends more clock time running the GC, defrag and wear leveling tasks) smartly in order to extend the life of flash.
Also the i/o profile of the workloads, payload size, freq. of payload etc., will determine how quickly the flash media wears out. So profiling the ind. app (Desktop daily stuff) and background workload use (Autonomi) is important in determining overall how long you flash storage will last in a combo use setup.
Just sayin, hope this helps when you are picking out your flash systems to run Autonomi.
1 Like
ok when the DAS or NAS array is dedicated with its own system cpu clock cycles, threads and ram, sure CEPH will run ok, its when you have the combo use case where the desktop apps are competing with the CEPH module you will not see it work so well. I am thinkin small businesses running a ‘distributed.net’ type of setup ‘i-house’ where Autonomi nodes are running in the background competing with Ceph nodes in a few or maybe all cases, and they are using a private ‘close’ group for their sensitive data, and pushing the rest of their public content out to Autonomi for one time write and read many use. Early days we are in, re: different use case scenarios…
imo Its all about the flash controller
That’s how Pure Storages makes the insanest of insanest fast storage arrays, they make their own boards with qlc but instead of the controller of the bread and butter ssd’s that us mere mortals can buy, they have a proprietary xeon based flash controller. Last time I checked they did 1,9 PB in a 4u box. Insane.
2 Likes
Pure charges typically 22-25% per year to service/support the box with drive replacement included, by year 5 you have paid for another box, keep in mind your are talking about a box which costs about US $10M, so over 5 years that is to be kind let’s say US 18M for say 2PB to round up. , that is approx. US $9000.00 per TB of ‘half fast’ storage (which includes at least 1 cycle of complete replacement if the volume of transactions is read 70%/write 30% 4K random high , meaning the composite workload payload OLTP data size is 4K, typical DB Server stuff … ouch…
note that Pure laid off a bunch of staff recently, as their pricing model starts to hurt them…
1 Like
I will go on record is their Storage is not fast. It’s average. There are some high end vendors out there which are way faster, but less known. Its hard to crack in to tht space, most of the newbies like VAST or Hammerspace are aimed at NETAPP NAS customers doing a lot of HPC file/object store which is 8X to 10X slower than block storage on any media doing the triplicate copy thing to three different locations in the SAN array, usually full of spinning rust storing Hadoop Spark and now Lakehouse datalake unstructured files and objects used to train up/iteratively tune AI LLMs these days so the attached genAI assistants actually get better engineered prompt response scores.
1 Like
Which block vendors would you say are faster than pure ?
Add in hdw controllers GRAID and Pliops are two worth looking at, BUT you need a really big commercial reason to have a lot of storage on hand to use those hardware accelerators. There are others in that category.
yeah ok but these are client side hardware accelerators. I mean, even a higher end consumer nvme will most likely beat a Pure array in terms of latency and maybe even throughput but it’s not the same thing.
Building a performant and always available san that serves thousands of bladeservers running vmware over fibrechannel is a whole different game than squeezing insane performance out of a single node pci card.
Weka is very fast for file access storage. They have written their own network and disk drivers which replace the ones installed by Linux. Combined with specific models of NVMe drives and 100Gb Ethernet cards (or Infiniband, but that’s much less common) they form a high performance storage cluster whose performance scales linearly with size. For the best speed you need to run Weka software on the client which then becomes part of the cluster and is able to target whichever of the hosts have whichever of the chunks being accessed. Filesystems can be backed by object storage which as long as it is performant enough allows low latency at huge scale with the NVMe drives acting as a large cache.
This is completely different to VAST which makes use of much cheaper SSDs and keeps everything at the same tier. VAST is great if you know that all your data is being accessed at much the same frequency. Weka has the upper hand if you know some of it isn’t being accessed as frequently and can’t predict what it will be.
A certain large manufacturer of electric cars uses Weka for its store of telemetry data from their vehicles. Lots of data being ingested all the time and lots of analysis of it being done.
It’s big in the science and AI space as well.
But we’ve departed a long way from the topic of ‘Low tech examples of making profit from farming’. I think we need a topic split to something like ‘Comparisons to other high capacity and bandwidth systems’.
Of course there are insights about drive hardware and storage networking we have that are applicable to Safe which is how this all started! I’m sure it’s interesting to a lot of folk and not just storage propellorheads. It’s just it’s a lot to wade through and no longer has much to do with Raspberry Pi and consumer drives!
1 Like
Thats for sure, haha. I for one like higher end hardware (> $50 dollars), provided its at the right price…
The one thing that keeps me away from low cost devices is not having a dedicated IPMI chipset. I don’t want to physically have to be present at the location of the device to reset its power and not get raw console access over https etc, granted there is ways around that, but that means even more devices and adhoc arrangements to make it work, instead of it being embedded in the motherboard etc.
Some things just cost a little more, but the benefit can be huge (convenience and disaster/recovery scenarios considered in as well) 
.
1 Like
I run several at home projects (lot of VMs and LXCs), as well as farming or mining (providing storage) for other coins (for now), and everything is stored on CEPH. I just won’t go back to direct attached individually managed devices / file systems per physical host (those days are in the past). I do make sure I do have enough CPU and RAM to handle CEPH’s needs, and non-CEPH resources as well.
CEPH requirements per TB of storage to run properly are steep… and if you don’t meet them, I agree with you that you are inviting tons of trouble for yourself.
Worst case, I will stick with ZFS if needed in scenarios where a CEPH cluster cannot be created at a certain location.
2 Likes
RAID 10 will deliver the fastest performance over a SAN Array, no question, (this is not NAS = NetApp, VAST or Pure in NAS mode).
My own experience tells me Hitachi, Fujitsu and IBM all have solutions which out perform Pure in the RAID10 space operating as appliances connected via iSCSi over Ethernet (Dell/EMC is the slowpoke) The bottleneck is always write speed.
The other ‘new emerging’ approach is to use PCIEgen5 and CXL 3.0 fabric manager to go even faster (than 400mbs/sec+ ) , but this is new territory and you wont see any commercial solutions for this until next year, earliest. The HPC guys have been employing PCIegen4.0 and CXL 1.0 using LIQID and Gigaio, and then there is Rockport Networks doing the fiber loom any to any non-blocking mesh solution for the same market segment. all of that stuff is typically 100GBSpine and dual 25/40/56 Leaf Eth networking using TOR low latency switching (Arista) to create a SAN Array RAID10, sometimes RAID5.
If you are looking to use SuperMicro or Celesitca OTS ‘merchant’ Storage hardware that is 40/56 or even 100GB Leaf Ethernet connected, its possible to configure ROCE(Linux) or iwarp(W11) Systems and marry the container/vm workload to connect to system software controllers doing same that handle the write to NVME 4.0 SSDs over Ethernet NICS (Mellanox for ROCE, Chelsio for iWARP), where the SPINE might be 2X 100/200GB and the LEAF connections to the storage appliances (typically 2U and up to 24 SSDs, 1/2/4 TNB sized drives are always faster at write speeds) from the TOR switch are say dual 40/56 GBit ETH to the OTS Storage Hdw ,
if that is what you are looking for, then in my biased opinion,
CloudProx is the only fast storage software controller on the market that has the FTL and Compression implemented in their appliance system Linux OS installed as a Linux Kernel Module which allows you to distribute cost of Intel’s system i/o compression across all drives in the box and use really cheap SSDs with no compression and FTL config and monitoring logic and supporting memory on board amortizing the cost of same across ALL the drives in the system, which is at least 50% cheaper per SSD drive.
CloudProx’s patented code is an evolution of the version code that powered Whiptail back in the day, before SANDisk/WDC delivered the v2 code to Whiptail that didn’t work well, so Cisco after they bought Whiptail, gave up and shut them down. The Same guy who authored that v2 Whiptail code that was not performant was Dave Flynn the original CTO of FusionIO before SANdisk bought them, he is now CEO over at Hammerspace doing a load balanced NFS play positioned as “AI Storage” lol. Flynn’s original FusionIO code was no match in terms of Write IOPs in SAN array vs. the orginal Whiptail v1 code (2U Accela box), so FusionIO were never able to win any SAN Array biz, Whiptail owned that early SAN Array FLASH market space circa 2009-2012 with the v1 code (running on SATA). CloudProx today sells appliances which are built to order using SuperMicro platforms using a vastly sped up version of that original v1 code which works on SATA/NVME as well as SMR and CMR HDDs. These types of Linux ‘soft’ controllers typically use 1GB RAM for every TB stored. In a DB Server DAS configuration using their Integrated Software offer, well you need to reserve 20% of the DB Server system CPU clock, so size the RAM accordingly.
You might also want to look at the Linux SEF project if you are really into engineering per SSD card performance… from Linux…
https://softwareenabledflash.org/ (This was opened up in October 2023 when XIOXIA (Toshiba Memories America invented FLASH) finally delivered their cli, sdk and api to the SEF.
1 Like
what about ssh into a Pi with the PoE hat?
no PSU and no display…just one chubby ethernet cable.
2 Likes