I had 200 nodes on 8TB SMR drive without any problems (just data, logging was redirected elsewhere).
$70-80
3 Likes
Its the cache size on the disk (and OS) that determines when it slows down.
Maybe for Safe with its rarely write way of working and then no faster than the internet can work does not ever tax the cache. So maybe that advise of mine might be unnecessary
BTW what sized PC were you running 200 nodes on if I can ask
2 Likes
but if you have 8TB and run only 200 nodes, that means you dont use all capacity of this 8GB no?
1 Like
Old HP Microserver with 4core Xeon and 12GB RAM.
CPU and internet speed is the limitation.
I think on something like AMD 7900X you could run maybe 2k+ nodes, but you would need 10Gbit line.
7 Likes
can somebody give final answer, if SSD is good or isnt good for data storing (node) thx
1 Like
Direct Attached Storage will always outperform NAS or SAN Array, the big issue though is single point of failure, so you could add RAID and UPS to the DAS array of SSDs to beef that up a bit.
Any NAS running ZFS is both slow and hugely wasteful of capacity because of the large amount of write amplification going on, and the fact all those writes will burn out your SSDs fast. Heck ext3 or ext4 or btrfs is a better choice,
Remember that NAS and SAN use the network commuications stack of your Host OS to route data writes (the slowest operation) over Ethernet to your Storage, which means there are at least 2X more ‘layers’ to pass through, than PCIe 4.0 motherboard signalling of the TLP ‘transaction layer protocol’ running across say 2 x4 physical motherboard bus lanes connected to your NVME SSDs or to your embedded SATA controller speaking to SATA SSDs ( 6X slower than NVME but still way faster than 15,000 or 7500rpm hard disk drives). Your Node reputation score for reliability and speed will always be higher on DAS, provided you have decent CPU core/thread count and enough RAM to run everything you need daily on your Desktop and the Node. Keep in mind USB and Thunderbolt3/4 use the motherboard or bus stack (TLP over PCIe GEN3.0/4.0 for most Gen5.0 just starting to show up this year which is 2X faster than Gen4 and 4X faster than Gen3), so it will be faster than most home Ethernet NAS setups running at 10, 25, 40/56 and even 100Gbit/sec
2 Likes
There is a Raspberry Pi 5 now! But it is expensive compared to a RPi4 and looks over the top in terms of CPU needed to run a decent number of nodes. I can comfortably run 20 on a RPi4.
Your network connection will likely be the limiting factor as to how many nodes you can run.
1 Like
ZFS is the perfect filesystem for a world that no longer exists. Just don’t ever use it with ssd’s (except for the l2arc and zil). The issue with zfs and ssd’s isn’t so much the write amplification, but the copy-on-write mechanism that is superfast until your ssd reaches full-ish. If you are blessed with a tlc/qlc ssd with borked garbage collection (yes I’m looking at you Micron !) your performance will drop to zero until you manually do a deallocate (=sata trim=scsi unmap).
I’ve seen quite promising demo’s that would outperform sas based DAS from Nvme-over-fibrechannel networks years ago, and it seems that it’s dumb tcp/ip based brother nvme-of is finding it’s way into the more adventurous enterprise it setups nowadays though it probably needs a beefy network(admin) to function properly.
2 Likes
I am sceptical about the long term prospect of running safenodes direct on the MicroSD drive that the Pi’s OS is on. MicroSDs just aren’t as durable as proper SSDs. I think the safenods will burn it out quite quickly and then the OS, the safenode install and your encryption keys are gone with it. Ok, so you can have a stack of them ready to replace it with and get at the earnings if you saved the keys off system but it’s still a hassle that can be avoided.
I think way to go is with a SSD or M2 drive attached for the actual safenodes mounted at:-
/home/<user>/.local/share/safe/node
2 Likes
RPI isn’t good bang for the buck anymore. It was once a cheap low power alternative but nowadays you can find a thin client on ebay with 4gb of ram and 16gb ssd builtin that runs circles around an rpi performance wise.
Check hp t520,t530,t620,t630,t820,t830 on ebay, Lenovo and fujitsu have similar models that are dumped by the kilogram.
1 Like
I don’t think the speed of storage is going to be an issue. Remember that the network as a whole will expect to be able to get a response for reads and writes in a reasonable time but the major factor will be the fact that you are at the end of a consumer grade internet connection through an ISP. That’s probably dozens of milliseconds. So an extra few milliseconds for reads and writes to slow storage is the least of anyone’s worries. I think it will matter not at all what the storage performance is so magnetic disks, disks on a NAS, slow SSDs or MicroSDs will be fine. Anything as long as it’s not tape!
I think the issue with MicroSDs will be their durability. But I will test this and report back. I’m prepared to be proved wrong! But I doubt it.
3 Likes
Beauty of Safe is that it is the “RAID”/“MIRRORING” and your rig can fail without consequences. So for the Safe Network the cheapest reasonable solution is the best. As long as it runs Nodes nominally the nodes will receive the same new chunks whether the node is 100mS faster or slower. And if it fails the loss is time taken to replace. It still ends up cheaper to just have direct attached storage than to setup a new NAS or other storage appliance.
I am making comments here in respect to being suitable for the Safe Network.
Yes I agree with much of what you say if not all, but for Safe much is not really needed.
For a reasonable priced SSD then no better or worse. Safe is mostly a write few read many application so the life should end up reasonable. Rotating media is also good and shouldn’t have a reduced lifetime either. So take your pick and choose the best drive for your device/setup
3 Likes
Also a number of 8TB drives are SMR drives and they can be extremely slow at writing if you write more than the cache size in the drive. And I mean slow.
Yes, SMR drives are total dogshit. Slow for random reads, slow for random writes, slow for sequential writes and when the buffer is full they just hang. So you are bust for most use cases already. The only thing they are kind of ok for is sequential reads but they are slower than anything you can think of, including tape. And if your use case is sequential reads you probably had to write the data sequentially and that’s right out. So, backup and archiving usage is off the table.
So they just aren’t usable for anything you might want to use them for.
I know this to my cost having bought some Western Digital drives that weren’t described as being SMR and they killed the array performance and then crapped out in a cascading fail. The performance dropped so low the system thought the drive had failed and started a RAID rebuild which caused so much activity that the others crapped out a few hours later. And that is why I hate Western Digital. But all the drive manufacturers are See You Next Tuesdays anyway.
But I’m actually thinking that Safe might be a genuine application for the flawed concept of SMR. The writes and reads to the drive will be throttled by network bandwidth anyway in most home setups so the things might have a chance of surviving.
I intend testing this. With some SMR drives I have sitting on a shelf…
2 Likes
RPI isn’t good bang for the buck anymore. It was once a cheap low power alternative but nowadays you can find a thin client on ebay with 4gb of ram and 16gb ssd builtin that runs circles around an rpi performance wise.
Check hp t520,t530,t620,t630,t820,t830 on ebay, Lenovo and fujitsu have similar models that are dumped by the kilogram.
ok, yes, I see your point. But does the CPU need to outperform the RPi4? Or even a RPi3? If you can still run the number of nodes on the RPi that will max out your home internet connection you don’t need the extra CPU horsepower and resulting electricity consumption and space it will take up. Unless of course you are using the HP thing for something else anyway.
These small computers are just more than better CPU, they have the improved ecosystem, form factor etc. And about the same power draw to 1 or 4 RPi
I estimate that you are running 2 to 4 times the nodes on one small computer for 100-200$ new compared to a RPi. The RPi for a person who while into hardware and computers are not into bare board tinkering which RPi still is even though it maybe easy.
For the tinkerer, running a single RPi with a drive is a great way to go, they can tinker to get best performance and enjoy doing it. But for the nerd who prefers a setup and forget type of safe nodes then I’d definitely say go for a small computer, even an Oddsey w/case, or however Seeed spell it, will be much better.
For Safe I think user preferences and ease of setup (their perception) will be way more important than +/- $50 in setup costs. After user preference then running costs (electricity) will play a part and for the most part there is little difference between 2 to 4 RPi and a decent small computer (cost, power, ability) except maybe only one drive is needed for the small computer solution (setup cost)
3 Likes
It’s likely not as cost effective to use microSD - I just like the compactness/tidyness of using them with the Pi-zero – which are still pretty cheap.
As for durability, if you have plenty of spare capacity on them, and in my price calcs above I would have massive spare cap., then durability won’t be an issue because all of these drives have wear-leveling software built-in.
But only if you have my aesthetic for tidyness would you probably want to go with microSD, as would be cheaper to get alternative external USB drive.
1 Like
Ok, you and @Erwin have convinced me to at least try one of these things. Maybe I’ve become over fixated on RPi 3 and 4s. I do see that there are plenty of HP TC Series things on eBay at low prices.
The use case is to give out to friends and family so it has to be small, unobtrusive, zero noise and low power. Without all these the concept will be a hard sell. Some of them I may have to post so that’s where a Pi has the edge but I suppose the shipping costs won’t be much more for one of these HP things.
Edit:
I see an issue with them for my use case: they seem to use Display Port (although they do have a VGA port as well). If one of my non technical friends needs to do some config they can just plug a Pi into their TV with the HDMI port. I can’t guarantee they will have a monitor with with VGA or Display Port or that their TV will have either. It’s much more likely they will have a monitor or TV with an HDMI port.
Edit:
I don’t like the PSUs. If one goes on a Pi it will be easy to tell people what to get to replace it and they may have one in the house already. If one of these HP PSUs goes it will be much more hassle.
There’s a catch with the rpi usb adapters. There’s a design flaw in the rpi4 that makes it require a special kind of usb adapter or it won’t work. So you’ll be stuck with an external psu either way.
Also, if you want to hook it up to a tv, you will need a special mini-hdmi cable as well, so it doesn’t really matter if you have to get a mini-hdmi->hdmi or displayport->hdmi cable.
Good point about the Pi PSU. You can’t use just a normal Micro USB wallplug. Still easier to obtain than one of these HP PSUs though.
For the Pi I intend putting them in a case that holds an SSD drive and has a full size HDMI port as well so an external adapter won’t be needed.
I think you’re underestimating how much of these thinclients are being dumped on the market 
Also keep in mind that these are built to last. They’re built for large enterprises that won’t accept high failure rates. Same with enterprise laptops, that’s why you still see 10 year old dell latitudes and lenovo T-series still being dumped by the kilogram. It’s a different level of engineering than for example asus laptops.
Have you made a rough calculation how much the rpi4 (if you can even find them), case, extensions cards and ssd will cost you ?