RAIDZ or MIRROR on SSD pool?

New to OpenZFS on OS X (Or ZFS in general)? Ask your questions here!

RAIDZ or MIRROR on SSD pool?

Postby roemer » Sat Feb 11, 2023 10:57 pm

I am about to migrate an HDD-based ZFS pool to a SSD-based pool.
On the existing pool, I emphasized data protection and used RAIDZ2 over 4 hard drives.

For the new SSD-based pool, I am wondering whether RAIDZ1 or MIRROR is better.
A lot of documentation about ZFS pools seems to indicate that RAIDZ is better for throughput, especially for large files such as photos or videos.
But for better latency, especially on smaller I/O requests such as done by databases or VMs, MIRROR is said to have an advantage.
Then there is also the issue of a potential device failure where a RAIDZ puts extra burden on all remaining drives during re-silvering, while MIRROR only involves one drive top copy from.

But all the information I found was basically written for hard drives, where I understand the reasoning.
In fact I was originally planning for a MIRROR setup with my new pool...
... until I started benchmarking ;)

It seems that in my case (4 x Crucial P3 NVMe SSDs), a RAIDZ1 pool is faster than a 2xMIRROR for large file access, and at least comparable in speed for 4K access. My use case will involve mainly large file access. This leaves the argument about the re-silvering stress on the remaining drives in case of a vdev failure. But I can't see this a huge issue for SSDs anymore, as reading from an (old) SSD should not really have a burden on them. It is the writes which wears down SSDs, but during re-silvering the writing is mainly done on the new replacement drive.

Does anyone else have experience with an SSD-based zfs pool?
Do you use RAIDZ or MIRROR on your SSD pool?
roemer
 
Posts: 73
Joined: Sat Mar 15, 2014 2:32 pm

Re: RAIDZ or MIRROR on SSD pool?

Postby Bottacco » Thu Mar 16, 2023 11:31 pm

While researching ZFS on Mac I maid several trials with 4 SATA3 SSDs to compare against SATA3 HDDs. I talked to several BSD sysadmins that use ZFS in a regular basis and all of them recommended a stripped mirror zpool (2 mirrors stripped like in RAID 10), but they said that would make more sense with more mirrors (several mirrors striped). With that configuration you get good latency and fast access (the more mirrors you have, the better). The only downside is the storage efficiency, that is 50%.

For small VDEVs it all comes down to your main purpose, but being NVMe you won't have much of a problem in the throughput department, so between a RAIDZ2 and a pair of stripped mirrors I would go with the second option. But this is just my opinion based on all the videos, docs and post that I have researched.
Bottacco
 
Posts: 10
Joined: Thu Feb 09, 2023 1:27 pm

Re: RAIDZ or MIRROR on SSD pool?

Postby roemer » Thu Mar 23, 2023 5:46 am

Many thanks for your feedback, Bottacco.
When you say that you talked to "several BSD sysadmins", I assume they are running production-size ZFS installations for multiple users and mainly using HDDs?
That would be quite different from my setting where I only look at 4 disks, and all of them SSDs.
I also note that you opted in you own setup, which you described in another post (viewtopic.php?f=11&t=3784), for ZRAID1 yourself...

My dilemma is that all advice I can find is either quite old from pre-SSD times, or assumes a large installation with multiple stripped mirrors to serve multiple users.

In my case, it is a personal RAID with one or max two users at the same time, and the workload consisting of mainly sequential file reads and writes. No VM, no major database workload, except perhaps Lightroom whose library file is indeed a huge sqlite database.
I understand that stripped mirrors a great for a lot of small IO request, and also the benefit for resilvering after a HDD crash in case of hard drives.
But I think this is simply not my scenario...
roemer
 
Posts: 73
Joined: Sat Mar 15, 2014 2:32 pm

Re: RAIDZ or MIRROR on SSD pool?

Postby Bottacco » Thu Mar 23, 2023 9:19 am

roemer wrote:Many thanks for your feedback, Bottacco.
When you say that you talked to "several BSD sysadmins", I assume they are running production-size ZFS installations for multiple users and mainly using HDDs?
That would be quite different from my setting where I only look at 4 disks, and all of them SSDs.

This days many sysadmins are configuring pure SSD vdevs with enterprise grade NVMe based SSDs (Kioxia for example), or hybrid systems with many HDDs with ZIL, SLOG and L2ARC on SSDs to accelerate the system. So now it is rarely just plain old HDDs and many chasis for 2,5" U.2/U.3 drives are appearing, which makes the use of pure SSD vdevs more common in that world.

roemer wrote:My dilemma is that all advice I can find is either quite old from pre-SSD times, or assumes a large installation with multiple stripped mirrors to serve multiple users.

I have been watching many tutorials and advice videos in Youtube. They are not obvious to find but try the ones from 45Drives, Kiara Systems, Level1Techs and some others where they are dealing with new configurations with SSDs.

roemer wrote:In my case, it is a personal RAID with one or max two users at the same time, and the workload consisting of mainly sequential file reads and writes. No VM, no major database workload, except perhaps Lightroom whose library file is indeed a huge sqlite database.
I understand that stripped mirrors a great for a lot of small IO request, and also the benefit for resilvering after a HDD crash in case of hard drives.
But I think this is simply not my scenario...

In your case the difference between two striped MIRRORs and a RAIDZ2 for the use case you are giving would not matter much, because you loose the same number of drives for the total capacity. I did choose RAIDZ1 because I just had four HDDs but it was a test setup. When I receive all the new drives for the final server it will be a RAIDZ2 of 8 HDDs and probably the ZIL will go on a pair of mirrored NVMe M.2 drives on PCIe card to accelerate the access of 15 people.
Bottacco
 
Posts: 10
Joined: Thu Feb 09, 2023 1:27 pm


Return to Absolute Beginners Section

Who is online

Users browsing this forum: No registered users and 11 guests

cron