Growing Mirrors, One Disk At a Time

Moderators: jhartley, MSR734, nola

Growing Mirrors, One Disk At a Time

Post by ilovezfs » Tue Mar 19, 2013 7:46 pm

So I am wondering if there is any reason not to do the following:

A1 = Disk1, Part1 = 0.5 of Disk1
B1 = Disk1, Part2 = 0.5 of Disk1

A2 = Disk2, Part1 = 0.5 of Disk2
B2 = Disk2, Part2 = 0.5 of Disk2

zpool create zA mirror A1 A2
zpool create zB mirror B1 B2

As a picture:
AB
BA

Now if I want to add a new disk, move A1 and B2 to the new disk.
So I have
A1 = Disk3, Part1
B2 = Disk3, Part2

Disk1, Part1 is now free, and Disk2, Part2 is now free.
C1 = Disk1, Part1.
C2 = Disk2, Part2.

And now,
zpool create zC mirror C1 C2

As a picture,
CAA
BCB

And a fourth disk could result in this picture (or others):
CDAA
BCDB

If one only ever reads or writes to one of zA, zB, or zC at a time, is there any downside to this configuration?

It seems like I can always grow one disk at time if I do it this way, instead of always having to buy in pairs.
ilovezfs Online


 
Posts: 249
Joined: Sun Feb 10, 2013 9:02 am

Re: Growing Mirrors, One Disk At a Time

Post by scasady » Wed Mar 20, 2013 11:10 am

Write performance and the possibility of massive confusion :)
scasady Offline


 
Posts: 45
Joined: Sat Sep 15, 2012 8:00 am

Re: Growing Mirrors, One Disk At a Time

Post by ilovezfs » Wed Mar 20, 2013 8:30 pm

Thanks for your reply. If I understand the performance situation correctly, this is basically just the performance difference between RAID 1 and RAID 10, correct?
Last edited by ilovezfs on Fri Mar 22, 2013 12:20 am, edited 1 time in total.
ilovezfs Online


 
Posts: 249
Joined: Sun Feb 10, 2013 9:02 am

Re: Growing Mirrors, One Disk At a Time

Post by raattgift » Thu Mar 21, 2013 10:06 am

When you give zfs (generically, not just the zevo port) a slice rather than a disk, it believes you know what you're doing.

It doesn't try to warn you that you are going to do a lot more seeking as it schedules concurrent I/O across all the slices.
You are probably halving your device IOPS with each new same-pool slice, and many workloads are IOPS-limited.
For different-pool slices, it will depend on the concurrency of the workload. if you are using zA and zB at the same time,
you'll get a similar IOPS fall-off.

So the terse response by scasady is right on the mark. It won't fail, but don't you think you're trading away a lot of performance for the savings of, what, [$€£] 100-200 for a good external drive of the right size?

Also do not underestimate the risk of accidentally making zA and zB wholly unimportable !

Do you really want two pools in this case, rather than a bigger single zAB ? The latter gives you much finer grained control of data ceilings (quotas, refreservations, etc.). It doesn't change the grow-by-adding-a-new-mirrored-vdev plan, though. You would just grow zAB rather than making a third or fourth pool...

PS: In extremis you could use file vdevs within a big enough pool to create little pools. Performance is ugly, but people have actually done, mainly that to facilitate migrations where version compatibility is a problem (zfs send into the file-vdev-based-pool of older zpool version; zpool export it; copy it to other media; ...; copy it to importable media; import it) over the years.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: Growing Mirrors, One Disk At a Time

Post by ilovezfs » Fri Mar 22, 2013 12:00 am

Yes, I see how one large pool is more flexible than several smaller pools and less susceptible to human error. And yes I would prefer the convenience of a single zAB. However, separating into zA and zB at least allows the possibility of avoiding concurrent IO accessing multiple slices on one disk. Growing one large zAB one disk at a time using slices will basically guarantee concurrent IO accessing multiple same-pool slices on every disk, with nearly every IO.

Besides the convenience of being able to grow one disk at a time by using different-pool slices on each disk, I am also concerned about the reliability of RAID 10. On the one hand, I have read that RAID 10 (stripes of vdev mirrors) is significantly more reliable than RAID 5 or even RAID 6 (i.e., raidz or raidz2) configurations for even a moderately sized pool due to exposure to disk failure and likely URE during raidz/raidz2 rebuilds, which are inevitably long, hammer all of the disks, and kill performance until the resilver is completed. On the other hand, it seems to me that one large RAID 10 is much more risky than several smaller RAID 1 pools (vdev mirrors of just two disks or just two, different-pool slices, without striping). With RAID 10, if a disk fails and its mirror counterpart fails during the resilver, then the entire large pool is dead. If instead I use several smaller RAID 1 pools, then only two smaller pools are at risk during a resilver, rather than the entire large RAID 10 pool. And a second disk failure of one of the two "wrong" disks (the disks containing the mirror counterparts of each of the two slices on the first failed disk) would only kill one smaller pool rather than the entire large zpool.

Am I being overly paranoid about RAID 10? Given that RAID is not a backup solution, presumably there is a backup of the entire large pool. And the likelihood of a mirror counterpart failing during a RAID 10 resilver should be very low, since RAID 10 resilvers are very fast (simple copy of one disk).

Are there any reliability benefits of RAID 10 over RAID 1? Or are the benefits of RAID 10 only the flexibility of having a single large pool, and the performance benefits of being able to spread IO over several disks?
ilovezfs Online


 
Posts: 249
Joined: Sun Feb 10, 2013 9:02 am

Re: Growing Mirrors, One Disk At a Time

Post by raattgift » Fri Mar 22, 2013 8:43 am

I'd avoid using standard RAID levels in the context of zfs. They are only very vaguely analogous to zfs's various vdevs.

I don't think we disagree on much other than terminology for most of your comment above.

However, on the point of "one big zAB", I meant giving both disks to a single pool, rather than giving it two slices on each of the disks. Grow "zAB" by adding two more disks at some point. It's safer than any approach that involves fiddling around with GPTs. It's pretty easy to wind up with alignment problems that will hurt performance, and insufficiently difficult to wind up making all the pools wholly unimportable, or winding up with a pool whose layout is not what you wanted and which cannot be changed (time for zpool destroy, whee). Also, it's much harder to keep the data in the pools available and replicated during the grows.

That said, well, it's your data and system, and you can actually do what you propose with zfs. It may even be the right set of tradeoffs for you.

I think growing a pool with mirror vdevs is almost always going to be a better tradeoff for a home user who wants to grow pools like you, as argued here :

http://constantin.glez.de/blog/2010/01/ ... still-best

On the point of pools that are sets of mirror vdevs, yes, pools that are one or more raidz2 will almost always have higher data availability figures, and raidz3 even more so.

*Availability* is the key here : it's a question of whether you can still use your data, live, as devices fail partially or wholly.
No pool, no matter how redundant, should be treated as "so safe I don't need backups", since you are unprotected from
(for example) accidentally supplying the wrong dataset, snapshot or clone to a "zfs destroy".

(On the other hand, multisided mirrors improve reliability as well; a pool with each vdev being a 4-disk mirror has even higher availability figures than raidz3, but it's hard to imagine any home user building even a single 4-disk mirror vdev).

It's unlikely that a home user would need that much of an increase in availability, unless the backup strategy is to have a large (and probably relatively slow) second system with a raidz3 (say) be the target of zfs send fastpool/dataset@snap | ssh ... | zfs receive -u safebutslowpool/dataset pipelines.

There is nothing wrong with having only numerous small pools either, other than your free space being scattered around a number of different pools and attendant "where am I going to put this great big blob of data now" and "where am I going to put this pool's backups" problems.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm


Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 1 guest

cron