First Time Expanding a Pool

New to OpenZFS on OS X (Or ZFS in general)? Ask your questions here!

First Time Expanding a Pool

Postby bro708 » Fri Apr 03, 2020 4:46 am

Hi all,

I have a raidz2 pool that I created a few years back (4 * 3TB SATA 3.0 disks in an external thunderbolt enclosure) and I'm beginning to run low on space so am planning how to proceed.

Ideally what I'd like to do is replace two of my disks with 8TB ones as those seem to offer the best balance of capacity/affordability right now and leave the other two 3TB disks in place until I run low again, by which time I hope that 12 or maybe even 16TB disks will be more affordable.

It's my understanding that I need to replace the disks one at a time and allow them to fully resilver before finally expanding the pool.

I've identified the following steps as hopefully all that is needed:

1. List the IDs of all disks in the pool...

Code: Select all
zpool status "External Raid"


This currently gives me the following:

Code: Select all
  pool: External Raid
 state: ONLINE
  scan: none requested
config:

   NAME                                            STATE     READ WRITE CKSUM
   External Raid                                   ONLINE       0     0     0
     raidz2-0                                      ONLINE       0     0     0
       media-BAA31399-5116-4045-A5C3-FC4221F15928  ONLINE       0     0     0
       media-22261235-A526-AC46-918F-DE1475BB6E32  ONLINE       0     0     0
       media-AA2B3A84-64D4-9848-8654-11D9E6525FC8  ONLINE       0     0     0
       media-DADC56B1-AED9-1141-96E4-EEAD8CE059B8  ONLINE       0     0     0


2. Power down and replace disk then boot back up and repeat to determine which disk has been replaced...

Code: Select all
zpool status "External Raid"


I'd now expect to see something along the lines of this (the disk marked UNAVAIL being the one removed obviously):

Code: Select all
  pool: External Raid
 state: ONLINE
  scan: none requested
config:

   NAME                                            STATE     READ WRITE CKSUM
   External Raid                                   DEGRADED     0     0     0
     raidz2-0                                      DEGRADED     0     0     0
       media-BAA31399-5116-4045-A5C3-FC4221F15928  UNAVAIL      0     0     0
       media-22261235-A526-AC46-918F-DE1475BB6E32  ONLINE       0     0     0
       media-AA2B3A84-64D4-9848-8654-11D9E6525FC8  ONLINE       0     0     0
       media-DADC56B1-AED9-1141-96E4-EEAD8CE059B8  ONLINE       0     0     0


3. Determine the disk number of the new blank disk

Code: Select all
diskutil list


4. Get the ID of the new disk

Code: Select all
ls -l /var/run/disk/by-id | grep disk<numberfromdiskutil>


5. Tell O3X to replace the missing disk with the new one...

Code: Select all
zpool replace "External Raid" <missingdiskID> /var/run/disk/by-id/<newdiskID>


6. Monitor the resilvering process with...

Code: Select all
zpool status "External Raid"


7. Once complete repeat steps 1-6 with the next disk.

8. Check the new space is available under EXPANDSZ

Code: Select all
zpool list


9. Determine the disk numbers of all disks in the array

Code: Select all
diskutil list


10. Export the pool

Code: Select all
zpool export "External Raid"


11. Re-import the pool using disk number references

Code: Select all
zpool import -d /dev "External Raid"


12. Bring the first disk in the array back online using all available space

Code: Select all
zpool online -e "External Raid" /dev/disk<disknumber>


13. Repeat step 12 for all disks in the array.

14. Check that EXPANDSZ is now empty and the pool size has increased.

Code: Select all
zpool list


15. Export the pool and re-import with the usual media identifier references

Code: Select all
zpool export "External Raid"
zpool import "External Raid"


16. Check that finder reports the new size correctly.

Can anyone see anything wrong with the above? Am I correct in thinking that my 2 * 8 + 2 * 3 TB disks will now provide me with an 11TB of redundant storage as opposed to the 6TB I have now?

Thanks so much in advance for any help or advice offered.
bro708
 
Posts: 4
Joined: Fri Apr 03, 2020 4:28 am

Re: First Time Expanding a Pool

Postby Jimbo » Sun Apr 05, 2020 3:04 am

AFAIK, disks need to be the same size in a RAIDZ or all drives are limited to the smallest drive capacity (you can use different size disks in a stripe and get the extra space, but no redundancy).

I’ve only ever done expansion on mirrored pairs. I *think* what you have there would work, but only if you replace all the drives.

Cheers!
James
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: First Time Expanding a Pool

Postby bro708 » Sun Apr 05, 2020 4:10 am

Jimbo wrote:AFAIK, disks need to be the same size in a RAIDZ or all drives are limited to the smallest drive capacity (you can use different size disks in a stripe and get the extra space, but no redundancy).

I’ve only ever done expansion on mirrored pairs. I *think* what you have there would work, but only if you replace all the drives.

Cheers!
James


Thanks for the reply James, I've done a ton of reading both before and after making this post and had kind of figured this out for myself already. Luckily I think I have enough space between a spare 3TB drive I have laying around as well as an external 1TB I can borrow to back everything up.

I'm then planning on destroying my existing pool and creating a new striped pool composed of two pairs of mirrors, one of them with the 2 new 8TB drives and the other with 2 of the existing 3TB drives to give me the 11TB I wanted and that should do me for the next few years hopefully. Then I'll look to do the one disk at a time resilvering upgrade thing on a per mirror basis as and when space runs low.

Seems to me this is the most cost effective way to go about future upgrades? I'd picked RAIDZ2 originally because of the double parity and hadn't really twigged that that meant my disks would all need to always be the same size. Live and learn I guess.

Thanks so much for taking the time to read and reply anyway, much appreciated.
bro708
 
Posts: 4
Joined: Fri Apr 03, 2020 4:28 am

Re: First Time Expanding a Pool

Postby Jimbo » Mon Apr 06, 2020 4:42 am

Yeah, I’ve done that - two mirrored pairs of differing sizes in a stripe. Not ideal, but works fine and can be rolling upgraded.

Alternatively, you can also stripe RAIDZ vdevs, so for example, two 4 disk RAIDZ vdevs which will get you somewhat better performance than a single RAIDZ at slightly lower capacity; as with everything, it depends what your requirements and end goals are. Mirrors tend to be the most performant and flexible, but not necessarily cost, capacity or physical space efficient.

I started with mirrored pairs back in the day but currently run a single RAIDZ of 4 big drives. Some days I miss the performance... but not enough to drop lots of cash.

Cheers!
James
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: First Time Expanding a Pool

Postby bro708 » Mon Apr 06, 2020 5:25 am

Perfect, Thankyou.

It's flexibility for future upgrades that's most important to me really, the machine is primarily a media server so will need to house an ever growing collection as larger storage becomes more affordable but only really needs to write as fast as Handbrake can encode on my ageing i5 system and read as fast as the bitrate on my 1080p (or maybe eventually 4k) encodes.

I'm restricted to the four bays in my little thunderbolt enclosure unfortunately as it's attached to an iMac that already has an SSD squeezed in alongside the original 3.5" drive which is now my Time Machine backup disk.

So it sounds like a pair of mirrors is probably the best option for my needs, unless you can see any benefit to using a pair of two disk RAIDZ1 vdevs over a pair of mirror vdevs?

While I have your attention, if you don't mind another question am I right in thinking that for a pool that's going to contain almost exclusive multimedia content I should be using ashift=12, recordsize=1M and compression=lz4? Want to make sure I get it right this time...
bro708
 
Posts: 4
Joined: Fri Apr 03, 2020 4:28 am

Re: First Time Expanding a Pool

Postby Jimbo » Wed Apr 08, 2020 3:50 am

RAIDZ1 will require a minimum of three disks for a vdev. That will leave you two drives shy if you’re limited to 4 bays.

Almost all drives should be using ashift=12 these days, lz4 can just generally be enabled (it probably won’t do anything for you) as the overhead will be negligible and, to be honest, don’t mess with the recordsize unless you have a real need (again, probably not going to get you anything).

Cheers!
James
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: First Time Expanding a Pool

Postby bro708 » Wed Apr 08, 2020 4:10 am

Perfect, then that's exactly what I'll do. Pair of mirrors, ashift=12, compression=lz4.

Thanks so much for all your help, greatly appreciated.
bro708
 
Posts: 4
Joined: Fri Apr 03, 2020 4:28 am


Return to Absolute Beginners Section

Who is online

Users browsing this forum: No registered users and 8 guests