cannot expand vdev with replaced disks

All your general support questions for OpenZFS on OS X.

cannot expand vdev with replaced disks

Postby Ikukuru » Mon Jan 27, 2020 7:09 am

I have replaced two mirrored 8TB disks with 10TB disks and expected to get an extra 2TB in my pool of 6x disks. The other 4 disks are 8TB.

What am I missing here? I was sure that mirrored VDEV can be expanded like this.

When I try to do zpool online I get the following error:
Code: Select all
 expand media-7E76C57D-B43C-284C-8AF2-080DBD91790U: cannot relabel '/dev/disk5': unable to open device: 13


Code: Select all
  pool: sanlux
 state: ONLINE
  scan: resilvered 7.87G in 0 days 05:01:41 with 0 errors on Mon Jan 27 22:38:01 2020
config:

   NAME                                            STATE     READ WRITE CKSUM
   sanlux                                        ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-1A76C57D-B43C-284C-8AF2-080DBD91234A            ONLINE       0     0     0
       media-96GG7619-9678-E241-8F94-71C4D4BB944B  ONLINE       0     0     0
     mirror-1                                      ONLINE       0     0     0
       media-7E76C57D-B43C-284C-8AF2-080DBD91790U  ONLINE       0     0     0
       media-61C3587C-1182-B24F-9217-3201CD02C59A  ONLINE       0     0     0
     mirror-2                                      ONLINE       0     0     0
       media-6CC72605-2C30-0A4D-8876-1F7305A50A21  ONLINE       0     0     0
       media-3EA2BB87-ADB7-B34C-9672-29956144875C  ONLINE       0     0     0

errors: No known data errors
Ikukuru
 
Posts: 26
Joined: Mon Sep 22, 2014 4:17 pm

Re: cannot expand vdev with replaced disks

Postby lundman » Tue Feb 04, 2020 10:57 pm

Try issuing "zpool online -r pool device" on each new disk to make it check for size increase. When using replace command, it is best to use GUID for existing disks (zpool status -g) and /dev/diskX names for new disks. After that, use export/import to make device names be by-id again.
User avatar
lundman
 
Posts: 1327
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: cannot expand vdev with replaced disks

Postby nathansvt » Wed Apr 01, 2020 11:16 am

I have a similar issue. I tried online -e which didn't throw any errors but also didn't expand. Tried export/import and also no expansion. autoexpand=on is set for the pool.

I was hoping the -r suggestion above would help but I get this:
Code: Select all
$ zpool status
  pool: zmirror
 state: ONLINE
  scan: resilvered 1.35T in 0 days 17:10:34 with 0 errors on Wed Apr  1 13:24:41 2020
config:

   NAME                                            STATE     READ WRITE CKSUM
   zmirror                                         ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-AFFC40BF-63E9-3542-A952-3E423BF4011C  ONLINE       0     0     0
       media-01648D9C-E219-684B-8CE8-34D07BBFA582  ONLINE       0     0     0
     mirror-1                                      ONLINE       0     0     0
       media-4C079649-39FB-0F4D-A575-F8F4FA677122  ONLINE       0     0     0
       media-2E4F7E78-F21E-5A49-87BF-E506C98CE911  ONLINE       0     0     0
   cache
     disk8                                         ONLINE       0     0     0

errors: No known data errors
$ sudo zpool online -r zmirror media-4C079649-39FB-0F4D-A575-F8F4FA677122
invalid option 'r'
usage:
   online <pool> <device> ...



What do I make of invalid option 'r'
usage:
online <pool> <device> ...
nathansvt
 
Posts: 7
Joined: Mon Jan 19, 2015 10:25 am

Re: cannot expand vdev with replaced disks

Postby lundman » Wed Apr 01, 2020 11:44 pm

Ugh so sorry, its supposed to be "online -e"
User avatar
lundman
 
Posts: 1327
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: cannot expand vdev with replaced disks

Postby dmz » Mon Dec 06, 2021 6:05 pm

I'm having exactly this problem now, but with current OpenZFS (2.1.0) on Monterey. I have a pool that looks like this:

Code: Select all
   NAME        STATE     READ WRITE CKSUM
   ZFS         ONLINE       0     0     0
     mirror-0  ONLINE       0     0     0
       disk4   ONLINE       0     0     0
       disk1   ONLINE       0     0     0
     mirror-1  ONLINE       0     0     0
       disk3   ONLINE       0     0     0
       disk5   ONLINE       0     0     0

where mirror-0 is two 10TB disks and mirror-1 used to be two 10TB disks, but is now two 14TB disks. The way I migrated mirror-1 - which may have been suboptimal, but I thought would still work - was to zpool attach the two new 14TB disks to mirror-1, let them resilver, then zpool detach the two 10TB disks from mirror-1. I did not have autoexpand=on set initially, but I did set it after I removed the smaller disks, and after an export/import (which I did as recommended in some other thread, to allow zpool online -e to work), zpool list reports:

Code: Select all
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ZFS   18.1T  14.7T  3.46T        -     3.62T    35%    80%  1.00x    ONLINE  -

(note the 3.62T "EXPANDSZ", which seems like the right amount of additional space for an extra 4 TB per drive on mirror-1)

I've tried to do zpool online -e ZFS /dev/disk3, but get the error "cannot expand /dev/disk3: cannot relabel '/dev/disk3': unable to open device: 16" (and the same for /dev/disk4).

If I absolutely have to, I can attach the old 10TB drives to mirror-1 again, resilver them, detach the 14TB drives, and then use zfs replace instead, in the hope that it will do the right thing... but I'd really rather not. Any suggestions for how to get OpenZFS to expand the pool without doing that?
dmz
 
Posts: 19
Joined: Thu Mar 13, 2014 10:03 pm

Re: cannot expand vdev with replaced disks

Postby dmz » Sat Jan 01, 2022 10:04 pm

Just to follow up on this, I ended up having to do exactly what I didn't want to do: attach the old 10TB drives, detach the 14TB drives, and then zfs replace the 10TB drives one at a time. Then the pool expanded. But man, the intermediate resilvering was slow, since the only way I could hook up the old drives was over USB2...
dmz
 
Posts: 19
Joined: Thu Mar 13, 2014 10:03 pm

Re: cannot expand vdev with replaced disks

Postby lundman » Mon Jan 03, 2022 5:41 pm

This does sound a bit like regression and a fix should be applied - it certainly used to work.
User avatar
lundman
 
Posts: 1327
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan


Return to General Help

Who is online

Users browsing this forum: No registered users and 8 guests