zpool replace yields "disk busy"

All your general support questions for OpenZFS on OS X.

zpool replace yields "disk busy"

Postby haer22 » Sat Aug 29, 2015 2:04 am

I encountered a disk problem and now I want to replace the disk:
Code: Select all
[ihecc:~] root# zpool status zeus
  pool: zeus
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
   invalid.  Sufficient replicas exist for the pool to continue
   functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 13,2M in 0h0m with 0 errors on Sat Aug 29 11:11:59 2015
config:

   NAME                     STATE     READ WRITE CKSUM
   zeus                     DEGRADED     0     0     0
     raidz1-0               DEGRADED     0     0     0
       disk1                ONLINE       0     0     0
       disk2                ONLINE       0     0     0
       9516495821961256675  UNAVAIL      0     0     0  was /dev/disk4s1
       disk3                ONLINE       0     0     0
     raidz1-1               ONLINE       0     0     0
       disk14               ONLINE       0     0     0
       disk9                ONLINE       0     0     0
       disk8                ONLINE       0     0     0
       disk12               ONLINE       0     0     0
   logs
     disk4s4                ONLINE       0     0     0
   cache
     disk4s5                ONLINE       0     0     0

errors: No known data errors

When I try to replace to culprit I get:

Code: Select all
[ihecc:~] root# zpool replace -f zeus 9516495821961256675 disk13
cannot replace 9516495821961256675 with disk13: disk13 is busy


At the same time I get the Apple prompt that MacOSX cannot read the disk, Do I want to format?

Somehow I guess the Finder blocks the device while it asks me about formatting and therefor zpool complains.

When I check the partitioning it seems like ifs has done it's thing, i.e. partitioned the disk:
Code: Select all
/dev/disk13
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk13
   1:                        ZFS                         3.0 TB     disk13s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk13s9

Just upgraded to 1.3.1-rc2 and I am running Yosemite 10.10.5

Any suggestions? It is 12 TB so I'd rather not rebuild everything from my backup...
haer22
 
Posts: 123
Joined: Sun Mar 23, 2014 2:13 am

Re: zpool replace yields "disk busy"

Postby lundman » Sat Aug 29, 2015 3:42 pm

If your disk13 is busy, something has it open. I assume it is not listed in zpool status so you could try just telling diskutil to unmount it.

diskutil unmountdisk disk13

and see how that goes
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: zpool replace yields "disk busy"

Postby haer22 » Sun Aug 30, 2015 4:42 am

disk13 was already Unmounted by DiskUtil so nobody should be using it and there was no prompt from MacOS whether I want it formatted.

But when I do the zpool replace... I get the prompt.

I guess zpool starts with an initialize of disk13 according to ZFS style and that is when MacOS sees a "new disk" and prompts me for formatting.

Worked many time before but that was version 1.3.0
haer22
 
Posts: 123
Joined: Sun Mar 23, 2014 2:13 am

Re: zpool replace yields "disk busy"

Postby haer22 » Sun Aug 30, 2015 9:17 am

After messing around some more I gave up and restored a backup from TeMachine 10 days ago. Well before the strange stuff. So far so good. I will let te resilvering run its course and then I will slowly move forward in versions and updates to see who/what was the culprit.
haer22
 
Posts: 123
Joined: Sun Mar 23, 2014 2:13 am

Re: zpool replace yields "disk busy"

Postby Brendon » Sun Aug 30, 2015 3:57 pm

Not sure what is happening for you, but I replaced a failed HDD in a RAIDZ running recent code just the other day without issue.

Cheers
Brendon
Brendon
 
Posts: 286
Joined: Thu Mar 06, 2014 12:51 pm

Re: zpool replace yields "disk busy"

Postby jdwhite » Sat Oct 17, 2015 8:37 pm

I'm running into this exact problem with OS X 10.11 and OpenZFS 1.4.2. I even tried using Onyx to enable the Quit Finder option and then quit Finder, but still got the prompt to initialize when using 'zpool replace' even though Finder is not running and did not start as a result of the 'zpool replace' command. Both 'lsof' and 'fuser' show nothing using the disk. Not sure where to go from here.
jdwhite
 
Posts: 11
Joined: Sat May 10, 2014 6:04 pm

Re: zpool replace yields "disk busy"

Postby jdwhite » Sun Oct 18, 2015 4:01 pm

jdwhite wrote:I'm running into this exact problem with OS X 10.11 and OpenZFS 1.4.2. I even tried using Onyx to enable the Quit Finder option and then quit Finder, but still got the prompt to initialize when using 'zpool replace' even though Finder is not running and did not start as a result of the 'zpool replace' command. Both 'lsof' and 'fuser' show nothing using the disk. Not sure where to go from here.


I've looked at https://openzfsonosx.org/wiki/Suppressi ... ing_pop-up and verified, using gpart, that the partition type GUID was for zfs. I noticed that partition 9 is set to a non-zfs type, so I even set that one to zfs, but the process of replacing changes the partition type to BF07 from BF01 and I'm wondering if that's causing the "not readable" popup.

Code: Select all
mmc:~ root# gdisk -l /dev/disk2
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/disk2: 3907029168 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 3A3F518C-4EAF-424C-B5FB-06E77E24FD90
Partition table holds up to 9 entries
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2157 sectors (1.1 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      3907012607   1.8 TiB     BF01  zfs
   9      3907012608      3907028991   8.0 MiB     BF01


Both partition types are now BF01, so now try and replace:

Code: Select all
mmc:~ root# zpool replace Qx2 11299653104960740877 /dev/disk2
cannot replace 11299653104960740877 with /dev/disk2: /dev/disk2 is busy


Now note that partition 9 is changed to type BF07.

Code: Select all
mmc:~ root# gdisk -l /dev/disk2
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/disk2: 3907029168 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 274CE1C0-D837-A747-BDF7-DA1E223FE4BA
Partition table holds up to 9 entries
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2157 sectors (1.1 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      3907012607   1.8 TiB     BF01  zfs
   9      3907012608      3907028991   8.0 MiB     BF07


I also tried to send the stop signal to /usr/libexec/diskarbitrationd, but the 'zpool replace' command just hung until I sent the continue signal.

My options at this point seem to be:
  • destroy and recreate the pool
  • spin up a different OS that supports ZFS and doesn't have this annoyance to run the zpool replace command

Any other ideas?
jdwhite
 
Posts: 11
Joined: Sat May 10, 2014 6:04 pm

Re: zpool replace yields "disk busy"

Postby Brendon » Sun Oct 18, 2015 8:17 pm

Well this is your lucky day, after yet another seagate HDD failure, I will be replacing one tomorrow, I suspect. I will document whatever works for me here or on the wiki or something.

Cheers
Brendon
Brendon
 
Posts: 286
Joined: Thu Mar 06, 2014 12:51 pm

Re: zpool replace yields "disk busy"

Postby jdwhite » Mon Oct 19, 2015 6:54 am

Brendon wrote:Well this is your lucky day, after yet another seagate HDD failure, I will be replacing one tomorrow, I suspect. I will document whatever works for me here or on the wiki or something.

Cheers
Brendon


I look forward to your reply. I did upgrade to 1.4.5 this morning, but made no difference.

-Jason
jdwhite
 
Posts: 11
Joined: Sat May 10, 2014 6:04 pm

Re: zpool replace yields "disk busy"

Postby ilovezfs » Fri Oct 23, 2015 8:09 pm

The gpt hacks to suppress the popup have not been needed for a long, long time. So you're barking up the wrong tree there.

The reason for the popup in this case is going to be that it IS marked ZFS, and therefore the probing is handed off to "us," by which I mean /Library/Filesystem/zfs.fs (1.4.5 10.11 version) or /System/Library/Filesystems/zfs.fs (< 1.4.5 or 1.4.5 with < OS X 10.11). We look at your disk and say, "There is no pool there so we don't recognize jack but thanks for asking us since it was marked ZFS after all." The popup itself is harmless and simply OS X trying to be helpful.

There was also probably no reason at all to restore from Time Machine, and you should not expect that to make any difference in this case.

There is no reason for you to be interacting with diskarbitrationd directly.

Messing with Onyx and Finder is also going to be beside the point. Finder has nothing to do with the price of bread unless there's a mounted file system.

First can we see the output of

Code: Select all
sudo zdb -l /dev/disk#s1

It's possible you need to do a labelclear (and also possibly a gpt destroy), but one step at a time.
ilovezfs
 
Posts: 232
Joined: Thu Mar 06, 2014 7:58 am

Next

Return to General Help

Who is online

Users browsing this forum: No registered users and 30 guests