How to fix "Device removal detected"?

[roemer]

Unfortunately, I encountered the "Device removal detected" error again.
I have an external RAID enclosure connected to my Mac Mini via a Kanex Thunderbolt-to-SATA adapter.
The Mac Mini is running in headless mode, and I put the machine to sleep when not needed (say during the night).
And I am on O3X 1.5.2

Every now and then (I haven't found what triggers it) the hard drives do not correctly register with the Mac Mini,
typically after a wake after sleep. In this case, on I find error messages such as these in the system.log:

Code: Select all

Jun 22 08:31:20 obelix kernel[0]: [ PCI configuration begin ]
Jun 22 08:31:20 obelix kernel[0]: [ PCI configuration end, bridges 13, devices 18 ]
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk4'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk2'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk3'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk5'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk4s1'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device '/private/var/run/disk/by-id/media-CBA9F7FB-3767-2B48-BDB7-38D107DF96E5' removal requested
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk4s9'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk2s1'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device '/private/var/run/disk/by-id/media-5DF749B8-6E58-1241-B618-6A248F2F5621' removal requested
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk2s9'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk3s1'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device '/private/var/run/disk/by-id/media-69A9EB7E-97D8-3E40-96C5-089E09C6142A' removal requested
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk3s9'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk5s1'
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device '/private/var/run/disk/by-id/media-9D6A15C8-29DA-8D49-9969-79D6A4B8AE31' removal requested
Jun 22 08:31:20 obelix kernel[0]: ZFS: Device removal detected: 'disk5s9'
Jun 22 08:31:20 obelix kernel[0]: 226743.496522 AppleUSBXHCIPCI@00000000: AppleUSBHostController::setPowerStateGated: going to state 1 returned 0xe00002e9


Interestingly, in the above case the server was actually already running for quite some time, but I guess it was the HDD sleep/wake cycle which failed?

Of course, I would love to know on how to avoid this in the first place.
But secondly, how can I fixe this "Device removal detected" situation after it occurred?
I would like to clear and re-activate the pool after I re-attached the drives without rebooting the server.
Is this possible? (I couldn't find corresponding information in the Wiki or in ZFS-8000-HC)

A bit more information on the system status in this case:

Code: Select all

zpool status
  pool: ztank
 state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://zfsonlinux.org/msg/ZFS-8000-HC
config:
   NAME                                            STATE     READ WRITE CKSUM
   ztank                                           UNAVAIL      0     0     0  insufficient replicas
     raidz2-0                                      UNAVAIL      0     0     0  insufficient replicas
       media-CBA9F7FB-3767-2B48-BDB7-38D107DF96E5  REMOVED      0     0     0
       media-5DF749B8-6E58-1241-B618-6A248F2F5621  REMOVED      0     0     0
       media-69A9EB7E-97D8-3E40-96C5-089E09C6142A  REMOVED      0     0     0
       media-9D6A15C8-29DA-8D49-9969-79D6A4B8AE31  REMOVED      0     0     0
   logs
     media-B07E3B9A-6ED6-4439-9951-B4FEE9FC508E    ONLINE       0     0     0
   cache
     media-4A353723-4D0F-4008-B064-114BD53DC931    ONLINE       0     0     0


diskutil list does not show the removed disk drives.

The corresponding disks also do not show in /dev, so I don't know which arguments to give to hdiutil for re-attachment:

Code: Select all

obelix:~ admin$ ls /dev/disk
disk0    disk0s1  disk0s2  disk0s3  disk0s4  disk0s5  disk1    disk1s1  disk1s2  disk1s3  disk6


[Ikukuru]

I don't have much advice, but I have experienced similar problems in the past. For me, it was flaky eSATA, and I would get quite a few errors, etc.

I upgraded the computer, stilling using the same disks, and haven't seen a single error nor had the problem of a device mysteriously disconnecting.

Fact is, from the looks of things, the adapter and its driver are probably not stable. There is a setting under System Preferenes > Power Saving to disable HDD sleep. You could give it a go. It never helped me.

You can of course offline certain disk and online them again, if they are still visible. zpool offline <poolname> <device-id> and then zpool online <poolname> <dewvice-id> It should do a momentary resilver.

Someone more knowledgeable may be able to help more.