I just got the following panic:
System was doing a scrub after I pulled the esate power cable by mistake. The drives came back up after I did a clear and were subbing.
After Booboo:
sh-3.2# zpool status
pool: BigDog
state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see:
http://zfsonlinux.org/msg/ZFS-8000-HC scan: resilvered 60.1G in 0h47m with 0 errors on Fri Mar 4 15:14:14 2016
config:
NAME STATE READ WRITE CKSUM
BigDog UNAVAIL 0 0 0 insufficient replicas
raidz1-0 UNAVAIL 0 2 0 insufficient replicas
media-371AB90C-5BF2-6843-A062-BAB8FFF49673 FAULTED 0 1 0 too many errors
media-9D7B1EE9-82D4-F543-A8B6-F0A8F47088CD FAULTED 0 1 0 too many errors
media-68D1475A-7161-A349-B356-949486CAD2DB FAULTED 0 1 0 too many errors
media-DD8EE5C7-FFAB-1443-8E45-924EE23FDA2A FAULTED 0 1 0 too many errors
cache
disk2 ONLINE 1 55.1K 0
errors: 6 data errors, use '-v' for a list
------------------------------
My actions to fix booboo
sh-3.2# zpool clear BigDog
sh-3.2# zpool status
pool: BigDog
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Mar 9 09:05:35 2016
968K scanned out of 321G at 484K/s, 193h28m to go
572K resilvered, 0.00% done
config:
NAME STATE READ WRITE CKSUM
BigDog ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
media-371AB90C-5BF2-6843-A062-BAB8FFF49673 ONLINE 0 0 0 (resilvering)
media-9D7B1EE9-82D4-F543-A8B6-F0A8F47088CD ONLINE 0 0 0 (resilvering)
media-68D1475A-7161-A349-B356-949486CAD2DB ONLINE 0 0 0 (resilvering)
media-DD8EE5C7-FFAB-1443-8E45-924EE23FDA2A ONLINE 0 0 0 (resilvering)
cache
disk2 ONLINE 7 55.7K 0
few min later
scan: resilvered 192K in 0h0m with 0 errors on Wed Mar 9 09:05:39 2016
config:
NAME STATE READ WRITE CKSUM
BigDog ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
media-371AB90C-5BF2-6843-A062-BAB8FFF49673 ONLINE 0 0 0
media-9D7B1EE9-82D4-F543-A8B6-F0A8F47088CD ONLINE 0 0 0
media-68D1475A-7161-A349-B356-949486CAD2DB ONLINE 0 0 0
media-DD8EE5C7-FFAB-1443-8E45-924EE23FDA2A ONLINE 0 0 0
cache
disk2 ONLINE 81 122K 0
errors: No known data errors
-----
After system crashed and rebooted:
zpool status
pool: BigDog
state: ONLINE
scan: resilvered 192K in 0h0m with 0 errors on Wed Mar 9 09:05:39 2016
config:
NAME STATE READ WRITE CKSUM
BigDog ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
media-371AB90C-5BF2-6843-A062-BAB8FFF49673 ONLINE 0 0 0
media-9D7B1EE9-82D4-F543-A8B6-F0A8F47088CD ONLINE 0 0 0
media-68D1475A-7161-A349-B356-949486CAD2DB ONLINE 0 0 0
media-DD8EE5C7-FFAB-1443-8E45-924EE23FDA2A ONLINE 0 0 0
cache
disk2 ONLINE 0 0 0
errors: No known data errors
-----------
Here is the output from panic to send to Apple:
Anonymous UUID: 2CD697E9-9B5E-89C5-86D4-0316A2FCB615
Wed Mar 9 09:47:11 2016
*** Panic Report ***
panic(cpu 5 caller 0xffffff800d1ce40a): Kernel trap at 0xffffff7f8d8590b0, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000000, CR3: 0x00000002a9905115, CR4: 0x00000000000626e0
RAX: 0x00000000e00002c0, RBX: 0x0000000000000009, RCX: 0xffffff802c6a6a08, RDX: 0x0000000000000001
RSP: 0xffffff92496bb000, RBP: 0xffffff92496bb080, RSI: 0x0000000000000004, RDI: 0x0000000000000000
R8: 0x000000000000ffff, R9: 0x0000000000000001, R10: 0x000000000900014b, R11: 0x00000000e0000000
R12: 0xffffff802c74e510, R13: 0xffffff80441741e0, R14: 0x0000000000000000, R15: 0xffffff800d920c70
RFL: 0x0000000000010286, RIP: 0xffffff7f8d8590b0, CS: 0x0000000000000008, SS: 0x0000000000000000
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0x5, PL: 0
Backtrace (CPU 5), Frame : Return Address
0xffffff92496bac90 : 0xffffff800d0dab12 mach_kernel : _panic + 0xe2
0xffffff92496bad10 : 0xffffff800d1ce40a mach_kernel : _kernel_trap + 0x91a
0xffffff92496baef0 : 0xffffff800d1ec273 mach_kernel : _return_from_trap + 0xe3
0xffffff92496baf10 : 0xffffff7f8d8590b0 com.apple.iokit.IOStorageFamily : _dkioctl + 0x2bd
0xffffff92496bb080 : 0xffffff800d338da5 mach_kernel : _spec_ioctl + 0x1f5
0xffffff92496bb0c0 : 0xffffff800d326d0e mach_kernel : _VNOP_IOCTL + 0x16e
0xffffff92496bb150 : 0xffffff7f8d9267ee net.lundman.zfs : _vdev_disk_io_start + 0x197
0xffffff92496bb180 : 0xffffff7f8d96900f net.lundman.zfs : _zio_vdev_io_start + 0x258
0xffffff92496bb1d0 : 0xffffff7f8d96603e net.lundman.zfs : _zio_nowait + 0x185
0xffffff92496bb230 : 0xffffff7f8d965e3f net.lundman.zfs : _zio_ioctl + 0xe8
0xffffff92496bb2d0 : 0xffffff7f8d9663b4 net.lundman.zfs : _zio_flush + 0x29
0xffffff92496bb2f0 : 0xffffff7f8d963164 net.lundman.zfs : _zil_commit + 0x909
0xffffff92496bb410 : 0xffffff7f8d951b95 net.lundman.zfs : _zfs_fsync + 0xf1
0xffffff92496bb440 : 0xffffff800d2ff077 mach_kernel : _vfs_addtrigger + 0x1ba7
0xffffff92496bb4c0 : 0xffffff800d2fed26 mach_kernel : _vfs_addtrigger + 0x1856
0xffffff92496bb500 : 0xffffff800d2f8a91 mach_kernel : _vnode_recycle + 0x51
0xffffff92496bb520 : 0xffffff7f8d95065a net.lundman.zfs : _zfs_remove + 0x524
0xffffff92496bb620 : 0xffffff7f8d9563bd net.lundman.zfs : _zfs_vnop_remove + 0x73
0xffffff92496bbad0 : 0xffffff800d32715e mach_kernel : _VNOP_REMOVE + 0x5e
0xffffff92496bbb50 : 0xffffff800d311fd8 mach_kernel : _vfs_purge + 0x3f18
0xffffff92496bbf60 : 0xffffff800d627f91 mach_kernel : _unix_syscall64 + 0x251
0xffffff92496bbfb0 : 0xffffff800d1eca76 mach_kernel : _hndl_unix_scall64 + 0x16
Kernel Extensions in backtrace:
com.apple.iokit.IOStorageFamily(2.1)[DC1AAB7C-F417-3238-BB3F-2A5B84D67B90]@0xffffff7f8d84c000->0xffffff7f8d872fff
net.lundman.zfs(1.5)[DC12A26F-A90C-3D97-8D88-E7B61B6330EA]@0xffffff7f8d8cb000->0xffffff7f8db0bfff
dependency: com.apple.iokit.IOStorageFamily(2.1)[DC1AAB7C-F417-3238-BB3F-2A5B84D67B90]@0xffffff7f8d84c000
dependency: net.lundman.spl(1.5.0)[B3517941-D80C-3026-B2F5-2F9B60388BDB]@0xffffff7f8d87c000
BSD process name corresponding to current thread: DesktopServicesH
Boot args: keepsyms=1
Mac OS version:
15E49a
Kernel version:
Darwin Kernel Version 15.4.0: Fri Feb 12 15:56:47 PST 2016; root:xnu-3248.40.183~6/RELEASE_X86_64
Kernel UUID: C4053AC4-EDC1-36CC-9A25-0DC576152473
Kernel slide: 0x000000000ce00000
Kernel text base: 0xffffff800d000000
__HIB text base: 0xffffff800cf00000
System model name: Macmini5,3 (Mac-7BA5B2794B2CDB12)
System uptime in nanoseconds: 47395643202311
last loaded kext at 42683183765379: com.apple.driver.AppleXsanScheme 3 (addr 0xffffff7f900db000, size 32768)
last unloaded kext at 41917314439345: com.apple.driver.AppleUSBMergeNub 900.4.1 (addr 0xffffff7f900d8000, size 12288)
loaded kexts:
com.sophos.kext.sav 9.4.52
com.sophos.nke.swi 9.4.52
org.virtualbox.kext.VBoxNetAdp 5.0.10
org.virtualbox.kext.VBoxNetFlt 5.0.10
com.nvidia.CUDA 1.1.0
org.virtualbox.kext.VBoxUSB 5.0.10
org.virtualbox.kext.VBoxDrv 5.0.10
net.lundman.zfs 1.5.0
net.lundman.spl 1.5.0
com.dvdfab.kext.fabio 1
com.makemkv.kext.daspi 1.5
com.apple.filesystems.afpfs 11.0
com.apple.nke.asp-tcp 8.0.0
com.apple.filesystems.smbfs 3.0.0
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.driver.AppleBluetoothMultitouch 90.3
com.apple.filesystems.autofs 3.0
com.apple.driver.ApplePlatformEnabler 2.6.0d0
com.apple.driver.AGPM 110.21.18
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleMikeyHIDDriver 124
com.apple.driver.AppleUpstreamUserClient 3.6.1
com.apple.driver.AppleHDA 274.7
com.apple.driver.AppleMCCSControl 1.2.13
com.apple.driver.AppleMikeyDriver 274.7
com.apple.driver.pmtelemetry 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.iokit.IOBluetoothSerialManager 4.4.4f4
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleIntelHD3000Graphics 10.0.0
com.apple.driver.AppleThunderboltIP 3.0.8
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 4.4.4f4
com.apple.driver.AppleHV 1
com.apple.driver.AppleSMCPDRC 1.0.0
com.apple.driver.AppleLPC 3.1
com.apple.driver.AppleIntelSNBGraphicsFB 10.0.0
com.apple.driver.AudioAUUC 1.70
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleIRController 327.5
com.apple.iokit.SCSITaskUserClient 3.7.7
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.BootCache 38
com.apple.iokit.AppleIntelI210Ethernet 2.2.1b3
com.apple.driver.AppleFWOHCI 5.5.2
com.apple.iokit.IOAHCIBlockStorage 2.8.5
com.apple.driver.AirPort.Brcm4331 800.20.24
com.apple.driver.AppleSDXC 1.7.0
com.apple.iokit.AppleBCM5701Ethernet 10.2.0
com.apple.driver.AppleAHCIPort 3.1.8
com.apple.driver.usb.AppleUSBEHCIPCI 1.0.1
com.apple.driver.AppleACPIButtons 4.0
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleACPIEC 4.0
com.apple.driver.AppleAPIC 1.7
com.apple.driver.AppleIntelCPUPowerManagementClient 218.0.0
com.apple.nke.applicationfirewall 163
com.apple.security.quarantine 3
com.apple.security.TMSafetyNet 8
com.apple.driver.AppleIntelCPUPowerManagement 218.0.0
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleUSBMergeNub 900.4.1
com.apple.security.SecureRemotePassword 1.0
com.apple.driver.AppleBluetoothHIDKeyboard 181
com.apple.driver.AppleHIDKeyboard 181
com.apple.driver.IOBluetoothHIDDriver 4.4.4f4
com.apple.driver.AppleMultitouchDriver 304.10.2
com.apple.driver.AppleHIDTransport 5
com.apple.iokit.IOUSBUserClient 900.4.1
com.apple.kext.triggers 1.0
com.apple.AppleGraphicsDeviceControl 3.12.6
com.apple.driver.DspFuncLib 274.7
com.apple.kext.OSvKernDSPLib 525
com.apple.iokit.IOSurface 108.2.1
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSMBusController 1.0.14d1
com.apple.driver.CoreCaptureResponder 1
com.apple.driver.AppleSMBusPCI 1.0.14d1
com.apple.iokit.IOBluetoothHostControllerUSBTransport 4.4.4f4
com.apple.iokit.IOBluetoothFamily 4.4.4f4
com.apple.driver.AppleHDAController 274.7
com.apple.iokit.IOHDAFamily 274.7
com.apple.iokit.IOFireWireIP 2.2.6
com.apple.iokit.IONDRVSupport 2.4.1
com.apple.iokit.IOGraphicsFamily 2.4.1
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d7
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOSCSIMultimediaCommandsDevice 3.7.7
com.apple.iokit.IOBDStorageFamily 1.8
com.apple.iokit.IODVDStorageFamily 1.8
com.apple.iokit.IOCDStorageFamily 1.8
com.apple.driver.AppleThunderboltDPOutAdapter 4.1.3
com.apple.driver.AppleThunderboltPCIUpAdapter 2.0.2
com.apple.driver.AppleThunderboltDPInAdapter 4.1.3
com.apple.driver.AppleThunderboltDPAdapterFamily 4.1.3
com.apple.driver.AppleThunderboltPCIDownAdapter 2.0.2
com.apple.iokit.IOUSBHIDDriver 900.4.1
com.apple.iokit.IOSCSIBlockCommandsDevice 3.7.7
com.apple.driver.AppleUSBAudio 303.3
com.apple.iokit.IOAudioFamily 204.3
com.apple.vecLib.kext 1.2.0
com.apple.iokit.IOUSBMassStorageDriver 1.0.0
com.apple.iokit.IOSCSIArchitectureModelFamily 3.7.7
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.0.1
com.apple.driver.usb.AppleUSBHub 1.0.1
com.apple.driver.usb.AppleUSBXHCIPCI 1.0.1
com.apple.driver.usb.AppleUSBXHCI 1.0.1
com.apple.driver.AppleThunderboltNHI 4.0.4
com.apple.iokit.IOThunderboltFamily 6.0.2
com.apple.iokit.IOFireWireFamily 4.6.0
com.apple.iokit.IOEthernetAVBController 1.0.3b3
com.apple.iokit.IO80211Family 1110.26
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.2
com.apple.driver.corecapture 1.0.4
com.apple.iokit.IOAHCIFamily 2.8.1
com.apple.driver.usb.AppleUSBEHCI 1.0.1
com.apple.iokit.IOUSBFamily 900.4.1
com.apple.iokit.IOUSBHostFamily 1.0.1
com.apple.driver.AppleUSBHostMergeProperties 1.0.1
com.apple.driver.AppleEFINVRAM 2.0
com.apple.driver.AppleEFIRuntime 2.0
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.DiskImages 417.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOReportFamily 31
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleACPIPlatform 4.0
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.Libm 1
com.apple.kec.pthread 1
com.apple.kec.corecrypto 1.0
Model: Macmini5,3, BootROM MM51.0077.B13, 4 processors, Intel Core i7, 2 GHz, 16 GB, SMC 1.77f0
Graphics: Intel HD Graphics 3000, Intel HD Graphics 3000, Built-In
Memory Module: BANK 0/DIMM0, 8 GB, DDR3, 1333 MHz, 0x8502, 0x505341333847313333333253000000000000
Memory Module: BANK 1/DIMM0, 8 GB, DDR3, 1333 MHz, 0x8502, 0x505341333847313333333253000000000000
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0xE4), Broadcom BCM43xx 1.0 (5.106.98.100.24)
Bluetooth: Version 4.4.4f4 17685, 3 services, 27 devices, 1 incoming serial ports
Network Service: Ethernet, Ethernet, en3
PCI Card: ethernet, Ethernet Controller, Thunderbolt@194,0,0
PCI Card: pci1b73,1100, USB eXtensible Host Controller, Thunderbolt@193,0,0
PCI Card: pci1b21,612, AHCI Controller, Thunderbolt@192,0,0
Serial ATA Device: Samsung SSD 850 EVO 500GB, 500.11 GB
Serial ATA Device: ST9500420ASG, 500.11 GB
Serial ATA Device: spsata_pm_name
USB Device: USB 2.0 Bus
USB Device: Hub
USB Device: SD-3102V-U3
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
USB Device: USB 2.0 Bus
USB Device: Hub
USB Device: IR Receiver
USB Device: USB 3.0 Bus
USB Device: Expansion Desk
USB Device: USB audio CODEC
USB Device: iPad
Thunderbolt Bus: Mac mini, Apple Inc., 22.2
Thunderbolt Device: Thunderbolt Station 2, CalDigit, Inc., 3, 25.1