Kernel Panic when connected through Thunderbolt and corrupt

[murraypetera]

The issue:
- After the pool get in a bad state, bad unmount, etc. trying to import it, when connected to eSATA or USB3 through Thunderbolt adapter, the system panics and reboots. This is not recoverable until I connect the drives directly to the Mac Mini's USB2 connector. At this point I am able to zpool import -R BigDog (the pool name), pool scrub, etc. all is happy when done.

If I try to do this while connected to the Thunderbolt config after a short time ~5 min of the disks blinking away and several zfs process running: zfs.util, zed, zconfigd, etc. the box panics and reboots.

My configuration

ZFS 1.4.5
Mac Mini late 2010 OSX 10.11.1 (did this with 10.10.? as well)
Vance HX4 external 4 Disk enclosure with USB3 and eSATA connections
SSD connected by USB

4 disk ZRAID + SSD Cache

Tested with Kanex and CalDigit adapter with both eSATA and USB3.

The Mini seems to really dislike the eSATA and I think this is what initiated my issue.
I wanted to benchmark under eSATA and then USB3 and see which did better.

How I got corrupted and the above issue:
1. system was connected, working running for weeks directly to USB2 on mini
2. Got Thuderbold adapter for speed
3. did unmount of pool...this does not seem to prevent the need to scrub after re-connecting. I think I do not know how to properly shut down a zpool.. I digress
4. attached through Thunderbolt eSATA, corrupted scrub starts
5. Stop scrub to do speed test
6. unmount pool, so I think
7. plugin USB connector through Thunderbolt - crashes system
7.1 not immediately... I don't remember exactly now my steps , ~ pool did not want to mount, disks not seen with diskutil list,
7.2 turn off power to enclosure
7.3 SSD not seen, unplug
7.4 Crash (I think)
8. connect all to USB3 through Thunderbolt .. not able to recover, system keeps panicking.
9. connect all disks to mini's USB2 and in the progress of scrubbing for many many hours over slow USB2

zpool status
pool: BigDog
state: ONLINE
scan: scrub in progress since Thu Oct 29 20:40:24 2015
1.11G scanned out of 2.70T at 386K/s, (scan is slow, no estimated time)
0 repaired, 0.04% done
config:

NAME STATE READ WRITE CKSUM
BigDog ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
media-7B75F17D-E7EA-4C3F-B02A-12BB040D68D1 ONLINE 0 0 0
media-35BA1A84-36F3-064D-9808-863434B4B5F5 ONLINE 0 0 0
media-012B10E6-2873-C944-AF72-91A8252D024E ONLINE 0 0 0
media-1F0796E7-210D-8744-BD76-C84B4B091B72 ONLINE 0 0 0
cache
media-574F4C76-458E-EE4E-BBB7-7A5E020D59A1 ONLINE 0 0 0

[murraypetera]

First Panic

Anonymous UUID: 2CD697E9-9B5E-89C5-86D4-0316A2FCB615

Thu Oct 29 20:58:15 2015

*** Panic Report ***
panic(cpu 6 caller 0xffffff8003dd6bba): Kernel trap at 0xffffff7f8445b892, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000010, CR3: 0x0000000007249000, CR4: 0x00000000000626e0
RAX: 0x0000000000000002, RBX: 0x00000000e00002d9, RCX: 0xffffff803b3b6608, RDX: 0x0000000000000001
RSP: 0xffffff91fbc4bc80, RBP: 0xffffff91fbc4bcc0, RSI: 0x0000000000000001, RDI: 0x0000000000000000
R8: 0x000000000000ffff, R9: 0x0000000000000001, R10: 0xffffff800f760540, R11: 0x0000000000000002
R12: 0x0000000000000000, R13: 0xffffff81e4666720, R14: 0x0000000000000000, R15: 0x000000000000000b
RFL: 0x0000000000010282, RIP: 0xffffff7f8445b892, CS: 0x0000000000000008, SS: 0x0000000000000010
Fault CR2: 0x0000000000000010, Error code: 0x0000000000000000, Fault CPU: 0x6, PL: 0

Backtrace (CPU 6), Frame : Return Address
0xffffff91fbc4b910 : 0xffffff8003ce5307
0xffffff91fbc4b990 : 0xffffff8003dd6bba
0xffffff91fbc4bb70 : 0xffffff8003df4313
0xffffff91fbc4bb90 : 0xffffff7f8445b892
0xffffff91fbc4bcc0 : 0xffffff7f84450de2
0xffffff91fbc4bd10 : 0xffffff7f84c67f47
0xffffff91fbc4bd70 : 0xffffff7f84c6bea6
0xffffff91fbc4bdc0 : 0xffffff7f846f4c81
0xffffff91fbc4be10 : 0xffffff7f846f4cec
0xffffff91fbc4be40 : 0xffffff7f84e1854d
0xffffff91fbc4be60 : 0xffffff80042b8538
0xffffff91fbc4bed0 : 0xffffff7f84e17684
0xffffff91fbc4bf00 : 0xffffff8003d1965a
0xffffff91fbc4bfb0 : 0xffffff8003dd15d7
Kernel Extensions in backtrace:
com.apple.iokit.IOStorageFamily(2.1)[956BA336-5C0A-37E9-B395-538D1BD95AC0]@0xffffff7f8444d000->0xffffff7f84473fff
com.apple.iokit.IOSCSIArchitectureModelFamily(3.7.7)[69F7C794-75F7-396C-9169-8CF2FDD29037]@0xffffff7f846f0000->0xffffff7f8471afff
com.apple.iokit.IOUSBMassStorageDriver(1.0)[E52C8824-B604-3741-997E-AE4089B296F1]@0xffffff7f84e0e000->0xffffff7f84e3afff
dependency: com.apple.iokit.IOUSBHostFamily(1.0.1)[CB073B0F-F3E0-37BD-8091-390367BEDAB0]@0xffffff7f8477f000
dependency: com.apple.iokit.IOStorageFamily(2.1)[956BA336-5C0A-37E9-B395-538D1BD95AC0]@0xffffff7f8444d000
dependency: com.apple.iokit.IOSCSIArchitectureModelFamily(3.7.7)[69F7C794-75F7-396C-9169-8CF2FDD29037]@0xffffff7f846f0000
com.apple.iokit.IOSCSIBlockCommandsDevice(3.7.7)[83CBB292-9DBE-380A-AC8D-9EA9429D5865]@0xffffff7f84c66000->0xffffff7f84c7dfff
dependency: com.apple.iokit.IOStorageFamily(2.1)[956BA336-5C0A-37E9-B395-538D1BD95AC0]@0xffffff7f8444d000
dependency: com.apple.iokit.IOSCSIArchitectureModelFamily(3.7.7)[69F7C794-75F7-396C-9169-8CF2FDD29037]@0xffffff7f846f0000

BSD process name corresponding to current thread: kernel_task
Boot args: kext-dev-mode=1

Mac OS version:
15B42

Kernel version:
Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64
Kernel UUID: AB5FC1B4-12E7-311E-8E6F-9023985D8C1D
Kernel slide: 0x0000000003a00000
Kernel text base: 0xffffff8003c00000
__HIB text base: 0xffffff8003b00000
System model name: Macmini5,3 (Mac-7BA5B2794B2CDB12)

System uptime in nanoseconds: 256759354319
last loaded kext at 70802255475: com.apple.iokit.SCSITaskUserClient 3.7.7 (addr 0xffffff7f8510e000, size 45056)
last unloaded kext at 147518290859: com.apple.driver.usb.AppleUSBUHCI 1.0.1 (addr 0xffffff7f84f19000, size 126976)
loaded kexts:
org.virtualbox.kext.VBoxNetAdp 4.3.30
org.virtualbox.kext.VBoxNetFlt 4.3.30
org.virtualbox.kext.VBoxUSB 4.3.30
com.nvidia.CUDA 1.1.0
org.virtualbox.kext.VBoxDrv 4.3.30
net.lundman.zfs 1.4.5
net.lundman.spl 1.4.5
com.apple.driver.AppleBluetoothMultitouch 90.3
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.filesystems.autofs 3.0
com.apple.driver.ApplePlatformEnabler 2.5.1d0
com.apple.driver.AGPM 110.20.21
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleMikeyHIDDriver 124
com.apple.driver.AppleHDA 272.50.31
com.apple.driver.AppleUpstreamUserClient 3.6.1
com.apple.driver.AppleMCCSControl 1.2.13
com.apple.driver.pmtelemetry 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.iokit.IOBluetoothSerialManager 4.4.2f1
com.apple.driver.AppleMikeyDriver 272.50.31
com.apple.driver.AppleIntelHD3000Graphics 10.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 4.4.2f1
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleSMCPDRC 1.0.0
com.apple.driver.AppleHV 1
com.apple.driver.AppleLPC 3.1
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.driver.AudioAUUC 1.70
com.apple.driver.AppleIntelSNBGraphicsFB 10.0.0
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleThunderboltIP 3.0.8
com.apple.driver.AppleIRController 327.5
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.BootCache 37
com.apple.iokit.AppleIntelI210Ethernet 2.2.1b3
com.apple.driver.AppleFWOHCI 5.5.2
com.apple.driver.AirPort.Brcm4331 800.20.24
com.apple.iokit.IOAHCIBlockStorage 2.8.0
com.apple.driver.AppleSDXC 1.7.0
com.apple.iokit.AppleBCM5701Ethernet 10.1.11
com.apple.driver.AppleAHCIPort 3.1.5
com.apple.driver.usb.AppleUSBEHCIPCI 1.0.1
com.apple.driver.AppleACPIButtons 4.0
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleACPIEC 4.0
com.apple.driver.AppleAPIC 1.7
com.apple.driver.AppleIntelCPUPowerManagementClient 218.0.0
com.apple.nke.applicationfirewall 163
com.apple.security.quarantine 3
com.apple.security.TMSafetyNet 8
com.apple.driver.AppleIntelCPUPowerManagement 218.0.0
com.apple.iokit.IOSCSIBlockCommandsDevice 3.7.7
com.apple.iokit.IOUSBMassStorageDriver 1.0.0
com.apple.driver.AppleBluetoothHIDKeyboard 181
com.apple.driver.AppleHIDKeyboard 181
com.apple.iokit.IOUSBUserClient 900.4.1
com.apple.driver.IOBluetoothHIDDriver 4.4.2f1
com.apple.driver.AppleMultitouchDriver 304.9
com.apple.driver.AppleHIDTransport 5
com.apple.kext.triggers 1.0
com.apple.AppleGraphicsDeviceControl 3.11.33b1
com.apple.driver.DspFuncLib 272.50.31
com.apple.kext.OSvKernDSPLib 525
com.apple.iokit.IOSurface 108.0.1
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleHDAController 272.50.31
com.apple.iokit.IOHDAFamily 272.50.31
com.apple.iokit.IOBluetoothHostControllerUSBTransport 4.4.2f1
com.apple.iokit.IOBluetoothFamily 4.4.2f1
com.apple.driver.AppleSMBusController 1.0.14d1
com.apple.driver.CoreCaptureResponder 1
com.apple.driver.AppleSMBusPCI 1.0.14d1
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d7
com.apple.iokit.IOFireWireIP 2.2.6
com.apple.iokit.IONDRVSupport 2.4.1
com.apple.iokit.IOGraphicsFamily 2.4.1
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOSCSIArchitectureModelFamily 3.7.7
com.apple.driver.usb.IOUSBHostHIDDevice 1.0.1
com.apple.iokit.IOUSBHIDDriver 900.4.1
com.apple.driver.AppleThunderboltDPInAdapter 4.1.2
com.apple.driver.AppleThunderboltDPOutAdapter 4.1.2
com.apple.driver.AppleThunderboltDPAdapterFamily 4.1.2
com.apple.driver.AppleThunderboltPCIUpAdapter 2.0.2
com.apple.driver.AppleThunderboltPCIDownAdapter 2.0.2
com.apple.driver.AppleUSBAudio 301.52
com.apple.iokit.IOAudioFamily 204.1
com.apple.vecLib.kext 1.2.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.0.1
com.apple.driver.usb.AppleUSBHub 1.0.1
com.apple.driver.usb.AppleUSBXHCIPCI 1.0.1
com.apple.driver.usb.AppleUSBXHCI 1.0.1
com.apple.driver.AppleThunderboltNHI 4.0.4
com.apple.iokit.IOThunderboltFamily 5.0.6
com.apple.iokit.IOFireWireFamily 4.5.8
com.apple.iokit.IOEthernetAVBController 1.0.3b3
com.apple.iokit.IO80211Family 1101.24
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.2
com.apple.driver.corecapture 1.0.4
com.apple.iokit.IOAHCIFamily 2.8.0
com.apple.driver.usb.AppleUSBEHCI 1.0.1
com.apple.iokit.IOUSBFamily 900.4.1
com.apple.iokit.IOUSBHostFamily 1.0.1
com.apple.driver.AppleUSBHostMergeProperties 1.0.1
com.apple.driver.AppleEFINVRAM 2.0
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.driver.AppleEFIRuntime 2.0
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.DiskImages 415
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOReportFamily 31
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleACPIPlatform 4.0
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.Libm 1
com.apple.kec.pthread 1
com.apple.kec.corecrypto 1.0

System Profile:
Graphics: Intel HD Graphics 3000, Intel HD Graphics 3000, Built-In
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0xE4), Broadcom BCM43xx 1.0 (5.106.98.100.24)
Bluetooth: Version 4.4.2f1 16391, 3 services, 27 devices, 1 incoming serial ports
PCI Card: pci1b73,1100, USB eXtensible Host Controller, Thunderbolt@193,0,0
PCI Card: pci1b21,612, AHCI Controller, Thunderbolt@192,0,0
PCI Card: ethernet, Ethernet Controller, Thunderbolt@194,0,0
Thunderbolt Bus: Mac mini, Apple Inc., 22.2
Thunderbolt Device: Thunderbolt Station 2, CalDigit, Inc., 3, 25.1
Memory Module: BANK 0/DIMM0, 8 GB, DDR3, 1333 MHz, 0x8502, 0x505341333847313333333253000000000000
Memory Module: BANK 1/DIMM0, 8 GB, DDR3, 1333 MHz, 0x8502, 0x505341333847313333333253000000000000
USB Device: USB 2.0 Bus
USB Device: Hub
USB Device: IR Receiver
USB Device: USB 2.0 Bus
USB Device: Hub
USB Device: USB Receiver
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
USB Device: USB 3.0 Bus
USB Device: USB audio CODEC
Serial ATA Device: Samsung SSD 850 EVO 500GB, 500.11 GB
Serial ATA Device: ST9500420ASG, 500.11 GB
Network Service: Ethernet, Ethernet, en0
Model: Macmini5,3, BootROM MM51.0077.B13, 4 processors, Intel Core i7, 2 GHz, 16 GB, SMC 1.77f0

[ilovezfs]

please enable keepsyms=1 in your boot-args

Code: Select all

sudo nvram boot-args="keepsyms=1"

and post panic report with symbols in it.

[lundman]

The panic report has nothing ZFS in it at all, which is strange.. Using a spare HDD, can you get HFS to work at all using esata or usb3?

I'm also wondering if your USB2 will present the device as 512 sectorsize, and the newer ESATA/USB3 as 4096, which would cause trouble. Don't remember how you would verify that, was it diskutil info?

[Brendon]

yep diskutil info. Compare when plugged in by USB vs ESATA

Cheers
Brendon

[murraypetera]

I have added flag to capture symbols so when it happens again I will have it.

Still scrubbing. This takes forever and back. After I got the disks back online I plugged cache in USB and 4 disk bay is on esate all through thunderbold adapter.

[murraypetera]

I gave up for a while due to work load. I would like to get my zfs disk back up.

I did look at the sector sizes and ssd is not the same as physical disks 4096 vs 512.
I have been looking for a way to reformat the either the physical or ssd disk to make them symmetric but have not been able to find such a comma for OS X or a too to do this.

Back in the day this was easy but not it seems to be a factory only process.

Setups is thunderbolt enclosure with 4 disk 5tb seagate sector Size 512 ZRAID + USB3 180GB SSD sector size 4096 Cache on Mac mini 2010 late connected through CalDigit TS2

Latest OS X version

Thanks in advance

[murraypetera]

Sort of answering my own question but not tested.

Ashift=12 seems to be the answer but I thought had used this and still get the panic

zpool create -d \
-o feature@async_destroy=enabled \
-o feature@empty_bpobj=enabled \
-o feature@lz4_compress=enabled \
-O compression=lz4 \
-O atime=off \
-o ashift=12 \
-O normalization=formD \
BigDog raidz \

Is there a command for me to verify how i created the pool or do I just need to start from scratch?

Thanks

[macz]

as sudo type

zpool history <poolname>

should tell you cradle to grave what you have done to it...

[lundman]

zdb command will output the pools current ashift value. It can only be set at creation time