Kernel Panic when connected through Thunderbolt and corrupt

All your general support questions for OpenZFS on OS X.

Re: Kernel Panic when connected through Thunderbolt and corr

Postby murraypetera » Mon Mar 07, 2016 2:55 pm

Well since I have no data on the zpool I just recreated everything. I also now have 4 matching 5tb disks and this time all,are formatted in zfs which somehow I messed up last time.

Mac Mini server late 2010 16gb ram 2 ssd's internal - same as earlier post.
4 5tb seagate disks - esate over thunderbolt
1 180gb ssd cache - separate UBS channel
1 raidz on single zpool

I have not had a crash yet over the past few days but have had to reboot when trying to do some tests due to system slowness.

I try to copy 3-400gb from another disk to the zpool and at first it screams getting through the first 100gb quickly but start to slow down to a point of not able finish over night.

It also slows down the entire system to a state of almost unsuitability after about an hour.
I have not tried bash copy yet but will.

IStats does not show zfs pools for monitoring as of now but memory and cpu does not look overloaded or out of ordinary.

It seems like the io channeled are potentially overloaded but am not sure the best way to show this on OS X. Solaris iostat

Any ideas on the massive slowdown with finder?

Thanks
murraypetera
 
Posts: 25
Joined: Wed Apr 22, 2015 2:18 pm

Re: Kernel Panic when connected through Thunderbolt and corr

Postby lundman » Mon Mar 07, 2016 4:46 pm

Generally slow down like that could be ARC taking too much memory for the caches. You could try setting the ARC max a bit lower to give your machine more breathing room. I assume you are running the latest version of O3X?
User avatar
lundman
 
Posts: 1337
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Kernel Panic when connected through Thunderbolt and corr

Postby murraypetera » Wed Mar 09, 2016 6:58 am

I just got the following panic:

System was doing a scrub after I pulled the esate power cable by mistake. The drives came back up after I did a clear and were subbing.

After Booboo:
sh-3.2# zpool status
pool: BigDog
state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: http://zfsonlinux.org/msg/ZFS-8000-HC
scan: resilvered 60.1G in 0h47m with 0 errors on Fri Mar 4 15:14:14 2016
config:

NAME STATE READ WRITE CKSUM
BigDog UNAVAIL 0 0 0 insufficient replicas
raidz1-0 UNAVAIL 0 2 0 insufficient replicas
media-371AB90C-5BF2-6843-A062-BAB8FFF49673 FAULTED 0 1 0 too many errors
media-9D7B1EE9-82D4-F543-A8B6-F0A8F47088CD FAULTED 0 1 0 too many errors
media-68D1475A-7161-A349-B356-949486CAD2DB FAULTED 0 1 0 too many errors
media-DD8EE5C7-FFAB-1443-8E45-924EE23FDA2A FAULTED 0 1 0 too many errors
cache
disk2 ONLINE 1 55.1K 0

errors: 6 data errors, use '-v' for a list
------------------------------
My actions to fix booboo

sh-3.2# zpool clear BigDog
sh-3.2# zpool status
pool: BigDog
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Mar 9 09:05:35 2016
968K scanned out of 321G at 484K/s, 193h28m to go
572K resilvered, 0.00% done
config:

NAME STATE READ WRITE CKSUM
BigDog ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
media-371AB90C-5BF2-6843-A062-BAB8FFF49673 ONLINE 0 0 0 (resilvering)
media-9D7B1EE9-82D4-F543-A8B6-F0A8F47088CD ONLINE 0 0 0 (resilvering)
media-68D1475A-7161-A349-B356-949486CAD2DB ONLINE 0 0 0 (resilvering)
media-DD8EE5C7-FFAB-1443-8E45-924EE23FDA2A ONLINE 0 0 0 (resilvering)
cache
disk2 ONLINE 7 55.7K 0

few min later
scan: resilvered 192K in 0h0m with 0 errors on Wed Mar 9 09:05:39 2016
config:

NAME STATE READ WRITE CKSUM
BigDog ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
media-371AB90C-5BF2-6843-A062-BAB8FFF49673 ONLINE 0 0 0
media-9D7B1EE9-82D4-F543-A8B6-F0A8F47088CD ONLINE 0 0 0
media-68D1475A-7161-A349-B356-949486CAD2DB ONLINE 0 0 0
media-DD8EE5C7-FFAB-1443-8E45-924EE23FDA2A ONLINE 0 0 0
cache
disk2 ONLINE 81 122K 0

errors: No known data errors

-----
After system crashed and rebooted:

zpool status
pool: BigDog
state: ONLINE
scan: resilvered 192K in 0h0m with 0 errors on Wed Mar 9 09:05:39 2016
config:

NAME STATE READ WRITE CKSUM
BigDog ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
media-371AB90C-5BF2-6843-A062-BAB8FFF49673 ONLINE 0 0 0
media-9D7B1EE9-82D4-F543-A8B6-F0A8F47088CD ONLINE 0 0 0
media-68D1475A-7161-A349-B356-949486CAD2DB ONLINE 0 0 0
media-DD8EE5C7-FFAB-1443-8E45-924EE23FDA2A ONLINE 0 0 0
cache
disk2 ONLINE 0 0 0

errors: No known data errors

-----------
Here is the output from panic to send to Apple:

Anonymous UUID: 2CD697E9-9B5E-89C5-86D4-0316A2FCB615

Wed Mar 9 09:47:11 2016

*** Panic Report ***
panic(cpu 5 caller 0xffffff800d1ce40a): Kernel trap at 0xffffff7f8d8590b0, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000000, CR3: 0x00000002a9905115, CR4: 0x00000000000626e0
RAX: 0x00000000e00002c0, RBX: 0x0000000000000009, RCX: 0xffffff802c6a6a08, RDX: 0x0000000000000001
RSP: 0xffffff92496bb000, RBP: 0xffffff92496bb080, RSI: 0x0000000000000004, RDI: 0x0000000000000000
R8: 0x000000000000ffff, R9: 0x0000000000000001, R10: 0x000000000900014b, R11: 0x00000000e0000000
R12: 0xffffff802c74e510, R13: 0xffffff80441741e0, R14: 0x0000000000000000, R15: 0xffffff800d920c70
RFL: 0x0000000000010286, RIP: 0xffffff7f8d8590b0, CS: 0x0000000000000008, SS: 0x0000000000000000
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0x5, PL: 0

Backtrace (CPU 5), Frame : Return Address
0xffffff92496bac90 : 0xffffff800d0dab12 mach_kernel : _panic + 0xe2
0xffffff92496bad10 : 0xffffff800d1ce40a mach_kernel : _kernel_trap + 0x91a
0xffffff92496baef0 : 0xffffff800d1ec273 mach_kernel : _return_from_trap + 0xe3
0xffffff92496baf10 : 0xffffff7f8d8590b0 com.apple.iokit.IOStorageFamily : _dkioctl + 0x2bd
0xffffff92496bb080 : 0xffffff800d338da5 mach_kernel : _spec_ioctl + 0x1f5
0xffffff92496bb0c0 : 0xffffff800d326d0e mach_kernel : _VNOP_IOCTL + 0x16e
0xffffff92496bb150 : 0xffffff7f8d9267ee net.lundman.zfs : _vdev_disk_io_start + 0x197
0xffffff92496bb180 : 0xffffff7f8d96900f net.lundman.zfs : _zio_vdev_io_start + 0x258
0xffffff92496bb1d0 : 0xffffff7f8d96603e net.lundman.zfs : _zio_nowait + 0x185
0xffffff92496bb230 : 0xffffff7f8d965e3f net.lundman.zfs : _zio_ioctl + 0xe8
0xffffff92496bb2d0 : 0xffffff7f8d9663b4 net.lundman.zfs : _zio_flush + 0x29
0xffffff92496bb2f0 : 0xffffff7f8d963164 net.lundman.zfs : _zil_commit + 0x909
0xffffff92496bb410 : 0xffffff7f8d951b95 net.lundman.zfs : _zfs_fsync + 0xf1
0xffffff92496bb440 : 0xffffff800d2ff077 mach_kernel : _vfs_addtrigger + 0x1ba7
0xffffff92496bb4c0 : 0xffffff800d2fed26 mach_kernel : _vfs_addtrigger + 0x1856
0xffffff92496bb500 : 0xffffff800d2f8a91 mach_kernel : _vnode_recycle + 0x51
0xffffff92496bb520 : 0xffffff7f8d95065a net.lundman.zfs : _zfs_remove + 0x524
0xffffff92496bb620 : 0xffffff7f8d9563bd net.lundman.zfs : _zfs_vnop_remove + 0x73
0xffffff92496bbad0 : 0xffffff800d32715e mach_kernel : _VNOP_REMOVE + 0x5e
0xffffff92496bbb50 : 0xffffff800d311fd8 mach_kernel : _vfs_purge + 0x3f18
0xffffff92496bbf60 : 0xffffff800d627f91 mach_kernel : _unix_syscall64 + 0x251
0xffffff92496bbfb0 : 0xffffff800d1eca76 mach_kernel : _hndl_unix_scall64 + 0x16
Kernel Extensions in backtrace:
com.apple.iokit.IOStorageFamily(2.1)[DC1AAB7C-F417-3238-BB3F-2A5B84D67B90]@0xffffff7f8d84c000->0xffffff7f8d872fff
net.lundman.zfs(1.5)[DC12A26F-A90C-3D97-8D88-E7B61B6330EA]@0xffffff7f8d8cb000->0xffffff7f8db0bfff
dependency: com.apple.iokit.IOStorageFamily(2.1)[DC1AAB7C-F417-3238-BB3F-2A5B84D67B90]@0xffffff7f8d84c000
dependency: net.lundman.spl(1.5.0)[B3517941-D80C-3026-B2F5-2F9B60388BDB]@0xffffff7f8d87c000

BSD process name corresponding to current thread: DesktopServicesH
Boot args: keepsyms=1

Mac OS version:
15E49a

Kernel version:
Darwin Kernel Version 15.4.0: Fri Feb 12 15:56:47 PST 2016; root:xnu-3248.40.183~6/RELEASE_X86_64
Kernel UUID: C4053AC4-EDC1-36CC-9A25-0DC576152473
Kernel slide: 0x000000000ce00000
Kernel text base: 0xffffff800d000000
__HIB text base: 0xffffff800cf00000
System model name: Macmini5,3 (Mac-7BA5B2794B2CDB12)

System uptime in nanoseconds: 47395643202311
last loaded kext at 42683183765379: com.apple.driver.AppleXsanScheme 3 (addr 0xffffff7f900db000, size 32768)
last unloaded kext at 41917314439345: com.apple.driver.AppleUSBMergeNub 900.4.1 (addr 0xffffff7f900d8000, size 12288)
loaded kexts:
com.sophos.kext.sav 9.4.52
com.sophos.nke.swi 9.4.52
org.virtualbox.kext.VBoxNetAdp 5.0.10
org.virtualbox.kext.VBoxNetFlt 5.0.10
com.nvidia.CUDA 1.1.0
org.virtualbox.kext.VBoxUSB 5.0.10
org.virtualbox.kext.VBoxDrv 5.0.10
net.lundman.zfs 1.5.0
net.lundman.spl 1.5.0
com.dvdfab.kext.fabio 1
com.makemkv.kext.daspi 1.5
com.apple.filesystems.afpfs 11.0
com.apple.nke.asp-tcp 8.0.0
com.apple.filesystems.smbfs 3.0.0
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.driver.AppleBluetoothMultitouch 90.3
com.apple.filesystems.autofs 3.0
com.apple.driver.ApplePlatformEnabler 2.6.0d0
com.apple.driver.AGPM 110.21.18
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleMikeyHIDDriver 124
com.apple.driver.AppleUpstreamUserClient 3.6.1
com.apple.driver.AppleHDA 274.7
com.apple.driver.AppleMCCSControl 1.2.13
com.apple.driver.AppleMikeyDriver 274.7
com.apple.driver.pmtelemetry 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.iokit.IOBluetoothSerialManager 4.4.4f4
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleIntelHD3000Graphics 10.0.0
com.apple.driver.AppleThunderboltIP 3.0.8
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 4.4.4f4
com.apple.driver.AppleHV 1
com.apple.driver.AppleSMCPDRC 1.0.0
com.apple.driver.AppleLPC 3.1
com.apple.driver.AppleIntelSNBGraphicsFB 10.0.0
com.apple.driver.AudioAUUC 1.70
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleIRController 327.5
com.apple.iokit.SCSITaskUserClient 3.7.7
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.BootCache 38
com.apple.iokit.AppleIntelI210Ethernet 2.2.1b3
com.apple.driver.AppleFWOHCI 5.5.2
com.apple.iokit.IOAHCIBlockStorage 2.8.5
com.apple.driver.AirPort.Brcm4331 800.20.24
com.apple.driver.AppleSDXC 1.7.0
com.apple.iokit.AppleBCM5701Ethernet 10.2.0
com.apple.driver.AppleAHCIPort 3.1.8
com.apple.driver.usb.AppleUSBEHCIPCI 1.0.1
com.apple.driver.AppleACPIButtons 4.0
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleACPIEC 4.0
com.apple.driver.AppleAPIC 1.7
com.apple.driver.AppleIntelCPUPowerManagementClient 218.0.0
com.apple.nke.applicationfirewall 163
com.apple.security.quarantine 3
com.apple.security.TMSafetyNet 8
com.apple.driver.AppleIntelCPUPowerManagement 218.0.0
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleUSBMergeNub 900.4.1
com.apple.security.SecureRemotePassword 1.0
com.apple.driver.AppleBluetoothHIDKeyboard 181
com.apple.driver.AppleHIDKeyboard 181
com.apple.driver.IOBluetoothHIDDriver 4.4.4f4
com.apple.driver.AppleMultitouchDriver 304.10.2
com.apple.driver.AppleHIDTransport 5
com.apple.iokit.IOUSBUserClient 900.4.1
com.apple.kext.triggers 1.0
com.apple.AppleGraphicsDeviceControl 3.12.6
com.apple.driver.DspFuncLib 274.7
com.apple.kext.OSvKernDSPLib 525
com.apple.iokit.IOSurface 108.2.1
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSMBusController 1.0.14d1
com.apple.driver.CoreCaptureResponder 1
com.apple.driver.AppleSMBusPCI 1.0.14d1
com.apple.iokit.IOBluetoothHostControllerUSBTransport 4.4.4f4
com.apple.iokit.IOBluetoothFamily 4.4.4f4
com.apple.driver.AppleHDAController 274.7
com.apple.iokit.IOHDAFamily 274.7
com.apple.iokit.IOFireWireIP 2.2.6
com.apple.iokit.IONDRVSupport 2.4.1
com.apple.iokit.IOGraphicsFamily 2.4.1
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d7
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOSCSIMultimediaCommandsDevice 3.7.7
com.apple.iokit.IOBDStorageFamily 1.8
com.apple.iokit.IODVDStorageFamily 1.8
com.apple.iokit.IOCDStorageFamily 1.8
com.apple.driver.AppleThunderboltDPOutAdapter 4.1.3
com.apple.driver.AppleThunderboltPCIUpAdapter 2.0.2
com.apple.driver.AppleThunderboltDPInAdapter 4.1.3
com.apple.driver.AppleThunderboltDPAdapterFamily 4.1.3
com.apple.driver.AppleThunderboltPCIDownAdapter 2.0.2
com.apple.iokit.IOUSBHIDDriver 900.4.1
com.apple.iokit.IOSCSIBlockCommandsDevice 3.7.7
com.apple.driver.AppleUSBAudio 303.3
com.apple.iokit.IOAudioFamily 204.3
com.apple.vecLib.kext 1.2.0
com.apple.iokit.IOUSBMassStorageDriver 1.0.0
com.apple.iokit.IOSCSIArchitectureModelFamily 3.7.7
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.0.1
com.apple.driver.usb.AppleUSBHub 1.0.1
com.apple.driver.usb.AppleUSBXHCIPCI 1.0.1
com.apple.driver.usb.AppleUSBXHCI 1.0.1
com.apple.driver.AppleThunderboltNHI 4.0.4
com.apple.iokit.IOThunderboltFamily 6.0.2
com.apple.iokit.IOFireWireFamily 4.6.0
com.apple.iokit.IOEthernetAVBController 1.0.3b3
com.apple.iokit.IO80211Family 1110.26
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.2
com.apple.driver.corecapture 1.0.4
com.apple.iokit.IOAHCIFamily 2.8.1
com.apple.driver.usb.AppleUSBEHCI 1.0.1
com.apple.iokit.IOUSBFamily 900.4.1
com.apple.iokit.IOUSBHostFamily 1.0.1
com.apple.driver.AppleUSBHostMergeProperties 1.0.1
com.apple.driver.AppleEFINVRAM 2.0
com.apple.driver.AppleEFIRuntime 2.0
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.DiskImages 417.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOReportFamily 31
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleACPIPlatform 4.0
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.Libm 1
com.apple.kec.pthread 1
com.apple.kec.corecrypto 1.0
Model: Macmini5,3, BootROM MM51.0077.B13, 4 processors, Intel Core i7, 2 GHz, 16 GB, SMC 1.77f0
Graphics: Intel HD Graphics 3000, Intel HD Graphics 3000, Built-In
Memory Module: BANK 0/DIMM0, 8 GB, DDR3, 1333 MHz, 0x8502, 0x505341333847313333333253000000000000
Memory Module: BANK 1/DIMM0, 8 GB, DDR3, 1333 MHz, 0x8502, 0x505341333847313333333253000000000000
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0xE4), Broadcom BCM43xx 1.0 (5.106.98.100.24)
Bluetooth: Version 4.4.4f4 17685, 3 services, 27 devices, 1 incoming serial ports
Network Service: Ethernet, Ethernet, en3
PCI Card: ethernet, Ethernet Controller, Thunderbolt@194,0,0
PCI Card: pci1b73,1100, USB eXtensible Host Controller, Thunderbolt@193,0,0
PCI Card: pci1b21,612, AHCI Controller, Thunderbolt@192,0,0
Serial ATA Device: Samsung SSD 850 EVO 500GB, 500.11 GB
Serial ATA Device: ST9500420ASG, 500.11 GB
Serial ATA Device: spsata_pm_name
USB Device: USB 2.0 Bus
USB Device: Hub
USB Device: SD-3102V-U3
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
USB Device: USB 2.0 Bus
USB Device: Hub
USB Device: IR Receiver
USB Device: USB 3.0 Bus
USB Device: Expansion Desk
USB Device: USB audio CODEC
USB Device: iPad
Thunderbolt Bus: Mac mini, Apple Inc., 22.2
Thunderbolt Device: Thunderbolt Station 2, CalDigit, Inc., 3, 25.1
murraypetera
 
Posts: 25
Joined: Wed Apr 22, 2015 2:18 pm

Re: Kernel Panic when connected through Thunderbolt and corr

Postby murraypetera » Wed Mar 09, 2016 8:49 am

I should have also thanked and replied to Lundman, I have lowered ARC to 3G ram and while the system does not get killed by slowness the copy never got going through finder where before it got through ~200gb of the 300+gb.

I have filed to find a good guide on how to tune ARC_MAX...
i.e. I have 16GB Ram and use the system as a desktop so I do not want ZFS to take over the box.

It would be nice if it would autotune based off the system it is running on.
murraypetera
 
Posts: 25
Joined: Wed Apr 22, 2015 2:18 pm

Re: Kernel Panic when connected through Thunderbolt and corr

Postby murraypetera » Wed Apr 06, 2016 3:54 pm

Well I finally got some time to try again..
I upgraded to the latest build posted someplace on the site here...
- It would be nice if there were a central location for all builds not just promoted builds...nightly build etc...

The system seemed to still crash with ESATA but then I switched to USB3 and have not had an issues since.
I ran zpool instatus but was not get very good number and Black Magic would be super fast at first and then freeze up after a while. I read on performance and and it seemed that a poorly sized cache could cause issues and a write log might work better.

Fist I took off the cache (SSD 180G) and my IO shot up with my best number ~90M up from a max of ~20M with the cache...
I then added the SSD back as a log and number went as high as 200M.

After some large copies through finder I seem to have settled closer to ~50M

More importantly the rest of the system is more usable than before....occasional rainbow spinning disk but for short periods but this is when doing large copies to the ZPool.
murraypetera
 
Posts: 25
Joined: Wed Apr 22, 2015 2:18 pm

Previous

Return to General Help

Who is online

Users browsing this forum: No registered users and 3 guests

cron