kernel crash

[jojo45]

I just had two kernel crashes occur over several days. Running 1.9.0. Was rock sold with 1.8 series. Here is the crash dump:

Sun Jul 14 20:46:25 2019

*** Panic Report ***
panic(cpu 2 caller 0xffffff800eadb92d): Kernel trap at 0xffffff7f90a88f87, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000018, CR3: 0x0000000014194000, CR4: 0x00000000001626e0
RAX: 0xffffff800f20ee40, RBX: 0x0000000000000000, RCX: 0x0000000000000000, RDX: 0x0000000001000000
RSP: 0xffffff81fafdbe40, RBP: 0xffffff81fafdbe60, RSI: 0xffffff92b6b77f00, RDI: 0x0000000000000000
R8: 0x0000000000000000, R9: 0x0000000000989680, R10: 0x0000000000000000, R11: 0xffffff800f3f2cc0
R12: 0xffffff81fc15eff0, R13: 0xffffff92b6b77ea0, R14: 0x0000000000000020, R15: 0xffffff7f90af44e8
RFL: 0x0000000000010246, RIP: 0xffffff7f90a88f87, CS: 0x0000000000000008, SS: 0x0000000000000000
Fault CR2: 0x0000000000000018, Error code: 0x0000000000000000, Fault CPU: 0x2, PL: 0, VF: 1

Backtrace (CPU 2), Frame : Return Address
0xffffff81fafdb910 : 0xffffff800e9aea2d
0xffffff81fafdb960 : 0xffffff800eae9e95
0xffffff81fafdb9a0 : 0xffffff800eadb70a
0xffffff81fafdba10 : 0xffffff800e95bb40
0xffffff81fafdba30 : 0xffffff800e9ae447
0xffffff81fafdbb50 : 0xffffff800e9ae293
0xffffff81fafdbbc0 : 0xffffff800eadb92d
0xffffff81fafdbd30 : 0xffffff800e95bb40
0xffffff81fafdbd50 : 0xffffff7f90a88f87
0xffffff81fafdbe60 : 0xffffff7f909b8b34
0xffffff81fafdbea0 : 0xffffff7f909d220d
0xffffff81fafdbee0 : 0xffffff7f909d2c58
0xffffff81fafdbf10 : 0xffffff7f8f791ed5
0xffffff81fafdbfa0 : 0xffffff800e95b0ce
Kernel Extensions in backtrace:
net.lundman.spl(1.9)[C6A1ACB1-D9AD-3D42-B5F6-10EBD56F0850]@0xffffff7f8f784000->0xffffff7f90978fff
net.lundman.zfs(1.9)[D4D4F072-304D-3ABE-92BE-87931D504865]@0xffffff7f909a9000->0xffffff7f90cc9fff
dependency: com.apple.iokit.IOStorageFamily(2.1)[D43332F2-2D40-3DF1-8F77-D633BCDAE660]@0xffffff7f90979000
dependency: net.lundman.spl(1.9.0)[C6A1ACB1-D9AD-3D42-B5F6-10EBD56F0850]@0xffffff7f8f784000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
18F132

Kernel version:
Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64
Kernel UUID: 7C8BB636-E593-3CE4-8528-9BD24A688851
Kernel slide: 0x000000000e600000
Kernel text base: 0xffffff800e800000
__HIB text base: 0xffffff800e700000
System model name: MacBookPro11,4 (Mac-06F11FD93F0323C5)

System uptime in nanoseconds: 141970212706081
last loaded kext at 141075044450866: com.apple.driver.AppleUSBAudio 315.6 (addr 0xffffff7f93814000, size 413696)
last unloaded kext at 141144553910806: com.apple.driver.usb.cdc 5.0.0 (addr 0xffffff7f9380d000, size 28672)
loaded kexts:
com.zerotier.tap 1.0
com.logmein.hamachi 1.0.0
com.birj.sbdsf 2.21
net.sf.tuntaposx.tap 1.0
com.paragon-software.filesystems.ntfs 318.3.14
net.sf.tuntaposx.tun 1.0
com.paragon-software.filesystems.extfs 109.1.10
com.Cycling74.driver.Soundflower 2
com.paragon-software.kext.VDMounter 118.2
net.lundman.zfs 1.9.0
net.lundman.spl 1.9.0
at.obdev.nke.LittleSnitch 5405
com.apple.fileutil 20.036.15
com.apple.filesystems.autofs 3.0
com.apple.driver.AGPM 110.25.10
com.apple.driver.ApplePlatformEnabler 2.7.0d0
com.apple.driver.X86PlatformShim 1.0.0
com.apple.driver.AppleUpstreamUserClient 3.6.5
com.apple.driver.AudioAUUC 1.70
com.apple.driver.AppleGraphicsDevicePolicy 3.40.16
com.apple.AGDCPluginDisplayMetrics 3.40.16
com.apple.driver.AppleHDA 282.54
com.apple.driver.AppleHV 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.iokit.IOBluetoothSerialManager 6.0.12f1
com.apple.driver.pmtelemetry 1
com.apple.driver.AppleIntelHD5000Graphics 12.0.9
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleBacklight 170.12.11
com.apple.driver.eficheck 1
com.apple.iokit.BroadcomBluetooth20703USBTransport 6.0.12f1
com.apple.driver.AppleThunderboltIP 3.1.2
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleMCCSControl 1.5.9
com.apple.driver.AppleLPC 3.1
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleSMCLMU 212
com.apple.driver.AppleIntelFramebufferAzul 12.0.9
com.apple.driver.AppleCameraInterface 6.8.0
com.apple.driver.AppleTopCaseHIDEventDriver 138.3
com.apple.driver.AppleUSBTopCaseDriver 138.3
com.apple.driver.AppleUSBCardReader 456.260.3
com.apple.filesystems.apfs 945.260.7
com.apple.driver.AirPort.BrcmNIC 1400.1.1
com.apple.driver.AppleVirtIO 2.1.3
com.apple.filesystems.hfs.kext 407.200.4
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleSystemPolicy 1.0
com.apple.private.KextAudit 1.0
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleACPIButtons 6.1
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleACPIEC 6.1
com.apple.driver.AppleAPIC 1.7
com.apple.nke.applicationfirewall 201
com.apple.security.TMSafetyNet 8
com.apple.kext.triggers 1.0
com.apple.iokit.IOAVBFamily 740.1
com.apple.plugin.IOgPTPPlugin 740.2
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.iokit.IOSkywalkFamily 1
com.apple.driver.DspFuncLib 282.54
com.apple.kext.OSvKernDSPLib 528
com.apple.driver.AppleSSE 1.0
com.apple.AppleGPUWrangler 3.40.16
com.apple.driver.AppleBacklightExpert 1.1.0
com.apple.iokit.IONDRVSupport 530.51
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 6.0.12f1
com.apple.iokit.IOBluetoothHostControllerUSBTransport 6.0.12f1
com.apple.iokit.IOBluetoothHostControllerTransport 6.0.12f1
com.apple.driver.AppleHDAController 282.54
com.apple.iokit.IOHDAFamily 282.54
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.driver.AppleSMBusController 1.0.18d1
com.apple.driver.AppleGraphicsControl 3.40.16
com.apple.AppleGraphicsDeviceControl 3.40.16
com.apple.iokit.IOAcceleratorFamily2 404.9
com.apple.iokit.IOGraphicsFamily 530.66
com.apple.iokit.IOSurface 255.5.4
com.apple.driver.X86PlatformPlugin 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d8
com.apple.driver.AppleActuatorDriver 2450.1
com.apple.driver.AppleHSBluetoothDriver 138.3
com.apple.driver.IOBluetoothHIDDriver 6.0.12f1
com.apple.iokit.IOBluetoothFamily 6.0.12f1
com.apple.driver.AppleHIDKeyboard 208
com.apple.driver.AppleMultitouchDriver 2450.1
com.apple.driver.AppleInputDeviceSupport 2440.2
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.iokit.IOAudioFamily 206.5
com.apple.vecLib.kext 1.2.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.AppleThunderboltDPInAdapter 5.6.4
com.apple.driver.AppleThunderboltDPAdapterFamily 5.6.4
com.apple.driver.AppleThunderboltPCIDownAdapter 2.1.4
com.apple.driver.AppleThunderboltNHI 4.7.6
com.apple.iokit.IOThunderboltFamily 6.8.4
com.apple.iokit.IO80211Family 1200.12.2
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.driver.corecapture 1.0.4
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.iokit.IOSerialFamily 11
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOUSBFamily 900.4.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.AppleEFINVRAM 2.1
com.apple.driver.AppleEFIRuntime 2.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.security.quarantine 3
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.DiskImages 493.0.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleUSBTDM 456.260.3
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.iokit.IOUSBMassStorageDriver 145.200.2
com.apple.iokit.IOSCSIBlockCommandsDevice 408.250.3
com.apple.iokit.IOSCSIArchitectureModelFamily 408.250.3
com.apple.iokit.IOStorageFamily 2.1
com.apple.kext.CoreTrust 1
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleBusPowerController 1.0
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.IOSlaveProcessor 1
com.apple.iokit.IOReportFamily 47
com.apple.iokit.IOTimeSyncFamily 740.2
com.apple.iokit.IONetworkingFamily 3.4
com.apple.driver.AppleACPIPlatform 6.1
com.apple.driver.AppleSMC 3.1.9
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.pthread 1
com.apple.kec.corecrypto 1.0
com.apple.kec.Libm 1

EOF

[jojo45]

Forgot to mention: on Mojave with all latest updates as of July 14, 2019.

[lundman]

Could I convince you to enable "keepsyms=1" to the boot-args, so we get function names?

[jojo45]

I got around this problem for a while by increasing the RAM allocated to zfs and by rebooting the system every day.

Just had another crash today and have uploaded the report below with the keepsyms boot option enabled:

Anonymous UUID: 8C9974C6-880D-F57A-F865-9DCD81F4AFF1

Thu Oct 3 22:45:34 2019

*** Panic Report ***
panic(cpu 2 caller 0xffffff80020dbadd): Kernel trap at 0xffffff7f82d891cb, type 14=page fault, registers:
CR0: 0x0000000080010033, CR2: 0x0000000000000010, CR3: 0x0000000007937000, CR4: 0x00000000001626e0
RAX: 0x0000000000000001, RBX: 0xffffff92d63de5d4, RCX: 0x0000000000000000, RDX: 0x00000000ffffffff
RSP: 0xffffff822e83be88, RBP: 0xffffff822e83bec0, RSI: 0xffffff935ccce758, RDI: 0xffffff81f00a2560
R8: 0xffffff935ccce758, R9: 0x0000000000000001, R10: 0x0000000000000000, R11: 0x0000000000000001
R12: 0xffffff92d63de5d0, R13: 0xffffff92d63de5d0, R14: 0xffffff81f00a2560, R15: 0xffffff7f82d9cb2c
RFL: 0x0000000000010202, RIP: 0xffffff7f82d891cb, CS: 0x0000000000000008, SS: 0x0000000000000010
Fault CR2: 0x0000000000000010, Error code: 0x0000000000000000, Fault CPU: 0x2, PL: 0, VF: 1

Backtrace (CPU 2), Frame : Return Address
0xffffff822e83b950 : 0xffffff8001fae6ed mach_kernel : _handle_debugger_trap + 0x47d
0xffffff822e83b9a0 : 0xffffff80020ea185 mach_kernel : _kdp_i386_trap + 0x155
0xffffff822e83b9e0 : 0xffffff80020db8ba mach_kernel : _kernel_trap + 0x50a
0xffffff822e83ba50 : 0xffffff8001f5bb40 mach_kernel : _return_from_trap + 0xe0
0xffffff822e83ba70 : 0xffffff8001fae107 mach_kernel : _panic_trap_to_debugger + 0x197
0xffffff822e83bb90 : 0xffffff8001fadf53 mach_kernel : _panic + 0x63
0xffffff822e83bc00 : 0xffffff80020dbadd mach_kernel : _kernel_trap + 0x72d
0xffffff822e83bd70 : 0xffffff8001f5bb40 mach_kernel : _return_from_trap + 0xe0
0xffffff822e83bd90 : 0xffffff7f82d891cb net.lundman.spl : _avl_rotation + 0x3e
0xffffff822e83bec0 : 0xffffff7f82d89574 net.lundman.spl : _avl_remove + 0x196
0xffffff822e83bf10 : 0xffffff7f83fbd233 net.lundman.zfs : _dbuf_destroy + 0x150
0xffffff822e83bf50 : 0xffffff7f83fbffc6 net.lundman.zfs : _dbuf_evict_one + 0x9e
0xffffff822e83bf70 : 0xffffff7f83fbbf41 net.lundman.zfs : _dbuf_evict_thread + 0xbb
0xffffff822e83bfa0 : 0xffffff8001f5b0ce mach_kernel : _call_continuation + 0x2e
Kernel Extensions in backtrace:
net.lundman.spl(1.9.2)[EAA28CC7-9F6A-3C7B-BB90-691EBDC3A258]@0xffffff7f82d88000->0xffffff7f83f7cfff
net.lundman.zfs(1.9.2)[4C34A112-866A-3499-B5EA-4DF7CDF1FFF4]@0xffffff7f83fad000->0xffffff7f84366fff
dependency: com.apple.iokit.IOStorageFamily(2.1)[71BB22B0-3075-35A1-B04E-FBAC574DA80D]@0xffffff7f83f7d000
dependency: net.lundman.spl(1.9.2)[EAA28CC7-9F6A-3C7B-BB90-691EBDC3A258]@0xffffff7f82d88000

BSD process name corresponding to current thread: kernel_task
Boot args: -v keepsyms=1

Mac OS version:
18G95

Kernel version:
Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64
Kernel UUID: C41337A1-0EC3-3896-A954-A1F85E849D53
Kernel slide: 0x0000000001c00000
Kernel text base: 0xffffff8001e00000
__HIB text base: 0xffffff8001d00000
System model name: MacBookPro11,4 (Mac-06F11FD93F0323C5)

System uptime in nanoseconds: 79062488404066
last loaded kext at 67934245749695: com.apple.driver.AppleMikeyHIDDriver 131 (addr 0xffffff7f86fbb000, size 16384)
last unloaded kext at 67521509408349: com.apple.driver.AppleMikeyHIDDriver 131 (addr 0xffffff7f86fbb000, size 16384)
loaded kexts:
com.bitdefender.TMProtection 5.0.0
com.bitdefender.FileProtect 1
com.zerotier.tap 1.0
com.logmein.hamachi 1.0.0
org.pqrs.driver.Karabiner.VirtualHIDDevice.v061000 6.10.0
com.birj.sbdsf 2.21
net.sf.tuntaposx.tun 1.0
net.sf.tuntaposx.tap 1.0
com.paragon-software.filesystems.extfs 109.1.10
com.paragon-software.filesystems.ntfs 318.3.14
com.Cycling74.driver.Soundflower 2
com.paragon-software.kext.VDMounter 118.2
com.bitdefender.SelfProtect 1.2.12
net.lundman.zfs 1.9.2
net.lundman.spl 1.9.2
at.obdev.nke.LittleSnitch 5422
com.apple.driver.AppleMikeyHIDDriver 131
com.apple.filesystems.smbfs 3.3.2
com.apple.fileutil 20.036.15
com.apple.filesystems.autofs 3.0
com.apple.driver.AGPM 110.25.11
com.apple.driver.ApplePlatformEnabler 2.7.0d0
com.apple.driver.X86PlatformShim 1.0.0
com.apple.driver.AppleUpstreamUserClient 3.6.5
com.apple.driver.AppleHDA 282.54
com.apple.driver.AudioAUUC 1.70
com.apple.driver.AppleGraphicsDevicePolicy 3.50.12
com.apple.AGDCPluginDisplayMetrics 3.50.12
com.apple.driver.AppleHV 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.iokit.IOBluetoothSerialManager 6.0.14d3
com.apple.driver.pmtelemetry 1
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleIntelHD5000Graphics 12.1.0
com.apple.driver.eficheck 1
com.apple.driver.AppleSMCLMU 212
com.apple.driver.AppleBacklight 170.12.12
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleThunderboltIP 3.1.2
com.apple.iokit.BroadcomBluetooth20703USBTransport 6.0.14d3
com.apple.driver.AppleLPC 3.1
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleIntelFramebufferAzul 12.1.0
com.apple.driver.AppleMCCSControl 1.5.9
com.apple.driver.AppleCameraInterface 6.8.0
com.apple.driver.AppleTopCaseHIDEventDriver 138.4
com.apple.driver.AppleUSBTopCaseDriver 138.4
com.apple.driver.AppleUSBCardReader 456.260.3
com.apple.filesystems.apfs 945.275.7
com.apple.iokit.AppleBCM5701Ethernet 10.3.3
com.apple.driver.AirPort.BrcmNIC 1400.1.1
com.apple.driver.AppleVirtIO 2.1.3
com.apple.filesystems.hfs.kext 407.200.4
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleSystemPolicy 1.0
com.apple.private.KextAudit 1.0
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleACPIButtons 6.1
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleACPIEC 6.1
com.apple.driver.AppleAPIC 1.7
com.apple.nke.applicationfirewall 201
com.apple.security.TMSafetyNet 8
com.apple.kext.triggers 1.0
com.apple.driver.DspFuncLib 282.54
com.apple.kext.OSvKernDSPLib 528
com.apple.iokit.IOAVBFamily 760.6
com.apple.plugin.IOgPTPPlugin 740.2
com.apple.iokit.IOSkywalkFamily 1
com.apple.driver.AppleSSE 1.0
com.apple.AppleGPUWrangler 3.50.12
com.apple.driver.AppleBacklightExpert 1.1.0
com.apple.iokit.IONDRVSupport 530.51
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 6.0.14d3
com.apple.iokit.IOBluetoothHostControllerUSBTransport 6.0.14d3
com.apple.iokit.IOBluetoothHostControllerTransport 6.0.14d3
com.apple.driver.AppleGraphicsControl 3.50.12
com.apple.AppleGraphicsDeviceControl 3.50.12
com.apple.iokit.IOAcceleratorFamily2 404.14
com.apple.iokit.IOSurface 255.6.1
com.apple.driver.X86PlatformPlugin 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d8
com.apple.driver.AppleSMBusController 1.0.18d1
com.apple.driver.AppleHDAController 282.54
com.apple.iokit.IOGraphicsFamily 530.66
com.apple.iokit.IOHDAFamily 282.54
com.apple.driver.AppleThunderboltDPInAdapter 5.6.9
com.apple.driver.AppleThunderboltDPAdapterFamily 5.6.9
com.apple.driver.AppleThunderboltPCIUpAdapter 2.1.5
com.apple.driver.AppleThunderboltPCIDownAdapter 2.1.5
com.apple.driver.AppleHSBluetoothDriver 138.4
com.apple.driver.IOBluetoothHIDDriver 6.0.14d3
com.apple.iokit.IOBluetoothFamily 6.0.14d3
com.apple.driver.AppleActuatorDriver 2450.1
com.apple.driver.AppleHIDKeyboard 208
com.apple.driver.AppleMultitouchDriver 2450.1
com.apple.driver.AppleInputDeviceSupport 2440.2
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.iokit.IOAudioFamily 206.5
com.apple.vecLib.kext 1.2.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.driver.AppleThunderboltNHI 4.7.9
com.apple.iokit.IOThunderboltFamily 6.8.6
com.apple.iokit.IO80211Family 1200.12.2
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.driver.corecapture 1.0.4
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.iokit.IOSerialFamily 11
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOUSBFamily 900.4.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.AppleEFINVRAM 2.1
com.apple.driver.AppleEFIRuntime 2.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.security.quarantine 3
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.DiskImages 493.0.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleUSBTDM 456.260.3
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.iokit.IOUSBMassStorageDriver 145.200.2
com.apple.iokit.IOSCSIBlockCommandsDevice 408.250.3
com.apple.iokit.IOSCSIArchitectureModelFamily 408.250.3
com.apple.iokit.IOStorageFamily 2.1
com.apple.kext.CoreTrust 1
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleBusPowerController 1.0
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.IOSlaveProcessor 1
com.apple.iokit.IOReportFamily 47
com.apple.iokit.IOTimeSyncFamily 740.2
com.apple.iokit.IONetworkingFamily 3.4
com.apple.driver.AppleACPIPlatform 6.1
com.apple.driver.AppleSMC 3.1.9
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.pthread 1
com.apple.kec.corecrypto 1.0
com.apple.kec.Libm 1

EOF

[lundman]

_avl_rotation ? Hmm well, that's a new one. That code is pretty straight forward, so possibly suggesting the avl tree for dbufs are bad somehow.

[jojo45]

Just had another kernel panic. It looks like it occurs somewhere else:

Anonymous UUID: 8C9974C6-880D-F57A-F865-9DCD81F4AFF1

Tue Oct 8 05:59:58 2019

*** Panic Report ***
panic(cpu 6 caller 0xffffff7f8ea63090): VERIFY3( rwlp->rw_pad == 0x012345678 ) failed (0 == 305419896)

Backtrace (CPU 6), Frame : Return Address
0xffffff81fa5e3b70 : 0xffffff800ddae6ed mach_kernel : _handle_debugger_trap + 0x47d
0xffffff81fa5e3bc0 : 0xffffff800deea185 mach_kernel : _kdp_i386_trap + 0x155
0xffffff81fa5e3c00 : 0xffffff800dedb8ba mach_kernel : _kernel_trap + 0x50a
0xffffff81fa5e3c70 : 0xffffff800dd5bb40 mach_kernel : _return_from_trap + 0xe0
0xffffff81fa5e3c90 : 0xffffff800ddae107 mach_kernel : _panic_trap_to_debugger + 0x197
0xffffff81fa5e3db0 : 0xffffff800ddadf53 mach_kernel : _panic + 0x63
0xffffff81fa5e3e20 : 0xffffff7f8ea63090 net.lundman.spl : _rw_destroy + 0x38
0xffffff81fa5e3e40 : 0xffffff7f8fca4626 net.lundman.zfs : _dnode_dest + 0x12
0xffffff81fa5e3e70 : 0xffffff7f8ea604ac net.lundman.spl : _kmem_magazine_destroy + 0x59
0xffffff81fa5e3eb0 : 0xffffff7f8ea5b8b6 net.lundman.spl : _kmem_depot_ws_reap + 0x6c
0xffffff81fa5e3ee0 : 0xffffff7f8ea5fa2e net.lundman.spl : _kmem_cache_reap + 0x66
0xffffff81fa5e3f10 : 0xffffff7f8ea64f6b net.lundman.spl : _taskq_thread + 0x1b9
0xffffff81fa5e3fa0 : 0xffffff800dd5b0ce mach_kernel : _call_continuation + 0x2e
Kernel Extensions in backtrace:
net.lundman.spl(1.9.2)[EAA28CC7-9F6A-3C7B-BB90-691EBDC3A258]@0xffffff7f8ea57000->0xffffff7f8fc4bfff
net.lundman.zfs(1.9.2)[4C34A112-866A-3499-B5EA-4DF7CDF1FFF4]@0xffffff7f8fc7c000->0xffffff7f90035fff
dependency: com.apple.iokit.IOStorageFamily(2.1)[71BB22B0-3075-35A1-B04E-FBAC574DA80D]@0xffffff7f8fc4c000
dependency: net.lundman.spl(1.9.2)[EAA28CC7-9F6A-3C7B-BB90-691EBDC3A258]@0xffffff7f8ea57000

BSD process name corresponding to current thread: kernel_task
Boot args: -v keepsyms=1

Mac OS version:
18G95

[lundman]

Code: Select all

panic(cpu 6 caller 0xffffff7f8ea63090): VERIFY3( rwlp->rw_pad == 0x012345678 ) failed (0 == 305419896)


Checking rw_pad against that number is the DEBUG (always on in rwlock) code checking that the rwlock has been initialised before use. The
number "305419896" is unknown, so it would appear to be random memory (and not the case where rwlock was in-use then
destroyed, as that has another clean number).

This implies the dnode_dest() code is destroying random memory, and not a real dnode. All triggered by a memory reap
due to memory pressure.

Which again is from eviction. It would suggest random memory changes, which is notoriously hard to track down.

[leeb]

FWIW I just saw my first KP in a very long time after upgrading to 1.9.2, on a 2010 Mac Pro running 10.13.6. It had a fair amount of uptime first though, at least a few weeks, so might be something random and unrelated. Unfortunately it's been long enough that I didn't still have any boot-args enabled (and had even forgotten SIP means you need to go to recovery to set them nowadays) but now have keepsyms=1 and debug=0x124, so if it comes up again maybe I can supply another data point.

[jojo45]

I have found that disabling the tunable "kstat.zfs.darwin.tunable.zfs_arc_max" has a dramatic effect. Since removing this from zsysctl.conf, I have had zero crashes. The system is perfectly stable now, and I have not had a single panic since my last post. I had set the tunable parameter anywhere from 4 to 12 GB, gradually increasing it until I could go at least a day between kernel panics. Removing it completely makes the system rock solid.