Kernel Panic Issues after installing 2.0.0 RC5 on Big Sur

[dguisinger01]

Thanks @jimbo

Ok...so....what to say about this........

Turning on keepsyms and setting that debug value suddenly made my system boot with ZFS, I can't explain it....
Obviously that shouldn't be a requirement, thoughts?

[Jimbo]

Yes, I mentioned this a few times earlier in this thread.

I'm testing on two bits of kit:

• Parallels VM on an iMac Pro (CPU issues with AVX) but boots fine with no boot-args; and
• An older MacBook Pro (MacBookPro11,5) that panics on boot without boot-args and does successfully boot with keepsyms=1 (the debug=0x146 isn't needed to boot successfully)

I'm not familiar enough wth macOS Kernel Extension development to really comment on boot-args="keepsyms=1" having an impact on panics, but it certainly smells funny.

@lundman - do you have any thoughts on this? It seems to certainly be reproducable by @dguisinger and myself.

Of course, enabling keepsyms to... keepsyms in a panic... but effectivly preventing the on-boot panic is... umm... not helpful macOS.

Cheers!
James

[Jimbo]

@lundman

This is different. Tesing Rev8.

Copy XCode to dataset, delete and then empty trash - about ~350k files and then:

Code: Select all

panic(cpu 0 caller 0xffffff8008ebfa66):  initproc exited -- exit reason namespace 2 subcode 0xa description: none

uuid info:
 0x114086000   uuid = <2705f0d8-c104-3de9-beb5-b1ef6e28656d>
 0x108990000   uuid = <abc55c09-7b42-3275-9c84-c661a79db594>

Thread 1 crashed

RAX: 0x0000000000000000, RBX: 0x00000001089efb68, RCX: 0x00000001089ec000, RDX: 0x00000001089efb88
RSP: 0x0000700007dc2290, RBP: 0x0000700007dc2390, RSI: 0x00000001089efb78, RDI: 0x0000000108990000
R8:  0x0000000108990b30, R9:  0x0000700007dc2330, R10: 0x000000010899000e, R11: 0x0000000000000202
R12: 0x0000000100020ef6, R13: 0x0000000108990000, R14: 0x00000001089f2740, R15: 0x0000700007dc23b0
RFL: 0x0000000000010246, RIP: 0x00007fff203352c3, CS:  0x000000000000002b, SS:  0x0000000000000023

Thread 0: 0xffffff86a5b0ce30
   0x00007fff202ed622
   0x00007fff2018251b
   0x0000000000000000

Thread 1: 0xffffff86acf5dbf0
   0x00007fff203352c3
   0x00007fff203350e4
   0x00007fff20334fd0
   0x00007fff2026e21d
   0x00000001089b0ef6
   0x00007fff201717c7
   0x00007fff20174195
   0x00007fff20184550
   0x00007fff201774a7
   0x00007fff201780cb
   0x00007fff20181c5d
   0x00007fff20319499
   0x00007fff20318467
   0x0000000000000000

Thread 2: 0xffffff86b0d83480
   0x00007fff202e953e
   0x00007fff20318467
   0x0000000000000000

Thread 3: 0xffffff86b0d82360
   0x00007fff202e953e
   0x00007fff20318467
   0x0000000000000000


Mac OS version:
20B29

Kernel version:
Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:11 PDT 2020; root:xnu-7195.50.7~2/RELEASE_X86_64
Kernel UUID: 84C6DC45-6B02-335F-9439-5D2A9BC385A4
System model name: MacBookPro11,5 (Mac-06F11F11946D27C5)
System shutdown begun: NO
Panic diags file available: YES (0x0)


Update:

• Also still kernel panics on boot if I remove keepsyms=1.
• Also still racey importing ZVOLs (volume appears as uninitialized some times).

Update 2:
Doh! Just saw 9th rev is up... grabbing that.

[lundman]

Well hmmm, we could try the other lookups, atos and lldb. But has to be done with the exact zfs binary and machine to work.

Code: Select all

 # xcrun atos -arch x86_64 -l 0xffffff7fbd4e5000 -o  /Library/Extensions/zfs.kext/Contents/macOS/zfs   0xffffff801c0bc66d 0xffffff801c1ff073 0xffffff801c1ef6aa


Where you get them from the panic report:

Code: Select all

Backtrace (CPU 6), Frame : Return Address
0xffffffa12cc6b6d0 : 0xffffff801c0bc66d
0xffffffa12cc6b720 : 0xffffff801c1ff073
0xffffffa12cc6b760 : 0xffffff801c1ef6aa
0xffffffa12cc6b7b0 : 0xffffff801c061a2f
0xffffffa12cc6b7d0 : 0xffffff801c0bbf0d
0xffffffa12cc6b8f0 : 0xffffff801c0bc1f8
0xffffffa12cc6b960 : 0xffffff801c8bee1a
0xffffffa12cc6b9d0 : 0xffffff801c1efa76
0xffffffa12cc6bb50 : 0xffffff801c1ef75d
0xffffffa12cc6bba0 : 0xffffff801c061a2f
0xffffffa12cc6bbc0 : 0xffffff7fbd77f702

Kernel Extensions in backtrace:
net.lundman.zfs(2.0)[BCA3C767-0F77-3C82-8314-77E397B605B2]@0xffffff7fbd4e5000->0xffffff7fbd814fff


So copy in the load address, then many addresses from the stack list.


An alternate way, if you have the KDK

Code: Select all

(lldb) target create --no-dependents --arch /Library/Extensions/zfs.kext/Contents/macOS/zfs
(lldb) target modules load --file zfs __TEXT 0xffffff7fbd4e5000
(lldb) image lookup --verbose --address 0xffffff801c0bc66d



[Jimbo]

Hi @lundman,

Are you referring to the on-boot panic report?

I'm just recreating my test scenario for deletes (takes about 20 minutes) and I'll then retest the delete panic. After that, I'll see about the on-boot panic...

Cheers!
James

[lundman]

Sure, carry on. It is for the boot panic, since keepsyms can not be set on.

[Jimbo]

Ouch! That was quick to panic deleting XCode from the Trash:

Code: Select all

panic(cpu 0 caller 0xffffff8018befa76): Kernel trap at 0xffffff7fba238631, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000611, CR3: 0x000000046d85f03c, CR4: 0x00000000001626e0
RAX: 0x0000000000000500, RBX: 0x00000000000005f1, RCX: 0x00000000d10cebe0, RDX: 0x00000000000000a5
RSP: 0xffffffb11b59aed0, RBP: 0xffffffb11b59af00, RSI: 0xffffff7fba25944d, RDI: 0x00000000000005f1
R8:  0x0000000000000557, R9:  0x123456789abcdef0, R10: 0x000000044b82fa08, R11: 0x0000000000000000
R12: 0x0000000000000000, R13: 0xffffffb12be47b60, R14: 0x00000000000000a5, R15: 0xffffff7fba25944d
RFL: 0x0000000000010282, RIP: 0xffffff7fba238631, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000611, Error code: 0x0000000000000000, Fault CPU: 0x0, PL: 0, VF: 0

Backtrace (CPU 0), Frame : Return Address
0xffffffb11b59a8f0 : 0xffffff8018abc66d mach_kernel : _handle_debugger_trap + 0x3dd
0xffffffb11b59a940 : 0xffffff8018bff073 mach_kernel : _kdp_i386_trap + 0x143
0xffffffb11b59a980 : 0xffffff8018bef6aa mach_kernel : _kernel_trap + 0x55a
0xffffffb11b59a9d0 : 0xffffff8018a61a2f mach_kernel : _return_from_trap + 0xff
0xffffffb11b59a9f0 : 0xffffff8018abbf0d mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffb11b59ab10 : 0xffffff8018abc1f8 mach_kernel : _panic_trap_to_debugger + 0x268
0xffffffb11b59ab80 : 0xffffff80192bee1a mach_kernel : _panic + 0x54
0xffffffb11b59abf0 : 0xffffff8018befa76 mach_kernel : _sync_iss_to_iks + 0x2c6
0xffffffb11b59ad70 : 0xffffff8018bef75d mach_kernel : _kernel_trap + 0x60d
0xffffffb11b59adc0 : 0xffffff8018a61a2f mach_kernel : _return_from_trap + 0xff
0xffffffb11b59ade0 : 0xffffff7fba238631 net.lundman.zfs : _spl_mutex_enter + 0x21
0xffffffb11b59af00 : 0xffffff7fba029e0b net.lundman.zfs : _rrw_enter_read_impl + 0x2b
0xffffffb11b59af50 : 0xffffff7fba0baa66 net.lundman.zfs : _zfs_znode_asyncwait + 0x46
0xffffffb11b59af90 : 0xffffff7fba0c03ae net.lundman.zfs : _zfs_zget_ext + 0x1fe
0xffffffb11b59b040 : 0xffffff7fba08f417 net.lundman.zfs : _zfs_rmnode + 0x107
0xffffffb11b59b230 : 0xffffff7fba0b8640 net.lundman.zfs : _zfs_vnop_reclaim + 0x180
0xffffffb11b59b260 : 0xffffff8018d40f93 mach_kernel : _vnode_getiocount + 0x1233
0xffffffb11b59b2e0 : 0xffffff8018d40590 mach_kernel : _vnode_getiocount + 0x830
0xffffffb11b59b320 : 0xffffff8018d417e9 mach_kernel : _vnode_rele_internal + 0x749
0xffffffb11b59b350 : 0xffffff8018d43ac4 mach_kernel : _vfs_init_io_attributes + 0x7f4
0xffffffb11b59b3d0 : 0xffffff7fba0ba815 net.lundman.zfs : _zfs_znode_getvnode + 0x1b5
0xffffffb11b59b470 : 0xffffff7fba0c057c net.lundman.zfs : _zfs_zget_ext + 0x3cc
0xffffffb11b59b520 : 0xffffff7fba08ecae net.lundman.zfs : _zfs_dirent_lock + 0x40e
0xffffffb11b59b5b0 : 0xffffff7fba08eeb6 net.lundman.zfs : _zfs_dirlook + 0xe6
0xffffffb11b59b620 : 0xffffff7fba0ad941 net.lundman.zfs : _zfs_lookup + 0x151
0xffffffb11b59b680 : 0xffffff7fba0b4a5a net.lundman.zfs : _zfs_vnop_lookup + 0x17a
0xffffffb11b59b820 : 0xffffff8018d3ad8e mach_kernel : _lookup + 0x36e
0xffffffb11b59b980 : 0xffffff8018d3a0b4 mach_kernel : _namei + 0x1144
0xffffffb11b59bba0 : 0xffffff8018d21581 mach_kernel : _getattrlistbulk + 0xac1
0xffffffb11b59bde0 : 0xffffff8018d211fc mach_kernel : _getattrlistbulk + 0x73c
0xffffffb11b59bf40 : 0xffffff8019169ceb mach_kernel : _unix_syscall64 + 0x27b
0xffffffb11b59bfa0 : 0xffffff8018a621f6 mach_kernel : _hndl_unix_scall64 + 0x16
      Kernel Extensions in backtrace:
         net.lundman.zfs(2.0)[9FBFB11A-8D78-3C25-A9D3-04961EC4FCB4]@0xffffff7fb9f9f000->0xffffff7fba2cefff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[3FE3BE82-782C-3D93-8EFA-F06A4594D17C]@0xffffff801b537000->0xffffff801b548fff

Process name corresponding to current thread: mds
Boot args: keepsyms=1

Mac OS version:
20B29

Kernel version:
Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:11 PDT 2020; root:xnu-7195.50.7~2/RELEASE_X86_64
Kernel UUID: 84C6DC45-6B02-335F-9439-5D2A9BC385A4
KernelCache slide: 0x0000000018800000
KernelCache base:  0xffffff8018a00000
Kernel slide:      0x0000000018810000
Kernel text base:  0xffffff8018a10000
__HIB  text base: 0xffffff8018900000
System model name: MacBookPro11,5 (Mac-06F11F11946D27C5)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 1910211898460
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000001bcc1807015
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x000000098069f56a 0x0000000000000000
last started kext at 64112037720: org.virtualbox.kext.VBoxNetAdp   6.1.16 (addr 0xffffff7fb9e89000, size 8192)
loaded kexts:
org.virtualbox.kext.VBoxNetAdp   6.1.16
org.virtualbox.kext.VBoxNetFlt   6.1.16
org.virtualbox.kext.VBoxUSB   6.1.16
org.virtualbox.kext.VBoxDrv   6.1.16
net.lundman.zfs   2.0.0
com.intel.driver.EnergyDriver   3.7.0
@kext.AMDRadeonServiceManager   4.0.0
>AudioAUUC   1.70
>!APlatformEnabler   2.7.0d0
@kext.AMDFramebuffer   4.0.0
>AGPM   119
>X86PlatformShim   1.0.0
@filesystems.autofs   3.0
@kext.AMDRadeonX4000   4.0.0
@fileutil   20.036.15
>!AGraphicsDevicePolicy   6.1.27
>!AUpstreamUserClient   3.6.8
@AGDCPluginDisplayMetrics   6.1.27
>pmtelemetry   1
>!AHDAHardwareConfigDriver   283.15
>LuaHardwareAccess   1.0.16
|IOUserEthernet   1.0.1
>!AHDA   283.15
>usb.!UUserHCI   1
|IO!BSerialManager   8.0.1f5
@Dont_Steal_Mac_OS_X   7.0.0
>eficheck   1
>!AHV   1
>!ADiskImages2   1
>!ALPC   3.1
>!ACameraInterface   7.6.0
>!AMuxControl   6.1.27
|Broadcom!B20703USBTransport   8.0.1f5
>!AFIVRDriver   4.1.0
>!ASMCLMU   212
>!A!IHD5000Graphics   16.0.0
>AGDCBacklightControl   6.1.27
>!AThunderboltIP   4.0.3
>!ABacklight   180.3
>ACPI_SMC_PlatformPlugin   1.0.0
>!A!ISlowAdaptiveClocking   4.0.0
@kext.AMD7000!C   4.0.0
>!A!IFramebufferAzul   16.0.0
>!AMCCSControl   1.14
>!ATopCaseHIDEventDriver   4000.27
>!UTopCaseDriver   4000.27
|IO!BUSBDFU   8.0.1f5
|SCSITaskUserClient   436.40.6
>!UCardReader   511.40.9
@filesystems.apfs   1677.50.1
>AirPort.BrcmNIC   1400.1.1
>!AAHCIPort   346
>!AFileSystemDriver   3.0.1
@filesystems.tmpfs   1
@filesystems.hfs.kext   556.41.1
@BootCache   40
@!AFSCompression.!AFSCompressionTypeZlib   1.0.0
@!AFSCompression.!AFSCompressionTypeDataless   1.0.0d1
@private.KextAudit   1.0
|!ABCM5701Ethernet   11.0.0
>!ASmartBatteryManager   161.0.0
>!ARTC   2.0
>!AACPIButtons   6.1
>!AHPET   1.8
>!ASMBIOS   2.1
>!AACPIEC   6.1
>!AAPIC   1.7
@!ASystemPolicy   2.0.0
@nke.applicationfirewall   310
|IOKitRegistryCompatibility   1
|EndpointSecurity   1
|IOUSBUserClient   900.4.2
@plugin.IOAVBDiscoveryPlugin   900.12
>!UAudio   401.4
@kext.AMDRadeonX4030HWLibs   1.0
@kext.triggers   1.0
@kext.AMDRadeonX4000HWServices   4.0.0
>DspFuncLib   283.15
@kext.OSvKernDSPLib   529
|IOSerial!F   11
|IOAVB!F   900.12
>!AGraphicsControl   6.1.27
|Broadcom!BHost!CUSBTransport   8.0.1f5
|IO!BHost!CUSBTransport   8.0.1f5
|IO!BHost!CTransport   8.0.1f5
@plugin.IOgPTPPlugin   900.11
|IOEthernetAVB!C   1.1.0
>!ASMBusPCI   1.0.14d1
@!AGPUWrangler   6.1.27
>!AThunderboltEDMSink   5.0.3
>!AThunderboltDPOutAdapter   8.1.4
>!ABacklightExpert   1.1.0
>IOPlatformPluginLegacy   1.0.0
>X86PlatformPlugin   1.0.0
>IOPlatformPlugin!F   6.0.0d8
|IONDRVSupport   585
|IOSlowAdaptiveClocking!F   1.0.0
@kext.AMDSupport   4.0.0
|IOAccelerator!F2   439.35.4
@!AGraphicsDeviceControl   6.1.27
>!ASMBus!C   1.0.18d1
>!AHDA!C   283.15
|IOHDA!F   283.15
|IOGraphics!F   585
|IOAudio!F   300.6.1
@vecLib.kext   1.2.0
>!AHS!BDriver   4000.27
>IO!BHIDDriver   8.0.1f5
>!AActuatorDriver   4400.28
>!AMultitouchDriver   4400.28
>!AInputDeviceSupport   4400.35
>!AHIDKeyboard   222
>usb.IOUSBHostHIDDevice   1.2
>usb.cdc   5.0.0
>usb.networking   5.0.0
>usb.!UHostCompositeDevice   1.2
>!AXsanScheme   3
|IOAHCIBlock!S   332
|IO80211!F   1200.12.2b1
>corecapture   1.0.4
>!A!ILpssGspi   3.0.60
|IOAHCI!F   294
>usb.!UXHCIPCI   1.2
>usb.!UXHCI   1.2
>!AThunderboltDPInAdapter   8.1.4
>!AThunderboltDPAdapter!F   8.1.4
>!AThunderboltPCIUpAdapter   4.1.1
>!AThunderboltPCIDownAdapter   4.1.1
>!ABSDKextStarter   3
|IOSurface   289.3
|IOSkywalk!F   1
@filesystems.hfs.encodings.kext   1
>usb.!UHostPacketFilter   1.0
|IOUSB!F   900.4.2
>mDNSOffloadUserClient   1.0.1b8
>!AThunderboltNHI   7.2.8
|IOThunderbolt!F   9.3.2
>!AEFINVRAM   2.1
>!AEFIRuntime   2.1
|IOSMBus!F   1.1
|IOHID!F   2.0.0
$!AImage4   3.0.0
|IOTimeSync!F   900.11
|IONetworking!F   3.4
>DiskImages   493.0.0
|IO!B!F   8.0.1f5
|IOReport!F   47
|IO!BPacketLogger   8.0.1f5
$quarantine   4
$sandbox   300.0
@kext.!AMatch   1.0.0d1
|CoreAnalytics!F   1
>!ASSE   1.0
>!AKeyStore   2
>!UTDM   511.40.9
|IOUSBMass!SDriver   184.40.6
|IOSCSIBlockCommandsDevice   436.40.6
|IO!S!F   2.1
|IOSCSIArchitectureModel!F   436.40.6
>!AMobileFileIntegrity   1.0.5
@kext.CoreTrust   1
>!AFDEKeyStore   28.30
>!AEffaceable!S   1.0
>!ACredentialManager   1.0
>KernelRelayHost   1
|IOUSBHost!F   1.2
>!UHostMergeProperties   1.2
>usb.!UCommon   1.0
>!ABusPower!C   1.0
>!ASEPManager   1.0.1
>IOSlaveProcessor   1
>!AACPIPlatform   6.1
>!ASMC   3.1.9
|IOPCI!F   2.9
|IOACPI!F   1.4
>watchdog   1
@kec.pthread   1
@kec.



FWIW, some of these panics seem to knock out my bluetooth and I need to shutdown to get that back...

[lundman]

dangit, stop finding panics

[Jimbo]

Not intentional, I assure you.

Boot panic, I might need a bit more direction here.

Code: Select all

xcrun atos -arch x86_64 -l 0xffffff7fa8b9f000 -o /Library/Extensions/zfs.kext/Contents/macOS/zfs 0xffffff80076bc66d 0xffffff80077ff073 0xffffff80077ef6aa 0xffffff8007661a2f 0xffffff80076bbf0d 0xffffff80076bc1f8 0xffffff8007ebee1a 0xffffff80077efa76 0xffffff80077ef75d 0xffffff8007661a2f 0xffffff7fa8e39782 0xffffff7fa8e39031 0xffffff7fa8e38dcd 0xffffff7fa8ca3296 0xffffff8007def09b 0xffffff8007deec46 0xffffff8007deddcf 0xffffff8007df0b46 0xffffff800766113e

Produces:

Code: Select all

0xffffff80076bc66d
0xffffff80077ff073
0xffffff80077ef6aa
0xffffff8007661a2f
0xffffff80076bbf0d
0xffffff80076bc1f8
0xffffff8007ebee1a
0xffffff80077efa76
0xffffff80077ef75d
0xffffff8007661a2f
macho_find_symbol (in zfs) + 242
spl_loadsymbols (in zfs) + 241
spl_start (in zfs) + 429
net_lundman_zfs_zvol::start(IOService*) (in zfs) + 70
0xffffff8007def09b
0xffffff8007deec46
0xffffff8007deddcf
0xffffff8007df0b46
0xffffff800766113e


Is that helpful?

Cheers!
James

[lundman]

orly! now that is interesting