2.2.0rc2 crash on Ventura

[tangent]

I just installed the latest release candidate build on my i9 iMac running macOS 13.6 and got an immediate panic on reimporting the pools:

Code: Select all

panic(cpu 0 caller 0xffffff800f7b60f3): Kernel trap at 0xffffff7fa63f3272, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000000, CR3: 0x000000002393e000, CR4: 0x00000000003626e0
RAX: 0x0000000000000001, RBX: 0x0000000000000000, RCX: 0x0000000000000010, RDX: 0x0000000000000001
RSP: 0xffffffc4d2357810, RBP: 0xffffffc4d2357850, RSI: 0xffffffe6aeaaffb0, RDI: 0xffffffe6a815b040
R8:  0xffffffe6ad536c00, R9:  0xb72a829bb39ffd00, R10: 0xffffffc4d2357b00, R11: 0x0000000000000000
R12: 0xffffffe6aeaaffb0, R13: 0x0000000000000001, R14: 0x0000000000000000, R15: 0xffffffe6a815b040
RFL: 0x0000000000010206, RIP: 0xffffff7fa63f3272, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0x0, PL: 0, VF: 1

Panicked task 0xffffff9a0b48b698: 662 threads: pid 0: kernel_task
Backtrace (CPU 0), panicked thread: 0xffffff9a0a805598, Frame : Return Address
0xffffffc4d23571f0 : 0xffffff800f67205d
0xffffffc4d2357240 : 0xffffff800f7c6144
0xffffffc4d2357280 : 0xffffff800f7b5c57
0xffffffc4d23572d0 : 0xffffff800f612951
0xffffffc4d23572f0 : 0xffffff800f67233d
0xffffffc4d23573e0 : 0xffffff800f6719e7
0xffffffc4d2357440 : 0xffffff800fddb32b
0xffffffc4d2357530 : 0xffffff800f7b60f3
0xffffffc4d23576b0 : 0xffffff800f7b5ddd
0xffffffc4d2357700 : 0xffffff800f612951
0xffffffc4d2357720 : 0xffffff7fa63f3272
0xffffffc4d2357850 : 0xffffff7fa63ecf91
0xffffffc4d2357990 : 0xffffff7fa652873e
0xffffffc4d23579d0 : 0xffffff7fa64054a7
0xffffffc4d2357a00 : 0xffffff7fa640b2cd
0xffffffc4d2357a30 : 0xffffff7fa640a0d3
0xffffffc4d2357af0 : 0xffffff7fa6421eb1
0xffffffc4d2357c30 : 0xffffff7fa644b3b4
0xffffffc4d2357ce0 : 0xffffff7fa642cff0
0xffffffc4d2357d10 : 0xffffff7fa64b99d5
0xffffffc4d2357d60 : 0xffffff7fa64855a4
0xffffffc4d2357dc0 : 0xffffff7fa64be637
0xffffffc4d2357e20 : 0xffffff7fa64c2c8f
0xffffffc4d2357ea0 : 0xffffff7fa64c2e7e
0xffffffc4d2357ec0 : 0xffffff7fa63fc145
0xffffffc4d2357f60 : 0xffffff7fa63fc6bb
0xffffffc4d2357fa0 : 0xffffff800f61219e
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.2)[CCB0DB09-A9A4-38CE-BAC6-860E585E7170]@0xffffff7fa63ea000->0xffffff7fa6750fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[70B6B0B8-8059-3529-A544-D33C65534409]@0xffffff80121e1000->0xffffff80121f8fff

Process name corresponding to current thread (0xffffff9a0a805598): kernel_task

Mac OS version:
22G120

Kernel version:
Darwin Kernel Version 22.6.0: Fri Sep 15 13:39:52 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_X86_64
Kernel UUID: F75FC53F-FC1A-3AB6-8980-EF66A83DD51D
roots installed: 0
KernelCache slide: 0x000000000f200000
KernelCache base:  0xffffff800f400000
Kernel slide:      0x000000000f2dc000
Kernel text base:  0xffffff800f4dc000
__HIB  text base: 0xffffff800f300000
System model name: iMac20,2 (Mac-AF89B6D9451A490B)
System shutdown begun: NO
Hibernation exit count: 0

System uptime in nanoseconds: 469698422668
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x0000006d5c3529bf
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000016086768ae 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff806fe60000 - 0xffffffa06fe60000
  . PGZ   : 0xffffff806fe60000 - 0xffffff8078e61000
  . VM    : 0xffffff8078e61000 - 0xffffff8544594000
  . RO    : 0xffffff8544594000 - 0xffffff86dd7fa000
  . GEN0  : 0xffffff86dd7fa000 - 0xffffff8ba8f2d000
  . GEN1  : 0xffffff8ba8f2d000 - 0xffffff9074660000
  . GEN2  : 0xffffff9074660000 - 0xffffff953fd93000
  . GEN3  : 0xffffff953fd93000 - 0xffffff9a0b4c6000
  . DATA  : 0xffffff9a0b4c6000 - 0xffffffa06fe60000
  Metadata: 0xffffffcc1b72a000 - 0xffffffcc3b72a000
  Bitmaps : 0xffffffcc3b72a000 - 0xffffffcc48f2a000
  Extra   : 0 - 0

last started kext at 345359233417: |IOAHCIBlock!S   350.140.3 (addr 0xffffff8011c58000, size 94208)
loaded kexts:
fi.dungeon.driver.SATSMARTDriver   0.10
org.openzfsonosx.zfs   2.2.0
>!AAHCIPort   378
>AudioAUUC   1.70
>!APlatformEnabler   2.7.0d0
>AGPM   131
>X86PlatformShim   1.0.0
@filesystems.autofs   3.0
@kext.AMDRadeonServiceManager   4.1.4
>!AHIDALSService   1
>!AUpstreamUserClient   3.6.9
@kext.AMDRadeonX6000   4.1.4
>!ABridgeAudio!C   340.1
>!AGFXHDA   240.1
>!AGraphicsDevicePolicy   7.1.18
@AGDCPluginDisplayMetrics   7.1.18
>!A!IKBLGraphics   20.0.6
>pmtelemetry   1
@filesystems.nfs   1
|IOUserEthernet   1.0.1
>usb.!UUserHCI   1
>!AMuxControl2   7.1.18
>!AHV   1
>!ADiskImages2   198.100.13
>AGDCBacklightControl   7.1.18
>!A!ICFLGraphicsFramebuffer   20.0.6
>!ABacklight   180.8
>!AFIVRDriver   4.1.0
@UVCService   1
>ACPI_SMC_PlatformPlugin   1.0.0
>!A!IPCHPMC   2.0.1
>!AMCCSControl   1.16
>BridgeAudioCommunication   340.1
>!AAVEBridge   6.1
>!AThunderboltIP   4.0.3
>!A!ISlowAdaptiveClocking   4.0.0
>PioneerSuperDrive   554
>!UODD   554
|SCSITaskUserClient   482.120.2
>!UMass!SInterfaceNub   554
>!UCardReader   554
@!AEthernetAquantiaAqtionFirmware   1.0.36
>BCMWLANFirmware4388.Hashstore   1
>BCMWLANFirmware4387.Hashstore   1
>BCMWLANFirmware4378.Hashstore   1
>BCMWLANFirmware4377.Hashstore   1
>BCMWLANFirmware4364.Hashstore   1
>BCMWLANFirmware4355.Hashstore   1
>!AFileSystemDriver   3.0.1
@filesystems.tmpfs   1
@filesystems.lifs   1
@filesystems.hfs.kext   627.100.6
@BootCache   40
@!AFSCompression.!AFSCompressionTypeZlib   1.0.0
@!AFSCompression.!AFSCompressionTypeDataless   1.0.0d1
@filesystems.apfs   2142.140.9
>!ABCMWLANBusInterfacePCIeMac   1
@private.KextAudit   1.0
>!AACPIButtons   6.1
>!ASMBIOS   2.1
>!AACPIEC   6.1
>!AAPIC   1.7
$!AUserConsent   1
@!ASystemPolicy   2.0.0
@nke.applicationfirewall   404
|IOKitRegistryCompatibility   1
|EndpointSecurity   1
@Dont_Steal_Mac_OS_X   7.0.0
@kec.Compression   1
@kec.!AEncryptedArchive   1
|IOAHCIBlock!S   350.140.3
|IOAHCI!F   301.100.1
>!AThunderboltPCIUpAdapter   4.1.1
>!AHIDKeyboard   231
@kext.AMDRadeonX6200HWLibs   1.0
@kext.AMDRadeonX6000HWServices   4.1.4
>!UAudio   560.2
>!AAudioClockLibs   240.1
>!AHDA!C   440.2
|IOHDA!F   440.2
|IOAudio!F   440.2
@vecLib.kext   1.2.0
>!A!ILpssUARTv1   3.0.60
>!A!ILpssUARTCommon   3.0.60
>!AOnboardSerial   1.0
>!ASMBusPCI   1.0.14d1
@kext.triggers   1.0
>IOHIDPowerSource   1
>!ACallbackPowerSource   1
|IO!BSerialManager   9.0.0
|IO!BPacketLogger   9.0.0
|IO!BHost!CUSBTransport   9.0.0
|IO!BHost!CUARTTransport   9.0.0
|IO!BHost!CTransport   9.0.0
>IO!BHost!CPCIeTransport   9.0.0
|IOAVB!F   1140.6
|CSR!BHost!CUSBTransport   9.0.0
|Broadcom!BHost!CUSBTransport   9.0.0
|Broadcom!B20703USBTransport   9.0.0
>!ARSMChannel   1
|IORSM!F   1
>!AIPAppender   1.0
>!AGraphicsControl   7.1.18
|IOAccelerator!F2   475.40.6
@kext.AMDRadeonX6000Framebuffer   4.1.4
@kext.AMDSupport   4.1.4
>!ABacklightExpert   1.1.0
|IONDRVSupport   597
>IOPlatformPluginLegacy   1.0.0
>X86PlatformPlugin   1.0.0
@plugin.IOgPTPPlugin   1150.2
>IOPlatformPlugin!F   6.0.0d8
>!ASMBus!C   1.0.18d1
>!AThunderboltEDMSink   5.0.3
>!AThunderboltDPOutAdapter   8.5.1
@!AGPUWrangler   7.1.18
@!AGraphicsDeviceControl   7.1.18
|IOGraphics!F   597
|IOSlowAdaptiveClocking!F   1.0.0
>driverkit.serial   6.0.0
|IOSCSIMultimediaCommandsDevice   482.120.2
|IOBD!S!F   1.8
|IODVD!S!F   1.8
|IOCD!S!F   1.8
>usb.!UHub   1.2
>usb.IOUSBHostHIDDevice   1.2
>usb.cdc.ecm   5.0.0
>usb.cdc.ncm   5.0.0
>usb.cdc   5.0.0
>usb.networking   5.0.0
>usb.!UHostCompositeDevice   1.2
>!AThunderboltPCIDownAdapter   4.1.1
>!AThunderboltDPInAdapter   8.5.1
>!AThunderboltDPAdapter!F   8.5.1
>!AHPM   3.4.4
>!A!ILpssI2C!C   3.0.60
>!A!ILpssI2C   3.0.60
>!A!ILpssDmac   3.0.60
>!ABSDKextStarter   3
|IOSurface   336.60.1
@filesystems.hfs.encodings.kext   1
>!ASyntheticGame!C   10.7.1
>!AXsanScheme   3
>!AThunderboltNHI   7.2.81
|IOThunderbolt!F   9.3.3
>usb.!UVHCIBCE   1.2
>usb.!UVHCICommonBCE   1.0
>usb.!UVHCI   1.2
>usb.!UVHCICommon   1.0
>!AEffaceableNOR   1.0
|IOBufferCopy!C   1.1.0
|IOBufferCopyEngine!F   1
|IONVMe!F   2.1.0
>!AEthernetAquantiaAqtion   1.0.64
>!ABCMWLANCoreMac   1.0.0
|IO80211!F   1200.13.0
>IOImageLoader   1.0.0
>!AOLYHALMac   1
|IOSerial!F   11
>corecapture   1.0.4
>usb.!UHostPacketFilter   1.0
|IOUSB!F   900.4.2
>usb.!UXHCIPCI   1.2
>usb.!UXHCI   1.2
>!AEFINVRAM   2.1
>!AEFIRuntime   2.1
>!ASMCRTC   1.0
|IOSMBus!F   1.1
|IOHID!F   2.0.0
|IOTimeSync!F   1150.2
|IOSkywalk!F   1.0
>mDNSOffloadUserClient   1.0.1b8
|IONetworking!F   3.4
>DiskImages   493.0.0
|IO!B!F   9.0.0
|IOReport!F   47
$quarantine   4
$sandbox   300.0
@kext.!AMatch   1.0.0d1
>!ASSE   1.0
>!AKeyStore   2
>!UTDM   554
|IOUSBMass!SDriver   235.140.2
|IOSCSIBlockCommandsDevice   482.120.2
|IO!S!F   2.1
|IOSCSIArchitectureModel!F   482.120.2
>!AFDEKeyStore   28.30
>!AEffaceable!S   1.0
>!ACyrus   1
>!AMobileFileIntegrity   1.0.5
$!AImage4   5.0.0
@kext.CoreTrust   1
>!ACredentialManager   1.0
|CoreAnalytics!F   1
>KernelRelayHost   1
|IOUSBHost!F   1.2
>!UHostMergeProperties   1.2
>usb.!UCommon   1.0
>!ABusPower!C   1.0
>!ASEPManager   1.0.1
>IOSlaveProcessor   1
>!AACPIPlatform   6.1
>!ASMC   3.1.9
|IOPCI!F   2.9
|IOACPI!F   1.4
>watchdog   1
@kec.pthread   1
@kec.Libm   1
@kec.corecrypto   12.0


I was previously using a build of 2.1.99 something, which worked fine for months.

Is the "Catalina 10.15" build supposed to work on this newer OS, or do I need a Ventura-specific build to test with?

I'm also trying to upgrade to Sonoma as part of a general system refresh. Is that going to exacerbate this problem or help?

[Jimbo]

I just posted in Development. Same thing in Sonoma iMac Pro. For me, it seems to be the mounting that triggers panics, though more testing is needed.

[tangent]

Rolling back to 2.1.7rc3 fixes the symptom.

[tangent]

I took this opportunity to do a clean reinstall of the OS, something that's been waiting for an excuse for a long time. I've now got the same machine upgraded to Sonoma 14.1.1, and the same 2.2.0rc2 build is also panicking on "sudo zpool import -a".

In case it's materially different and that matters in debugging this, here's the new panic report:

Code: Select all

panic(cpu 2 caller 0xffffff8011588fb3): Kernel trap at 0xffffff7fa72cc7e3, type 13=general protection, registers:
CR0: 0x000000008001003b, CR2: 0x0000700005e7de58, CR3: 0x0000000eff20b1a6, CR4: 0x00000000003626e0
RAX: 0x0040261025002001, RBX: 0x000000000000106f, RCX: 0x0000000000000010, RDX: 0x0040261025002001
RSP: 0xffffffd441d1b130, RBP: 0xffffffd441d1b130, RSI: 0xffffffe6a815b188, RDI: 0x0040261025002001
R8:  0xffffffe2dff90c90, R9:  0x28b3a00d00733f00, R10: 0xffffffd441d1b400, R11: 0x0000000000000100
R12: 0x0000000000000000, R13: 0xffffffe6a815b040, R14: 0xffffffe6a815b180, R15: 0x0000000000000000
RFL: 0x0000000000010202, RIP: 0xffffff7fa72cc7e3, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0x2, PL: 0, VF: 0

Panicked task 0xffffffa044346008: 1 threads: pid 1684: zpool
Backtrace (CPU 2), panicked thread: 0xffffff9b774c5b30, Frame : Return Address
0xffffff801110d140 : 0xffffff801143aca1
0xffffff801110d190 : 0xffffff80115993c0
0xffffff801110d1d0 : 0xffffff8011588b0c
0xffffff801110d250 : 0xffffff80113d7971
0xffffff801110d270 : 0xffffff801143af8d
0xffffff801110d360 : 0xffffff801143a633
0xffffff801110d3c0 : 0xffffff8011bd957b
0xffffff801110d4b0 : 0xffffff8011588fb3
0xffffff801110d630 : 0xffffff8011588c97
0xffffff801110d6b0 : 0xffffff80113d7971
0xffffff801110d6d0 : 0xffffff7fa72cc7e3
0xffffffd441d1b130 : 0xffffff7fa72a2d76
0xffffffd441d1b270 : 0xffffff7fa73de73e
0xffffffd441d1b2b0 : 0xffffff7fa72bb4a7
0xffffffd441d1b2e0 : 0xffffff7fa72c12cd
0xffffffd441d1b310 : 0xffffff7fa72c00d3
0xffffffd441d1b3d0 : 0xffffff7fa72db79a
0xffffffd441d1b410 : 0xffffff7fa72dafa0
0xffffffd441d1b530 : 0xffffff7fa72db9b8
0xffffffd441d1b550 : 0xffffff7fa72e39c5
0xffffffd441d1b5a0 : 0xffffff7fa736769d
0xffffffd441d1b7b0 : 0xffffff7fa7356a25
0xffffffd441d1b920 : 0xffffff7fa7355215
0xffffffd441d1b970 : 0xffffff7fa7354d15
0xffffffd441d1ba10 : 0xffffff7fa73bb5a0
0xffffffd441d1ba50 : 0xffffff7fa73b4f9f
0xffffffd441d1bb20 : 0xffffff7fa73f892d
0xffffffd441d1bb60 : 0xffffff801165a688
0xffffffd441d1bb90 : 0xffffff801164ca04
0xffffffd441d1bbf0 : 0xffffff801163f3eb
0xffffffd441d1be00 : 0xffffff80118f2f64
0xffffffd441d1be30 : 0xffffff80119559a4
0xffffffd441d1bf40 : 0xffffff8011a541cd
0xffffffd441d1bfa0 : 0xffffff80113d7db6
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.2)[CCB0DB09-A9A4-38CE-BAC6-860E585E7170]@0xffffff7fa72a0000->0xffffff7fa7606fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[A96D850A-038F-31D7-9F3D-A6775064548E]@0xffffff8013859000->0xffffff8013870fff

Process name corresponding to current thread (0xffffff9b774c5b30): zpool

Mac OS version:
23B81

Kernel version:
Darwin Kernel Version 23.1.0: Mon Oct  9 21:27:27 PDT 2023; root:xnu-10002.41.9~6/RELEASE_X86_64
Kernel UUID: 7C8FE931-57C6-38AD-9EBB-CC5572293AE6
roots installed: 0
KernelCache slide: 0x0000000011000000
KernelCache base:  0xffffff8011200000
Kernel slide:      0x00000000110e0000
Kernel text base:  0xffffff80112e0000
__HIB  text base: 0xffffff8011100000
System model name: iMac20,2 (Mac-AF89B6D9451A490B)
System shutdown begun: NO
Hibernation exit count: 0

System uptime in nanoseconds: 307631606798
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x00000047a045ceab
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x0000001359f35a72 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff90406e7000 - 0xffffffb0406e7000
  . PGZ   : 0xffffff90406e7000 - 0xffffff90496e8000
  . VM    : 0xffffff90496e8000 - 0xffffff9514e1b000
  . RO    : 0xffffff9514e1b000 - 0xffffff96ae081000
  . GEN0  : 0xffffff96ae081000 - 0xffffff9b797b4000
  . GEN1  : 0xffffff9b797b4000 - 0xffffffa044ee7000
  . GEN2  : 0xffffffa044ee7000 - 0xffffffa51061a000
  . GEN3  : 0xffffffa51061a000 - 0xffffffa9dbd4d000
  . DATA  : 0xffffffa9dbd4d000 - 0xffffffb0406e7000
  Metadata: 0xffffff80f69a1000 - 0xffffff81169a1000
  Bitmaps : 0xffffff81169a1000 - 0xffffff81241a1000
  Extra   : 0 - 0

last started kext at 279832083728: |IOAHCIBlock!S   355 (addr 0xffffff80132d0000, size 94208)
loaded kexts:
org.openzfsonosx.zfs   2.2.0
>!AAHCIPort   380
>AudioAUUC   1.70
@kext.AMDRadeonServiceManager   5.0.1
>!AHIDALSService   1
>!APlatformEnabler   2.7.0d0
>AGPM   135
>X86PlatformShim   1.0.0
>!AUpstreamUserClient   3.6.11
@filesystems.autofs   3.0
@kext.AMDRadeonX6000   5.0.1
>!ABridgeAudio!C   400.1
>!AGFXHDA   300.1
>!A!IKBLGraphics   22.0.1
>!AGraphicsDevicePolicy   8.1.9
@AGDCPluginDisplayMetrics   8.1.9
>pmtelemetry   1
@filesystems.nfs   1
|IOUserEthernet   1.0.1
>usb.!UUserHCI   1
>!ASunrise   1
>!AHV   1
>!AMuxControl2   8.1.9
>AGDCBacklightControl   8.1.9
>!AFIVRDriver   4.1.0
>!ABacklight   180.9
>!AMCCSControl   1.17
>!A!IPCHPMC   2.0.1
>BridgeAudioCommunication   400.1
>!AAVEBridge   6.1
>!A!ICFLGraphicsFramebuffer   22.0.1
@UVCService   1
>!AThunderboltIP   4.0.3
>ACPI_SMC_PlatformPlugin   1.0.0
>!A!ISlowAdaptiveClocking   4.0.0
>PioneerSuperDrive   556
>!UODD   556
|SCSITaskUserClient   492
>!UCardReader   556
>!UMass!SInterfaceNub   556
@!AEthernetAquantiaAqtionFirmware   1.0.36
>BCMWLANFirmware4388.Hashstore   1
>BCMWLANFirmware4387.Hashstore   1
>BCMWLANFirmware4378.Hashstore   1
>BCMWLANFirmware4377.Hashstore   1
>BCMWLANFirmware4364.Hashstore   1
>BCMWLANFirmware4355.Hashstore   1
>!AFileSystemDriver   3.0.1
@filesystems.tmpfs   1
@filesystems.lifs   1
@filesystems.hfs.kext   650.0.2
@BootCache   40
@!AFSCompression.!AFSCompressionTypeZlib   1.0.0
@!AFSCompression.!AFSCompressionTypeDataless   1.0.0d1
@filesystems.apfs   2235.41.1
>!ABCMWLANBusInterfacePCIeMac   1
@private.KextAudit   1.0
>!AACPIButtons   6.1
>!ASMBIOS   2.1
>!AACPIEC   6.1
>!AAPIC   1.7
@!ASystemPolicy   2.0.0
@nke.applicationfirewall   404
|IOKitRegistryCompatibility   1
|EndpointSecurity   1
@Dont_Steal_Mac_OS_X   7.0.0
>!ADiskImages2   273
@kec.!AEncryptedArchive   1
|IOAHCIBlock!S   355
|IOAHCI!F   304
>!AThunderboltPCIUpAdapter   4.1.1
>!AHIDKeyboard   7400.2
@kext.AMDRadeonX6200HWLibs   1.0
>!UAudio   610.3
>!AAudioClockLibs   300.1
@kext.AMDRadeonX6000HWServices   5.0.1
>!AHDA!C   500.3
|IOHDA!F   500.3
|IOAudio!F   500.4
@vecLib.kext   1.2.0
>!A!ILpssUARTv1   3.0.60
>!A!ILpssUARTCommon   3.0.60
>!AOnboardSerial   1.0
>!ASMBusPCI   1.0.14d1
@kext.triggers   1.0
>IOHIDPowerSource   1
>!ACallbackPowerSource   1
|IOAVB!F   1200.18
>!ARSMChannel   1
|IORSM!F   1
>!AIPAppender   1.0
>!AGraphicsControl   8.1.9
>!ABacklightExpert   1.1.0
>!ASMBus!C   1.0.18d1
@kext.AMDRadeonX6000Framebuffer   5.0.1
@kext.AMDSupport   5.0.1
|IONDRVSupport   598
|IOAccelerator!F2   481.0.1
>!AThunderboltEDMSink   5.0.3
>!AThunderboltDPOutAdapter   8.5.1
@plugin.IOgPTPPlugin   1200.91
>IOPlatformPluginLegacy   1.0.0
>X86PlatformPlugin   1.0.0
>IOPlatformPlugin!F   6.0.0d8
@!AGPUWrangler   8.1.9
@!AGraphicsDeviceControl   8.1.9
|IOGraphics!F   598
|IOSlowAdaptiveClocking!F   1.0.0
>driverkit.serial   6.0.0
|IOSCSIMultimediaCommandsDevice   492
|IOBD!S!F   1.8
|IODVD!S!F   1.8
|IOCD!S!F   1.8
>usb.!UHub   1.2
>usb.IOUSBHostHIDDevice   1.2
>usb.cdc.ecm   5.0.0
>usb.cdc.ncm   5.0.0
>usb.cdc   5.0.0
>usb.networking   5.0.0
>usb.!UHostCompositeDevice   1.2
>!AThunderboltDPInAdapter   8.5.1
>!AThunderboltDPAdapter!F   8.5.1
>!AThunderboltPCIDownAdapter   4.1.1
>!AHPM   3.4.4
>!A!ILpssI2C!C   3.0.60
>!A!ILpssI2C   3.0.60
>!A!ILpssDmac   3.0.60
>!ABSDKextStarter   3
|IOSurface   352.0.3
@filesystems.hfs.encodings.kext   1
>!ASyntheticGame!C   11.1.9
>!AXsanScheme   3
>!AThunderboltNHI   7.2.81
|IOThunderbolt!F   9.3.3
>usb.!UVHCIBCE   1.2
>usb.!UVHCICommonBCE   1.0
>usb.!UVHCI   1.2
>usb.!UVHCICommon   1.0
>!AEffaceableNOR   1.0
|IOBufferCopy!C   1.1.0
|IOBufferCopyEngine!F   1
|IONVMe!F   2.1.0
>!AEthernetAquantiaAqtion   1.0.64
>!ABCMWLANCoreMac   1.0.0
|IO80211!F   1200.13.0
>IOImageLoader   1.0.0
>!AOLYHALMac   1
|IOSerial!F   11
>corecapture   1.0.4
>usb.!UHostPacketFilter   1.0
|IOUSB!F   900.4.2
>usb.!UXHCIPCI   1.2
>usb.!UXHCI   1.2
>!AEFINVRAM   2.1
>!AEFIRuntime   2.1
>!ASMCRTC   1.0
|IOSMBus!F   1.1
|IOHID!F   2.0.0
|IOTimeSync!F   1200.91
|IOSkywalk!F   1.0
>mDNSOffloadUserClient   1.0.1b8
|IONetworking!F   3.4
>DiskImages   493.0.0
|IO!B!F   9.0.0
|IOReport!F   47
$quarantine   4
$sandbox   300.0
@kext.!AMatch   1.0.0d1
>!ASSE   1.0
>!ALockdownMode   1
>!AKeyStore   2
>!UTDM   556
|IOUSBMass!SDriver   243
|IOSCSIBlockCommandsDevice   492
|IOSCSIArchitectureModel!F   492
>!AFDEKeyStore   28.30
>!AEffaceable!S   1.0
|IO!S!F   2.1
>!ACyrus   1
>!ACredentialManager   1.0
>KernelRelayHost   1
|IOUSBHost!F   1.2
>!UHostMergeProperties   1.2
>usb.!UCommon   1.0
>!AMobileFileIntegrity   1.0.5
$!AImage4   5.0.0
@kext.CoreTrust   1
|CoreAnalytics!F   1
>!ABusPower!C   1.0
>!ASEPManager   1.0.1
>IOSlaveProcessor   1
>!AACPIPlatform   6.1
>!ASMC   3.1.9
|IOPCI!F   2.9
|IOACPI!F   1.4
>watchdog   1
@kec.pthread   1
@kec.XrtHostedXnu   1
@kec.Libm   1
@kec.Compression   1.0
@kec.corecrypto   14.0


The pool configuration is fairly complex, having many snapshots and many filesystems, some of which are nested, but I have not yet tried enabling encryption on this pool.

[lundman]

It would have helped immensely if keepsyms=1 was on, since it is Intel you are crashing on?

Let me see if I can translate the addresses

[lundman]

First panic is

Code: Select all

0xffffff800f612951
kmem_slab_alloc_impl (in zfs) (spl-kmem.c:1303)
kmem_cache_alloc (in zfs) (spl-kmem.c:2252)
abd_alloc_chunks (in zfs) (abd_os.c:261)
abd_alloc (in zfs) (abd.c:195)
arc_hdr_alloc_abd (in zfs) (arc.c:3175)
arc_read (in zfs) (arc.c:5772)
dbuf_read (in zfs) (dbuf.c:1817)
dnode_hold_impl (in zfs) (dnode.c:1470)
dmu_bonus_hold (in zfs) (dmu.c:407)
space_map_open (in zfs) (space_map.c:831)
metaslab_init (in zfs) (metaslab.c:2728)
vdev_metaslab_init (in zfs) (vdev.c:1541)
vdev_load (in zfs) (vdev.c:3737)
vdev_load_child (in zfs) (vdev.c:1823)
taskq_thread (in zfs) (spl-taskq.c:2144)
spl_thread_setup (in zfs) (spl-thread.c:128)


[lundman]

Second panic is

Code: Select all

avl_first (in zfs) (avl.c:0)
kmem_cache_alloc (in zfs) (spl-kmem.c:2252)
abd_alloc_chunks (in zfs) (abd_os.c:261)
abd_alloc (in zfs) (abd.c:195)
arc_hdr_alloc_abd (in zfs) (arc.c:3175)
arc_read (in zfs) (arc.c:5772)
dbuf_issue_final_prefetch (in zfs) (dbuf.c:3373)
dbuf_prefetch_impl (in zfs) (dbuf.c:0)
dbuf_prefetch (in zfs) (dbuf.c:3624)


[lundman]

Ah this is why we do rc releases!

Oh I think I see, can you guys try this special build, if it is ok, we can do rc3.

OpenZFSonOsX-2.2.0rc2-Catalina-10.15.pkg

(16.55 MiB) Downloaded 293 times

[Jimbo]

Thanks, giving it a try now.

Hrm... installing that paniced my VM... will try again.

Code: Select all

panic(cpu 0 caller 0xffffff7fb30c2c5c): kernel heap corruption detected
Panicked task 0xffffff9a39e6f038: 5 threads: pid 152: kernelmanagerd
Backtrace (CPU 0), panicked thread: 0xffffff956e99c598, Frame : Return Address
0xfffffffe00a97300 : 0xffffff801c07207d mach_kernel : _handle_debugger_trap + 0x4ad
0xfffffffe00a97350 : 0xffffff801c1c6164 mach_kernel : _kdp_i386_trap + 0x114
0xfffffffe00a97390 : 0xffffff801c1b5c77 mach_kernel : _kernel_trap + 0x3b7
0xfffffffe00a973e0 : 0xffffff801c012951 mach_kernel : _return_from_trap + 0xc1
0xfffffffe00a97400 : 0xffffff801c07235d mach_kernel : _DebuggerTrapWithState + 0x5d
0xfffffffe00a974f0 : 0xffffff801c071a07 mach_kernel : _panic_trap_to_debugger + 0x1a7
0xfffffffe00a97550 : 0xffffff801c7db40b mach_kernel : _panic + 0x84
0xfffffffe00a97640 : 0xffffff7fb30c2c5c org.openzfsonosx.zfs : _kmem_error.cold.1 + 0x1c
0xfffffffe00a97650 : 0xffffff7fb2dfa611 org.openzfsonosx.zfs : _kmem_error + 0x581
0xfffffffe00a976c0 : 0xffffff7fb2dffa7b org.openzfsonosx.zfs : _kmem_magazine_destroy + 0x3b
0xfffffffe00a97700 : 0xffffff7fb2dff8d7 org.openzfsonosx.zfs : _kmem_depot_ws_reap + 0x157
0xfffffffe00a97750 : 0xffffff7fb2dfbb6c org.openzfsonosx.zfs : _kmem_cache_destroy + 0x6c
0xfffffffe00a97790 : 0xffffff7fb2f34f55 org.openzfsonosx.zfs : _abd_fini + 0x165
0xfffffffe00a977a0 : 0xffffff7fb2ec1dc6 org.openzfsonosx.zfs : _spa_fini + 0x36
0xfffffffe00a977e0 : 0xffffff7fb2f0cc6a org.openzfsonosx.zfs : _zfs_kmod_fini + 0x7a
0xfffffffe00a97810 : 0xffffff7fb2f4f4ed org.openzfsonosx.zfs : __ZN25org_openzfsonosx_zfs_zvol4stopEP9IOService + 0x3d
0xfffffffe00a97830 : 0xffffff801c6e3ede mach_kernel : __ZN9IOService10actionStopEPS_S0_PvS1_S1_ + 0xde
0xfffffffe00a97870 : 0xffffff801c71209e mach_kernel : __ZN10IOWorkLoop9runActionEPFiP8OSObjectPvS2_S2_S2_ES1_S2_S2_S2_S2_ + 0x3e
0xfffffffe00a978b0 : 0xffffff801c6e2f04 mach_kernel : __ZN9IOService15terminateWorkerEj + 0xa64
0xfffffffe00a97930 : 0xffffff801c6e2167 mach_kernel : __ZN9IOService23scheduleTerminatePhase2Ej + 0x147
0xfffffffe00a97990 : 0xffffff801c6e63c2 mach_kernel : __ZN9IOService15terminatePhase1Ej + 0xa92
0xfffffffe00a97a00 : 0xffffff801c70ad70 mach_kernel : __ZN11IOCatalogue16terminateDriversEP12OSDictionaryPcb + 0x260
0xfffffffe00a97a60 : 0xffffff801c70b4ac mach_kernel : __ZN11IOCatalogue25terminateDriversForModuleEP8OSStringbb + 0xec
0xfffffffe00a97ae0 : 0xffffff801c70b602 mach_kernel : __ZN11IOCatalogue25terminateDriversForModuleEPKcbb + 0x62
0xfffffffe00a97b20 : 0xffffff801c68131a mach_kernel : __ZN6OSKext10removeKextEPS_b + 0x23a
0xfffffffe00a97b60 : 0xffffff801c68ae28 mach_kernel : __ZN6OSKext13handleRequestEP4hostjPcjPS2_PjS3_S4_ + 0xbf8
0xfffffffe00a97be0 : 0xffffff801c69e9d6 mach_kernel : _kext_request + 0x1b6
0xfffffffe00a97c70 : 0xffffff801c0ec36e mach_kernel : _host_priv_server_routine + 0x16fe
0xfffffffe00a97cc0 : 0xffffff801c04c51a mach_kernel : _ipc_kmsg_send + 0x41a
0xfffffffe00a97d60 : 0xffffff801c0632a2 mach_kernel : _mach_msg_overwrite_trap + 0x592
0xfffffffe00a97dd0 : 0xffffff801c063927 mach_kernel : _mach_msg2_trap + 0x3c7
0xfffffffe00a97ef0 : 0xffffff801c19b533 mach_kernel : _mach_call_munger64 + 0x1f3
0xfffffffe00a97fa0 : 0xffffff801c012db6 mach_kernel : _hndl_mach_scall64 + 0x16
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.2)[CCB0DB09-A9A4-38CE-BAC6-860E585E7170]@0xffffff7fb2df6000->0xffffff7fb315cfff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[70B6B0B8-8059-3529-A544-D33C65534409]@0xffffff801ebe1000->0xffffff801ebf8fff

Process name corresponding to current thread (0xffffff956e99c598): kernelmanagerd
Boot args: keepsyms=1

Mac OS version:
22G91

Kernel version:
Darwin Kernel Version 22.6.0: Wed Jul  5 22:21:56 PDT 2023; root:xnu-8796.141.3~6/RELEASE_X86_64
Kernel UUID: 8295C77B-2AAB-3471-81B5-438F59B1A36D
roots installed: 0
KernelCache slide: 0x000000001bc00000
KernelCache base:  0xffffff801be00000
Kernel slide:      0x000000001bcdc000
Kernel text base:  0xffffff801bedc000
__HIB  text base: 0xffffff801bd00000
System model name: Parallels19,1 (Parallels Virtual Platform)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 1914795688415
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000001bdd2b78444
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000008476f4970 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff80a0794000 - 0xffffffa0a0794000
  . PGZ   : 0xffffff80a0794000 - 0xffffff80a2795000
  . VM    : 0xffffff80a2795000 - 0xffffff856ef94000
  . RO    : 0xffffff856ef94000 - 0xffffff8708794000
  . GEN0  : 0xffffff8708794000 - 0xffffff8bd4f94000
  . GEN1  : 0xffffff8bd4f94000 - 0xffffff90a1794000
  . GEN2  : 0xffffff90a1794000 - 0xffffff956df94000
  . GEN3  : 0xffffff956df94000 - 0xffffff9a3a794000
  . DATA  : 0xffffff9a3a794000 - 0xffffffa0a0794000
  Metadata: 0xffffffb0a114d000 - 0xffffffb0c114d000
  Bitmaps : 0xffffffb0c114d000 - 0xffffffb0c414d000
  Extra   : 0 - 0

last started kext at 1910828879586: com.highpoint-tech.kext.HighPointIOP   4.4.5 (addr 0xffffff7fb2d11000, size 45056)
loaded kexts:
com.highpoint-tech.kext.HighPointIOP   4.4.5
com.highpoint-tech.kext.HighPointRR   4.22.1
com.parallels.kext.video   19.1.0 54729
com.softraid.driver.SoftRAID   6.3.1
org.openzfsonosx.zfs   2.2.0
@filesystems.smbfs   5.0
>AudioAUUC   1.70
>!APlatformEnabler   2.7.0d0
>AGPM   131
>X86PlatformShim   1.0.0
@filesystems.autofs   3.0
>!AUpstreamUserClient   3.6.9
>!AMCCSControl   1.16
>!AHDAHardwareConfigDriver   440.2
>!AHDA   440.2
>!AGraphicsDevicePolicy   7.1.18
@AGDCPluginDisplayMetrics   7.1.18
>pmtelemetry   1
@filesystems.nfs   1
|IOUserEthernet   1.0.1
>usb.!UUserHCI   1
>!AHV   1
>!ADiskImages2   198.100.13
>!AFIVRDriver   4.1.0
>ACPI_SMC_PlatformPlugin   1.0.0
>!A!ISlowAdaptiveClocking   4.0.0
>!AParavirtGPU   13.4.13
>!UMass!SInterfaceNub   554
>!AFileSystemDriver   3.0.1
@filesystems.tmpfs   1
|SCSITaskUserClient   482.120.2
@filesystems.lifs   1
@filesystems.hfs.kext   627.100.6
@BootCache   40
@!AFSCompression.!AFSCompressionTypeZlib   1.0.0
@!AFSCompression.!AFSCompressionTypeDataless   1.0.0d1
@filesystems.apfs   2142.140.9
>!AAHCIPort   378
@private.KextAudit   1.0
>!APVPanic   1
>!AACPIButtons   6.1
>!AHPET   1.8
>!ARTC   2.0.1
>!ASMBIOS   2.1
>!AAPIC   1.7
$!AUserConsent   1
@!ASystemPolicy   2.0.0
@nke.applicationfirewall   404
|IOKitRegistryCompatibility   1
|EndpointSecurity   1
@Dont_Steal_Mac_OS_X   7.0.0
@kec.Compression   1
@kec.!AEncryptedArchive   1
|IOSCSIParallel!F   3.0.0
>!AHIDKeyboard   231
>!ASMBus!C   1.0.18d1
|IOSMBus!F   1.1
>!UAudio   560.2
>!AAudioClockLibs   240.1
>DspFuncLib   440.2
@kext.OSvKernDSPLib   530
>!AGraphicsControl   7.1.18
>!AHDA!C   440.2
|IOHDA!F   440.2
|IOAudio!F   440.2
@vecLib.kext   1.2.0
@kext.triggers   1.0
>IOHIDPowerSource   1
>!ACallbackPowerSource   1
|IO!BSerialManager   9.0.0
|IO!BPacketLogger   9.0.0
>!ASMBusPCI   1.0.14d1
|IO!BHost!CUSBTransport   9.0.0
|IO!BHost!CUARTTransport   9.0.0
|IO!BHost!CTransport   9.0.0
>IO!BHost!CPCIeTransport   9.0.0
|IOAVB!F   1140.6
|CSR!BHost!CUSBTransport   9.0.0
|Broadcom!BHost!CUSBTransport   9.0.0
|Broadcom!B20703USBTransport   9.0.0
>!ARSMChannel   1
|IORSM!F   1
>!AIPAppender   1.0
|IONDRVSupport   597
@plugin.IOgPTPPlugin   1150.2
@!AGPUWrangler   7.1.18
>IOPlatformPluginLegacy   1.0.0
>X86PlatformPlugin   1.0.0
>IOPlatformPlugin!F   6.0.0d8
|IOSlowAdaptiveClocking!F   1.0.0
>driverkit.serial   6.0.0
|IOAccelerator!F2   475.40.6
@!AGraphicsDeviceControl   7.1.18
|IOGraphics!F   597
>usb.IOUSBHostHIDDevice   1.2
>usb.cdc   5.0.0
>usb.networking   5.0.0
>usb.!UHostCompositeDevice   1.2
|IOSCSIMultimediaCommandsDevice   482.120.2
|IOBD!S!F   1.8
|IODVD!S!F   1.8
|IOCD!S!F   1.8
>!ABSDKextStarter   3
|IOSurface   336.60.1
@filesystems.hfs.encodings.kext   1
>!ASyntheticGame!C   10.7.1
>!AXsanScheme   3
|IOAHCISerialATAPI   272
|IOAHCIBlock!S   350.140.3
|IOAHCI!F   301.100.1
>usb.!UEHCIPCI   1.2
>usb.!UEHCI   1.2
>usb.!UXHCIPCI   1.2
>usb.!UXHCI   1.2
>!AVirtIO   188.140.3
|IOSerial!F   11
>usb.!UHostPacketFilter   1.0
|IOUSB!F   900.4.2
|IOHID!F   2.0.0
>!AEFINVRAM   2.1
>!AEFIRuntime   2.1
|IOTimeSync!F   1150.2
|IOSkywalk!F   1.0
>mDNSOffloadUserClient   1.0.1b8
|IONetworking!F   3.4
>DiskImages   493.0.0
|IO!B!F   9.0.0
|IOReport!F   47
$quarantine   4
$sandbox   300.0
@kext.!AMatch   1.0.0d1
>!ASSE   1.0
>!AKeyStore   2
>!UTDM   554
|IOUSBMass!SDriver   235.140.2
|IOSCSIBlockCommandsDevice   482.120.2
|IO!S!F   2.1
|IOSCSIArchitectureModel!F   482.120.2
>!AFDEKeyStore   28.30
>!AEffaceable!S   1.0
>!ACyrus   1
>!AMobileFileIntegrity   1.0.5
$!AImage4   5.0.0
@kext.CoreTrust   1
>!ACredentialManager   1.0
|CoreAnalytics!F   1
>KernelRelayHost   1
|IOUSBHost!F   1.2
>!UHostMergeProperties   1.2
>usb.!UCommon   1.0
>!ABusPower!C   1.0
>!ASEPManager   1.0.1
>IOSlaveProcessor   1
>!AACPIPlatform   6.1
>!ASMC   3.


[Jimbo]

Nope... will boot into recovery and remove manually...