2.2.3 KP on Sequoia 15.3.1

[BjarneDM]

Code: Select all

panic(cpu 1 caller 0xffffff800654601e): Kernel trap at 0xffffff7f9fa44f2b, type 6=invalid opcode, registers:
CR0: 0x0000000080010033, CR2: 0x00006000038fc000, CR3: 0x000000000aded000, CR4: 0x00000000000226e0
RAX: 0xffffff7f9fb001b0, RBX: 0xffffffad4f9cbb08, RCX: 0xffffff7f9faf5100, RDX: 0x0000000000000010
RSP: 0xffffffad4f9cb9e8, RBP: 0xffffffad4f9cba30, RSI: 0xffffffbbf25fc000, RDI: 0xffffffad4f9cbb08
R8:  0xffffffad4f9cbb08, R9:  0x0000000000000000, R10: 0xffffffbbeff0b020, R11: 0x0000000000000000
R12: 0xffffffbbf25fc000, R13: 0x0000000000000000, R14: 0x0000000000000400, R15: 0x0000000000000010
RFL: 0x0000000000010202, RIP: 0xffffff7f9fa44f2b, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0x1, PL: 0, VF: 0

Panicked task 0xffffff9bb7794d60: 345 threads: pid 0: kernel_task
Backtrace (CPU 1), panicked thread: 0xffffffa07fc09b40, Frame : Return Address
0xffffffad4f9cb2a0 : 0xffffff80063e8351 mach_kernel : _handle_debugger_trap + 0x4c1
0xffffffad4f9cb2f0 : 0xffffff800655622c mach_kernel : _kdp_i386_trap + 0x11c
0xffffffad4f9cb330 : 0xffffff8006545af3 mach_kernel : _kernel_trap + 0x763
0xffffffad4f9cb400 : 0xffffff800637d971 mach_kernel : _return_from_trap + 0xc1
0xffffffad4f9cb420 : 0xffffff80063e8647 mach_kernel : _DebuggerTrapWithState + 0x67
0xffffffad4f9cb520 : 0xffffff80063e7ce2 mach_kernel : _panic_trap_to_debugger + 0x1e2
0xffffffad4f9cb590 : 0xffffff8006bd2038 mach_kernel : _panic + 0x81
0xffffffad4f9cb680 : 0xffffff800654601e mach_kernel : _sync_iss_to_iks + 0x2ae
0xffffffad4f9cb800 : 0xffffff8006545d17 mach_kernel : _kernel_trap + 0x987
0xffffffad4f9cb8d0 : 0xffffff800637d971 mach_kernel : _return_from_trap + 0xc1
0xffffffad4f9cb8f0 : 0xffffff7f9fa44f2b org.openzfsonosx.zfs : .Loop_shani + 0x2b
0xffffffad4f9cba30 : 0xffffff7f9f844845 org.openzfsonosx.zfs : _sha_incremental + 0x15
0xffffffad4f9cba40 : 0xffffff7f9f7abb36 org.openzfsonosx.zfs : _abd_iterate_func + 0x156
0xffffffad4f9cbad0 : 0xffffff7f9f8447ba org.openzfsonosx.zfs : _abd_checksum_sha256 + 0x8a
0xffffffad4f9cbc10 : 0xffffff7f9f7c6291 org.openzfsonosx.zfs : _chksum_benchit + 0xb1
0xffffffad4f9cbc80 : 0xffffff7f9f7c5bf7 org.openzfsonosx.zfs : _chksum_init + 0x1c7
0xffffffad4f9cbcf0 : 0xffffff7f9f8642ec org.openzfsonosx.zfs : _spa_init + 0xfc
0xffffffad4f9cbd10 : 0xffffff7f9f8b4c80 org.openzfsonosx.zfs : _zfs_kmod_init + 0x20
0xffffffad4f9cbd30 : 0xffffff7f9f8fc2fe org.openzfsonosx.zfs : __ZN25org_openzfsonosx_zfs_zvol5startEP9IOService + 0x17e
0xffffffad4f9cbd60 : 0xffffff8006aca927 mach_kernel : __ZN9IOService14startCandidateEPS_ + 0xf7
0xffffffad4f9cbdd0 : 0xffffff8006aca3b4 mach_kernel : __ZN9IOService15probeCandidatesEP12OSOrderedSet + 0xf04
0xffffffad4f9cbec0 : 0xffffff8006ac932b mach_kernel : __ZN9IOService14doServiceMatchEj + 0x36b
0xffffffad4f9cbf20 : 0xffffff8006acc8c9 mach_kernel : __ZN15_IOConfigThread4mainEPvi + 0x159
0xffffffad4f9cbfa0 : 0xffffff800637d19e mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.2.3)[D8CF36D3-B8EE-3796-BEE1-C37EFE2827E8]@0xffffff7f9f78f000->0xffffff7f9faf5fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[4B0CF2F6-B2D4-37B7-BD27-A96CBA970155]@0xffffff80089c6000->0xffffff80089dcfff

Process name corresponding to current thread (0xffffffa07fc09b40): kernel_task
Boot args: keepsyms=1 debug=0x100 -lilubetaall -btlfxallowanyaddr ipc_control_port_options=0 -nokcmismatchpanic

Mac OS version:
24D70

Kernel version:
Darwin Kernel Version 24.3.0: Thu Jan  2 20:22:00 PST 2025; root:xnu-11215.81.4~3/RELEASE_X86_64
Kernel UUID: 0FA106B0-CE42-3EF5-98D6-610845FB1DE7
roots installed: 0
KernelCache slide: 0x0000000006000000
KernelCache base:  0xffffff8006200000
Kernel slide:      0x00000000060e4000
Kernel text base:  0xffffff80062e4000
__HIB  text base: 0xffffff8006100000
System model name: MacPro5,1 (Mac-27AD2F918AE68F61)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 24399238368
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x00000005ae4ee0fe
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000022af2e08f4 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff86e2447000 - 0xffffffa6e2447000
  . PGZ   : 0xffffff86e2447000 - 0xffffff86fa448000
  . VM    : 0xffffff86fa448000 - 0xffffff8bc377b000
  . RO    : 0xffffff8bc377b000 - 0xffffff8d5bde1000
  . GEN0  : 0xffffff8d5bde1000 - 0xffffff9225114000
  . GEN1  : 0xffffff9225114000 - 0xffffff96ee447000
  . GEN2  : 0xffffff96ee447000 - 0xffffff9bb777a000
  . GEN3  : 0xffffff9bb777a000 - 0xffffffa080aad000
  . DATA  : 0xffffffa080aad000 - 0xffffffa6e2447000
  Metadata: 0xffffffa7673f0000 - 0xffffffa7873f0000
  Bitmaps : 0xffffffa7873f0000 - 0xffffffa7973f0000
  Extra   : 0 - 0


[Sharko]

I have been getting pretty consistent boot failures/kernel panics on my Mac Pro 5,1 also. My 5,1 is also running Open Core (but it is the manual version of OpenCore, not the OCLP version). That machine is only running MacOS 12.7.x (latest, I forget the exact version). I'll post complete details this evening, if I have time.

What caught my eye in your panic report, however, was this line:

Code: Select all

0xffffffad4f9cb8f0 : 0xffffff7f9fa44f2b org.openzfsonosx.zfs : .Loop_shani + 0x2b

The reason it caught my eye is that I know that my panic report had that same symbol, Loop_shani.

I see from your signature that your Mac Pro 5,1 is a dual processor machine (like mine); I wonder if that is significant?

[Sharko]

Here is the kernel panic I keep seeing on boot: Mac Pro 5,1 with manual install of OpenCore (emulating a 2019 Mac Pro), on Monterey 12.7.6, zfs version 2.2.3 (Last Modified: 1/9/25, 9:10 PM). Note the .Loop_shani + 0x2b line

Code: Select all

panic(cpu 18 caller 0xffffff80069cc483): Kernel trap at 0xffffff7fa0ca5bab, type 6=invalid opcode, registers:
CR0: 0x0000000080010033, CR2: 0x000000010feda000, CR3: 0x000000000aef1000, CR4: 0x00000000000226e0
RAX: 0xffffff7fa0d5e230, RBX: 0x0000000000000400, RCX: 0xffffff7fa0d52f00, RDX: 0x0000000000000010
RSP: 0xffffffd54abb3a48, RBP: 0xffffffd54abb3a90, RSI: 0xfffffff378b66000, RDI: 0xffffffd54abb3b68
R8:  0xffffffd54abb3b68, R9:  0x00000000dc00054c, R10: 0x00000000cc60bd88, R11: 0x00000000a3306dfe
R12: 0x0000000000000000, R13: 0x0000000000000400, R14: 0xffffffd54abb3b68, R15: 0xfffffff378b66000
RFL: 0x0000000000010206, RIP: 0xffffff7fa0ca5bab, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0x12, PL: 0, VF: 0

Panicked task 0xffffff9a22a74670: 310 threads: pid 0: kernel_task
Backtrace (CPU 18), panicked thread: 0xffffff908a298aa0, Frame : Return Address
0xffffffd54abb33f0 : 0xffffff8006879a3d mach_kernel : _handle_debugger_trap + 0x41d
0xffffffd54abb3440 : 0xffffff80069dcd26 mach_kernel : _kdp_i386_trap + 0x116
0xffffffd54abb3480 : 0xffffff80069cc093 mach_kernel : _kernel_trap + 0x4d3
0xffffffd54abb34d0 : 0xffffff8006819a90 mach_kernel : _return_from_trap + 0xe0
0xffffffd54abb34f0 : 0xffffff8006879e0d mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffd54abb3610 : 0xffffff80068795c6 mach_kernel : _panic_trap_to_debugger + 0x2b6
0xffffffd54abb3670 : 0xffffff8007114a73 mach_kernel : _panic + 0x84
0xffffffd54abb3760 : 0xffffff80069cc483 mach_kernel : _sync_iss_to_iks + 0x2c3
0xffffffd54abb38e0 : 0xffffff80069cc166 mach_kernel : _kernel_trap + 0x5a6
0xffffffd54abb3930 : 0xffffff8006819a90 mach_kernel : _return_from_trap + 0xe0
0xffffffd54abb3950 : 0xffffff7fa0ca5bab org.openzfsonosx.zfs : .Loop_shani + 0x2b
0xffffffd54abb3a90 : 0xffffff7fa0a7c6a5 org.openzfsonosx.zfs : panic(cpu 18 caller 0xffffff80069cc483): Kernel trap at 0xffffff80069cc338, type 1253783440=panic(cpu 18 caller 0xffffff80069cc483): Kernel trap at 0x00000000000226e0, type 0=, registers:
CR0: 0x0000000080010033, CR2: 0x000000010feda000, CR3: 0x0000000000000000, CR4: 0xffffff80071d9608
RAX: 0x0000000000000006, RBX: 0x0000000000000000, RCX: 0xffffffd54abb3970, RDX: 0x0000000000000005
RSP: 0x0000000000000000, RBP: 0xffffffd54abb3930, RSI: 0xffffff80069cc166, RDI: 0x00000000d3a92a63
R8:  0xffffff7fa0ca5bab, R9:  0x000000001004889c, R10: 0x0000000000000006, R11: 0xffffffd54abb3950
R12: 0x0000000000000000, R13: 0x0000000000000000, R14: 0xffffffd54abb3960, R15: 0xffffffd54abb3950
RFL: 0xffffff8006819a90, RIP: 0xffffffd54abb3950, CS:  0x0000000000000000, SS:  0xffffffd54abb3a90
Fault CR2: 0xffffff7fa0ca5bab, Error code: 0x000000000000000f, Fault CPU: 0x202gf   j.g;rsn<:uO%RQh+YM`[~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J]:~J

Is there any other debugging info that would be helpful?

[jawbroken]

did you both deliberately override the default and set the checksum to sha256 or sha512? i'm presuming that the assembly code for these algorithms calls some instruction that it thinks should be valid but doesn't exist on your processor, perhaps because you're pretending to be a 2019 Mac Pro

[Sharko]

Thanks, @jawbroken, I think you're onto something. I didn't deliberately set up the pool to use SHA-256 anywhere; I use the standard invocation from the wiki when creating my pools:

Code: Select all

sudo zpool create -f -o ashift=12 -O compression=lz4 -O casesensitivity=insensitive -O atime=off -O normalization=formD tank mirror disk3 disk4

However, I asked CoPilot where OpenZFS might use SHA-256, and it came back with this:

OpenZFS uses the SHA-256 algorithm primarily for data integrity and security purposes. Here are some specific uses:

1. **Checksum Verification**: OpenZFS uses SHA-256 to generate checksums for data blocks. These checksums are used to verify the integrity of data, ensuring that it has not been corrupted or tampered with. When data is read, the checksum is recalculated and compared to the stored checksum to detect any discrepancies¹(https://arstechnica.com/gadgets/2021/06 ... ncryption/).

2. **Deduplication**: SHA-256 is used in the deduplication process to identify and eliminate duplicate data blocks. By generating a unique hash for each block, OpenZFS can efficiently determine if a block has already been stored, reducing storage requirements¹(https://arstechnica.com/gadgets/2021/06 ... ncryption/).

3. **Encryption**: While OpenZFS primarily uses AES for data encryption, SHA-256 is used in the key derivation process. It helps to securely generate encryption keys from passphrases or other input data²(https://blog.elcomsoft.com/2021/11/prot ... -compared/).

These uses of SHA-256 help ensure the reliability, efficiency, and security of data stored in OpenZFS. If you have any more questions or need further information, feel free to ask!

¹(https://arstechnica.com/gadgets/2021/06 ... ncryption/): [Ars Technica](https://arstechnica.com/gadgets/2021/06 ... ncryption/)
²(https://blog.elcomsoft.com/2021/11/prot ... -compared/): [ElcomSoft Blog](https://blog.elcomsoft.com/2021/11/prot ... -compared/)

Source: Conversation with Copilot, 3/3/2025
(1) A quick-start guide to OpenZFS native encryption - Ars Technica. https://arstechnica.com/gadgets/2021/06 ... ncryption/.
(2) Protecting Linux and NAS Devices: LUKS, eCryptFS and Native ZFS .... https://blog.elcomsoft.com/2021/11/prot ... -compared/.
(3) A quick-start guide to OpenZFS native encryption. https://arstechnica.com/civis/threads/a ... n.1477586/.

My kernel panics tend to be intermittent, and only during start-up (once I get Monterey up and running I don't think I've seen this kernel panic). It's a little weird, because I booted the machine yesterday and got the kernel panic as usual, but somehow it seems to get through the boot after a couple of tries. The other weird thing is that I don't think I even have a pool in that machine at present.

[jawbroken]

don't really know what i'm talking about, but SHA-NI probably refers to https://en.wikipedia.org/wiki/SHA_instruction_set https://github.com/armfazh/flo-shani-aesni etc which i don't think your hardware supports. i know there's code in OpenZFS to test which implementation of the various checksum, etc. algorithms is fastest and use that, so perhaps that is happening at startup. e.g. it's trying to determine which SHA256 implementation is fastest on your hardware, and kernel panicking on the invalid op code. i think there's some way to override this and pick a specific implementation instead of "fastest", so that might help if you can find it (though i don't know if it will still run the test to see which is fastest if you have something else selected).

[Sharko]

I think that you have put your finger precisely on the problem... when I look at your original link to the .shani_lp code I see a bunch of references to xmm0, xmm1, etc registers. According to this link:

https://en.wikipedia.org/wiki/Advanced_ ... Extensions

those xmm registers are all associated with AVX instructions. I'll have to do some digging to see if there is a kstat tunable that can control whether OpenZFS tries to use AVX.