2.2.2 Panics on 14.2.1

[nathansvt]

Mac mini 2.3 GHz Quad-Core Intel Core i7, 16GB RAM

Consistently panics during long writes. 3x memtest reports OK.

Code: Select all

panic(cpu 2 caller 0xffffff7fa1a8e500): VERIFY3(0 == zap_add(os, parent_obj, name, sizeof (uint64_t), 1, &new_obj, tx)) failed (0 == 22)

Panicked task 0xffffff901f673df8: 5 threads: pid 44927: DesktopServicesH
Backtrace (CPU 2), panicked thread: 0xffffff8b57de70c8, Frame : Return Address
0xfffffff40e9d7110 : 0xffffff800ba36c41 mach_kernel : _handle_debugger_trap + 0x4b1
0xfffffff40e9d7160 : 0xffffff800bb95310 mach_kernel : _kdp_i386_trap + 0x110
0xfffffff40e9d71a0 : 0xffffff800bb84a5c mach_kernel : _kernel_trap + 0x55c
0xfffffff40e9d7220 : 0xffffff800b9d3971 mach_kernel : _return_from_trap + 0xc1
0xfffffff40e9d7240 : 0xffffff800ba36f2d mach_kernel : _DebuggerTrapWithState + 0x5d
0xfffffff40e9d7330 : 0xffffff800ba365d3 mach_kernel : _panic_trap_to_debugger + 0x1e3
0xfffffff40e9d7390 : 0xffffff800c1d8bcb mach_kernel : _panic + 0x84
0xfffffff40e9d7480 : 0xffffff7fa1a8e500 org.openzfsonosx.zfs : _cmn_err
0xfffffff40e9d75f0 : 0xffffff7fa1b9f2ac org.openzfsonosx.zfs : _zap_create_link + 0x7c
0xfffffff40e9d7630 : 0xffffff7fa1b41c5e org.openzfsonosx.zfs : _sa_add_layout_entry + 0xce
0xfffffff40e9d76b0 : 0xffffff7fa1b4576a org.openzfsonosx.zfs : _sa_find_layout + 0x1aa
0xfffffff40e9d77a0 : 0xffffff7fa1b4491d org.openzfsonosx.zfs : _sa_build_layouts + 0x8cd
0xfffffff40e9d78a0 : 0xffffff7fa1b45373 org.openzfsonosx.zfs : _sa_modify_attrs + 0x3f3
0xfffffff40e9d7970 : 0xffffff7fa1b4303f org.openzfsonosx.zfs : _sa_attr_op + 0x4bf
0xfffffff40e9d7a10 : 0xffffff7fa1b44d5d org.openzfsonosx.zfs : _sa_bulk_update_impl + 0x6d
0xfffffff40e9d7a60 : 0xffffff7fa1b44efa org.openzfsonosx.zfs : _sa_bulk_update + 0x3a
0xfffffff40e9d7aa0 : 0xffffff7fa1bc6f1a org.openzfsonosx.zfs : _zfs_sa_set_xattr + 0x29a
0xfffffff40e9d7b80 : 0xffffff7fa1c1640e org.openzfsonosx.zfs : _zpl_xattr_set_sa + 0x1ae
0xfffffff40e9d7c00 : 0xffffff7fa1c15f74 org.openzfsonosx.zfs : _zpl_xattr_set + 0xc4
0xfffffff40e9d7c60 : 0xffffff7fa1c0fa2d org.openzfsonosx.zfs : _zfs_vnop_setxattr + 0x27d
0xfffffff40e9d7d80 : 0xffffff800bc4adf9 mach_kernel : _VNOP_SETXATTR + 0x49
0xfffffff40e9d7dd0 : 0xffffff800bc402ac mach_kernel : _vn_setxattr + 0x10c
0xfffffff40e9d7e20 : 0xffffff800bc343b1 mach_kernel : _fsetxattr + 0x331
0xfffffff40e9d7f40 : 0xffffff800c052c9d mach_kernel : _unix_syscall64 + 0x1dd
0xfffffff40e9d7fa0 : 0xffffff800b9d3db6 mach_kernel : _hndl_unix_scall64 + 0x16
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.2.2)[44A09F80-8ABD-3CA5-B2AC-E553033E9179]@0xffffff7fa1a8d000->0xffffff7fa1e19fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[A96D850A-038F-31D7-9F3D-A6775064548E]@0xffffff800de8c000->0xffffff800dea3fff

Process name corresponding to current thread (0xffffff8b57de70c8): DesktopServicesHelper
Boot args: keepsyms=1 debug=0x100 ipc_control_port_options=0 -nokcmismatchpanic amfi=0x80

Mac OS version:
23C71

Kernel version:
Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:10 PST 2023; root:xnu-10002.61.3~2/RELEASE_X86_64
Kernel UUID: F73CFE5A-7EF6-3C21-AFA7-742645B733CC
roots installed: 0
KernelCache slide: 0x000000000b600000
KernelCache base:  0xffffff800b800000
Kernel slide:      0x000000000b6e0000
Kernel text base:  0xffffff800b8e0000
__HIB  text base: 0xffffff800b700000
System model name: Macmini6,2 (Mac-F65AE981FFA204ED)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 143493885056854
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x00008281c60bdeb9
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000008f171f541 0x0000000000000000
Compressor Info: 5% of compressed pages limit (OK) and 10% of segments limit (OK) with 1 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff8024540000 - 0xffffffa024540000
  . PGZ   : 0xffffff8024540000 - 0xffffff8026541000
  . VM    : 0xffffff8026541000 - 0xffffff84f2d40000
  . RO    : 0xffffff84f2d40000 - 0xffffff868c540000
  . GEN0  : 0xffffff868c540000 - 0xffffff8b58d40000
  . GEN1  : 0xffffff8b58d40000 - 0xffffff9025540000
  . GEN2  : 0xffffff9025540000 - 0xffffff94f1d40000
  . GEN3  : 0xffffff94f1d40000 - 0xffffff99be540000
  . DATA  : 0xffffff99be540000 - 0xffffffa024540000
  Metadata: 0xffffffb064790000 - 0xffffffb084790000
  Bitmaps : 0xffffffb084790000 - 0xffffffb087790000
  Extra   : 0 - 0





[nathansvt]

Code: Select all

panic(cpu 6 caller 0xffffff7fa328e500): VERIFY3(0 == zap_add(os, parent_obj, name, sizeof (uint64_t), 1, &new_obj, tx)) failed (0 == 22)

Panicked task 0xffffff8b55c35218: 11 threads: pid 492: Finder
Backtrace (CPU 6), panicked thread: 0xffffff9023d070c8, Frame : Return Address
0xffffffcfe882f0c0 : 0xffffff800d236c41 mach_kernel : _handle_debugger_trap + 0x4b1
0xffffffcfe882f110 : 0xffffff800d395310 mach_kernel : _kdp_i386_trap + 0x110
0xffffffcfe882f150 : 0xffffff800d384a5c mach_kernel : _kernel_trap + 0x55c
0xffffffcfe882f1d0 : 0xffffff800d1d3971 mach_kernel : _return_from_trap + 0xc1
0xffffffcfe882f1f0 : 0xffffff800d236f2d mach_kernel : _DebuggerTrapWithState + 0x5d
0xffffffcfe882f2e0 : 0xffffff800d2365d3 mach_kernel : _panic_trap_to_debugger + 0x1e3
0xffffffcfe882f340 : 0xffffff800d9d8bcb mach_kernel : _panic + 0x84
0xffffffcfe882f430 : 0xffffff7fa328e500 org.openzfsonosx.zfs : _cmn_err
0xffffffcfe882f5a0 : 0xffffff7fa339f2ac org.openzfsonosx.zfs : _zap_create_link + 0x7c
0xffffffcfe882f5e0 : 0xffffff7fa3341c5e org.openzfsonosx.zfs : _sa_add_layout_entry + 0xce
0xffffffcfe882f660 : 0xffffff7fa334576a org.openzfsonosx.zfs : _sa_find_layout + 0x1aa
0xffffffcfe882f750 : 0xffffff7fa334491d org.openzfsonosx.zfs : _sa_build_layouts + 0x8cd
0xffffffcfe882f850 : 0xffffff7fa3345373 org.openzfsonosx.zfs : _sa_modify_attrs + 0x3f3
0xffffffcfe882f920 : 0xffffff7fa334303f org.openzfsonosx.zfs : _sa_attr_op + 0x4bf
0xffffffcfe882f9c0 : 0xffffff7fa3344d5d org.openzfsonosx.zfs : _sa_bulk_update_impl + 0x6d
0xffffffcfe882fa10 : 0xffffff7fa3344efa org.openzfsonosx.zfs : _sa_bulk_update + 0x3a
0xffffffcfe882fa50 : 0xffffff7fa33c6f1a org.openzfsonosx.zfs : _zfs_sa_set_xattr + 0x29a
0xffffffcfe882fb30 : 0xffffff7fa341640e org.openzfsonosx.zfs : _zpl_xattr_set_sa + 0x1ae
0xffffffcfe882fbb0 : 0xffffff7fa3415f74 org.openzfsonosx.zfs : _zpl_xattr_set + 0xc4
0xffffffcfe882fc10 : 0xffffff7fa340fc5d org.openzfsonosx.zfs : _zfs_vnop_removexattr + 0x8d
0xffffffcfe882fc50 : 0xffffff800d442106 mach_kernel : _vn_removexattr + 0xe6
0xffffffcfe882fcc0 : 0xffffff800d434693 mach_kernel : _removexattr + 0x293
0xffffffcfe882ff40 : 0xffffff800d852c9d mach_kernel : _unix_syscall64 + 0x1dd
0xffffffcfe882ffa0 : 0xffffff800d1d3db6 mach_kernel : _hndl_unix_scall64 + 0x16
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.2.2)[44A09F80-8ABD-3CA5-B2AC-E553033E9179]@0xffffff7fa328d000->0xffffff7fa3619fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[A96D850A-038F-31D7-9F3D-A6775064548E]@0xffffff800f68c000->0xffffff800f6a3fff

Process name corresponding to current thread (0xffffff9023d070c8): Finder
Boot args: keepsyms=1 debug=0x100 ipc_control_port_options=0 -nokcmismatchpanic amfi=0x80

Mac OS version:
23C71

Kernel version:
Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:10 PST 2023; root:xnu-10002.61.3~2/RELEASE_X86_64
Kernel UUID: F73CFE5A-7EF6-3C21-AFA7-742645B733CC
roots installed: 0
KernelCache slide: 0x000000000ce00000
KernelCache base:  0xffffff800d000000
Kernel slide:      0x000000000cee0000
Kernel text base:  0xffffff800d0e0000
__HIB  text base: 0xffffff800cf00000
System model name: Macmini6,2 (Mac-F65AE981FFA204ED)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 509133883025
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000000768abe8e0b
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x0000000ebc35ded1 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff8021413000 - 0xffffffa021413000
  . PGZ   : 0xffffff8021413000 - 0xffffff8023414000
  . VM    : 0xffffff8023414000 - 0xffffff84efc13000
  . RO    : 0xffffff84efc13000 - 0xffffff8689413000
  . GEN0  : 0xffffff8689413000 - 0xffffff8b55c13000
  . GEN1  : 0xffffff8b55c13000 - 0xffffff9022413000
  . GEN2  : 0xffffff9022413000 - 0xffffff94eec13000
  . GEN3  : 0xffffff94eec13000 - 0xffffff99bb413000
  . DATA  : 0xffffff99bb413000 - 0xffffffa021413000
  Metadata: 0xffffffa04b099000 - 0xffffffa06b099000
  Bitmaps : 0xffffffa06b099000 - 0xffffffa06e099000
  Extra   : 0 - 0




Similar panic, but on creation of a Finder alias (short write).

[Sharko]

What sort of history on the pool? Are you using encryption? Just to get some background.

[lundman]

Sounds like something is up with SA in your zap. I do believe this is a soft panic, so you should be able to enable "zfs_recover=1" and carry on.

[nathansvt]

Fixed by moving impacted files to a new zvol. xattr=sa was likely the issue, in conjunction with one of OSX's background media daemon processes.