2.2.0rc3 feedback

[o3x_prokyon]

Adding keepsyms=1 to /Library/Preferences/SystemConfiguration/com.apple.Boot.plist did't change anything for the panic report. It looks very familiar to the ones before, without that...

You're right, Mr. Lundman, ZFS seems not to be the primary reason for panic, but the kext is loaded, so I think it must have an impact on the crash. I can easily mount and unmount (or eject) any internal or external disk, ramdisks, VeraCrypt containers, network volumes and so on, without any panic, as long as no ZFS pool is imported. Exporting the ZFS pool prior to ejecting any other disk, is also no problem. I'm a little clueless about this phenomenon – nobody else seems to have this issue...

The other thing is: why does Carbon Copy Cloner (rsync) always copies every data from an internal volume to the external ZFS pool? I just tested this with a smaller directory (~10 GB) and even if I start one backup directly after the previous, all files are "updated" over and over again on the ZFS destination. I have no idea, why! The sizes are the same, the modification times are the same (even nanoseconds) and even checksums (sha256) are the same. What else can cause this behaviour? By the way – access-times are not written, neither on the apfs disks (mount-option: noatime), nor on the ZFS pool (atime=off). Even if these times would be written, only metadata should be updated, not the whole file. I think this is a problem of CCC or rsync, but some time ago, with an older ZFS version (and an older rsync?) incremental updates worked fine...

[lundman]

You're right, Mr. Lundman, ZFS seems not to be the primary reason for panic, but the kext is loaded, so I think it must have an impact on the crash. I can easily mount and unmount (or eject) any internal or external disk, ramdisks, VeraCrypt containers, network volumes and so on, without any panic, as long as no ZFS pool is imported. Exporting the ZFS pool prior to ejecting any other disk, is also no problem. I'm a little clueless about this phenomenon – nobody else seems to have this issue...

If you do happen to get a panic with ZFS listed in it, we can look it up. No reason for hfs to call us, I doubt you mount hfs on a zfs volume? Could call us via zfs_util in the FS-Bundle (/Library/Filesystems/zfs.fs) perhaps, could be renamed out and tested to see if it still crashes.

The other thing is: why does Carbon Copy Cloner (rsync) always copies every data from an internal volume to the

Could be date-timestamps don't match or gets changed. Could be xattrs are in different order in ZFS to hfs (the order can be relied on in hfs, but not zfs). Could be we handle xattrs differently, for example, wrt to finderinfo. See if it produces on files with no xattrs at all

[o3x_prokyon]

...you would see a big questionmark in my face...

— Disk utility recognises the mounted zfs pool as "ZFS Dataset", the raw disks are not active.

— The "mount" command states:

Code: Select all

/dev/disk13s1 on /Volumes/EXT_Backup_ZFS (apfs, local, journaled, noatime)

Perhaps this is the result of the zfs setting: com.apple.mimic=apfs (?)

— The command to import the ZFS pool is:

Code: Select all

sudo zpool import -d /var/run/disk/by-id "EXT_Backup_ZFS"

I have no idea about zfs_util and do not intentionally mount hfs or apfs on a zfs volume (maybe I do anyway?).


Backup issue:
The timestamps seem to be absolutely equal (stat -f %Fm …; checked via ls and Finder). Files without any xattrs are copied, files with xattrs are copied. It's very strange. I think, I will try my old backup scripts using rsync directly to check its output. Maybe this will help... Maybe his behaviour is related to com.apple.mimic=apfs, too?


I re-enabled some analytic processes, now I will force another panic to see if I get a better panic-report...

[o3x_prokyon]

So, the „new“ panic report is almost the same as before (the crash happened on another core, that's all). I have no Idea, why this happens, but there is a workaround:

Do not eject any other mounted disk while the ZFS pool is imported...!

The backup-problem seems to be a problem of rsync writing to ZFS. Why? No idea! Maybe the rsync developers have...? Yesterday, I started a backup with another app, called „FreeFileSync“, and everything was fine! FFS found and corrected a lot of permission-issues on the ZFS pool and did only copy new or changed files. CCC would be a little more convenient, but I can live with that for now.

[niket]

Hello everyone, this is my first post in the forum, I am new to ZFS and this is my first time using it on a mac. And I am facing the same issue mentioned here. I have set com.apple.mimic=apfs and I am able to reproduce the kernel panic when unmounting a .dmg file or creating .sparsebundle file from disk utility.

Specs:
M2 Mac Mini, 16GB Ram

macOS version: 14.1.2

Code: Select all

❯ zfs version
zfs-macOS-2.2.0-1
zfs-kmod-2.2.0-1


Any app that I try to install with homebrew will also cause a kernel panic when it has to mount and unmount a dmg file

Code: Select all

brew install --cask <cask-name>

If zfs pools were already exported then this issue doesn’t happen. But exporting and importing them everything is not particular as the storage is critical to be available at all the times.

And my workaround for now is to set com.apple.mimic=hfs which prevent this issue.

I am a bit surprised that this issue is not widely reported as some common tasks like installing a homebrew cask causes a kernel panic when using com.apple.mimic=apfs. I need to use that because I am running this Mac mimi as able content caching server and also as time machine backup server.

If someone can look into this issue then it will be greatly appreciated!

Code: Select all

panic(cpu 5 caller 0xfffffe001bf8b32c): Kernel data abort. at pc 0xfffffe001e6e39f0, lr 0xfffffe001e6e39ec (saved state: 0xfffffe49c84139f0)
     x0:  0xfffffe6038b26000 x1:  0xfffffe001abf4c08  x2:  0x000000000000a93c  x3:  0x0000000000000014
     x4:  0x0000000000000031 x5:  0x0000000000000035  x6:  0x0000000000000067  x7:  0x0000000000000b10
     x8:  0x0000000000000012 x9:  0x0000000000000000  x10: 0x0000000000000000  x11: 0x000000000000001d
     x12: 0x0000000000000018 x13: 0x0000000000000000  x14: 0xffffffffffffffff  x15: 0xffffffffffffffff
     x16: 0xd7877e001e6e39a4 x17: 0x000000000000e4fe  x18: 0x0000000000000000  x19: 0xfffffe49c8413e58
     x20: 0xfffffe29a8c56084 x21: 0xfffffe29a8c56000  x22: 0xfffffe24d9317000  x23: 0x000000000000000f
     x24: 0xfffffe29ab1ab300 x25: 0xfffffe001ee1f000  x26: 0xfffffe29a8c569a4  x27: 0xfffffe29a8c56000
     x28: 0x000000000000000b fp:  0xfffffe49c8413d90  lr:  0xfffffe001e6e39ec  sp:  0xfffffe49c8413d40
     pc:  0xfffffe001e6e39f0 cpsr: 0x60401208         esr: 0x96000005          far: 0x00000000000000da

Debugger message: panic
Memory ID: 0xff
OS release type: User
OS version: 23B81
Kernel version: Darwin Kernel Version 23.1.0: Mon Oct  9 21:28:31 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8112
Fileset Kernelcache UUID: C230BD6947603CC30AE9B4D42D26F96C
Kernel UUID: 75A3938A-F8CA-3AFD-AE20-A3477CEA6FB2
Boot session UUID: F84714C0-47C2-4E05-98A5-12F3A129D5ED
iBoot version: iBoot-10151.41.12
secure boot?: YES
roots installed: 0
Paniclog version: 14
KernelCache slide: 0x00000000131e4000
KernelCache base:  0xfffffe001a1e8000
Kernel slide:      0x00000000131ec000
Kernel text base:  0xfffffe001a1f0000
Kernel text exec slide: 0x0000000014734000
Kernel text exec base:  0xfffffe001b738000
mach_absolute_time: 0x1f23110ec6a
Epoch Time:        sec       usec
  Boot    : 0x6563dd0a 0x000a0858
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x65653949 0x0004f028

Zone info:
  Zone map: 0xfffffe10107f0000 - 0xfffffe30107f0000
  . VM    : 0xfffffe10107f0000 - 0xfffffe14dd4bc000
  . RO    : 0xfffffe14dd4bc000 - 0xfffffe1676e54000
  . GEN0  : 0xfffffe1676e54000 - 0xfffffe1b43b20000
  . GEN1  : 0xfffffe1b43b20000 - 0xfffffe20107ec000
  . GEN2  : 0xfffffe20107ec000 - 0xfffffe24dd4b8000
  . GEN3  : 0xfffffe24dd4b8000 - 0xfffffe29aa184000
  . DATA  : 0xfffffe29aa184000 - 0xfffffe30107f0000
  Metadata: 0xfffffe49bb70c000 - 0xfffffe49c370c000
  Bitmaps : 0xfffffe49c370c000 - 0xfffffe49c6520000
  Extra   : 0 - 0

CORE 0 recently retired instr at 0xfffffe001b8dc538
CORE 1 recently retired instr at 0xfffffe001b8dc538
CORE 2 recently retired instr at 0xfffffe001b8dc538
CORE 3 recently retired instr at 0xfffffe001b8dc538
CORE 4 recently retired instr at 0xfffffe001b8dc538
CORE 5 recently retired instr at 0xfffffe001b8dad98
CORE 6 recently retired instr at 0xfffffe001b8dc538
CORE 7 recently retired instr at 0xfffffe001b8dc538
TPIDRx_ELy = {1: 0xfffffe24d9317000  0: 0x0000000000001005  0ro: 0x0000000000000000 }
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 0: PC=0x000000018c905040, LR=0x000000018c904f70, FP=0x000000016b750480
CORE 1: PC=0xfffffe001bdea410, LR=0xfffffe001bdb0978, FP=0xfffffe49c860b600
CORE 2: PC=0xfffffe001b7c1520, LR=0xfffffe001b7c1520, FP=0xfffffe49c81eba10
CORE 3: PC=0xfffffe001b7fd610, LR=0xfffffe001b783564, FP=0xfffffe49c85b7d10
CORE 4: PC=0x000000018ed08954, LR=0x000000018ed09020, FP=0x000000016db79df0
CORE 5 is the one that panicked. Check the full backtrace for details.
CORE 6: PC=0x0000000103e327dc, LR=0x0000000102608bfc, FP=0x000000016e069760
CORE 7: PC=0xfffffe001b7c7dd0, LR=0xfffffe001b7c7dd0, FP=0xfffffe49c8653ef0
Compressor Info: 2% of compressed pages limit (OK) and 1% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe29aa082668: 0 pages, 625 threads: pid 0: kernel_task
Panicked thread: 0xfffffe24d9317000, backtrace: 0xfffffe49c8413120, tid: 573221
        lr: 0xfffffe001b78e9fc  fp: 0xfffffe49c84131b0
        lr: 0xfffffe001b8d37dc  fp: 0xfffffe49c8413220
        lr: 0xfffffe001b8d1c64  fp: 0xfffffe49c8413310
        lr: 0xfffffe001b73f808  fp: 0xfffffe49c8413320
        lr: 0xfffffe001b78e2f0  fp: 0xfffffe49c84136d0
        lr: 0xfffffe001bf80aa8  fp: 0xfffffe49c84136f0
        lr: 0xfffffe001bf8b32c  fp: 0xfffffe49c8413870
        lr: 0xfffffe001b8d3638  fp: 0xfffffe49c84138e0
        lr: 0xfffffe001b8d1f50  fp: 0xfffffe49c84139d0
        lr: 0xfffffe001b73f808  fp: 0xfffffe49c84139e0
        lr: 0xfffffe001e6e39ec  fp: 0xfffffe49c8413d90
        lr: 0xfffffe001b96a3dc  fp: 0xfffffe49c8413e20
        lr: 0xfffffe001e6e38e4  fp: 0xfffffe49c8413f20
        lr: 0xfffffe001b748be4  fp: 0x0000000000000000
      Kernel Extensions in backtrace:
         com.apple.filesystems.apfs(2235.41.1)[E17C57DD-EE74-3808-B998-A02BCE5DACA7]@0xfffffe001e6b5f40->0xfffffe001e816eef
            dependency: com.apple.driver.AppleEffaceableStorage(1.0)[6381C046-1D51-311C-A825-0D736F0B139A]@0xfffffe001c7d4920->0xfffffe001c7d993b
            dependency: com.apple.iokit.CoreAnalyticsFamily(1)[0F366549-E9FC-3B68-9765-ED75460DC0B0]@0xfffffe001d3934e0->0xfffffe001d39b2cb
            dependency: com.apple.iokit.IOStorageFamily(2.1)[AAF21051-80CE-3C4D-8B6F-B91FEDD4F18B]@0xfffffe001df62430->0xfffffe001df8410f
            dependency: com.apple.kec.corecrypto(14.0)[4EAE32F2-4DA7-3078-AE3D-6E4CBCE4739B]@0xfffffe001e857440->0xfffffe001e8a778b
            dependency: com.apple.security.AppleImage4(5.0.0)[DB08D649-F4D7-3D40-BCC9-BD9008ECAB95]@0xfffffe001cae38a0->0xfffffe001cafc547

last started kext at 2831078183: com.apple.filesystems.smbfs   5.0 (addr 0xfffffe001ac2dd90, size 81955)
loaded kexts:
.
.
.


[abc123]

Hello everyone, this is my first post in the forum, I am new to ZFS and this is my first time using it on a mac. And I am facing the same issue mentioned here. I have set com.apple.mimic=apfs and I am able to reproduce the kernel panic when unmounting a .dmg file or creating .sparsebundle file from disk utility.

Think I'm seeing this also. I've had the system panic when unmounting DMGs since 2.2.0 but also upgraded to latest macOS around same time. Not 100% of the time but more often than not. This is on an M1 now running 14.1.2

[o3x_prokyon]

And my workaround for now is to set com.apple.mimic=hfs which prevent this issue.

Very interesting aspect! I never considered this to be the reason for the panic. I will change this setting and see what's happening. In previous ZFS versions I played around with this parameter, but didn't find any real difference, so I left it at "apfs". My external rotating ZFS disks are only for backup, so I think it doesn't matter, if they are seen as "hfs" or "apfs".

[niket]

so I think it doesn't matter, if they are seen as "hfs" or "apfs".

Yeah I don't know the difference between apfs vs hfs for this property. I also use external rotational disks for zfs and amongst other things they are also used for content caching and time machine server for backups, that's why I need to set this property. I guess time machine backs-up differently when it's on APFS but I have not noticed any difference after switching it back to HFS since the backups for clients happen over the network anyways.

[dmz]

I have _exactly_ this problem with disk image unmounting/creation, and I've had it since 2.2.0 (and still have it on 2.2.2). If I have any pool imported with a dataset that has com.apple.mimic=apfs, I can _always_ panic my machine, with a log that looks like the one below, by doing the following: (1) open Disk Utility; (2) create a 1 TB read/write disk image (why 1 TB? because there's some minimum size below which the crash doesn't happen and I haven't figured out what that is, but it always happens with 1 TB). The image creation process involves mounting and unmounting the image, and at the end of the process the machine panics. Similarly, mounting any existing large image (1 TB or more definitely, some smaller ones probably) and subsequently unmounting it causes the same panic.

This does not occur with com.apple.mimic=hfs. So what this says to me is that when a ZFS dataset is presenting itself to the OS as APFS, it is not responding properly (in some way) to something that the APFS filesystem code is doing when a disk image is unmounted. My guess is that this is why the ZFS kernel extension doesn't actually appear in the backtrace; it's something in the APFS code that's failing because of some unexpected/incorrect property of the ZFS filesystem masquerading as APFS.

Fortunately, I can switch to com.apple.mimic=hfs on all my filesystems with no apparent ill effects. So I can avoid the crash for now. But it's still lurking out there...

Backtrace example:

Code: Select all

panic(cpu 1 caller 0xfffffe0015cb2008): Kernel data abort. at pc 0xfffffe001848c2b0, lr 0xfffffe001848c2ac (saved state: 0xfffffe664b01b970)
     x0:  0xfffffe6c67e92000 x1:  0xfffffe00148e729e  x2:  0x000000000001e0dd  x3:  0x0000000000000014
     x4:  0x0000000000000000 x5:  0xfeedfacefeedfad3  x6:  0x0000000000000067  x7:  0xfffffe001cba6800
     x8:  0x0000000000000012 x9:  0x0000000000000000  x10: 0x0000000000000000  x11: 0x000000000000001e
     x12: 0x0000000000000018 x13: 0x0000400000000000  x14: 0xffffffffffffffff  x15: 0xffffffffffffffff
     x16: 0x000002003fab5dd8 x17: 0x000000000000e4fe  x18: 0x0000000000000000  x19: 0xfffffe664b01be58
     x20: 0xfffffe2997bcc084 x21: 0xfffffe2997bcc000  x22: 0xfffffe24bd252800  x23: 0x0000000000000013
     x24: 0xfffffe2ffd83ee00 x25: 0xfffffe0018bef000  x26: 0xfffffe2997bcc9a4  x27: 0xfffffe2997bcc000
     x28: 0x000000000000000f fp:  0xfffffe664b01bd90  lr:  0xfffffe001848c2ac  sp:  0xfffffe664b01bcc0
     pc:  0xfffffe001848c2b0 cpsr: 0x60401208         esr: 0x96000005          far: 0x00000000000000da

Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 23E214
Kernel version: Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:43 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 8E2B18A405A53944BF38EFAC03047CB4
Kernel UUID: FF85748F-8743-36DA-9559-6332F0650C53
Boot session UUID: B8B57E73-0D91-4551-87F6-8DAEB5D89A13
iBoot version: iBoot-10151.101.3
secure boot?: YES
roots installed: 0
Paniclog version: 14
KernelCache slide: 0x000000000ceec000
KernelCache base:  0xfffffe0013ef0000
Kernel slide:      0x000000000cef4000
Kernel text base:  0xfffffe0013ef8000
Kernel text exec slide: 0x000000000e428000
Kernel text exec base:  0xfffffe001542c000
mach_absolute_time: 0x2effe37b1
Epoch Time:        sec       usec
  Boot    : 0x66146109 0x00005f5e
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x6614630d 0x0007324b

Zone info:
  Zone map: 0xfffffe10004d0000 - 0xfffffe30004d0000
  . VM    : 0xfffffe10004d0000 - 0xfffffe14cd19c000
  . RO    : 0xfffffe14cd19c000 - 0xfffffe1666b34000
  . GEN0  : 0xfffffe1666b34000 - 0xfffffe1b33800000
  . GEN1  : 0xfffffe1b33800000 - 0xfffffe20004cc000
  . GEN2  : 0xfffffe20004cc000 - 0xfffffe24cd198000
  . GEN3  : 0xfffffe24cd198000 - 0xfffffe2999e64000
  . DATA  : 0xfffffe2999e64000 - 0xfffffe30004d0000
  Metadata: 0xfffffe6c42448000 - 0xfffffe6c4a448000
  Bitmaps : 0xfffffe6c4a448000 - 0xfffffe6c5a448000
  Extra   : 0 - 0

TPIDRx_ELy = {1: 0xfffffe24bd252800  0: 0x0000000000000001  0ro: 0x0000000000000000 }
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0x000000019bc90030, LR=0x000000019bc83fd8, FP=0x000000016d221eb0
CORE 1 is the one that panicked. Check the full backtrace for details.
CORE 2: PC=0x000000018ec90950, LR=0x000000018eeca7f4, FP=0x000000016b85c9d0
CORE 3: PC=0x000000018ecc8f54, LR=0x000000018ecc8f54, FP=0x000000016b5a6a70
CORE 4: PC=0xfffffe00154fee74, LR=0xfffffe0015498d48, FP=0xfffffe664f6fb480
CORE 5: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664b8c7ef0
CORE 6: PC=0x000000018ec6be0c, LR=0x000000018ec6b7dc, FP=0x000000016f51d960
CORE 7: PC=0xfffffe001543e414, LR=0xfffffe00154ffaa0, FP=0xfffffe664f68f8e0
CORE 8: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664f7c7ef0
CORE 9: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664ed9fef0
CORE 10: PC=0xfffffe00154694bc, LR=0xfffffe001546977c, FP=0xfffffe664f1e3940
CORE 11: PC=0xfffffe00154f942c, LR=0xfffffe00154f942c, FP=0xfffffe664e257cf0
CORE 12: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664c353ef0
CORE 13: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664f513ef0
CORE 14: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664dbd7ef0
CORE 15: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664ea87ef0
CORE 16: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664f46bef0
CORE 17: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664c4dfef0
CORE 18: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664e2ffef0
CORE 19: PC=0xfffffe00154c18bc, LR=0xfffffe00154c18bc, FP=0xfffffe664f5d7ef0
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe2999da2678: 0 pages, 1342 threads: pid 0: kernel_task
Panicked thread: 0xfffffe24bd252800, backtrace: 0xfffffe664b01b100, tid: 42365
        lr: 0xfffffe0015485f70  fp: 0xfffffe664b01b190
        lr: 0xfffffe00155d37f4  fp: 0xfffffe664b01b200
        lr: 0xfffffe00155d1aec  fp: 0xfffffe664b01b2c0
        lr: 0xfffffe0015433ba8  fp: 0xfffffe664b01b2d0
        lr: 0xfffffe001548584c  fp: 0xfffffe664b01b680
        lr: 0xfffffe0015ca73ec  fp: 0xfffffe664b01b6a0
        lr: 0xfffffe0015cb2008  fp: 0xfffffe664b01b820
        lr: 0xfffffe00155d361c  fp: 0xfffffe664b01b890
        lr: 0xfffffe00155d1ca8  fp: 0xfffffe664b01b950
        lr: 0xfffffe0015433ba8  fp: 0xfffffe664b01b960
        lr: 0xfffffe001848c2ac  fp: 0xfffffe664b01bd90
        lr: 0xfffffe001566cb08  fp: 0xfffffe664b01be20
        lr: 0xfffffe001848c178  fp: 0xfffffe664b01bf20
        lr: 0xfffffe001543cf08  fp: 0x0000000000000000
      Kernel Extensions in backtrace:
         com.apple.filesystems.apfs(2236.101.1)[6510A98B-70D7-3B7E-BF65-5965CF196234]@0xfffffe001845db90->0xfffffe00185c265b
            dependency: com.apple.driver.AppleEffaceableStorage(1.0)[10C4F9B7-E1DB-35A7-B1CD-3294E0840C37]@0xfffffe00165033b0->0xfffffe00165083c7
            dependency: com.apple.iokit.CoreAnalyticsFamily(1)[988E6C15-3630-34F3-B3AE-4FBA1B2ECBC3]@0xfffffe0017100b50->0xfffffe0017108937
            dependency: com.apple.iokit.IOStorageFamily(2.1)[4F209F9C-383F-34E2-9514-27D0FF687FEA]@0xfffffe0017ce6960->0xfffffe0017d07fd7
            dependency: com.apple.kec.corecrypto(14.0)[382760DA-F216-3EE0-BD95-5BE034AFCAD3]@0xfffffe0018602940->0xfffffe0018653afb
            dependency: com.apple.security.AppleImage4(6.3.0)[CB824FFE-F685-3788-89AC-C16D90308E30]@0xfffffe00167fdd90->0xfffffe001681fd6f

last started kext at 3207991746: com.apple.filesystems.smbfs   5.1 (addr 0xfffffe0014920870, size 82354)
loaded kexts:
org.openzfsonosx.zfs   2.2.2
com.apple.filesystems.smbfs   5.1
com.apple.filesystems.autofs   3.0
com.apple.AppleEthernetAquantiaAqtionFirmware   1.0.36
com.apple.driver.AppleBiometricServices   1
com.apple.driver.CoreKDL   1
com.apple.driver.AppleAHCIPort   380
com.apple.driver.usb.AppleUSBHostBillboardDevice   1.0
com.apple.driver.AppleTopCaseHIDEventDriver   7440.8
com.apple.driver.AppleUSBTopCaseDriver   7440.8
com.apple.iokit.SCSITaskUserClient   495
com.apple.driver.AppleUSBODD   556
com.apple.driver.DiskImages.ReadWriteDiskImage   493.0.0
com.apple.driver.DiskImages.UDIFDiskImage   493.0.0
com.apple.driver.DiskImages.RAMBackingStore   493.0.0
com.apple.driver.DiskImages.FileBackingStore   493.0.0
com.apple.driver.AppleUSBMassStorageInterfaceNub   556
com.apple.driver.BCMWLANFirmware4387.Hashstore   1
com.apple.driver.AppleUSBDeviceNCM   5.0.0
com.apple.driver.AppleFileSystemDriver   3.0.1
com.apple.nke.l2tp   1.9
com.apple.filesystems.tmpfs   1
com.apple.filesystems.nfs   1
com.apple.filesystems.lifs   1
com.apple.filesystems.apfs   2236.101.1
com.apple.IOTextEncryptionFamily   1.0.0
com.apple.filesystems.hfs.kext   650.0.2
com.apple.security.BootPolicy   1
com.apple.BootCache   40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib   1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless   1.0.0d1
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.driver.SEPHibernation   1
com.apple.driver.AppleThunderboltIP   4.0.3
com.apple.driver.ApplePMP   1
com.apple.driver.AppleBCMWLANBusInterfacePCIe   1
com.apple.driver.AppleBluetoothModule   1
com.apple.driver.ApplePMPFirmware   1
com.apple.driver.AppleEventLogHandler   1
com.apple.driver.AppleAOPVoiceTrigger   340.42
com.apple.driver.AppleAVE2   760.23.1
com.apple.driver.AppleAVD   740
com.apple.AppleEmbeddedSimpleSPINORFlasher   1
com.apple.driver.AppleSN012776Amp   740.41
com.apple.driver.AppleSmartIO2   1
com.apple.driver.AppleCS42L84Audio   740.41
com.apple.driver.AppleT6000SOCTuner   1
com.apple.driver.AppleT6002CLPC   1
com.apple.driver.AppleProResHW   350.47.0
com.apple.driver.AppleT6001PMGR   1
com.apple.driver.AppleS5L8960XNCO   1
com.apple.driver.AppleSPIMC   1
com.apple.driver.AppleJPEGDriver   6.6.2
com.apple.driver.AppleSamsungSerial   1.0.0d1
com.apple.driver.AppleSerialShim   1
com.apple.driver.AppleMCDP29XXUpdateSupport   1
com.apple.driver.AudioDMAController-T600x   340.58
com.apple.driver.AppleInterruptControllerV2   1.0.0d1
com.apple.driver.AppleMobileDispT600X-DCP   140.0
com.apple.driver.AppleS8000DWI   1.0.0d1
com.apple.driver.AppleS8000AES   1
com.apple.driver.usb.AppleSynopsysUSB40XHCI   1
com.apple.AGXG13X   280.3.9
com.apple.driver.AppleS5L8920XPWM   1.0.0d1
com.apple.driver.AppleT8110DART   1
com.apple.driver.AppleSDXC   3.5.1
com.apple.driver.AppleS5L8940XI2C   1.0.0d2
com.apple.driver.AppleT6000DART   1
com.apple.driver.AppleT6000   1
com.apple.driver.AppleM68Buttons   1.0.0d1
com.apple.iokit.IOUserEthernet   1.0.1
com.apple.driver.usb.AppleUSBUserHCI   1
com.apple.iokit.IOKitRegistryCompatibility   1
com.apple.iokit.EndpointSecurity   1
com.apple.driver.AppleUIO   1
com.apple.driver.AppleDiskImages2   276.100.16
com.apple.AppleSystemPolicy   2.0.0
com.apple.nke.applicationfirewall   404
com.apple.kec.InvalidateHmac   1
com.apple.kec.AppleEncryptedArchive   1
com.apple.driver.AppleBluetoothHIDKeyboard   7440.3
com.apple.iokit.IOAHCIBlockStorage   359
com.apple.driver.AppleUSBMergeNub   900.4.2
com.apple.driver.driverkit.serial   6.0.0
com.apple.driver.usb.cdc.ncm   5.0.0
com.apple.driver.usb.cdc.ecm   5.0.0
com.apple.driver.usb.cdc.acm   5.0.0
com.apple.driver.usb.serial   6.0.0
com.apple.driver.AppleMesaSEPDriver   100.99
com.apple.iokit.IOBiometricFamily   1
com.apple.iokit.IOAHCIFamily   304
com.apple.driver.AppleThunderboltPCIUpAdapter   4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter   8.5.1
com.apple.driver.AppleEthernetAquantiaAqtion   1.0.64
com.apple.driver.AppleHSBluetoothDriver   7440.8
com.apple.driver.IOBluetoothHIDDriver   9.0.0
com.apple.driver.AppleActuatorDriver   7440.9
com.apple.driver.AppleMultitouchDriver   7440.9
com.apple.driver.AppleInputDeviceSupport   7440.1
com.apple.driver.AppleHIDKeyboard   7440.3
com.apple.driver.usb.IOUSBHostHIDDevice   1.2
com.apple.iokit.IOSCSIMultimediaCommandsDevice   495
com.apple.iokit.IOBDStorageFamily   1.8
com.apple.iokit.IODVDStorageFamily   1.8
com.apple.iokit.IOCDStorageFamily   1.8
com.apple.driver.DiskImages.KernelBacked   493.0.0
com.apple.driver.usb.cdc   5.0.0
com.apple.driver.AppleUSBAudio   640.12
com.apple.iokit.IOAudioFamily   540.3
com.apple.vecLib.kext   1.2.0
com.apple.driver.AppleXsanScheme   3
com.apple.driver.AppleUVDMDriver   1.0.0
com.apple.driver.AppleUVDM   1.0.0
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI   1
com.apple.driver.usb.AppleUSBXHCIPCI   1.2
com.apple.driver.AppleConvergedIPCOLYBTControl   1
com.apple.driver.AppleConvergedPCI   1
com.apple.driver.AppleBluetoothDebug   1
com.apple.driver.usb.networking   5.0.0
com.apple.nke.ppp   1.9
com.apple.driver.AppleBSDKextStarter   3
com.apple.kext.triggers   1.0
com.apple.filesystems.hfs.encodings.kext   1
com.apple.driver.AppleSyntheticGameController   11.4.11
com.apple.driver.AppleBTM   1.0.1
com.apple.driver.IOHIDPowerSource   1
com.apple.driver.AppleCallbackPowerSource   1
com.apple.driver.AppleSEPHDCPManager   1.0.1
com.apple.driver.AppleTrustedAccessory   1
com.apple.iokit.AppleSEPGenericTransfer   1
com.apple.driver.AppleStockholmControl   1.0.0
com.apple.driver.AppleThunderboltPCIDownAdapter   4.1.1
com.apple.driver.AppleThunderboltUSBDownAdapter   1.0.4
com.apple.driver.AppleThunderboltDPInAdapter   8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily   8.5.1
com.apple.driver.AppleBluetoothDebugService   1
com.apple.driver.AppleBCMWLANCore   1.0.0
com.apple.iokit.IO80211Family   1200.13.0
com.apple.driver.IOImageLoader   1.0.0
com.apple.driver.AppleOLYHAL   1
com.apple.driver.AppleMultiFunctionManager   1
com.apple.driver.corecapture   1.0.4
com.apple.driver.AppleSPMIPMU   1.0.1
com.apple.driver.AppleDialogPMU   1.0.1
com.apple.driver.AppleSPMI   1.0.1
com.apple.driver.AppleT600xPCIe   1
com.apple.driver.AppleAOPAudio   340.4
com.apple.driver.AppleARMWatchdogTimer   1
com.apple.driver.AppleHPM   3.4.4
com.apple.driver.AppleDiagnosticDataAccessReadOnly   1.0.0
com.apple.driver.AppleSPU   1
com.apple.driver.AppleI2CEthernetAquantia   1.0.0
com.apple.driver.AppleDCPDPTXProxy   1.0.0
com.apple.driver.DCPDPFamilyProxy   1
com.apple.driver.AppleCSEmbeddedAudio   740.41
com.apple.driver.AppleEmbeddedAudio   740.41
com.apple.iokit.AppleARMIISAudio   340.16
com.apple.driver.IISAudioIsolatedStreamECProxy   340.16
com.apple.driver.ExclavesAudioKext   1
com.apple.driver.ApplePassthroughPPM   3.0
com.apple.driver.ApplePMGR   1
com.apple.iokit.IONVMeFamily   2.1.0
com.apple.driver.AppleNANDConfigAccess   1.0.0
com.apple.driver.AppleSART   1
com.apple.driver.DCPAVFamilyProxy   1
com.apple.AGXFirmwareKextG13XRTBuddy   280.3.9
com.apple.AGXFirmwareKextRTBuddy64   280.3.9
com.apple.iokit.IOMobileGraphicsFamily-DCP   343.0.0
com.apple.driver.AppleDCP   1
com.apple.driver.AppleFirmwareKit   1
com.apple.iokit.IOMobileGraphicsFamily   343.0.0
com.apple.driver.AppleM2ScalerCSCDriver   265.0.0
com.apple.driver.AppleT6000TypeCPhy   1
com.apple.driver.AppleT8103TypeCPhy   1
com.apple.driver.AppleThunderboltNHI   7.2.81
com.apple.driver.AppleUSBXDCIARM   1.0
com.apple.driver.AppleUSBXDCI   1.0
com.apple.iokit.IOUSBDeviceFamily   2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI   1
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.driver.AppleEmbeddedUSBHost   1
com.apple.driver.usb.AppleUSBHub   1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice   1.2
com.apple.driver.AppleT6000PCIeC   1
com.apple.driver.ApplePIODMA   1
com.apple.iokit.IOGPUFamily   93.40.3
com.apple.driver.AppleMCA2-T600x   840.3
com.apple.driver.AppleEmbeddedAudioLibs   340.8
com.apple.driver.AppleFirmwareUpdateKext   1
com.apple.driver.AppleA7IOP-M3Wrap-v2   1.0.2
com.apple.driver.AppleDisplayCrossbar   1.0.0
com.apple.iokit.IOThunderboltFamily   9.3.3
com.apple.iokit.IODisplayPortFamily   1.0.0
com.apple.driver.AppleTypeCPhy   1
com.apple.iokit.IOPortFamily   1.0
com.apple.iokit.IOAVBFamily   1220.1
com.apple.plugin.IOgPTPPlugin   1240.15
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor   1.0.0
com.apple.driver.AppleEmbeddedPCIE   1
com.apple.driver.AppleA7IOP-ASCWrap-v4   1.0.2
com.apple.driver.AppleGPIOICController   1.0.2
com.apple.driver.AppleFireStormErrorHandler   1
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.driver.AppleMobileApNonce   1
com.apple.iokit.IOTimeSyncFamily   1240.15
com.apple.driver.DiskImages   493.0.0
com.apple.iokit.IOGraphicsFamily   598
com.apple.iokit.IOBluetoothFamily   9.0.0
com.apple.driver.AppleSSE   1.0
com.apple.driver.AppleSEPKeyStore   2
com.apple.driver.AppleUSBTDM   556
com.apple.iokit.IOUSBMassStorageDriver   245
com.apple.iokit.IOPCIFamily   2.9
com.apple.iokit.IOUSBHostFamily   1.2
com.apple.driver.AppleUSBHostMergeProperties   1.2
com.apple.driver.usb.AppleUSBCommon   1.0
com.apple.driver.AppleSMC   3.1.9
com.apple.driver.RTBuddy   1.0.0
com.apple.driver.AppleEmbeddedTempSensor   1.0.0
com.apple.driver.AppleARMPMU   1.0
com.apple.iokit.IOAccessoryManager   1.0.0
com.apple.driver.AppleOnboardSerial   1.0
com.apple.iokit.IOSerialFamily   11
com.apple.iokit.IOSCSIBlockCommandsDevice   495
com.apple.iokit.IOSCSIArchitectureModelFamily   495
com.apple.driver.AppleRSMChannel   1
com.apple.iokit.IORSMFamily   1
com.apple.driver.AppleLockdownMode   1
com.apple.driver.AppleIPAppender   1.0
com.apple.iokit.IOSkywalkFamily   1.0
com.apple.driver.mDNSOffloadUserClient   1.0.1b8
com.apple.iokit.IONetworkingFamily   3.4
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage   1.0
com.apple.driver.AppleCredentialManager   1.0
com.apple.driver.AppleSEPManager   1.0.1
com.apple.driver.AppleA7IOP   1.0.2
com.apple.driver.IOSlaveProcessor   1
com.apple.driver.AppleBiometricSensor   2
com.apple.iokit.IOHIDFamily   2.0.0
com.apple.driver.AppleANELoadBalancer   7.400.0
com.apple.driver.AppleH11ANEInterface   7.400.0
com.apple.driver.IODARTFamily   1
com.apple.AUC   1.0
com.apple.iokit.IOSurface   352.41.7
com.apple.iokit.IOAVFamily   1.0.0
com.apple.iokit.IOHDCPFamily   1.0.0
com.apple.iokit.IOCECFamily   1
com.apple.iokit.IOAudio2Family   1.0
com.apple.driver.AppleIISController   340.1
com.apple.driver.AppleAudioClockLibs   340.8
com.apple.driver.FairPlayIOKit   71.10.0
com.apple.driver.AppleARMPlatform   1.0.2
com.apple.iokit.IOSlowAdaptiveClockingFamily   1.0.0
com.apple.iokit.IOReportFamily   47
com.apple.security.quarantine   4
com.apple.security.sandbox   300.0
com.apple.iokit.IOStorageFamily   2.1
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.iokit.CoreAnalyticsFamily   1
com.apple.security.AppleImage4   6.3.0
com.apple.kext.CoreTrust   1
com.apple.iokit.IOCryptoAcceleratorFamily   1.0.1
com.apple.kec.pthread   1
com.apple.kec.Libm   1
com.apple.kec.Compression   1.0
com.apple.kec.corecrypto   14.0

** Stackshot Succeeded ** Bytes Traced 936397 (Uncompressed 2541232) **


[incumbent]

i started a new thread about what i think is the same possible issue; i had not noticed dmg behavior being involved but i got it back from the vendor and i had to bootstrap it again so it was running `brew bundle install` and slurping data out of Arq backups. i absolutely use mimic APFS on the datasets on my NVME pool, i am going to restart with openzfs 2.2.2 again, change my attributes from mimic'ing APFS to HFS and see what happens.

i'm really glad i found this thread, mine is here; viewtopic.php?f=26&t=3888 and also implicates APFS. i will say that after a dozen panics or so First Aid was fixing things on my boot volume. a month ago this thing would panic while running first aid, i nuked the whole thing without enabling booting from external devices and internet recovery didn't work until apple factory issue 8GB of memory was installed which makes no sense to me.

they charged me $60 for installing macOS off the internet :eyeroll: but still no idea why it's panic'ing all the time. my diagnostics look quite similar though so i am going to give the HFS mimic a try.