zfs-macOS-2.3.0 RC1 issues

[hamiid]

I am testing zfs-macOS-2.3.0 RC1 on a mac mini (Apple M2). the pool has only 1 file system that is on an SSD attached to the mac mini using a USB.
The pool is just used to save VM machine related files such as the ones that are created by Lima/Docker.
The pool gets suspended with corruption errors after the mac mini goes to sleep while the VM was active, and I get messages similar to this:

Code: Select all

▶ zpool status && zpool list
  pool: tank
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
config:

   NAME                                          STATE     READ WRITE CKSUM
   tank                                          ONLINE       0     0     0
    media-DDD3EC81-B5D5-C047-9575-8FE146345ACB  ONLINE       3     0     0

errors: 7 data errors, use '-v' for a list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   928G  29.6G   898G        -         -     0%     3%  1.00x  SUSPENDED  -

~
▶ sudo zpool clear tank
▶ sudo zpool status -x -v
Password:
  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
   corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
   entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
config:

   NAME                                          STATE     READ WRITE CKSUM
   tank                                          ONLINE       0     0     0
    media-DDD3EC81-B5D5-C047-9575-8FE146345ACB  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <metadata>:<0x0>
        <metadata>:<0x81>
        tank/lima@zbk-daily-20250203-002026:/.Spotlight-V100/Store-V2/23513C02-75E9-4A3A-AD37-E985763E03DE/dbStr-1.map.offsets
        tank/lima@zbk-daily-20250203-002026:/default/lima.yaml
        tank/lima@zbk-daily-20250203-002026:/default/diffdisk
        tank/lima@zbk-daily-20250201-023905:/.Spotlight-V100/Store-V2/23513C02-75E9-4A3A-AD37-E985763E03DE/dbStr-1.map.offsets
        tank/lima@zbk-daily-20250201-023905:/default/lima.yaml
        tank/lima@zbk-hourly-20250201-025329:/.Spotlight-V100/Store-V2/23513C02-75E9-4A3A-AD37-E985763E03DE/dbStr-1.map.offsets
        tank/lima@zbk-hourly-20250201-025329:/default/lima.yaml
        /Volumes/tank/lima/.Spotlight-V100/Store-V2/23513C02-75E9-4A3A-AD37-E985763E03DE/dbStr-1.map.offsets
        /Volumes/tank/lima/default/lima.yaml

[hamiid]

Getting random corruptions as well

Code: Select all

panic(cpu 2 caller 0xfffffe0023864264): kernel heap corruption detected
Debugger message: panic
Memory ID: 0xff
OS release type: User
OS version: 24C101
Kernel version: Darwin Kernel Version 24.2.0: Fri Dec  6 18:51:28 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T8112
Fileset Kernelcache UUID: 0418246E11F390EC454186F6A7C8FE78
Kernel UUID: EB273378-6BA7-3F65-8805-348FF13D784A
Boot session UUID: 664716D9-2F71-4DA4-A42A-413C30E3CBFA
iBoot version: iBoot-11881.61.3
secure boot?: YES
roots installed: 0
Paniclog version: 14
KernelCache slide: 0x000000001c984000
KernelCache base:  0xfffffe0023988000
Kernel slide:      0x000000001c98c000
Kernel text base:  0xfffffe0023990000
Kernel text exec slide: 0x000000001e088000
Kernel text exec base:  0xfffffe002508c000
mach_absolute_time: 0x89d1772f486
Epoch Time:        sec       usec
  Boot    : 0x679b8fc8 0x0001ed97
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x67a1953b 0x000aa439

Zone info:
  Zone map: 0xfffffe1000040000 - 0xfffffe3000040000
  . VM    : 0xfffffe1000040000 - 0xfffffe14ccd0c000
  . RO    : 0xfffffe14ccd0c000 - 0xfffffe16666a4000
  . GEN0  : 0xfffffe16666a4000 - 0xfffffe1b33370000
  . GEN1  : 0xfffffe1b33370000 - 0xfffffe200003c000
  . GEN2  : 0xfffffe200003c000 - 0xfffffe24ccd08000
  . GEN3  : 0xfffffe24ccd08000 - 0xfffffe29999d4000
  . DATA  : 0xfffffe29999d4000 - 0xfffffe3000040000
  Metadata: 0xfffffe49c175c000 - 0xfffffe49c975c000
  Bitmaps : 0xfffffe49c975c000 - 0xfffffe49cc574000
  Extra   : 0 - 0

CORE 0 recently retired instr at 0xfffffe0025244978
CORE 1 recently retired instr at 0xfffffe0025244978
CORE 2 recently retired instr at 0xfffffe0025243228
CORE 3 recently retired instr at 0xfffffe0025244978
CORE 4 recently retired instr at 0xfffffe0025244978
CORE 5 recently retired instr at 0xfffffe0025244978
CORE 6 recently retired instr at 0xfffffe0025244978
CORE 7 recently retired instr at 0xfffffe0025244978
TPIDRx_ELy = {1: 0xfffffe1b32038fd0  0: 0x0000000000000002  0ro: 0x0000000000000000 }
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 0: PC=0x000000019253bc44, LR=0x000000019809bdac, FP=0x000000016d54d640
CORE 1: PC=0xfffffe0025121e4c, LR=0xfffffe0025121e4c, FP=0xfffffe5ddd2bfed0
CORE 2 is the one that panicked. Check the full backtrace for details.
CORE 3: PC=0xfffffe002523f124, LR=0xfffffe002523f120, FP=0xfffffe5ddd733e40
CORE 4: PC=0xfffffe0025121e50, LR=0xfffffe0025121e4c, FP=0xfffffe5ddcecfed0
CORE 5: PC=0xfffffe0025121e50, LR=0xfffffe0025121e4c, FP=0xfffffe5ddb47bed0
CORE 6: PC=0xfffffe0025121e50, LR=0xfffffe0025121e4c, FP=0xfffffe5ddd407ed0
CORE 7: PC=0xfffffe0025121e50, LR=0xfffffe0025121e4c, FP=0xfffffe5ddba9fed0
Compressor Info: 9% of compressed pages limit (OK) and 11% of segments limit (OK) with 2 swapfiles and OK swap space
Panicked task 0xfffffe24cce1eb58: 0 pages, 616 threads: pid 0: kernel_task
Panicked thread: 0xfffffe1b32038fd0, backtrace: 0xfffffe5ddb8776e0, tid: 3341
        lr: 0xfffffe00250e6670  fp: 0xfffffe5ddb877770
        lr: 0xfffffe002523b158  fp: 0xfffffe5ddb8777e0
        lr: 0xfffffe0025239530  fp: 0xfffffe5ddb877890
        lr: 0xfffffe00250938b0  fp: 0xfffffe5ddb8778a0
        lr: 0xfffffe00250e5f88  fp: 0xfffffe5ddb877c70
        lr: 0xfffffe002593d514  fp: 0xfffffe5ddb877c90
        lr: 0xfffffe0023864264  fp: 0xfffffe5ddb877ca0
        lr: 0xfffffe00236137cc  fp: 0xfffffe5ddb877d40
        lr: 0xfffffe0023618c30  fp: 0xfffffe5ddb877d80
        lr: 0xfffffe0023618ab8  fp: 0xfffffe5ddb877df0
        lr: 0xfffffe00236188ec  fp: 0xfffffe5ddb877e20
        lr: 0xfffffe0023622658  fp: 0xfffffe5ddb877ec0
        lr: 0xfffffe0023622bfc  fp: 0xfffffe5ddb877f20
        lr: 0xfffffe002509cbe8  fp: 0x0000000000000000
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.3)[1B15B167-3FC9-3862-92A5-85D58F2B6FE3]@0xfffffe0023610000->0xfffffe002386585b
            dependency: com.apple.iokit.IOStorageFamily(2.1)[664C0AD0-B154-31C7-8A33-BFE1336774B1]@0xfffffe0027a0c1b0->0xfffffe0027a2d8ab

last started kext at 1505086638258: com.apple.driver.usb.cdc.ecm   5.0.0 (addr 0xfffffe002405ebb0, size 2532)
loaded kexts:
org.openzfsonosx.zfs   2.3.0
com.apple.filesystems.autofs   3.0
com.apple.driver.AppleBiometricServices   1
com.apple.driver.CoreKDL   1
com.apple.iokit.SCSITaskUserClient   500
com.apple.driver.AppleUSBMassStorageInterfaceNub   556
com.apple.driver.usb.AppleUSBHostBillboardDevice   1.0
com.apple.iokit.AppleBCM5701Ethernet   11.0.0
com.apple.driver.DiskImages.ReadWriteDiskImage   493.0.0
com.apple.driver.DiskImages.UDIFDiskImage   493.0.0
com.apple.driver.DiskImages.RAMBackingStore   493.0.0
com.apple.driver.DiskImages.FileBackingStore   493.0.0
com.apple.driver.SEPHibernation   1
com.apple.driver.AppleUSBDeviceNCM   5.0.0
com.apple.driver.AppleThunderboltIP   4.0.3
com.apple.driver.AppleFileSystemDriver   3.0.1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib   1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless   1.0.0d1
com.apple.nke.l2tp   1.9
com.apple.filesystems.tmpfs   1
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.filesystems.nfs   1
com.apple.filesystems.lifs   1
com.apple.filesystems.apfs   2317.61.2
com.apple.IOTextEncryptionFamily   1.0.0
com.apple.filesystems.hfs.kext   677.60.1
com.apple.security.BootPolicy   1
com.apple.BootCache   40
com.apple.driver.AppleSmartIO2   1
com.apple.AppleEmbeddedSimpleSPINORFlasher   1
com.apple.driver.ApplePMP   1
com.apple.driver.AppleAOPVoiceTrigger   420.7
com.apple.driver.AppleMobileDispH14G-DCP   140.0
com.apple.driver.ApplePMPFirmware   1
com.apple.driver.AppleSPIMC   1
com.apple.driver.AudioDMAController-T8112   420.19
com.apple.driver.usb.AppleSynopsysUSB40XHCI   1
com.apple.driver.AppleJPEGDriver   7.1.5
com.apple.driver.AppleAVE2   803.48.1
com.apple.driver.AppleAVD   808
com.apple.driver.AppleProResHW   426.1
com.apple.AGXG14G   323.15
com.apple.driver.AppleSN012776Amp   820.18
com.apple.driver.AppleCS42L84Audio   820.18
com.apple.driver.AppleTypeCRetimer   1.0.0
com.apple.driver.AppleEventLogHandler   1
com.apple.driver.AppleSamsungSerial   1.0.0d1
com.apple.driver.AppleSerialShim   1
com.apple.driver.AppleT8020SOCTuner   1
com.apple.driver.AppleT8112CLPC   1
com.apple.driver.AppleS5L8960XNCO   1
com.apple.driver.AppleT8112PMGR   1
com.apple.driver.AppleS8000AES   1
com.apple.driver.AppleS5L8920XPWM   1.0.0d1
com.apple.driver.AppleS8000DWI   1.0.0d1
com.apple.driver.AppleInterruptControllerV2   1.0.0d1
com.apple.driver.AppleBluetoothModule   1
com.apple.driver.AppleBCMWLANBusInterfacePCIe   1
com.apple.driver.AppleT8110DART   1
com.apple.driver.AppleS5L8940XI2C   1.0.0d2
com.apple.driver.AppleM68Buttons   1.0.0d1
com.apple.driver.AppleT8110   1
com.apple.iokit.IOUserEthernet   1.0.1
com.apple.driver.usb.AppleUSBUserHCI   1
com.apple.iokit.IOKitRegistryCompatibility   1
com.apple.iokit.EndpointSecurity   1
com.apple.driver.AppleDiskImages2   379.60.1
com.apple.AppleSystemPolicy   2.0.0
com.apple.kec.InvalidateHmac   1
com.apple.kec.AppleEncryptedArchive   1
com.apple.driver.usb.cdc.ecm   5.0.0
com.apple.driver.usb.cdc.acm   5.0.0
com.apple.driver.usb.serial   6.0.0
com.apple.driver.AppleHIDKeyboard   8010.1
com.apple.driver.driverkit.serial   6.0.0
com.apple.driver.AppleMesaSEPDriver   100.99
com.apple.iokit.IOBiometricFamily   1
com.apple.driver.usb.IOUSBHostHIDDevice   1.2
com.apple.driver.usb.cdc   5.0.0
com.apple.driver.AppleUSBAudio   720.24
com.apple.iokit.IOAudioFamily   600.2
com.apple.vecLib.kext   1.2.0
com.apple.driver.AppleUVDMDriver   1.0.0
com.apple.driver.AppleUVDM   1.0.0
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI   1
com.apple.driver.usb.AppleUSBXHCIPCI   1.2
com.apple.driver.AppleConvergedIPCOLYBTControl   1
com.apple.driver.AppleConvergedPCI   1
com.apple.driver.AppleBluetoothDebug   1
com.apple.driver.DiskImages.KernelBacked   493.0.0
com.apple.driver.AppleXsanScheme   3
com.apple.driver.AppleSEPHDCPManager   1.0.1
com.apple.driver.AppleTrustedAccessory   1
com.apple.iokit.AppleSEPGenericTransfer   1
com.apple.driver.usb.networking   5.0.0
com.apple.driver.AppleThunderboltDPInAdapter   8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily   8.5.1
com.apple.driver.AppleThunderboltPCIDownAdapter   4.1.1
com.apple.driver.AppleThunderboltUSBDownAdapter   1.0.4
com.apple.driver.AppleBTM   1.0.1
com.apple.driver.AppleSyntheticGameController   12.2.1
com.apple.driver.AppleDCPDPTXProxy   1.0.0
com.apple.driver.DCPDPFamilyProxy   1
com.apple.nke.ppp   1.9
com.apple.driver.AppleBSDKextStarter   3
com.apple.kext.triggers   1.0
com.apple.driver.IOHIDPowerSource   1
com.apple.driver.AppleCallbackPowerSource   1
com.apple.filesystems.hfs.encodings.kext   1
com.apple.driver.AppleStockholmControl   1.0.0
com.apple.driver.AppleDiagnosticDataAccessReadOnly   1.0.0
com.apple.AGXFirmwareKextG14GRTBuddy   1
com.apple.AGXFirmwareKextRTBuddy64   323.15
com.apple.driver.AppleSPMIPMU   1.0.1
com.apple.driver.AppleDialogPMU   1.0.1
com.apple.driver.AppleAOPAudio   420.2
com.apple.driver.AppleHPM   3.4.4
com.apple.iokit.IOMobileGraphicsFamily-DCP   343.0.0
com.apple.iokit.IOMobileGraphicsFamily   343.0.0
com.apple.driver.AppleDCP   1
com.apple.driver.AppleSPMI   1.0.1
com.apple.driver.AppleUSBXDCIARM   1.0
com.apple.driver.AppleUSBXDCI   1.0
com.apple.iokit.IOUSBDeviceFamily   2.0.0
com.apple.driver.AppleT8112TypeCPhy   1
com.apple.driver.AppleT8103TypeCPhy   1
com.apple.driver.usb.AppleSynopsysUSBXHCI   1
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.driver.AppleEmbeddedUSBHost   1
com.apple.driver.usb.AppleUSBHub   1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice   1.2
com.apple.driver.DCPAVFamilyProxy   1
com.apple.driver.AppleFirmwareKit   1
com.apple.driver.AppleM2ScalerCSCDriver   265.0.0
com.apple.driver.AppleMCA2-T8112   920.1
com.apple.iokit.IONVMeFamily   2.1.0
com.apple.driver.AppleNANDConfigAccess   1.0.0
com.apple.driver.AppleSPU   1
com.apple.driver.AppleSART   1
com.apple.driver.AppleH11ANEInterface   8.203.4
com.apple.iokit.IOGPUFamily   104.1.2
com.apple.driver.AppleCSEmbeddedAudio   820.18
com.apple.driver.AppleEmbeddedAudio   820.18
com.apple.iokit.AppleARMIISAudio   420.14
com.apple.driver.IISAudioIsolatedStreamECProxy   420.14
com.apple.driver.ExclavesAudioKext   220.24
com.apple.driver.AppleEmbeddedAudioLibs   420.3
com.apple.driver.AppleFirmwareUpdateKext   1
com.apple.driver.ApplePassthroughPPM   3.0
com.apple.driver.ApplePMGR   1
com.apple.driver.AppleA7IOP-ASCWrap-v4   1.0.2
com.apple.driver.AppleARMWatchdogTimer   1
com.apple.driver.AppleT8112PCIe   1
com.apple.driver.AppleA7IOP-MXWrap-v1   1.0.2
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.driver.AppleThunderboltNHI   7.2.81
com.apple.driver.AppleDisplayCrossbar   1.0.0
com.apple.iokit.IODisplayPortFamily   1.0.0
com.apple.driver.AppleTypeCPhy   1
com.apple.driver.AppleT8103PCIeC   1
com.apple.iokit.IOThunderboltFamily   9.3.3
com.apple.iokit.IOPortFamily   1.0
com.apple.driver.ApplePIODMA   1
com.apple.driver.AppleBluetoothDebugService   1
com.apple.driver.AppleBCMWLANCore   1.0.0
com.apple.iokit.IO80211Family   1200.13.1
com.apple.driver.IOImageLoader   1.0.0
com.apple.driver.AppleOLYHAL   1
com.apple.iokit.IOAVBFamily   1320.3
com.apple.plugin.IOgPTPPlugin   1320.8
com.apple.driver.AppleGPIOICController   1.0.2
com.apple.driver.AppleMobileApNonce   1
com.apple.driver.AppleFireStormErrorHandler   1
com.apple.driver.AppleMultiFunctionManager   1
com.apple.driver.corecapture   1.0.4
com.apple.driver.AppleEmbeddedPCIE   1
com.apple.iokit.IOTimeSyncFamily   1320.8
com.apple.driver.DiskImages   493.0.0
com.apple.iokit.IOGraphicsFamily   598
com.apple.iokit.IOBluetoothFamily   9.0.0
com.apple.driver.AppleUIO   1
com.apple.driver.AppleSSE   1.0
com.apple.driver.AppleSEPKeyStore   2
com.apple.driver.AppleUSBTDM   556
com.apple.iokit.IOUSBMassStorageDriver   250.60.2
com.apple.iokit.IOPCIFamily   2.9
com.apple.iokit.IOUSBHostFamily   1.2
com.apple.driver.AppleUSBHostMergeProperties   1.2
com.apple.driver.usb.AppleUSBCommon   1.0
com.apple.driver.AppleSMC   3.1.9
com.apple.driver.RTBuddy   1.0.0
com.apple.driver.AppleEmbeddedTempSensor   1.0.0
com.apple.driver.AppleARMPMU   1.0
com.apple.iokit.IOAccessoryManager   1.0.0
com.apple.driver.AppleOnboardSerial   1.0
com.apple.iokit.IOSerialFamily   11
com.apple.iokit.IOSCSIBlockCommandsDevice   500
com.apple.iokit.IOSCSIArchitectureModelFamily   500
com.apple.driver.AppleRSMChannel   1
com.apple.iokit.IORSMFamily   1
com.apple.driver.AppleLockdownMode   1
com.apple.driver.AppleIPAppender   1.0
com.apple.iokit.IOSkywalkFamily   1.0
com.apple.driver.mDNSOffloadUserClient   1.0.1b8
com.apple.iokit.IONetworkingFamily   3.4
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage   1.0
com.apple.driver.AppleCredentialManager   1.0
com.apple.driver.AppleSEPManager   1.0.1
com.apple.driver.IODARTFamily   1
com.apple.driver.AppleA7IOP   1.0.2
com.apple.driver.IOSlaveProcessor   1
com.apple.driver.AppleBiometricSensor   2
com.apple.iokit.IOHIDFamily   2.0.0
com.apple.AUC   1.0
com.apple.iokit.IOSurface   372.3.4
com.apple.iokit.IOAVFamily   1.0.0
com.apple.iokit.IOHDCPFamily   1.0.0
com.apple.iokit.IOCECFamily   1
com.apple.iokit.IOAudio2Family   1.0
com.apple.driver.AppleIISController   420.1
com.apple.driver.AppleAudioClockLibs   420.3
com.apple.driver.FairPlayIOKit   72.10.0
com.apple.driver.AppleARMPlatform   1.0.2
com.apple.iokit.IOSlowAdaptiveClockingFamily   1.0.0
com.apple.iokit.IOReportFamily   47
com.apple.security.quarantine   4
com.apple.security.sandbox   300.0
com.apple.iokit.IOStorageFamily   2.1
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.iokit.CoreAnalyticsFamily   1
com.apple.security.AppleImage4   7.0.0
com.apple.kext.CoreTrust   1
com.apple.iokit.IOCryptoAcceleratorFamily   1.0.1
com.apple.kec.pthread   1
com.apple.kec.Libm   1
com.apple.kec.Compression   1.0
com.apple.kec.corecrypto   14.0



** Stackshot Succeeded ** Bytes Traced 487412 (Uncompressed 1255744) **


[Sharko]

I think that the most relevant fact about your problem is that the pool is connected via USB. I'm still amazed at how poorly Apple handles the relationship between USB devices and the processor when the system is put to sleep.

My personal example for this is updating the songs and playlists on an iPad Nano 7th Generation. I connect it to my MacBook Pro via USB to update it. If I leave it connected too long because I walked away while the iPod was being updated, the laptop will go to sleep after a few minutes. When that happens I usually find that the iPod has been ungracefully ejected when I wake up the system. I suspect that something similar is happening with your system.

[hamiid]

Yea, I was suspecting that as well.
However the kernel panics happen during a fresh reboot of system and normal operations