I'm on a Mac Studio recently created a ZFS pool. For the most part, it works fine. But for some reason ZFS has trouble exporting pools. This manifests in two ways:
1) When exporting a pool via zfs export, the process often hangs without visible progress. It appears that it is stuck in a call to diskutil unmount while it hangs, but I wasn't able to figure out the details.
2) Often, the OS will hang during shutdown when a ZFS pool is still imported. On the next boot, macOS tells me that the computer was restarted because of a problem. The problem details are as follows:
- Code: Select all
panic(cpu 0 caller 0xfffffe00204fa9d4): watchdog timeout: no checkins from watchdogd in 139 seconds (20 totalcheckins since monitoring last enabled), shutdown in progress, vfs shutdown in progress, last vfs completion time: 124 seconds ago
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 22C65
Kernel version: Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 1E7F9A3BC6E4784ABF02268D42826B23
Kernel UUID: 02030AF5-8978-32DB-9D21-A487B7DB895F
Boot session UUID: E8520E6D-2F8A-400D-9895-978BCA4A148C
iBoot version: iBoot-8419.60.44
secure boot?: YES
roots installed: 0
Paniclog version: 14
KernelCache slide: 0x000000001770c000
KernelCache base: 0xfffffe001e710000
Kernel slide: 0x0000000018970000
Kernel text base: 0xfffffe001f974000
Kernel text exec slide: 0x0000000018a5c000
Kernel text exec base: 0xfffffe001fa60000
mach_absolute_time: 0x5b515cec7e
Epoch Time: sec usec
Boot : 0x63a45f51 0x0006fb25
Sleep : 0x63a4d4ce 0x000b8b0f
Wake : 0x63a4d862 0x0006d0ce
Calendar: 0x63a4da1f 0x00098418
Zone info:
Zone map: 0xfffffe1000664000 - 0xfffffe3000664000
. VM : 0xfffffe1000664000 - 0xfffffe14cd330000
. RO : 0xfffffe14cd330000 - 0xfffffe1666cc8000
. GEN0 : 0xfffffe1666cc8000 - 0xfffffe1b33994000
. GEN1 : 0xfffffe1b33994000 - 0xfffffe2000660000
. GEN2 : 0xfffffe2000660000 - 0xfffffe24cd32c000
. GEN3 : 0xfffffe24cd32c000 - 0xfffffe2999ff8000
. DATA : 0xfffffe2999ff8000 - 0xfffffe3000664000
Metadata: 0xfffffe4f95e9c000 - 0xfffffe4f9de9c000
Bitmaps : 0xfffffe4f9de9c000 - 0xfffffe4fa993c000
TPIDRx_ELy = {1: 0xfffffe24cd2d1030 0: 0x0000000000000000 0ro: 0x0000000000000000 }
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 0 is the one that panicked. Check the full backtrace for details.
CORE 1: PC=0xfffffe001fafcfac, LR=0xfffffe001fafcfac, FP=0xfffffe5a678cfda0
CORE 2: PC=0xfffffe001fafc1b4, LR=0xfffffe001fafc1b4, FP=0xfffffe5a68533f00
CORE 3: PC=0xfffffe001fafc1b4, LR=0xfffffe001fafc1b4, FP=0xfffffe5a680b3f00
CORE 4: PC=0xfffffe001fafc1b4, LR=0xfffffe001fafc1b4, FP=0xfffffe5a68c1ff00
CORE 5: PC=0xfffffe001fafc1b4, LR=0xfffffe001fafc1b4, FP=0xfffffe5a67f93f00
CORE 6: PC=0xfffffe001fafc1b4, LR=0xfffffe001fafc1b4, FP=0xfffffe5a68dc3f00
CORE 7: PC=0xfffffe001fafc1b4, LR=0xfffffe001fafc1b4, FP=0xfffffe5a68b53f00
CORE 8: PC=0xfffffe001fafc1b4, LR=0xfffffe001fafc1b4, FP=0xfffffe5a67d8bf00
CORE 9: PC=0xfffffe001fafc1b4, LR=0xfffffe001fafc1b4, FP=0xfffffe5a661c7f00
Compressor Info: 0% of compressed pages limit (OK) and 4% of segments limit (OK) with 0 swapfiles and OK swap space
Total cpu_usage: 11120382
Thread task pri cpu_usage
0xfffffe24cd2d1828 kernel_task 95 1196
0xfffffe24cd2d1030 kernel_task 0 3852061
0xfffffe24ccf0a020 kernel_task 0 4049871
0xfffffe24cce03808 kernel_task 0 0
0xfffffe24cce01030 kernel_task 0 0
Panicked task 0xfffffe24cd40d660: 0 pages, 775 threads: pid 0: kernel_task
Panicked thread: 0xfffffe24cd2d1030, backtrace: 0xfffffe4fafa7f840, tid: 102
lr: 0xfffffe001fabec14 fp: 0xfffffe4fafa7f8b0
lr: 0xfffffe001fabe9bc fp: 0xfffffe4fafa7f930
lr: 0xfffffe001fc23780 fp: 0xfffffe4fafa7f950
lr: 0xfffffe001fc14cf0 fp: 0xfffffe4fafa7f9c0
lr: 0xfffffe001fc125f4 fp: 0xfffffe4fafa7fa80
lr: 0xfffffe001fa677f8 fp: 0xfffffe4fafa7fa90
lr: 0xfffffe001fabe344 fp: 0xfffffe4fafa7fe40
lr: 0xfffffe0020304060 fp: 0xfffffe4fafa7fe60
lr: 0xfffffe00204fa9d4 fp: 0xfffffe4fafa7fed0
lr: 0xfffffe00204fa7f8 fp: 0xfffffe4fafa7ff10
lr: 0xfffffe00204f7e94 fp: 0xfffffe4fafa7ff30
lr: 0xfffffe0020d4c720 fp: 0xfffffe4fafa7ffd0
lr: 0xfffffe001fc1591c fp: 0xfffffe4fafa7ffe0
lr: 0xfffffe001fa6786c fp: 0xfffffe4fafa7fff0
lr: 0xfffffe001fafc1b4 fp: 0xfffffe5a6856ff00
lr: 0xfffffe001fafc3d4 fp: 0xfffffe5a6856ff20
lr: 0xfffffe001fa70e98 fp: 0x0000000000000000
Kernel Extensions in backtrace:
com.apple.driver.AppleInterruptControllerV2(1.0d1)[9673AF56-65B1-373F-8030-A240168F7681]@0xfffffe0020d4ac10->0xfffffe0020d4d42f
dependency: com.apple.driver.AppleARMPlatform(1.0.2)[00A0000C-A4BD-3590-A973-61FA2E8A1D63]@0xfffffe00204a8390->0xfffffe00204f68ff
com.apple.driver.AppleARMWatchdogTimer(1.0)[F9C14BA5-D791-362B-ADCC-9FA6782CB310]@0xfffffe00204f6900->0xfffffe00204fb6b7
dependency: com.apple.driver.AppleARMPlatform(1.0.2)[00A0000C-A4BD-3590-A973-61FA2E8A1D63]@0xfffffe00204a8390->0xfffffe00204f68ff
last started kext at 29612188776: com.apple.filesystems.autofs 3.0 (addr 0xfffffe001eee9d40, size 5904)
last stopped kext at 3250690457: com.jmicron.driver.jmPeripheralDevice 2.0.13 (addr 0xfffffe001e2e52f0, size 6745)
loaded kexts:
com.binaryfruit.driver.SATSMARTDriver 0.10.3
org.openzfsonosx.zfs 2.1.6
com.apple.filesystems.autofs 3.0
com.apple.driver.AppleALSColorSensor 1.0.0d1
com.apple.driver.AppleTopCaseHIDEventDriver 6000.22
com.apple.iokit.SCSITaskUserClient 476
com.apple.driver.AppleUSBMassStorageInterfaceNub 547
com.apple.AppleEthernetAquantiaAqtionFirmware 1.0.36
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL 1
com.apple.driver.usb.AppleUSBHostBillboardDevice 1.0
com.apple.driver.BCMWLANFirmware4387.Hashstore 1
com.apple.driver.DiskImages.ReadWriteDiskImage 493.0.0
com.apple.driver.DiskImages.UDIFDiskImage 493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore 493.0.0
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleUSBDeviceNCM 5.0.0
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.nke.l2tp 1.9
com.apple.filesystems.tmpfs 1
com.apple.driver.AppleGL3590FirmwareUpdater 1
com.apple.filesystems.nfs 1
com.apple.filesystems.lifs 1
com.apple.filesystems.apfs 2142.61.2
com.apple.IOTextEncryptionFamily 1.0.0
com.apple.filesystems.hfs.kext 627.40.1
com.apple.security.BootPolicy 1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.driver.ApplePMP 1
com.apple.driver.AppleSmartIO2 1
com.apple.driver.AppleCS42L84Audio 600.99
com.apple.AppleEmbeddedSimpleSPINORFlasher 1
com.apple.driver.AppleSN012776Amp 600.99
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6000CLPCv3 1
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleAVE2 640.4.1
com.apple.driver.AppleAVD 617
com.apple.AGXG13X 227.2.45
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleS5L8920XPWM 1.0.0d1
com.apple.driver.AppleAOPVoiceTrigger 200.4
com.apple.driver.ApplePMPFirmware 1
com.apple.driver.AppleProResHW 226.0.0
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AudioDMAController-T600x 220.1
com.apple.driver.AppleSPMIPMU 1.0.1
com.apple.driver.AppleMobileDispT600X-DCP 140.0
com.apple.driver.AppleJPEGDriver 5.2.6
com.apple.driver.AppleSDXC 3.3.0
com.apple.driver.AppleSerialShim 1
com.apple.driver.AppleEventLogHandler 1
com.apple.driver.AppleS5L8960XNCO 1
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleS8000AES 1
com.apple.driver.AppleS8000DWI 1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule 1
com.apple.driver.AppleBCMWLANBusInterfacePCIe 1
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleS5L8940XI2C 1.0.0d2
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons 1.0.0d1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.usb.AppleUSBUserHCI 1
com.apple.iokit.IOKitRegistryCompatibility 1
com.apple.iokit.EndpointSecurity 1
com.apple.driver.AppleDiskImages2 198.40.3
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall 403
com.apple.kec.InvalidateHmac 1
com.apple.kec.Compression 1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial 6.0.0
com.apple.driver.AppleMultitouchDriver 6400.44
com.apple.driver.AppleInputDeviceSupport 6400.40
com.apple.driver.AppleHSBluetoothDriver 6000.22
com.apple.driver.IOBluetoothHIDDriver 9.0.0
com.apple.driver.AppleHIDKeyboard 231
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.iokit.IOSCSIMultimediaCommandsDevice 476
com.apple.iokit.IOBDStorageFamily 1.8
com.apple.iokit.IODVDStorageFamily 1.8
com.apple.iokit.IOCDStorageFamily 1.8
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.AppleUSBAudio 500.2
com.apple.iokit.IOAudioFamily 400.3
com.apple.vecLib.kext 1.2.0
com.apple.iokit.IOAVBFamily 1120.2
com.apple.plugin.IOgPTPPlugin 1110.13
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily 1
com.apple.driver.AppleEthernetAquantiaAqtion 1.0.64
com.apple.driver.DiskImages.KernelBacked 493.0.0
com.apple.driver.AppleSEPHDCPManager 1.0.1
com.apple.driver.AppleTrustedAccessory 1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI 1
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleConvergedIPCOLYBTControl 1
com.apple.driver.AppleConvergedPCI 1
com.apple.driver.AppleBluetoothDebug 1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleThunderboltDPInAdapter 8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily 8.5.1
com.apple.nke.ppp 1.9
com.apple.driver.AppleBSDKextStarter 3
com.apple.kext.triggers 1.0
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleSyntheticGameController 10.3.5
com.apple.driver.AppleBTM 1.0.1
com.apple.driver.IOHIDPowerSource 1
com.apple.driver.AppleHPM 3.4.4
com.apple.driver.AppleDCPDPTXProxy 1.0.0
com.apple.driver.DCPDPFamilyProxy 1
com.apple.driver.AppleCSEmbeddedAudio 600.99
com.apple.driver.AppleI2CEthernetAquantia 1.0.0
com.apple.driver.AppleDiagnosticDataAccessReadOnly 1.0.0
com.apple.driver.ApplePassthroughPPM 3.0
com.apple.AGXFirmwareKextG13XRTBuddy 227.2.45
com.apple.AGXFirmwareKextRTBuddy64 227.2.45
com.apple.iokit.IOGPUFamily 65.0.22
com.apple.driver.AppleStockholmControl 1.0.0
com.apple.driver.AppleMCA2-T600x 720.1
com.apple.driver.AppleUSBXDCIARM 1.0
com.apple.driver.AppleUSBXDCI 1.0
com.apple.iokit.IOUSBDeviceFamily 2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI 1
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.driver.AppleEmbeddedUSBHost 1
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.AppleAOPAudio 200.45
com.apple.driver.AppleEmbeddedAudio 600.99
com.apple.iokit.AppleARMIISAudio 200.8
com.apple.driver.AppleEmbeddedAudioLibs 200.5
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleSPMI 1.0.1
com.apple.iokit.IOMobileGraphicsFamily-DCP 343.0.0
com.apple.iokit.IOMobileGraphicsFamily 343.0.0
com.apple.driver.AppleSPU 1
com.apple.driver.DCPAVFamilyProxy 1
com.apple.driver.AppleDCP 1
com.apple.driver.AppleFirmwareKit 1
com.apple.driver.AppleFirmwareUpdateKext 1
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.AppleNANDConfigAccess 1.0.0
com.apple.driver.AppleSART 1
com.apple.driver.ApplePMGR 1
com.apple.driver.AppleARMWatchdogTimer 1
com.apple.driver.AppleDisplayCrossbar 1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy 1
com.apple.driver.AppleThunderboltNHI 7.2.81
com.apple.driver.AppleT6000PCIeC 1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA 1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore 1.0.0
com.apple.iokit.IO80211Family 1200.13.0
com.apple.driver.IOImageLoader 1.0.0
com.apple.driver.AppleOLYHAL 1
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleGPIOICController 1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMultiFunctionManager 1
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleEmbeddedPCIE 1
com.apple.driver.AppleMobileApNonce 1
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOTimeSyncFamily 1110.13
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily 597
com.apple.iokit.IOBluetoothSerialManager 9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport 9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily 9.0.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE 1.0
com.apple.driver.AppleSEPKeyStore 2
com.apple.driver.AppleUSBTDM 547
com.apple.iokit.IOUSBMassStorageDriver 232
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice 476
com.apple.iokit.IOSCSIArchitectureModelFamily 476
com.apple.driver.AppleRSMChannel 1
com.apple.iokit.IORSMFamily 1
com.apple.driver.AppleIPAppender 1.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.RTBuddy 1.0.0
com.apple.driver.AppleEmbeddedTempSensor 1.0.0
com.apple.driver.AppleARMPMU 1.0
com.apple.iokit.IOAccessoryManager 1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.4
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor 1
com.apple.driver.AppleBiometricSensor 2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.driver.AppleANELoadBalancer 6.201.2
com.apple.driver.AppleH11ANEInterface 6.201.0
com.apple.AUC 1.0
com.apple.iokit.IOAVFamily 1.0.0
com.apple.iokit.IOHDCPFamily 1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family 1.0
com.apple.driver.AppleIISController 200.2
com.apple.driver.AppleAudioClockLibs 200.5
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface 334.0.1
com.apple.driver.IODARTFamily 1
com.apple.driver.FairPlayIOKit 68.19.0
com.apple.driver.AppleARMPlatform 1.0.2
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOReportFamily 47
com.apple.security.quarantine 4
com.apple.security.sandbox 300.0
com.apple.iokit.IOStorageFamily 2.1
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.security.AppleImage4 5.0.0
com.apple.kext.CoreTrust 1
com.apple.iokit.IOCryptoAcceleratorFamily 1.0.1
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 12.0
** Stackshot Succeeded ** Bytes Traced 71637 (Uncompressed 276000) **
This issue never appears when the ZFS pool is exported before shutdown and only sometimes appears when a ZFS pool is still imported. However, the issue seems to mainly manifest when the pool has been imported for a few hours.
Now for my setup: I have a mac Studio M1 Max with 32 GB of memory. The ZFS version is zfs-macOS-2.1.6-1/zfs-kmod-2.1.6-1. I use a mirrored ZFS pool that runs on two WD Red Plus 6 TB hard drives in a QNAP TR-002 USB-C enclosure. The only configuration change I've made to ZFS was setting kstat.zfs.darwin.tunable.zfs_arc.max=16106127360 so ZFS wouldn't crash during boot.
Does this behavior make sense to anyone? Is there some obvious thing I'm missing? I'll admit that I'm new to operating ZFS so I guess me missing something is quite likely.