Another 1.9.4 crash

Developer discussions.

Another 1.9.4 crash

Postby Sharko » Tue Nov 01, 2022 8:54 pm

Again, probably not too useful, but I consistently get a crash while working with this one pool, trying to do a raw send of a snapshot in the middle of a lot other snapshots built up over time (I was trying to do a replicate of a backup drive, and maintain the same encryption key):

Code: Select all
Anonymous UUID:       44F79C15-D3FC-74F8-04A1-7F6F7A246E32

Tue Nov  1 21:48:06 2022

*** Panic Report ***
panic(cpu 1 caller 0xffffff7f8be81f8d): VERIFY3( size != 0 ) failed (0 != 0)

Backtrace (CPU 1), Frame : Return Address
0xffffffa4132b38a0 : 0xffffff8009fad5cd mach_kernel : _handle_debugger_trap + 0x47d
0xffffffa4132b38f0 : 0xffffff800a0e9245 mach_kernel : _kdp_i386_trap + 0x155
0xffffffa4132b3930 : 0xffffff800a0da97a mach_kernel : _kernel_trap + 0x50a
0xffffffa4132b39a0 : 0xffffff8009f5a9d0 mach_kernel : _return_from_trap + 0xe0
0xffffffa4132b39c0 : 0xffffff8009facfe7 mach_kernel : _panic_trap_to_debugger + 0x197
0xffffffa4132b3ae0 : 0xffffff8009face33 mach_kernel : _panic + 0x63
0xffffffa4132b3b50 : 0xffffff7f8be81f8d net.lundman.zfs : _range_tree_remove_impl + 0x43
0xffffffa4132b3bf0 : 0xffffff7f8be7e356 net.lundman.zfs : _metaslab_alloc_dva + 0xa4e
0xffffffa4132b3d00 : 0xffffff7f8be7f59c net.lundman.zfs : _metaslab_alloc + 0xa2
0xffffffa4132b3d90 : 0xffffff7f8beffd1d net.lundman.zfs : _zio_dva_allocate + 0xf5
0xffffffa4132b3eb0 : 0xffffff7f8befd1d8 net.lundman.zfs : ___zio_execute + 0xf3
0xffffffa4132b3f10 : 0xffffff7f8ac0bf6f net.lundman.spl : _taskq_thread + 0x1b9
0xffffffa4132b3fa0 : 0xffffff8009f5a0ce mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         net.lundman.spl(1.9.4)[E8040162-4851-3943-ABB3-05BCA6FCCFC6]@0xffffff7f8abfe000->0xffffff7f8bdf2fff
         net.lundman.zfs(1.9.4)[413FC73C-7660-397A-A033-0EC2DFD09337]@0xffffff7f8be23000->0xffffff7f8c1dcfff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[E28FB7B6-F78E-340D-91AC-7B78EC5F7D8D]@0xffffff7f8bdf3000
            dependency: net.lundman.spl(1.9.4)[E8040162-4851-3943-ABB3-05BCA6FCCFC6]@0xffffff7f8abfe000

BSD process name corresponding to current thread: kernel_task
Boot args: keepsyms=y cwae=2

Mac OS version:
18G9323

Kernel version:
Darwin Kernel Version 18.7.0: Tue Jun 22 19:37:08 PDT 2021; root:xnu-4903.278.70~1/RELEASE_X86_64
Kernel UUID: 041B6A6D-CD16-36FA-88B9-E32FF46EF89F
Kernel slide:     0x0000000009c00000
Kernel text base: 0xffffff8009e00000
__HIB  text base: 0xffffff8009d00000
System model name: MacPro5,1 (Mac-F221BEC8)

System uptime in nanoseconds: 50117379321510
last loaded kext at 49889106100051: com.apple.driver.AppleXsanScheme   3 (addr 0xffffff7f9072c000, size 32768)
last unloaded kext at 49715085722866: com.apple.driver.usb.cdc   5.0.0 (addr 0xffffff7f9072c000, size 28672)
loaded kexts:
org.virtualbox.kext.VBoxNetAdp   6.1.40
org.virtualbox.kext.VBoxNetFlt   6.1.40
org.virtualbox.kext.VBoxUSB   6.1.40
org.virtualbox.kext.VBoxDrv   6.1.40
com.Cycling74.driver.Soundflower   1.6.7
org.dungeon.driver.SATSMARTDriver   0.8.1
net.lundman.zfs   1.9.4
net.lundman.spl   1.9.4
at.obdev.nke.LittleSnitch   5474
com.apple.driver.AppleBluetoothMultitouch   96
com.apple.fileutil   20.036.15
com.apple.driver.AppleUpstreamUserClient   3.6.5
com.apple.driver.AppleMCCSControl   1.5.9
com.apple.kext.AMDFramebuffer   2.1.1
com.apple.filesystems.autofs   3.0
com.apple.driver.AGPM   110.25.11
com.apple.driver.AppleMikeyHIDDriver   131
com.apple.driver.AppleMikeyDriver   282.54
com.apple.kext.AMDRadeonX4000   2.1.1
com.apple.driver.AppleHDA   282.54
com.apple.driver.AppleGraphicsDevicePolicy   3.50.16
com.apple.AGDCPluginDisplayMetrics   3.50.16
com.apple.driver.AppleHV   1
com.apple.iokit.IOUserEthernet   1.0.1
com.apple.iokit.IOBluetoothSerialManager   6.0.14d13
com.apple.kext.AMD9500Controller   2.1.1
com.apple.driver.pmtelemetry   1
com.apple.Dont_Steal_Mac_OS_X   7.0.0
com.apple.driver.AppleOSXWatchdog   1
com.apple.driver.ACPI_SMC_PlatformPlugin   1.0.0
com.apple.driver.AppleGFXHDA   100.1.414
com.apple.driver.AppleLPC   3.1
com.apple.driver.AppleIntelSlowAdaptiveClocking   4.0.0
com.apple.driver.AudioAUUC   1.70
com.apple.driver.AppleUSBDisplays   380
com.apple.filesystems.apfs   945.275.10
com.apple.driver.AppleVirtIO   2.1.3
com.apple.filesystems.hfs.kext   407.200.4
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless   1.0.0d1
com.apple.BootCache   40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib   1.0.0
com.apple.AppleSystemPolicy   1.0
com.apple.iokit.SCSITaskUserClient   408.250.3
com.apple.driver.AppleFWOHCI   5.6.0
com.apple.driver.Intel82574LEthernet   2.7.2
com.apple.driver.AirPort.BrcmNIC   1400.1.1
com.apple.private.KextAudit   1.0
com.apple.driver.AppleAHCIPort   329.260.5
com.apple.driver.AppleRTC   2.0
com.apple.driver.AppleHPET   1.8
com.apple.driver.AppleACPIButtons   6.1
com.apple.driver.AppleSMBIOS   2.1
com.apple.driver.AppleACPIEC   6.1
com.apple.driver.AppleAPIC   1.7
com.apple.driver.AppleIntelCPUPowerManagementClient   220.0.0
com.apple.nke.applicationfirewall   202
com.apple.security.TMSafetyNet   8
com.apple.driver.AppleIntelCPUPowerManagement   220.0.0
com.apple.driver.AppleXsanScheme   3
com.apple.driver.IOBluetoothHIDDriver   6.0.14d13
com.apple.driver.AppleMultitouchDriver   2450.1
com.apple.driver.AppleInputDeviceSupport   2440.4
com.apple.iokit.IOUSBUserClient   900.4.2
com.apple.kext.triggers   1.0
com.apple.kext.AMDRadeonX4000HWLibs   1.0
com.apple.iokit.IOAcceleratorFamily2   404.15
com.apple.kext.AMDRadeonX4000HWServices   2.1.1
com.apple.driver.DspFuncLib   282.54
com.apple.kext.OSvKernDSPLib   528
com.apple.driver.AppleGraphicsControl   3.50.16
com.apple.iokit.IOAVBFamily   760.6
com.apple.plugin.IOgPTPPlugin   740.2
com.apple.iokit.IOEthernetAVBController   1.1.0
com.apple.iokit.IOSkywalkFamily   1
com.apple.driver.AppleSSE   1.0
com.apple.iokit.IOSurface   255.6.1
com.apple.driver.AppleSMBusController   1.0.18d1
com.apple.kext.AMDSupport   2.1.1
com.apple.driver.AppleSMBusPCI   1.0.14d1
com.apple.driver.IOPlatformPluginLegacy   1.0.0
com.apple.driver.AppleHDAController   282.54
com.apple.iokit.IOHDAFamily   282.54
com.apple.iokit.IONDRVSupport   530.51
com.apple.driver.IOPlatformPluginFamily   6.0.0d8
com.apple.iokit.IOFireWireIP   2.3.0
com.apple.AppleGPUWrangler   3.50.16
com.apple.AppleGraphicsDeviceControl   3.50.16
com.apple.iokit.IOSlowAdaptiveClockingFamily   1.0.0
com.apple.iokit.IOGraphicsFamily   530.68
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport   6.0.14d13
com.apple.iokit.IOBluetoothHostControllerUSBTransport   6.0.14d13
com.apple.iokit.IOBluetoothHostControllerTransport   6.0.14d13
com.apple.iokit.IOBluetoothFamily   6.0.14d13
com.apple.driver.AppleHIDKeyboard   208
com.apple.iokit.IOAHCIBlockStorage   301.270.1
com.apple.driver.usb.AppleUSBHub   1.2
com.apple.driver.AppleUSBAudio   315.6
com.apple.driver.usb.IOUSBHostHIDDevice   1.2
com.apple.iokit.IOAudioFamily   206.5
com.apple.vecLib.kext   1.2.0
com.apple.driver.usb.networking   5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice   1.2
com.apple.iokit.IOSerialFamily   11
com.apple.filesystems.hfs.encodings.kext   1
com.apple.iokit.IOSCSIMultimediaCommandsDevice   408.250.3
com.apple.iokit.IOBDStorageFamily   1.8
com.apple.iokit.IODVDStorageFamily   1.8
com.apple.iokit.IOCDStorageFamily   1.8
com.apple.iokit.IOFireWireFamily   4.7.3
com.apple.iokit.IOAHCISerialATAPI   267.50.1
com.apple.iokit.IO80211Family   1200.12.2
com.apple.driver.mDNSOffloadUserClient   1.0.1b8
com.apple.driver.corecapture   1.0.4
com.apple.driver.usb.AppleUSBEHCIPCI   1.2
com.apple.driver.usb.AppleUSBUHCIPCI   1.2
com.apple.driver.usb.AppleUSBUHCI   1.2
com.apple.driver.usb.AppleUSBEHCI   1.2
com.apple.driver.usb.AppleUSBXHCIPCI   1.2
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.iokit.IOAHCIFamily   288
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.iokit.IOUSBFamily   900.4.2
com.apple.driver.AppleUSBHostMergeProperties   1.2
com.apple.driver.AppleEFINVRAM   2.1
com.apple.driver.AppleEFIRuntime   2.1
com.apple.iokit.IOSMBusFamily   1.1
com.apple.iokit.IOHIDFamily   2.0.0
com.apple.security.quarantine   3
com.apple.security.sandbox   300.0
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.DiskImages   493.0.0
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage   1.0
com.apple.driver.AppleKeyStore   2
com.apple.driver.AppleUSBTDM   456.260.3
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.iokit.IOUSBMassStorageDriver   145.200.2
com.apple.iokit.IOSCSIBlockCommandsDevice   408.250.3
com.apple.iokit.IOSCSIArchitectureModelFamily   408.250.3
com.apple.iokit.IOStorageFamily   2.1
com.apple.kext.CoreTrust   1
com.apple.driver.AppleCredentialManager   1.0
com.apple.driver.KernelRelayHost   1
com.apple.iokit.IOUSBHostFamily   1.2
com.apple.driver.usb.AppleUSBCommon   1.0
com.apple.driver.AppleBusPowerController   1.0
com.apple.driver.AppleSEPManager   1.0.1
com.apple.driver.IOSlaveProcessor   1
com.apple.iokit.IOReportFamily   47
com.apple.iokit.IOTimeSyncFamily   740.2
com.apple.iokit.IONetworkingFamily   3.4
com.apple.driver.AppleACPIPlatform   6.1
com.apple.driver.AppleSMC   3.1.9
com.apple.iokit.IOPCIFamily   2.9
com.apple.iokit.IOACPIFamily   1.4
com.apple.kec.pthread   1
com.apple.kec.corecrypto   1.0
com.apple.kec.Libm   1

EOF
Model: MacPro5,1, BootROM 144.0.0.0.0, 6 processors, 6-Core Intel Xeon, 3.46 GHz, 32 GB, SMC 1.39f11
Graphics: Radeon RX 560, Radeon RX 560, spdisplays_pcie_device, 4 GB
Memory Module: DIMM 1, 8 GB, DDR3 ECC, 1333 MHz, 0x85F7, 0x463732314755363547393333334700520000
Memory Module: DIMM 2, 8 GB, DDR3 ECC, 1333 MHz, 0x85F7, 0x463732314755363547393333334700520000
Memory Module: DIMM 3, 8 GB, DDR3 ECC, 1333 MHz, 0x85F7, 0x463732314755363547393333334700520000
Memory Module: DIMM 4, 8 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342314B37304448302D5948392020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x111), Broadcom BCM43xx 1.0 (7.77.61.3 AirPortDriverBrcmNIC-1305.12)
Bluetooth: Version 6.0.14d13, 3 services, 27 devices, 1 incoming serial ports
Network Service: Ethernet 1, Ethernet, en0
Network Service: Ethernet 2, Ethernet, en1
Network Service: Wi-Fi, AirPort, en2
PCI Card: pci1b21,612, AHCI Controller, Slot-2
PCI Card: Radeon RX 560, Display Controller, Slot-1
PCI Card: pci1002,aae0, Audio Device, Slot-1
PCI Card: PXS3, USB eXtensible Host Controller, Slot-3
PCI Card: pci1b21,625, AHCI Controller, Slot-4
Serial ATA Device: HL-DT-ST DVD-RW GH61N
Serial ATA Device: SanDisk SDSSDH32000G, 2 TB
Serial ATA Device: Micron_5100_MTFDDAK960TCB, 960.2 GB
Serial ATA Device: Micron_5100_MTFDDAK960TCB, 960.2 GB
Serial ATA Device: SPCC Solid State Disk, 120.03 GB
USB Device: USB 2.0 Bus
USB Device: USB 2.0 Bus
USB Device: Hub
USB Device: Keyboard Hub
USB Device: Apple Keyboard
USB Device: Hub
USB Device: Apple LED Cinema Display
USB Device: Display iSight
USB Device: Display Audio
USB Device: USB 3.0 Bus
USB Device: Scarlett 2i2 USB
USB Device: USB Bus
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB Bus
USB Device: Beats Flex
FireWire Device: built-in_hub, Up to 800 Mb/sec
Thunderbolt Bus:


Is this something that recover=1 can overcome? And is that something that I put on the command line when I import the pool? Can't quite remember...

The pool passes with a clean scrub, by the way.
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Another 1.9.4 crash

Postby lundman » Tue Nov 01, 2022 9:39 pm

Not seen that one before, so possibly not something zfs_recover can handle. You set it with sysctl.

Do something like:
sysctl -a | grep recover

To find what its called, then using that name:

sysctl kstat.zfs.darwin.tunable.zfs.recover=1

then import.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Another 1.9.4 crash

Postby Sharko » Tue Nov 01, 2022 10:07 pm

Thanks for your reminders on how to use recover. I'm just trying to switch over to the latest RC4 candidate under Monterey, but I like to have lots of duplicate backups in case things go south and I have to either go back to Mojave or start over in Monterey.
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Another 1.9.4 crash

Postby lundman » Tue Nov 01, 2022 10:31 pm

You can certainly try import with -N and -o readonly, then just copy data off 1.9.4 - either rsync or zfs send.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan


Return to OpenZFS on OS X Development

Who is online

Users browsing this forum: No registered users and 19 guests