2.1.0rc1 Kernel Panics during ZFS Receive

[spencerh]

I installed the 2.1.0rc1 build on a Catalina VM to see if it resolved my issues with send/receive. Kernel panic report follows:

Code: Select all

panic(cpu 1 caller 0xffffff801f248a5a): Kernel trap at 0xffffff801f241343, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000008, CR3: 0x000000018b121000, CR4: 0x00000000000406e0
RAX: 0x0000000000000000, RBX: 0x0000000000000000, RCX: 0x0000000000000000, RDX: 0x0000000001000000
RSP: 0xffffff812e7fb9a8, RBP: 0xffffff812e7fb9c0, RSI: 0x0000000000000000, RDI: 0x0000000000000000
R8:  0x0000000000000006, R9:  0x0000000000000000, R10: 0x0000000000000000, R11: 0x0000000000000001
R12: 0xffffff914a6e6180, R13: 0xffffff914a6e6160, R14: 0x0000000000000000, R15: 0xffffff914a6e5b00
RFL: 0x0000000000010282, RIP: 0xffffff801f241343, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000008, Error code: 0x0000000000000000, Fault CPU: 0x1 VMM, PL: 0, VF: 0

Backtrace (CPU 1), Frame : Return Address
0xffffff812e7fb400 : 0xffffff801f11c63d mach_kernel : _handle_debugger_trap + 0x49d
0xffffff812e7fb450 : 0xffffff801f256b25 mach_kernel : _kdp_i386_trap + 0x155
0xffffff812e7fb490 : 0xffffff801f2486ae mach_kernel : _kernel_trap + 0x4ee
0xffffff812e7fb4e0 : 0xffffff801f0c2a40 mach_kernel : _return_from_trap + 0xe0
0xffffff812e7fb500 : 0xffffff801f11bd07 mach_kernel : _DebuggerTrapWithState + 0x17
0xffffff812e7fb600 : 0xffffff801f11c0f7 mach_kernel : _panic_trap_to_debugger + 0x227
0xffffff812e7fb650 : 0xffffff801f8c0b3c mach_kernel : _panic + 0x54
0xffffff812e7fb6c0 : 0xffffff801f248a5a mach_kernel : _sync_iss_to_iks + 0x2aa
0xffffff812e7fb840 : 0xffffff801f248758 mach_kernel : _kernel_trap + 0x598
0xffffff812e7fb890 : 0xffffff801f0c2a40 mach_kernel : _return_from_trap + 0xe0
0xffffff812e7fb8b0 : 0xffffff801f241343 mach_kernel : _lck_mtx_lock_spin_always + 0x3
0xffffff812e7fb9c0 : 0xffffff7fa1a577cb org.openzfsonosx.zfs : _zfs_resume_fs + 0x25b
0xffffff812e7fba00 : 0xffffff7fa1a455f5 org.openzfsonosx.zfs : _zfs_ioc_rollback + 0xf5
0xffffff812e7fba40 : 0xffffff7fa1a42136 org.openzfsonosx.zfs : _zfsdev_ioctl_common + 0x596
0xffffff812e7fbb10 : 0xffffff7fa1a4c459 org.openzfsonosx.zfs : _zfsdev_ioctl + 0x69
0xffffff812e7fbb50 : 0xffffff801f3bb2b2 mach_kernel : _spec_ioctl + 0xb2
0xffffff812e7fbb80 : 0xffffff801f3af560 mach_kernel : _VNOP_IOCTL + 0xf0
0xffffff812e7fbc00 : 0xffffff801f3a086e mach_kernel : _utf8_normalizestr + 0xd2e
0xffffff812e7fbe00 : 0xffffff801f6489cb mach_kernel : _fo_ioctl + 0x7b
0xffffff812e7fbe30 : 0xffffff801f6a11e2 mach_kernel : _ioctl + 0x512
0xffffff812e7fbf40 : 0xffffff801f783cb7 mach_kernel : _unix_syscall64 + 0x287
0xffffff812e7fbfa0 : 0xffffff801f0c3206 mach_kernel : _hndl_unix_scall64 + 0x16
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.1)[C35DC396-3FF5-31A0-AB1F-6F4D327ADEB9]@0xffffff7fa1948000->0xffffff7fa30f9fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[E5EEA957-1FBA-354A-A334-78116167B91E]@0xffffff7f9fa65000

BSD process name corresponding to current thread: zfs
Boot args: keepsyms=1

Mac OS version:
19H1217

Kernel version:
Darwin Kernel Version 19.6.0: Thu May  6 00:48:39 PDT 2021; root:xnu-6153.141.33~1/RELEASE_X86_64
Kernel UUID: 0B6CD26B-B526-3646-9B69-02D22BBE9C52
Kernel slide:     0x000000001ee00000
Kernel text base: 0xffffff801f000000
__HIB  text base: 0xffffff801ef00000
System model name: iMacPro1,1 (Mac-7BA5B2D9E42DDD94)
System shutdown begun: NO
Panic diags file available: NO (0xe00002bc)

System uptime in nanoseconds: 641042899121
last loaded kext at 41783181084: @filesystems.smbfs   3.4.4 (addr 0xffffff7fa0bfc000, size 454656)
last unloaded kext at 324283014274: >usb.!UHostPacketFilter   1.0 (addr 0xffffff7fa0429000, size 24576)
loaded kexts:
org.openzfsonosx.zfs   2.1.0
as.vit9696.!AALC   1.5.1
as.vit9696.WhateverGreen   1.4.1
as.vit9696.Lilu   1.4.5
@filesystems.smbfs   3.4.4
@fileutil   20.036.15
>X86PlatformShim   1.0.0
>AGPM   111.4.4
>!APlatformEnabler   2.7.0d0
@filesystems.autofs   3.0
@AGDCPluginDisplayMetrics   5.2.7
>!AHV   1
|IOUserEthernet   1.0.1
>!AUpstreamUserClient   3.6.8
>!AMCCSControl   1.14
|IO!BSerialManager   7.0.6f8
@Dont_Steal_Mac_OS_X   7.0.0
>!A!ISlowAdaptiveClocking   4.0.0
@filesystems.apfs   1412.141.2
>!AVmxnet3Ethernet   1.0.10
>!AAHCIPort   341.140.1
>!AVirtIO   1.0
@filesystems.hfs.kext   522.100.5
@!AFSCompression.!AFSCompressionTypeDataless   1.0.0d1
@BootCache   40
@!AFSCompression.!AFSCompressionTypeZlib   1.0.0
@private.KextAudit   1.0
>!AACPIButtons   6.1
>!AHPET   1.8
>!ARTC   2.0
>!ASMBIOS   2.1
>!AAPIC   1.7
$!AImage4   1
@nke.applicationfirewall   303
$TMSafetyNet   8
@!ASystemPolicy   2.0.0
|EndpointSecurity   1
@kext.triggers   1.0
|IOAVB!F   850.1
>!ASMBus!C   1.0.18d1
|IOSMBus!F   1.1
|IO!B!F   7.0.6f8
|IO!BPacketLogger   7.0.6f8
>!AHDA!C   283.15
|IOHDA!F   283.15
>X86PlatformPlugin   1.0.0
>IOPlatformPlugin!F   6.0.0d8
@!AGPUWrangler   5.2.7
@!AGraphicsDeviceControl   5.2.7
|IOSlowAdaptiveClocking!F   1.0.0
|IONDRVSupport   576.1
|IOGraphics!F   576.1
@plugin.IOgPTPPlugin   840.3
|IOEthernetAVB!C   1.1.0
|IOSkywalk!F   1
>usb.networking   5.0.0
>usb.!UHostCompositeDevice   1.2
|IOAHCIBlock!S   316.100.5
|IOAHCI!F   290.0.1
>usb.!UEHCIPCI   1.2
>usb.!UUHCIPCI   1.2
>usb.!UUHCI   1.2
>usb.!UEHCI   1.2
|IOAudio!F   300.2
@vecLib.kext   1.2.0
|IOSerial!F   11
|IOSurface   269.11
@filesystems.hfs.encodings.kext   1
|IOUSB!F   900.4.2
>!AEFINVRAM   2.1
>!AEFIRuntime   2.1
|IOHID!F   2.0.0
$quarantine   4
$sandbox   300.0
@kext.!AMatch   1.0.0d1
>DiskImages   493.0.0
>!AFDEKeyStore   28.30
>!AEffaceable!S   1.0
>!ASSE   1.0
>!AKeyStore   2
>!UTDM   489.120.1
|IOSCSIBlockCommandsDevice   422.120.3
>!ACredentialManager   1.0
>KernelRelayHost   1
>!ASEPManager   1.0.1
>IOSlaveProcessor   1
|IOUSBMass!SDriver   157.140.1
|IOSCSIArchitectureModel!F   422.120.3
|IO!S!F   2.1
|IOUSBHost!F   1.2
>!UHostMergeProperties   1.2
>usb.!UCommon   1.0
>!ABusPower!C   1.0
|CoreAnalytics!F   1
>!AMobileFileIntegrity   1.0.5
@kext.CoreTrust   1
|IOTimeSync!F   840.3
|IONetworking!F   3.4
|IOReport!F   47
>!AACPIPlatform   6.1
>!ASMC   3.1.9
>watchdog   1
|IOPCI!F   2.9
|IOACPI!F   1.4
@kec.pthread   1
@kec.corecrypto   1.0
@kec.Libm   1


[lundman]

Do you know which commands you use with send and recv to trigger that so i can replicate it here?

Also, which macOS do you run so I can build you a test for a fix?

[spencerh]

I'm using syncoid to trigger the send/receive. It's a perl script that I pulled from here: https://github.com/jimsalterjrs/sanoid/blob/master/syncoid
I'm using the following invocation of syncoid:

Code: Select all

/usr/local/sbin/syncoid -r root@server1:rpool/srv/docker Backup/srv/docker

Here's the output from sw_vers on the machine:

Code: Select all

ProductName:    Mac OS X
ProductVersion: 10.15.7
BuildVersion:   19H1217

I'm executing syncoid from the Mac, not from the other machine.

EDIT: It also sometimes hangs indefinitely but doesn't panic.

Code: Select all

USER   PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
root  1836   0.0  0.0  4283116   2832   ??  U     6:35pm   0:04.09 zfs receive -s -F Backup/srv/docker/dataset
USER   PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
root  1837   0.0  0.0  4282668    516   ??  Ss    6:35pm   0:02.52 zfs receive -s -F Backup/srv/docker/dataset
USER   PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
root  3390   0.0  0.0  4272928   2896 s002  U+    6:58pm   0:00.05 zfs destroy Backup/srv/docker@snap


[lundman]

ok, I'll make a catalina test build for you, I think we know what it is.

[xenophon]

Sorry to barge in, but this looks quite similar to my incessant kernel panics with 2.1.0rc1 and znapzend.

They occur at times when a snapshot is expected to be created. Sample panic report follows:

Code: Select all

panic(cpu 8 caller 0xffffff80015c58f6): Kernel trap at 0xffffff80015bd833, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000008, CR3: 0x000000046f4c710c, CR4: 0x00000000003626e0
RAX: 0x0000000000000000, RBX: 0x0000000000000000, RCX: 0x0000000000000000, RDX: 0x0000000001000000
RSP: 0xffffffc7a1737558, RBP: 0xffffffc7a1737570, RSI: 0x0000000000000000, RDI: 0x0000000000000000
R8:  0x0000000000000000, R9:  0x0000000000000000, R10: 0x0000000000000000, R11: 0x0000000000000001
R12: 0xffffffc208212180, R13: 0xffffffc208212160, R14: 0x0000000000000000, R15: 0xffffffc208211b00
RFL: 0x0000000000010282, RIP: 0xffffff80015bd833, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000008, Error code: 0x0000000000000000, Fault CPU: 0x8, PL: 0, VF: 0

Backtrace (CPU 8), Frame : Return Address
0xffffffc7a1736f70 : 0xffffff800148e0dd mach_kernel : _handle_debugger_trap + 0x3fd
0xffffffc7a1736fc0 : 0xffffff80015d4f33 mach_kernel : _kdp_i386_trap + 0x143
0xffffffc7a1737000 : 0xffffff80015c552a mach_kernel : _kernel_trap + 0x55a
0xffffffc7a1737050 : 0xffffff8001432a2f mach_kernel : _return_from_trap + 0xff
0xffffffc7a1737070 : 0xffffff800148d8fd mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffc7a1737190 : 0xffffff800148dbf3 mach_kernel : _panic_trap_to_debugger + 0x273
0xffffffc7a1737200 : 0xffffff8001c9d81a mach_kernel : _panic + 0x54
0xffffffc7a1737270 : 0xffffff80015c58f6 mach_kernel : _sync_iss_to_iks + 0x2c6
0xffffffc7a17373f0 : 0xffffff80015c55dd mach_kernel : _kernel_trap + 0x60d
0xffffffc7a1737440 : 0xffffff8001432a2f mach_kernel : _return_from_trap + 0xff
0xffffffc7a1737460 : 0xffffff80015bd833 mach_kernel : _lck_mtx_lock_spin_always + 0x3
0xffffffc7a1737570 : 0xffffff7f9e04b9be org.openzfsonosx.zfs : _zfs_resume_fs + 0x26e
0xffffffc7a17375b0 : 0xffffff7f9e03fc86 org.openzfsonosx.zfs : _zfs_ioc_recv_impl + 0x9c6
0xffffffc7a1737770 : 0xffffff7f9e03e07e org.openzfsonosx.zfs : _zfs_ioc_recv + 0x25e
0xffffffc7a1737a60 : 0xffffff7f9e0363ba org.openzfsonosx.zfs : _zfsdev_ioctl_common + 0x57a
0xffffffc7a1737b30 : 0xffffff7f9e040678 org.openzfsonosx.zfs : _zfsdev_ioctl + 0x68
0xffffffc7a1737b70 : 0xffffff8001760ca5 mach_kernel : _spec_ioctl + 0x75
0xffffffc7a1737ba0 : 0xffffff8001755294 mach_kernel : _VNOP_IOCTL + 0x144
0xffffffc7a1737c30 : 0xffffff80017460fc mach_kernel : _utf8_normalizeOptCaseFoldAndMatchSubstring + 0xebc
0xffffffc7a1737e30 : 0xffffff8001a4eb4b mach_kernel : _ioctl + 0x51b
0xffffffc7a1737f40 : 0xffffff8001b3fc9e mach_kernel : _unix_syscall64 + 0x2ce
0xffffffc7a1737fa0 : 0xffffff80014331f6 mach_kernel : _hndl_unix_scall64 + 0x16
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.1)[5116548C-0A14-34BD-A9CE-BF135FCA92F6]@0xffffff7f9df3e000->0xffffff7f9e270fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[A0D72FE9-649B-316A-8B5C-934E295FF0B5]@0xffffff800406d000->0xffffff800407efff

Process name corresponding to current thread: zfs
Boot args: keepsyms=1 darkwake=3 shikigva=80 debug=0x100 igfxonln=1 igfxfw=2 forceRenderStandby=0 alcid=11 kernel_stack_pages=8

Mac OS version:
20F71

Kernel version:
Darwin Kernel Version 20.5.0: Sat May  8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
Kernel UUID: 52A1E876-863E-38E3-AC80-09BBAB13B752
KernelCache slide: 0x0000000001200000
KernelCache base:  0xffffff8001400000
Kernel slide:      0x0000000001210000
Kernel text base:  0xffffff8001410000
__HIB  text base: 0xffffff8001300000
System model name: iMac19,1 (Mac-AA95B1DDAB278B95)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 7585160525710
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000006e60ecc79d2
  Sleep   : 0x000006e2a4b4090a 0x000000012afb0cf7 0x0000033323733441
  Wake    : 0x000006e2bc9caf3b 0x000000012b3fa6a8 0x000006e2ae169a5d

My only customization is the "kernel_stack_pages=8" argument.

I've disabled znazpend and things get along smoothly, it seems.

Once again, thanks for your work!

[xenophon]

Code: Select all

% zpool version
zfs-macOS-2.1.0-rc2
zfs-kmod-2.1.0-rc2

Perhaps off-topic, but my really bad instability has disappeared with rc2 and znapzend running in the background. Having high hopes with >5d of uptime.

Once again, sorry to intrude but it seems that rc2 is on track with zfs command kernel panics.

Take care.

Xen

[lundman]

Yep, rc2 contained a fix for that specific problem, thanks for reporting back. I think we are a go for a 2.1 release