Watchdog reboots on Big Sur

All your general support questions for OpenZFS on OS X.

Watchdog reboots on Big Sur

Postby zandr » Tue Jan 05, 2021 11:44 am

I'm running 2.0.0-1 on 20D5029f (11.2 Beta) and seeing watchdogs under heavy I/O to a 4-drive Thunderbolt stripe. Also znapzend segfaults, but that's probably a different issue.

Here's the Panic:

Code: Select all
panic(cpu 2 caller 0xffffff802015fa25): userspace watchdog timeout: no successful checkins from com.apple.logd in 120 seconds
service returned not alive with context : unresponsive dispatch queue(s): com.apple.firehose.io-wl
service: com.apple.logd, total successful checkins since load (45141 seconds ago): 4503, last successful checkin: 120 seconds ago
service: com.apple.WindowServer, total successful checkins since load (45081 seconds ago): 4508, last successful checkin: 0 seconds ago

Backtrace (CPU 2), Frame : Return Address
0xffffffc11feab670 : 0xffffff801ceb3bbd mach_kernel : _handle_debugger_trap + 0x3dd
0xffffffc11feab6c0 : 0xffffff801cff67c3 mach_kernel : _kdp_i386_trap + 0x143
0xffffffc11feab700 : 0xffffff801cfe6dfa mach_kernel : _kernel_trap + 0x55a
0xffffffc11feab750 : 0xffffff801ce58a2f mach_kernel : _return_from_trap + 0xff
0xffffffc11feab770 : 0xffffff801ceb345d mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffc11feab890 : 0xffffff801ceb3748 mach_kernel : _panic_trap_to_debugger + 0x268
0xffffffc11feab900 : 0xffffff801d6b82a4 mach_kernel : _panic_with_thread_context
0xffffffc11feab970 : 0xffffff802015fa25 com.apple.driver.watchdog : __ZN10IOWatchdog14userspacePanicEP8OSObjectPvP25IOExternalMethodArguments.cold.1 + 0x27
0xffffffc11feab980 : 0xffffff802015f678 com.apple.driver.watchdog : __ZN10IOWatchdog35userspaceDisableUserspaceMonitoringEP8OSObjectPvP25IOExternalMethodArguments
0xffffffc11feab9a0 : 0xffffff801d63f53e mach_kernel : __ZN12IOUserClient14externalMethodEjP25IOExternalMethodArgumentsP24IOExternalMethodDispatchP8OSObjectPv + 0x1de
0xffffffc11feab9f0 : 0xffffff802015eab4 com.apple.driver.watchdog : __ZN20IOWatchdogUserClient14externalMethodEjP25IOExternalMethodArgumentsP24IOExternalMethodDispatchP8OSObjectPv + 0x7c
0xffffffc11feabb20 : 0xffffff801d64970b mach_kernel : _is_io_connect_method + 0x35b
0xffffffc11feabc80 : 0xffffff801cfa1e61 mach_kernel : _iokit_server_routine + 0x4d81
0xffffffc11feabd90 : 0xffffff801ceb91b7 mach_kernel : _ipc_kobject_server + 0x337
0xffffffc11feabe00 : 0xffffff801ce8f975 mach_kernel : _ipc_kmsg_send + 0x115
0xffffffc11feabe60 : 0xffffff801cea70d2 mach_kernel : _mach_msg_overwrite_trap + 0x2d2
0xffffffc11feabef0 : 0xffffff801cfcafa3 mach_kernel : _mach_call_munger64 + 0x273
0xffffffc11feabfa0 : 0xffffff801ce59216 mach_kernel : _hndl_mach_scall64 + 0x16
      Kernel Extensions in backtrace:
         com.apple.driver.watchdog(1.0)[331D9A05-47F3-38B2-B8EC-7B57AA619D62]@0xffffff802015e000->0xffffff802015ffff

Process name corresponding to current thread: watchdogd
Boot args: keepsyms=1

Mac OS version:
20D5029f

Kernel version:
Darwin Kernel Version 20.3.0: Mon Dec  7 22:04:02 PST 2020; root:xnu-7195.80.16.111.1~1/RELEASE_X86_64
Kernel UUID: 94121F23-610E-35B9-8961-C0811E01A897
KernelCache slide: 0x000000001cc00000
KernelCache base:  0xffffff801ce00000
Kernel slide:      0x000000001cc10000
Kernel text base:  0xffffff801ce10000
__HIB  text base: 0xffffff801cd00000
System model name: iMac15,1 (Mac-FA842E06C61E91C5)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 45155283628636
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x0000291188a3f5a1
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x0000002e4672c711 0x0000000000000000
last started kext at 37219616326928: >!AXsanScheme   3 (addr 0xffffff801e667000, size 12288)
last stopped kext at 39551837120600: >usb.!UUserHCI   1 (addr 0xffffff7fbd79c000, size 98304)
loaded kexts:
net.lundman.zfs   2.0.0
@filesystems.smbfs   3.4.1
@fsplaceholder   1.13.2
>!ATopCaseHIDEventDriver   4030.2
>!APlatformEnabler   2.7.0d0
>AGPM   119
>X86PlatformShim   1.0.0
@filesystems.autofs   3.0
@kext.AMDRadeonServiceManager   4.0.2
@fileutil   20.036.15
>AudioAUUC   1.70
>!AUpstreamUserClient   3.6.8
@kext.AMDFramebuffer   4.0.2
@kext.AMDRadeonX4000   4.0.2
>!AMikeyHIDDriver   131
>!AGraphicsDevicePolicy   6.2.2
>!A!IHD5000Graphics   16.0.1
>!AHDA   283.15
>eficheck   1
@AGDCPluginDisplayMetrics   6.2.2
>pmtelemetry   1
>LuaHardwareAccess   1.0.16
|IOUserEthernet   1.0.1
|IO!BSerialManager   8.0.3d2
@Dont_Steal_Mac_OS_X   7.0.0
>!AHV   1
>!ADiskImages2   1
>!ALPC   3.1
>!A!ISlowAdaptiveClocking   4.0.0
>!AMikeyDriver   283.15
>!ASMCLMU   212
>!AThunderboltIP   4.0.3
@kext.AMD9000!C   4.0.2
>!AMCCSControl   1.14
>!A!IFramebufferAzul   16.0.1
|SCSITaskUserClient   436.40.6
@filesystems.tmpfs   1
@filesystems.hfs.kext   556.60.1
@BootCache   40
@!AFSCompression.!AFSCompressionTypeZlib   1.0.0
@!AFSCompression.!AFSCompressionTypeDataless   1.0.0d1
@filesystems.apfs   1677.80.5
>!ASDXC   1.9.0
|!ABCM5701Ethernet   11.0.0
>AirPort.BrcmNIC   1400.1.1
>!AAHCIPort   346
@private.KextAudit   1.0
>!AACPIButtons   6.1
>!ARTC   2.0
>!AHPET   1.8
>!ASMBIOS   2.1
>!AACPIEC   6.1
>!AAPIC   1.7
@!ASystemPolicy   2.0.0
@nke.applicationfirewall   310
|IOKitRegistryCompatibility   1
|EndpointSecurity   1
>!AHIDKeyboard   223
>!AActuatorDriver   4400.28
>!AMultitouchDriver   4400.28
>!AInputDeviceSupport   4400.35
>!AHS!BDriver   4030.2
>IO!BHIDDriver   8.0.3d2
@kext.triggers   1.0
@kext.AMDRadeonX4070HWLibs   1.0
@kext.AMDRadeonX4000HWServices   4.0.2
>!AGraphicsControl   6.2.2
>DspFuncLib   283.15
@kext.OSvKernDSPLib   529
>X86PlatformPlugin   1.0.0
|IOSerial!F   11
@plugin.IOAVBDiscoveryPlugin   900.12
>IOPlatformPlugin!F   6.0.0d8
@!AGPUWrangler   6.2.2
|IOSlowAdaptiveClocking!F   1.0.0
>!AHDA!C   283.15
|IOHDA!F   283.15
|IOAVB!F   900.12
@plugin.IOgPTPPlugin   900.11
|IOEthernetAVB!C   1.1.0
|IONDRVSupport   585
>!AThunderboltEDMSink   5.0.3
@kext.AMDSupport   4.0.2
>!ASMBus!C   1.0.18d1
>!ASMBusPCI   1.0.14d1
|IOAccelerator!F2   439.52
@!AGraphicsDeviceControl   6.2.2
|IOGraphics!F   585
>!UAudio   401.4
|IOAudio!F   300.6.1
@vecLib.kext   1.2.0
>Core!S   554
|IOSCSIMultimediaCommandsDevice   436.40.6
|IOBD!S!F   1.8
|IODVD!S!F   1.8
|IOCD!S!F   1.8
|Broadcom!BHost!CUSBTransport   8.0.3d2
|IO!BHost!CUSBTransport   8.0.3d2
|IO!BHost!CTransport   8.0.3d2
>!AThunderboltDPInAdapter   8.1.4
>!AThunderboltDPOutAdapter   8.1.4
>!AThunderboltDPAdapter!F   8.1.4
>!AThunderboltPCIUpAdapter   4.1.1
>!AThunderboltPCIDownAdapter   4.1.1
>usb.networking   5.0.0
>usb.!UHostCompositeDevice   1.2
>usb.!UHub   1.2
>!ABSDKextStarter   3
|IOSurface   289.3
@filesystems.hfs.encodings.kext   1
|IOAHCIBlock!S   332
>!AThunderboltNHI   7.2.8
|IOThunderbolt!F   9.3.2
|IO80211!F   1200.12.2b1
|IOSkywalk!F   1
>mDNSOffloadUserClient   1.0.1b8
>corecapture   1.0.4
|IOAHCI!F   294
>usb.!UXHCIPCI   1.2
>usb.!UXHCI   1.2
|IOUSB!F   900.4.2
>!AEFINVRAM   2.1
>!AEFIRuntime   2.1
|IOSMBus!F   1.1
|IOHID!F   2.0.0
$!AImage4   3.0.0
|IOTimeSync!F   900.11
|IONetworking!F   3.4
>DiskImages   493.0.0
|IO!B!F   8.0.3d2
|IOReport!F   47
|IO!BPacketLogger   8.0.3d2
$quarantine   4
$sandbox   300.0
@kext.!AMatch   1.0.0d1
|CoreAnalytics!F   1
>!ASSE   1.0
>!AKeyStore   2
>!UTDM   511.60.2
|IOUSBMass!SDriver   184.40.6
|IOSCSIBlockCommandsDevice   436.40.6
|IO!S!F   2.1
|IOSCSIArchitectureModel!F   436.40.6
>!AMobileFileIntegrity   1.0.5
@kext.CoreTrust   1
>!AFDEKeyStore   28.30
>!AEffaceable!S   1.0
>!ACredentialManager   1.0
>KernelRelayHost   1
|IOUSBHost!F   1.2
>!UHostMergeProperties   1.2
>usb.!UCommon   1.0
>!ABusPower!C   1.0
>!ASEPManager   1.0.1
>IOSlaveProcessor   1
>!AACPIPlatform   6.1
>!ASMC   3.1.9
|IOPCI!F   2.9
|IOACPI!F   1.4
>watchdog   1
@kec.pthread   1
@kec.corecrypto   11.1
@kec.Libm   1


Pool info:
Code: Select all
zen:~ amilewski$ zpool --version
zfs-2.0.0-1
zfs-kmod-macOS-2.0.0-2-g0197ddc9c5-dirty
zen:~ amilewski$ zpool status
  pool: ThunderBay
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
   still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
   the pool may no longer be accessible by software that does not support
   the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 20:55:32 with 0 errors on Thu Dec 24 12:38:05 2020
config:

   NAME                                          STATE     READ WRITE CKSUM
   ThunderBay                                    ONLINE       0     0     0
     media-EB1ECB0F-1747-A549-90BA-E963F2CE5D78  ONLINE       0     0     0
     media-3F8FC7C1-088F-EA4C-82B3-27D05275B1EF  ONLINE       0     0     0
     media-94E52482-C2A9-5949-A55F-23255A4B868C  ONLINE       0     0     0
     media-181B1A3E-14D6-684F-B3BE-962282E0B591  ONLINE       0     0     0

errors: No known data errors
zen:~ amilewski$


This is easily repeatable, so please let me know what other info would help debug. Thanks!
-Zandr
zandr
 
Posts: 10
Joined: Thu Mar 19, 2020 6:19 am

Re: Watchdog reboots on Big Sur

Postby lundman » Wed Jan 06, 2021 4:33 pm

Hmm this one is new, and you can easily reproduce it? What do you do to make that happen?
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Watchdog reboots on Big Sur

Postby zandr » Wed Jan 06, 2021 10:14 pm

'Reproduce' might be too strong, but repeat, yes. To trigger it, I leave the machine on overnight.

BUT- I now think this is a false alarm. To test to see if it was OpenZFS, I exported the pool and let the machine run overnight. It hadn't panicked this morning, so I thought I had my smoking gun.
I got home after being out all day to find a 7 hour uptime, and the same panic.

Sorry about that. Off to figure out the znapzend SEGV. ;)
zandr
 
Posts: 10
Joined: Thu Mar 19, 2020 6:19 am


Return to General Help

Who is online

Users browsing this forum: No registered users and 34 guests

cron