(2012.09.23) still waiting on znode … REG ZIL … mds panic

Moderators: jhartley, MSR734, nola

(2012.09.23) still waiting on znode … REG ZIL … mds panic

Post by grahamperrin » Thu Oct 04, 2012 3:16 am

In brief … probably obscure, and I don't know whether it'll be easily reproducible …

ZEVO Community Edition 1.1.1 on OS X 10.8.2, MacBookPro5,2 with 8 GB memory and one USB device alone for a single pool (dataset with one child).

Repeatedly in Console:

Code: Select all
… kernel[0]: still waiting on znode 15740167 (0xffffff81792fab00) REG ZIL


GUIs, including the Force Quit Applications window of loginwindow, were unresponsive. I keyed the chord for sysdiagnose but before it completed, a kernel panic.

FWIW before the panic I recall attempting
    gpt show …
or
    gpt show -l
for at least that one USB device, which I thought was recently ejected, but eventually I realised it was present in Finder.

AFAIR I used Finder to eject/unmount both file systems. Initially those actions seemed to succeed (both disappeared from Finder) but the system began feeling sluggish in an I/O way and sure enough I observed strings of peculiarities in Console.

Code: Select all
panic(cpu 0 caller 0xffffff8012cb7bd5): Kernel trap at 0xffffff8013088d4f, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x00000000000000e8, CR3: 0x0000000020c54000, CR4: 0x0000000000000660
RAX: 0x0000000000000003, RBX: 0x0000000000000001, RCX: 0xffffff7f9341aa3f, RDX: 0xffffff8024bd8000
RSP: 0xffffff80fddcbec0, RBP: 0xffffff80fddcbef0, RSI: 0x0000000000000000, RDI: 0xffffff8031c1b260
R8:  0x0000000000000001, R9:  0x0000000000000000, R10: 0x0000000000000001, R11: 0x0000000000000000
R12: 0x0000000000000000, R13: 0xffffff80132e1f58, R14: 0x0000000000000000, R15: 0xffffff8028859e60
RFL: 0x0000000000010286, RIP: 0xffffff8013088d4f, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x00000000000000e8, Error code: 0x0000000000000000, Fault CPU: 0x0

Backtrace (CPU 0), Frame : Return Address
0xffffff80fddcbb60 : 0xffffff8012c1d626
0xffffff80fddcbbd0 : 0xffffff8012cb7bd5
0xffffff80fddcbda0 : 0xffffff8012cce4ed
0xffffff80fddcbdc0 : 0xffffff8013088d4f
0xffffff80fddcbef0 : 0xffffff8012d04e10
0xffffff80fddcbf50 : 0xffffff8012fe182a
0xffffff80fddcbfb0 : 0xffffff8012cced33

BSD process name corresponding to current thread: mds
Boot args: -v

Mac OS version:
12C54

Kernel version:
Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64
Kernel UUID: 69A5853F-375A-3EF4-9247-478FD0247333
Kernel slide:     0x0000000012a00000
Kernel text base: 0xffffff8012c00000
System model name: MacBookPro5,2 (Mac-F2268EC8)

System uptime in nanoseconds: 27106633911957
last loaded kext at 15103583380236: com.apple.driver.AppleBluetoothMultitouch   75.15 (addr 0xffffff7f951fc000, size 73728)
last unloaded kext at 27039917029456: com.apple.iokit.IOUSBMassStorageClass   3.5.0 (addr 0xffffff7f93d68000, size 69632)


For the .panic file in full, and other files (to follow maybe later today): http://www.wuala.com/grahamperrin/public/2012/10/04/a/
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

cross reference

Post by grahamperrin » Thu Oct 04, 2012 8:28 pm

Cross reference

gpt show versus zfs_pool_proxy, versus unmounted volumes

With mistreatment, expect the unexpected ;)
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: (2012.09.23) still waiting on znode … REG ZIL … mds pani

Post by dbrady » Sat Oct 06, 2012 2:49 am

Note that this is a generic panic that is impossible to decode due to 10.8 kernels no longer having static addresses. We can now only decode kext addresses. : (
If you have access to another Mac, the panic can be decoded from within gdb and would yield a basic stack trace (instead of a list of hex addresses).

The still waiting on znode... messages indicate an unexpected condition during vnode recycling.
dbrady Offline


 
Posts: 67
Joined: Wed Sep 12, 2012 12:43 am

Re: (2012.09.23) still waiting on znode … REG ZIL … mds pani

Post by grahamperrin » Sat Oct 06, 2012 3:15 am

There's a nearby PowerPC iMac that I used for a while, a few months ago, as a kernel core dump server for an Intel client. No longer running an Apple OS, notionally this PowerPC is set aside for me to experiment with other OSes' approaches to ZFS. OT, my most recent experiment was probably FreeBSD, but I hit some stumbling blocks in attempts to get a GUI on the PowerPC so …

… would it be useful to give that old PowerPC back to Mac OS X 10.5.8 for core dump service, then try to repeat the panic on this MacBookPro5,2?

Or should we prefer an Intel for core dump service? If so, would Snow Leopard be good enough?

----

Assuming that the panic is reproducible, I wouldn't know where to start with gdb (and to be honest, I wouldn't have the time to learn – sorry) but for someone at Don's level I'd happily grant, in private:

  • Wuala access to a copy of the dump
  • Apple Remote Desktop, ssh or VNC access to the core dump server
  • etc..

Bottom line: for me, this type of panic (from a peculiar use case) is least priority. Follow-ups only if it might help a broader user base.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: (2012.09.23) still waiting on znode … REG ZIL … mds pani

Post by grahamperrin » Mon Oct 08, 2012 9:58 pm

dbrady wrote:… a generic panic that is impossible to decode …


Afterthought: was my original excerpt from the .panic file too short?

Here's the full text (copied from the collection of files on Wuala):

Code: Select all
Thu Oct  4 08:56:29 2012
panic(cpu 0 caller 0xffffff8012cb7bd5): Kernel trap at 0xffffff8013088d4f, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x00000000000000e8, CR3: 0x0000000020c54000, CR4: 0x0000000000000660
RAX: 0x0000000000000003, RBX: 0x0000000000000001, RCX: 0xffffff7f9341aa3f, RDX: 0xffffff8024bd8000
RSP: 0xffffff80fddcbec0, RBP: 0xffffff80fddcbef0, RSI: 0x0000000000000000, RDI: 0xffffff8031c1b260
R8:  0x0000000000000001, R9:  0x0000000000000000, R10: 0x0000000000000001, R11: 0x0000000000000000
R12: 0x0000000000000000, R13: 0xffffff80132e1f58, R14: 0x0000000000000000, R15: 0xffffff8028859e60
RFL: 0x0000000000010286, RIP: 0xffffff8013088d4f, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x00000000000000e8, Error code: 0x0000000000000000, Fault CPU: 0x0

Backtrace (CPU 0), Frame : Return Address
0xffffff80fddcbb60 : 0xffffff8012c1d626
0xffffff80fddcbbd0 : 0xffffff8012cb7bd5
0xffffff80fddcbda0 : 0xffffff8012cce4ed
0xffffff80fddcbdc0 : 0xffffff8013088d4f
0xffffff80fddcbef0 : 0xffffff8012d04e10
0xffffff80fddcbf50 : 0xffffff8012fe182a
0xffffff80fddcbfb0 : 0xffffff8012cced33

BSD process name corresponding to current thread: mds
Boot args: -v

Mac OS version:
12C54

Kernel version:
Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64
Kernel UUID: 69A5853F-375A-3EF4-9247-478FD0247333
Kernel slide:     0x0000000012a00000
Kernel text base: 0xffffff8012c00000
System model name: MacBookPro5,2 (Mac-F2268EC8)

System uptime in nanoseconds: 27106633911957
last loaded kext at 15103583380236: com.apple.driver.AppleBluetoothMultitouch   75.15 (addr 0xffffff7f951fc000, size 73728)
last unloaded kext at 27039917029456: com.apple.iokit.IOUSBMassStorageClass   3.5.0 (addr 0xffffff7f93d68000, size 69632)
loaded kexts:
com.github.osxfuse.filesystems.osxfusefs   2.5.2
org.virtualbox.kext.VBoxNetAdp   4.2.0
org.virtualbox.kext.VBoxNetFlt   4.2.0
org.virtualbox.kext.VBoxUSB   4.2.0
org.virtualbox.kext.VBoxDrv   4.2.0
foo.tap   1.0
foo.tun   1.0
com.eltima.ElmediaPlayer.kext   1.0
org.openafs.filesystems.afs   1.6.2a1
net.telestream.driver.TelestreamAudio   1.0.5
com.Cycling74.driver.Soundflower   1.6.2
com.Greatdy.driver.SystemAudioCapture   1.0.0
com.getgreenbytes.driver.zfs   2012.09.14
com.getgreenbytes.filesystem.zfs   2012.09.23
at.obdev.nke.LittleSnitch   3876
com.apple.driver.AppleBluetoothMultitouch   75.15
com.apple.filesystems.smbfs   1.8
com.apple.driver.AppleHWSensor   1.9.5d0
com.apple.filesystems.autofs   3.0
com.apple.iokit.IOBluetoothSerialManager   4.0.9f33
com.apple.driver.AudioAUUC   1.60
com.apple.driver.AGPM   100.12.69
com.apple.driver.AppleHDA   2.3.1f2
com.apple.driver.AppleLPC   1.6.0
com.apple.driver.AppleUpstreamUserClient   3.5.10
com.apple.driver.AppleMikeyHIDDriver   122
com.apple.iokit.IOUserEthernet   1.0.0d1
com.apple.Dont_Steal_Mac_OS_X   7.0.0
com.apple.driver.AppleMCCSControl   1.0.33
com.apple.driver.AppleMikeyDriver   2.3.1f2
com.apple.driver.AppleSMCLMU   2.0.2d0
com.apple.driver.AppleMuxControl   3.2.11
com.apple.driver.ApplePolicyControl   3.2.11
com.apple.driver.ACPI_SMC_PlatformPlugin   1.0.0
com.apple.GeForce   8.0.0
com.apple.driver.SMCMotionSensor   3.0.2d6
com.apple.driver.AppleUSBTCButtons   235.4
com.apple.driver.AppleUSBTCKeyboard   235.4
com.apple.driver.AppleIRController   320.15
com.apple.iokit.SCSITaskUserClient   3.5.1
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless   1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib   1.0.0d1
com.apple.BootCache   34
com.apple.driver.XsanFilter   404
com.apple.iokit.IOAHCIBlockStorage   2.2.2
com.apple.driver.AirPort.Brcm4331   602.15.22
com.apple.driver.AppleFWOHCI   4.9.6
com.apple.driver.AppleUSBHub   5.2.5
com.apple.driver.AppleAHCIPort   2.4.1
com.apple.nvenet   2.0.19
com.apple.driver.AppleUSBEHCI   5.4.0
com.apple.driver.AppleUSBOHCI   5.2.5
com.apple.driver.AppleIntelCPUPowerManagementClient   196.0.0
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.driver.AppleEFINVRAM   1.6.1
com.apple.driver.AppleRTC   1.5
com.apple.driver.AppleHPET   1.7
com.apple.driver.AppleACPIButtons   1.6
com.apple.driver.AppleSMBIOS   1.9
com.apple.driver.AppleACPIEC   1.6
com.apple.driver.AppleAPIC   1.6
com.apple.nke.applicationfirewall   4.0.39
com.apple.security.quarantine   2
com.apple.driver.AppleIntelCPUPowerManagement   196.0.0
com.apple.driver.IOBluetoothHIDDriver   4.0.9f33
com.apple.driver.AppleMultitouchDriver   235.28
com.apple.kext.triggers   1.0
com.apple.iokit.IOSerialFamily   10.0.6
com.apple.driver.DspFuncLib   2.3.1f2
com.apple.iokit.IOSurface   86.0.3
com.apple.iokit.IOBluetoothFamily   4.0.9f33
com.apple.iokit.AppleBluetoothHCIControllerUSBTransport   4.0.9f33
com.apple.iokit.IOAudioFamily   1.8.9fc10
com.apple.kext.OSvKernDSPLib   1.6
com.apple.driver.AppleSMBusController   1.0.10d0
com.apple.iokit.IOFireWireIP   2.2.5
com.apple.driver.AppleHDAController   2.3.1f2
com.apple.iokit.IOHDAFamily   2.3.1f2
com.apple.driver.AppleBacklightExpert   1.0.4
com.apple.driver.AppleGraphicsControl   3.2.11
com.apple.driver.IOPlatformPluginLegacy   1.0.0
com.apple.driver.IOPlatformPluginFamily   5.2.0d16
com.apple.driver.AppleSMBusPCI   1.0.10d0
com.apple.nvidia.nv50hal   8.0.0
com.apple.NVDAResman   8.0.0
com.apple.iokit.IONDRVSupport   2.3.5
com.apple.iokit.IOGraphicsFamily   2.3.5
com.apple.driver.AppleSMC   3.1.4d2
com.apple.driver.AppleUSBMultitouch   235.7
com.apple.iokit.IOUSBHIDDriver   5.2.5
com.apple.driver.AppleUSBMergeNub   5.2.5
com.apple.driver.AppleUSBComposite   5.2.5
com.apple.driver.CoreStorage   274.7
com.apple.iokit.IOSCSIMultimediaCommandsDevice   3.5.1
com.apple.iokit.IOBDStorageFamily   1.7
com.apple.iokit.IODVDStorageFamily   1.7.1
com.apple.iokit.IOCDStorageFamily   1.7.1
com.apple.iokit.IOAHCISerialATAPI   2.5.0
com.apple.iokit.IOSCSIArchitectureModelFamily   3.5.1
com.apple.iokit.IO80211Family   500.15
com.apple.iokit.IOFireWireFamily   4.5.5
com.apple.iokit.IOUSBUserClient   5.2.5
com.apple.iokit.IOAHCIFamily   2.2.1
com.apple.iokit.IONetworkingFamily   3.0
com.apple.iokit.IOUSBFamily   5.4.0
com.apple.driver.NVSMU   2.2.9
com.apple.driver.AppleEFIRuntime   1.6.1
com.apple.iokit.IOHIDFamily   1.8.0
com.apple.iokit.IOSMBusFamily   1.1
com.apple.security.sandbox   220
com.apple.kext.AppleMatch   1.0.0d1
com.apple.security.TMSafetyNet   7
com.apple.driver.DiskImages   344
com.apple.iokit.IOStorageFamily   1.8
com.apple.driver.AppleKeyStore   28.21
com.apple.driver.AppleACPIPlatform   1.6
com.apple.iokit.IOPCIFamily   2.7.1
com.apple.iokit.IOACPIFamily   1.4
com.apple.kec.corecrypto   1.0
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Link to a report of another mds panic

Post by grahamperrin » Tue Oct 22, 2013 7:51 pm

dbrady wrote:… a basic stack trace (instead of a list of hex addresses).

The still waiting on znode... messages indicate an unexpected condition during vnode recycling.


Maybe of use: mds kernel panic, _zfsx_searchfs, _zfsx_search_harvest
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest

cron