Kernel Panic (idle state)

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

Re: Kernel Panic (idle state)

Post by BrianDieckman » Fri Oct 05, 2012 9:00 am

scasady wrote:did that got this response "extension .panic not allowed"



If you wrap the panic text in the {code}{/code} tags, that makes it easier to read and copy/paste. (just use the square bracket instead of the curly bracket)

Alternatively, you could rename the file with a different extension. (.txt would work)
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

cross reference

Post by grahamperrin » Fri Oct 05, 2012 1:41 pm

scasady wrote:… "extension .panic not allowed" …


viewtopic.php?p=657#p657 under General advice for newcomers to this phpBB-powered forum
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Kernel Panic (idle state)

Post by dbrady » Sat Oct 06, 2012 2:33 am

As far as I can tell, these zio_notify_parent panics only occur under 10.6.8. They don't appear to be related to RAM size.

We have not yet been able to reproduce this. Ideally we can uncover some reproducible steps so that we can debug this one.
dbrady Offline


 
Posts: 67
Joined: Wed Sep 12, 2012 12:43 am

Re: Kernel Panic (idle state)

Post by scasady » Sun Oct 07, 2012 3:34 pm

This panic was under stress. Copying and compareing gigibyte biles, some with different record sizes.
It happened after a couple of minutes of this.
zpool is raidz2 8 500 G disks areca 1300 SAS controller

Code: Select all
Sun Oct  7 15:22:01 2012
panic(cpu 16 caller 0xffffff7f826110da): "zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff81d0b65e30 0 0xffffff81d0b660b8\n"@/staging/zevo/src/uts/common/fs/zfs/zio.c:478
Backtrace (CPU 16), Frame : Return Address
0xffffff8155f13d70 : 0xffffff8000204d15
0xffffff8155f13e70 : 0xffffff7f826110da
0xffffff8155f13ec0 : 0xffffff7f8260fd31
0xffffff8155f13ef0 : 0xffffff7f8260d369
0xffffff8155f13f30 : 0xffffff7f8257dc39
0xffffff8155f13fa0 : 0xffffff80002c8527
      Kernel Extensions in backtrace (with dependencies):
         com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f82570000->0xffffff7f826affff
            dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f81472000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
10K549

Kernel version:
Darwin Kernel Version 10.8.0: Tue Jun  7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64
System model name: MacPro5,1 (Mac-F221BEC8)

System uptime in nanoseconds: 179656098383307
unloaded kexts:
com.apple.driver.AppleIntel8254XEthernet   2.1.3b1 (addr 0xffffff7f81606000, size 0x118784) - last unloaded 157451400722
loaded kexts:
com.vmware.kext.vmioplug.10.1.21   10.1.21 - last loaded 2737913484104
com.vmware.kext.vmx86   0082.54.49
com.vmware.kext.vmnet   0082.54.49
com.vmware.kext.vsockets   90.4.22
com.vmware.kext.vmci   90.4.18
com.nvidia.CUDA   1.1.0
com.bresink.driver.BRESINKx86Monitoring   8.0
com.dvdfab.kext.fabio   1
com.makemkv.kext.daspi   1
com.getgreenbytes.driver.zfs   2012.09.14
com.getgreenbytes.filesystem.zfs   2012.09.23
com.Logitech.Unifying.HID Driver   1.2.0
com.Logitech.Control Center.HID Driver   3.3.0
com.highpoint-tech.kext.rr64x   1.0.0
com.Areca.ArcSAS   2.0.3
com.rim.driver.BlackBerryUSBDriverInt   0.0.64
com.apple.filesystems.afpfs   9.7.1
com.apple.nke.asp_tcp   5.0
com.apple.filesystems.autofs   2.1.0
com.apple.driver.AppleHWSensor   1.9.3d0
com.apple.driver.AppleMikeyHIDDriver   1.2.0
com.apple.driver.AppleTyMCEDriver   1.0.2d2
com.apple.driver.AGPM   100.12.31
com.apple.driver.AppleUpstreamUserClient   3.5.7
com.apple.driver.AppleHDA   2.0.5f14
com.apple.kext.ATIFramebuffer   6.3.6
com.apple.driver.AppleMCCSControl   1.0.20
com.apple.driver.AppleMikeyDriver   2.0.5f14
com.apple.driver.AudioAUUC   1.57
com.apple.ATIRadeonX3000   6.3.6
com.apple.driver.AppleUSBDisplays   289
com.apple.GeForce   6.3.7
com.apple.Dont_Steal_Mac_OS_X   7.0.0
com.apple.driver.AudioIPCDriver   1.1.6
com.apple.driver.ACPI_SMC_PlatformPlugin   4.7.0a1
com.apple.driver.AppleLPC   1.5.1
com.apple.iokit.SCSITaskUserClient   2.6.8
com.apple.iokit.IOAHCIBlockStorage   1.6.4
com.apple.BootCache   31.1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib   1.0.0d1
com.apple.driver.AppleFWOHCI   4.7.3
com.apple.driver.AirPortBrcm43224   428.42.4
com.apple.driver.Intel82574L   2.1.8b1
com.apple.driver.AppleAHCIPort   2.1.7
com.apple.driver.AppleUSBHub   4.2.4
com.apple.driver.AppleUSBEHCI   4.2.4
com.apple.driver.AppleUSBUHCI   4.2.0
com.apple.driver.AppleEFINVRAM   1.4.0
com.apple.driver.AppleRTC   1.3.1
com.apple.driver.AppleHPET   1.5
com.apple.driver.AppleACPIButtons   1.3.6
com.apple.driver.AppleSMBIOS   1.7
com.apple.driver.AppleACPIEC   1.3.6
com.apple.driver.AppleAPIC   1.4
com.apple.driver.AppleIntelCPUPowerManagementClient   142.6.0
com.apple.security.sandbox   1
com.apple.security.quarantine   0
com.apple.nke.applicationfirewall   2.1.14
com.apple.iokit.CHUDUtils   364
com.apple.iokit.CHUDProf   366
com.apple.driver.AppleIntelCPUPowerManagement   142.6.0
com.apple.driver.AppleProfileReadCounterAction   17
com.apple.driver.DspFuncLib   2.0.5f14
com.apple.driver.AppleProfileTimestampAction   10
com.apple.driver.AppleProfileThreadInfoAction   14
com.apple.driver.AppleProfileRegisterStateAction   10
com.apple.driver.AppleProfileKEventAction   10
com.apple.driver.AppleProfileCallstackAction   20
com.apple.driver.AppleSMBusController   1.0.10d0
com.apple.kext.ATI5000Controller   6.3.6
com.apple.kext.ATISupport   6.3.6
com.apple.iokit.IOFireWireIP   2.0.3
com.apple.driver.AppleUSBAudio   2.7.6f6
com.apple.iokit.IOSurface   74.2
com.apple.iokit.IOBluetoothSerialManager   2.4.5f3
com.apple.iokit.IOSerialFamily   10.0.3
com.apple.iokit.IOAudioFamily   1.8.3fc2
com.apple.kext.OSvKernDSPLib   1.3
com.apple.driver.AppleHDAController   2.0.5f14
com.apple.iokit.IOHDAFamily   2.0.5f14
com.apple.iokit.AppleProfileFamily   41.4
com.apple.driver.AppleSMC   3.1.0d5
com.apple.driver.IOPlatformPluginFamily   4.7.0a1
com.apple.driver.AppleSMBusPCI   1.0.10d0
com.apple.nvidia.nvGF100hal   6.3.7
com.apple.NVDAResman   6.3.7
com.apple.iokit.IONDRVSupport   2.2.1
com.apple.iokit.IOGraphicsFamily   2.2.1
com.apple.iokit.IOSCSIMultimediaCommandsDevice   2.6.8
com.apple.iokit.IOBDStorageFamily   1.6
com.apple.iokit.IODVDStorageFamily   1.6
com.apple.iokit.IOCDStorageFamily   1.6.1
com.apple.driver.XsanFilter   402.1
com.apple.driver.AppleUSBHIDKeyboard   141.5
com.apple.driver.AppleHIDKeyboard   141.5
com.apple.iokit.IOUSBHIDDriver   4.2.0
com.apple.driver.BroadcomUSBBluetoothHCIController   2.4.5f3
com.apple.driver.AppleUSBBluetoothHCIController   2.4.5f3
com.apple.iokit.IOBluetoothFamily   2.4.5f3
com.apple.driver.AppleUSBMergeNub   4.2.4
com.apple.iokit.IOSCSIParallelFamily   2.1.1
com.apple.iokit.IOFireWireFamily   4.2.6
com.apple.iokit.IOAHCISerialATAPI   1.2.6
com.apple.iokit.IO80211Family   320.1
com.apple.iokit.IONetworkingFamily   1.10
com.apple.iokit.IOAHCIFamily   2.0.6
com.apple.iokit.IOUSBUserClient   4.2.4
com.apple.driver.AppleEFIRuntime   1.4.0
com.apple.iokit.IOHIDFamily   1.6.6
com.apple.iokit.IOSMBusFamily   1.1
com.apple.kext.AppleMatch   1.0.0d1
com.apple.security.TMSafetyNet   6
com.apple.iokit.IOUSBMassStorageClass   2.6.7
com.apple.driver.AppleUSBComposite   3.9.0
com.apple.iokit.IOSCSIBlockCommandsDevice   2.6.8
com.apple.iokit.IOSCSIArchitectureModelFamily   2.6.8
com.apple.iokit.IOUSBFamily   4.2.4
com.apple.iokit.CHUDKernLib   365
com.apple.driver.DiskImages   289
com.apple.iokit.IOStorageFamily   1.6.3
com.apple.driver.AppleACPIPlatform   1.3.6
com.apple.iokit.IOPCIFamily   2.6.5
com.apple.iokit.IOACPIFamily   1.3.0
 
Last edited by scasady on Mon Oct 08, 2012 8:34 am, edited 1 time in total.
scasady Offline


 
Posts: 45
Joined: Sat Sep 15, 2012 8:00 am

Re: Kernel Panic (idle state)

Post by grahamperrin » Mon Oct 08, 2012 3:13 am

(Consider this post deleted.)
Last edited by grahamperrin on Mon Oct 08, 2012 10:01 pm, edited 1 time in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Kernel Panic (idle state)

Post by grahamperrin » Mon Oct 08, 2012 3:27 am

dbrady wrote:… reproducible steps …


As scasady describes his kernel panic as under stress: here's a hunch, based on experiences with Mountain Lion. If we're lucky the hunch will translate to Snow Leopard.

Run Windows 7 in a VirtualBoxVM with that guest limited 1 to GB memory, store the .vdi on ZEVO on external USB. In the VM, download Remote Server Administration Tools for Windows 7 with Service Pack 1 (SP1) then begin installing. Have VirtualBoxVM in background but not hidden whilst you do things with any other app in front. Take as long as you like. Just work normally. If you have a choice of discrete or integrated GPU, maybe prefer discrete.

Depending on your physical machine configuration: the virtual machine may take a considerable time attempting the installation; and the physical machine may panic (hopefully with zio_notify_parent) before installation succeeds.

Don, on Wuala I don't have enough room for the .vdi that I'd use for this type of thing, but if you like I can arrange a copy some other way.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Kernel Panic (idle state)

Post by BrianDieckman » Mon Oct 08, 2012 3:32 pm

An update on this panic: I've installed the RAID admin software on the computer to which the Xserve RAID units are connected. The date/time on the units have been synchronized to the local system clock (so the RAID logs and Kernel Panic logs will have corresponding date/times).

To my shock and surprise, the kernel panics have been occurring before the Fibre Link alarms on the RAID units. In other words, the kernel panic is causing the machine to crash hard enough to prevent it from gracefully closing the link.

I've never experienced this before; in the past, the RAIDs have not recorded link alarms when shutting down/restarting/freezing.

This is happening frequently enough that I'm considering pulling Zevo CE off this machine and using another software RAID solution. I'm open to suggestions on how to resolve; would other logs or debug information be helpful in troubleshooting? I'm willing to run in a development environment or debug mode if that would be of assistance.
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

Re: Kernel Panic (idle state)

Post by BrianDieckman » Mon Oct 08, 2012 3:53 pm

Sorry, I meant to attach the Panic code...

Code: Select all
Interval Since Last Panic Report:  261290 sec
Panics Since Last Report:          6
Anonymous UUID:                    4A0E941A-70AD-4451-AA87-4AAB37B91A49

Mon Oct  8 16:13:23 2012
panic(cpu 7 caller 0xffffff7f80d840da): "zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80d17025d8 0 0xffffff80d1702860\n"@/staging/zevo/src/uts/common/fs/zfs/zio.c:478
Backtrace (CPU 7), Frame : Return Address
0xffffff80b1b7bd70 : 0xffffff8000204d15
0xffffff80b1b7be70 : 0xffffff7f80d840da
0xffffff80b1b7bec0 : 0xffffff7f80d82d31
0xffffff80b1b7bef0 : 0xffffff7f80d80369
0xffffff80b1b7bf30 : 0xffffff7f80cf0c39
0xffffff80b1b7bfa0 : 0xffffff80002c8527
      Kernel Extensions in backtrace (with dependencies):
         com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f80ce3000->0xffffff7f80e22fff
            dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f80884000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
10K549

Kernel version:
Darwin Kernel Version 10.8.0: Tue Jun  7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64
System model name: Xserve2,1 (Mac-F42289C8)

System uptime in nanoseconds: 74364240904889
unloaded kexts:
com.apple.iokit.IOAHCIFamily   2.0.6 (addr 0xffffff7f80994000, size 0x28672) - last unloaded 100703189738
loaded kexts:
com.aladdin.kext.aksfridge   1.0.2
com.sns.driver.SnsiSCSI   5.1.0
com.sns.driver.Xtarget   1.1.1
com.getgreenbytes.driver.zfs   2012.09.14
com.getgreenbytes.filesystem.zfs   2012.09.23
com.sns.driver.SNSArchitectureModel   1.1.3
com.apple.filesystems.afpfs   9.7.1 - last loaded 9122063094650
com.apple.nke.asp_tcp   5.0
com.apple.filesystems.autofs   2.1.0
com.apple.driver.AppleHWSensor   1.9.3d0
com.apple.Dont_Steal_Mac_OS_X   7.0.0
com.apple.iokit.CHUDUtils   364
com.apple.driver.AppleUpstreamUserClient   3.5.7
com.apple.iokit.CHUDProf   366
com.apple.driver.AppleMCCSControl   1.0.20
com.apple.driver.AppleIntelPenrynProfile   17.1
com.apple.kext.ATIFramebuffer   6.3.6
com.apple.driver.AudioIPCDriver   1.1.6
com.apple.ATIRadeonX1000   6.3.6
com.apple.driver.AppleIntel8254XEthernet   2.1.3b1
com.apple.driver.AppleSEP   1.4.2
com.apple.driver.AppleMCEDriver   1.1.9
com.apple.driver.AppleBMC   2.0.2
com.apple.driver.Apple16X50ACPI   3.0
com.apple.driver.ACPI_SMC_PlatformPlugin   4.7.0a1
com.apple.driver.AppleLPC   1.5.1
com.apple.driver.AppleRAID   4.0.6
com.apple.iokit.SCSITaskUserClient   2.6.8
com.apple.driver.AppleFWOHCI   4.7.3
com.apple.driver.AppleUSBHub   4.2.4
com.apple.BootCache   31.1
com.apple.driver.AppleLSIFusionMPT   2.5.0
com.apple.driver.AppleUSBEHCI   4.2.4
com.apple.driver.AppleEFINVRAM   1.4.0
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib   1.0.0d1
com.apple.driver.AppleIntelPIIXATA   2.5.1
com.apple.driver.AppleUSBUHCI   4.2.0
com.apple.driver.AppleACPIButtons   1.3.6
com.apple.driver.AppleRTC   1.3.1
com.apple.driver.AppleHPET   1.5
com.apple.driver.AppleSMBIOS   1.7
com.apple.driver.AppleACPIEC   1.3.6
com.apple.driver.AppleAPIC   1.4
com.apple.driver.AppleIntelCPUPowerManagementClient   142.6.0
com.apple.security.sandbox   1
com.apple.security.quarantine   0
com.apple.nke.applicationfirewall   2.1.14
com.apple.driver.AppleIntelCPUPowerManagement   142.6.0
com.apple.driver.AppleProfileReadCounterAction   17
com.apple.driver.AppleProfileTimestampAction   10
com.apple.driver.AppleProfileThreadInfoAction   14
com.apple.driver.AppleProfileRegisterStateAction   10
com.apple.driver.AppleProfileKEventAction   10
com.apple.driver.AppleProfileCallstackAction   20
com.apple.iokit.IOSurface   74.2
com.apple.iokit.IOBluetoothSerialManager   2.4.5f3
com.apple.iokit.CHUDKernLib   365
com.apple.iokit.IOAudioFamily   1.8.3fc2
com.apple.kext.OSvKernDSPLib   1.3
com.apple.iokit.IONDRVSupport   2.2.1
com.apple.iokit.AppleProfileFamily   41.4
com.apple.driver.Apple16X50Serial   3.0
com.apple.iokit.IOSerialFamily   10.0.3
com.apple.driver.AppleSMC   3.1.0d5
com.apple.driver.IOPlatformPluginFamily   4.7.0a1
com.apple.kext.ATI1300Controller   6.3.6
com.apple.kext.ATISupport   6.3.6
com.apple.iokit.IOGraphicsFamily   2.2.1
com.apple.iokit.IOFireWireIP   2.0.3
com.apple.iokit.IONetworkingFamily   1.10
com.apple.driver.XsanFilter   402.1
com.apple.iokit.IOSCSIBlockCommandsDevice   2.6.8
com.apple.iokit.IOSCSIMultimediaCommandsDevice   2.6.8
com.apple.iokit.IOBDStorageFamily   1.6
com.apple.iokit.IODVDStorageFamily   1.6
com.apple.iokit.IOCDStorageFamily   1.6.1
com.apple.iokit.IOATAPIProtocolTransport   2.5.1
com.apple.iokit.IOFireWireFamily   4.2.6
com.apple.iokit.IOUSBUserClient   4.2.4
com.apple.iokit.IOSCSIParallelFamily   2.1.1
com.apple.iokit.IOSCSIArchitectureModelFamily   2.6.8
com.apple.iokit.IOATAFamily   2.5.1
com.apple.iokit.IOUSBFamily   4.2.4
com.apple.driver.AppleEFIRuntime   1.4.0
com.apple.driver.AppleKeyswitch   1.0.5f4
com.apple.iokit.IOHIDFamily   1.6.6
com.apple.iokit.IOSMBusFamily   1.1
com.apple.security.TMSafetyNet   6
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.DiskImages   289.1
com.apple.iokit.IOStorageFamily   1.6.3
com.apple.driver.AppleACPIPlatform   1.3.6
com.apple.iokit.IOPCIFamily   2.6.5
com.apple.iokit.IOACPIFamily   1.3.0
Model: Xserve2,1, BootROM XS21.006C.B06, 8 processors, Quad-Core Intel Xeon, 3 GHz, 8 GB, SMC 1.26f3
Graphics: ATI Radeon X1300, ATY,RadeonX1300, PCIe, 64 MB
Memory Module: global_name
Network Service: Ethernet 1, Ethernet, en0
PCI Card: ATY,RadeonX1300, sppci_displaycontroller, Mezzanine
PCI Card: Apple 2 Port 4Gbps Fibre Channel Card, sppci_fibrechannel, Slot-1
PCI Card: Apple 2 Port 4Gbps Fibre Channel Card, sppci_fibrechannel, Slot-1
PCI Card: pci1000,646, sppci_fibrechannel, Slot-2@5,0,0
PCI Card: pci1000,646, sppci_fibrechannel, Slot-2@5,0,1
PCI Card: pci1000,646, sppci_fibrechannel, Slot-2@6,0,0
PCI Card: pci1000,646, sppci_fibrechannel, Slot-2@6,0,1
Parallel ATA Device: MATSHITADVD-R   UJ-85J
Fibre Channel Device: SCSI Target Device @ 0
Fibre Channel Device: SCSI Target Device @ 0
Fibre Channel Device: SCSI Target Device @ 0
Fibre Channel Device: SCSI Target Device @ 0
Fibre Channel Device: SCSI Target Device @ 0
Fibre Channel Device: SCSI Target Device @ 0
USB Device: Frontpanel Controller, 0x05ac  (Apple Inc.), 0x8261, 0x3d100000 / 2
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

Re: Kernel Panic (idle state)

Post by elfpltfn » Mon Oct 08, 2012 7:49 pm

FWIW- I have 2 drive controller cards (ASM1061), and get the same panic under load.

Don- Would a kernel dump be helpful in debugging? (I have gotten test dumps from my real macs, and if it will help I'll try to get one of them to panic and send the dump to another machine.)
elfpltfn Offline


 
Posts: 23
Joined: Fri Sep 14, 2012 9:52 pm

Re: Kernel Panic (idle state)

Post by BrianDieckman » Tue Oct 09, 2012 8:23 am

Some more info about this panic;

The drives are 500GB each, in two Xserve RAID enclosures. There are a total of 14 of them. Because the Xserve RAID controllers only handle 7 drives each, there are actually two drive controllers involved here. I'm wondering if having two controllers is causing an issue?

Another consideration; the drives are configured in what Apple calls an "Enhanced JBOD" configuration; basically it takes advantage of the controller's cacheing features. Could this be a factor?

Three settings are checked in the RAID configuration; "Use Controller Write Cache", "Allow Host Cache Flushing" and "Use Drive Write Cache". There's a fourth option "Use Steady Streaming Mode" that is unchecked.

Finally, I found an obscure knowledge base article on Apple's site that recommends setting the Fibre Channel link topology to "Point to Point" when connected directly to a FC card in the server. (Which I do... no switches involved) I've set that on both controllers to see if this ends my link issues.
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

PreviousNext

Return to Troubleshooting

Who is online

Users browsing this forum: hlxpgxmum, ilovezfs and 0 guests

cron