Kernel Panic (idle state)

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

relating kernel panics to fibre channel issues

Post by grahamperrin » Tue Oct 02, 2012 2:52 pm

BrianDieckman wrote:… When the FC Link went down …


No experience here with RAID-Z(2), neither have I used ZEVO with Snow Leopard, but I'm curious.

How closely can you relate (a) the time of the panic to (b) the time of the fibre channel link loss?

If not close to link down time, then (in your estimation) was panic closer to: (c) link up time, or (d) some other system event?

----

Very loosely speaking, with ZEVO CE on Mountain Lion … if I ungracefully disrupt the physical connection to part of a pool, then: the OS does continue to run; subsequent zfs and zpool commands may not run (and not respond to Control-Z); an attempt to shut down the OS may get to near completion but ultimately require force; if instead of shut down I attempt to eject/unmount an affected volume, then I *might* be wary of a panic.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: relating kernel panics to fibre channel issues

Post by BrianDieckman » Tue Oct 02, 2012 4:23 pm

grahamperrin wrote:How closely can you relate (a) the time of the panic to (b) the time of the fibre channel link loss?


The most recent event occurred "Mon Oct 1 14:46:54 2012"

One of the Xserve RAID units has link errors in that time frame:
Code: Select all
Warning   Lower Controller   10/01/12 02:47:47 PM   RAID Controller 2 Fibre LIP
    Lower Controller   10/01/12 02:47:47 PM   RAID Controller 2 Fibre Link Up
Warning   Lower Controller   10/01/12 02:47:47 PM   RAID Controller 2 Fibre LIP
Warning   Upper Controller   10/01/12 02:47:39 PM   RAID Controller 1 Fibre LIP
    Upper Controller   10/01/12 02:47:39 PM   RAID Controller 1 Fibre Link Up
Warning   Upper Controller   10/01/12 02:47:39 PM   RAID Controller 1 Fibre LIP
Warning   Lower Controller   10/01/12 02:47:13 PM   RAID Controller 2 Fibre Link Down
Warning   Upper Controller   10/01/12 02:47:11 PM   RAID Controller 1 Fibre Link Down
Warning   Lower Controller   10/01/12 02:47:11 PM   RAID Controller 2 Fibre LIP
Warning   Upper Controller   10/01/12 02:47:09 PM   RAID Controller 1 Fibre LIP


Now these units are not using a time server, and are synchronized to my local time (Via RAID Admin) so the time difference between my computer and the server is relevant. At this time, my clock reads 5:15:03 while the server reads 5:14:04. So the server is about a minute behind the RAID.

This would mean that the Link events in the RAID log happened just a few seconds before the kernel panic.

grahamperrin wrote:Very loosely speaking, with ZEVO CE on Mountain Lion … if I ungracefully disrupt the physical connection to part of a pool, then: the OS does continue to run; subsequent zfs and zpool commands may not run (and not respond to Control-Z); an attempt to shut down the OS may get to near completion but ultimately require force; if instead of shut down I attempt to eject/unmount an affected volume, then I *might* be wary of a panic.


Thanks for sharing your experience with Moutain Lion, Graham. Most OS X builds since Tiger have had shutdown/restart issues with drives that aren't properly attached. I remember early testing with eSATA cards in Mac Pros that refused to shut down because the drive controller just dropped the hard drive for no reason. This seems similar so I'm not surprised by your experience.

As I said, a more graceful handling of the link inconsistencies would be great, but I'm not too fussed by it. Especially in this case where the link is down, then up and down again within a span of only a few seconds. The root of the problem here is obviously the Link, which I've remedied.

Can you believe it was a simple cable incompatibility? While the Xserve RAID devices are 2GB max themselves, since I have them plugged in to a 4GB FC card, the Apple firmware requires a 4GB cable!

So anyway, I replaced the cables and I'll keep an eye on it.
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

Re: Kernel Panic (idle state)

Post by scasady » Tue Oct 02, 2012 7:17 pm

In answer to dbrady I am running 10.6.8 mac pro 16 G memory the pool is a v28 raidz2 of 8 500G disks
areca 1300 controller
scasady Offline


 
Posts: 45
Joined: Sat Sep 15, 2012 8:00 am

Re: Kernel Panic (idle state)

Post by elfpltfn » Tue Oct 02, 2012 9:49 pm

I think I have the same panic but no fiber.... Sorry- its a hack so I can't post the panic file,
KP IMAGE HERE and
Loaded KEXTS
I listed the kexts, then started a 12 GB file copy during a scrub, immediate panic. (which I can't repeat now)

10.6.8
uname -a
Darwin Server.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64
16 GB Ram
z68 chipset- (ga-z68a-d3-b3 if you care)
zevo 1.1

zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
happyland 9.09Ti 6.81Ti 2.28Ti 74% ONLINE -

Happy to post any more info
elfpltfn Offline


 
Posts: 23
Joined: Fri Sep 14, 2012 9:52 pm

OSx86, scrub, copy, panic et cetera

Post by grahamperrin » Wed Oct 03, 2012 4:55 am

elfpltfn wrote:… Loaded KEXTS
I listed the kexts, then started a 12 GB file copy during a scrub, immediate panic. (which I can't repeat now) … 


FakeSMC, I recognise from occasional use of Lion and Mountain Lion in VirtualBox without EFI on Apple hardware.

The other OSx86 and third party stuff is mostly unfamiliar to me.

com.bresink.driver.BRESINKx86Monitoring (8.0) catches my eye. Please, which of the range of products are used?

Postscript

Re viewtopic.php?p=287#p287 under Kernel Panic on Scrub- *ISSUE RESOLVED

Did you manage to suppress/stop the client of a kernel core dump server behaviour? Or is that to be expected with your OSx86 case?

Was the one unrepeatable panic, in this topic, before or after your resolution in the other topic?

Thanks
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Kernel Panic (idle state)

Post by scasady » Wed Oct 03, 2012 8:07 am

Another panic more less idle on the zfs disks, spotlight enabled.
macpro 10.6.8 16 G memory, 8 500G disks raidz2 on areca 1300 controller

this panic has happened a couple of times now, updated areca drivers after first one

One another topic it would be nice if the forum would allow attaching a .panic file rather than
having to cut and paste, copy etc.

panic(cpu 22 caller 0xffffff7f814d21ed): "/staging/zevo/src/uts/common/fs/zfs/zio.c:512 ZFS assertion failed: size <= (1ULL << 17) (0x5a6a00 <= 0x20000)"@/staging/zevo/src/uts/darwin/os/printf.c:62
Backtrace (CPU 22), Frame : Return Address
0xffffff8153463ba0 : 0xffffff8000204d15
0xffffff8153463ca0 : 0xffffff7f814d21ed
0xffffff8153463cc0 : 0xffffff7f81563191
0xffffff8153463d30 : 0xffffff7f81563924
0xffffff8153463dc0 : 0xffffff7f81525645
0xffffff8153463df0 : 0xffffff7f814e13c1
0xffffff8153463e30 : 0xffffff7f815251a5
0xffffff8153463f10 : 0xffffff7f8152c070
0xffffff8153463fa0 : 0xffffff80002c8527
Kernel Extensions in backtrace (with dependencies):
com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f814c7000->0xffffff7f81606fff
dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f80884000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
10K549

Kernel version:
Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64
System model name: MacPro5,1 (Mac-F221BEC8)

System uptime in nanoseconds: 69808662378841
unloaded kexts:
com.nvidia.CUDA 1.1.0 (addr 0xffffff7f80aec000, size 0x8192) - last unloaded 1260169567628
loaded kexts:
com.vmware.kext.vmioplug.10.1.21 10.1.21 - last loaded 11269746544196
com.vmware.kext.vmx86 0082.54.49
com.vmware.kext.vmnet 0082.54.49
com.vmware.kext.vsockets 90.4.22
com.vmware.kext.vmci 90.4.18
com.nvidia.CUDA 1.1.0
com.bresink.driver.BRESINKx86Monitoring 8.0
com.dvdfab.kext.fabio 1
com.makemkv.kext.daspi 1
com.getgreenbytes.driver.zfs 2012.09.14
com.getgreenbytes.filesystem.zfs 2012.09.23
com.Logitech.Unifying.HID Driver 1.2.0
com.Logitech.Control Center.HID Driver 3.3.0
com.highpoint-tech.kext.rr64x 1.0.0
com.Areca.ArcSAS 2.0.3
com.rim.driver.BlackBerryUSBDriverInt 0.0.64
com.apple.filesystems.afpfs 9.7.1
com.apple.nke.asp_tcp 5.0
com.apple.filesystems.autofs 2.1.0
com.apple.driver.AppleTyMCEDriver 1.0.2d2
com.apple.driver.AGPM 100.12.31
com.apple.driver.AppleHWSensor 1.9.3d0
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleUpstreamUserClient 3.5.7
com.apple.iokit.CHUDUtils 364
com.apple.driver.AppleMCCSControl 1.0.20
com.apple.iokit.CHUDProf 366
com.apple.driver.AppleMikeyHIDDriver 1.2.0
com.apple.GeForce 6.3.7
com.apple.kext.ATIFramebuffer 6.3.6
com.apple.driver.AudioAUUC 1.57
com.apple.driver.AppleMikeyDriver 2.0.5f14
com.apple.ATIRadeonX3000 6.3.6
com.apple.driver.Intel82574L 2.1.8b1
com.apple.driver.AudioIPCDriver 1.1.6
com.apple.driver.AppleHDA 2.0.5f14
com.apple.driver.IOBluetoothBNEPDriver 2.4.5f3
com.apple.driver.AppleUSBDisplays 289
com.apple.driver.AirPortBrcm43224 428.42.4
com.apple.driver.ACPI_SMC_PlatformPlugin 4.7.0a1
com.apple.driver.AppleLPC 1.5.1
com.apple.iokit.SCSITaskUserClient 2.6.8
com.apple.iokit.IOAHCIBlockStorage 1.6.4
com.apple.driver.AppleFWOHCI 4.7.3
com.apple.driver.AppleUSBHub 4.2.4
com.apple.BootCache 31.1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.driver.AppleAHCIPort 2.1.7
com.apple.driver.AppleUSBEHCI 4.2.4
com.apple.driver.AppleUSBUHCI 4.2.0
com.apple.driver.AppleEFINVRAM 1.4.0
com.apple.driver.AppleRTC 1.3.1
com.apple.driver.AppleHPET 1.5
com.apple.driver.AppleACPIButtons 1.3.6
com.apple.driver.AppleSMBIOS 1.7
com.apple.driver.AppleACPIEC 1.3.6
com.apple.driver.AppleAPIC 1.4
com.apple.driver.AppleIntelCPUPowerManagementClient 142.6.0
com.apple.security.sandbox 1
com.apple.security.quarantine 0
com.apple.nke.applicationfirewall 2.1.14
com.apple.driver.AppleIntelCPUPowerManagement 142.6.0
com.apple.driver.AppleProfileReadCounterAction 17
com.apple.driver.AppleProfileTimestampAction 10
com.apple.driver.AppleProfileThreadInfoAction 14
com.apple.driver.AppleProfileRegisterStateAction 10
com.apple.driver.AppleProfileKEventAction 10
com.apple.driver.AppleProfileCallstackAction 20
com.apple.iokit.IOSurface 74.2
com.apple.iokit.IOBluetoothSerialManager 2.4.5f3
com.apple.iokit.IOSerialFamily 10.0.3
com.apple.iokit.CHUDKernLib 365
com.apple.driver.DspFuncLib 2.0.5f14
com.apple.driver.AppleSMBusController 1.0.10d0
com.apple.kext.ATI5000Controller 6.3.6
com.apple.kext.ATISupport 6.3.6
com.apple.iokit.IOFireWireIP 2.0.3
com.apple.driver.AppleUSBAudio 2.7.6f6
com.apple.iokit.IOAudioFamily 1.8.3fc2
com.apple.kext.OSvKernDSPLib 1.3
com.apple.iokit.AppleProfileFamily 41.4
com.apple.driver.AppleHDAController 2.0.5f14
com.apple.iokit.IOHDAFamily 2.0.5f14
com.apple.iokit.IO80211Family 320.1
com.apple.iokit.IONetworkingFamily 1.10
com.apple.driver.AppleSMBusPCI 1.0.10d0
com.apple.driver.AppleSMC 3.1.0d5
com.apple.driver.IOPlatformPluginFamily 4.7.0a1
com.apple.nvidia.nvGF100hal 6.3.7
com.apple.NVDAResman 6.3.7
com.apple.iokit.IONDRVSupport 2.2.1
com.apple.iokit.IOGraphicsFamily 2.2.1
com.apple.iokit.IOSCSIMultimediaCommandsDevice 2.6.8
com.apple.iokit.IOBDStorageFamily 1.6
com.apple.iokit.IODVDStorageFamily 1.6
com.apple.iokit.IOCDStorageFamily 1.6.1
com.apple.driver.AppleUSBHIDKeyboard 141.5
com.apple.driver.AppleHIDKeyboard 141.5
com.apple.driver.BroadcomUSBBluetoothHCIController 2.4.5f3
com.apple.driver.AppleUSBBluetoothHCIController 2.4.5f3
com.apple.iokit.IOBluetoothFamily 2.4.5f3
com.apple.iokit.IOUSBHIDDriver 4.2.0
com.apple.driver.AppleUSBMergeNub 4.2.4
com.apple.driver.XsanFilter 402.1
com.apple.iokit.IOAHCISerialATAPI 1.2.6
com.apple.iokit.IOFireWireFamily 4.2.6
com.apple.iokit.IOSCSIParallelFamily 2.1.1
com.apple.iokit.IOUSBUserClient 4.2.4
com.apple.iokit.IOAHCIFamily 2.0.6
com.apple.driver.AppleEFIRuntime 1.4.0
com.apple.iokit.IOHIDFamily 1.6.6
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.TMSafetyNet 6
com.apple.kext.AppleMatch 1.0.0d1
com.apple.iokit.IOUSBMassStorageClass 2.6.7
com.apple.driver.AppleUSBComposite 3.9.0
com.apple.iokit.IOSCSIBlockCommandsDevice 2.6.8
com.apple.iokit.IOSCSIArchitectureModelFamily 2.6.8
com.apple.iokit.IOUSBFamily 4.2.4
com.apple.driver.DiskImages 289
com.apple.iokit.IOStorageFamily 1.6.3
com.apple.driver.AppleACPIPlatform 1.3.6
com.apple.iokit.IOPCIFamily 2.6.5
com.apple.iokit.IOACPIFamily 1.3.0
scasady Offline


 
Posts: 45
Joined: Sat Sep 15, 2012 8:00 am

Re: Kernel Panic (idle state)

Post by elfpltfn » Wed Oct 03, 2012 8:20 am

com.bresink.driver.BRESINKx86Monitoring (8.0) is the temp monitor: http://www.bresink.com/osx/TemperatureMonitor.html
Other than that its actually quite vanilla- native support for the drives, and the thing was rock steady using maczfs for the last year.. (but zfs was slower)
the client kernel core dump is expected... I boot with the 0x144 kernel flag so I get the text on screen- since I don;t have nvram its the only way to see the panic text.
FWIW I could use EFI on this board and get rid of the fakeSMC, but I have not had any trouble with stability in the past. Uptime is just time between apple updates.

The panic history with zevo goes like this:
zevo v1- One panic, which you linked to at: Re viewtopic.php?p=287#p287 under Kernel Panic on Scrub- *ISSUE RESOLVED
I have not had the same panic- (i think, the text is different) Happened during a scrub with no I/O that I'm aware of, So I doubled the ram to meet the 1Gb/ TB recommendation and was able to scrub twice with no errors, so life looked good.

zevo 1.1- 3 panics during scrub WITH I/O (I was actively writing to the pool) so I tried to see if I could induce the panic by starting a scrub and writing a large file to the pool. This is what you see above- I listed the kexts, and the next command was to copy the large file- Immediate panic with associated screenshot.
I tried to repeat the panic three times, by copying the file as the scrub completed, but no further panics.
elfpltfn Offline


 
Posts: 23
Joined: Fri Sep 14, 2012 9:52 pm

Re: Kernel Panic (idle state)

Post by BrianDieckman » Wed Oct 03, 2012 10:30 am

scasady: could you please start your own thread with your panic information. Based on the text of your panic, it has a different cause than mine.
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

off-topic: for attachments, use the full editor

Post by grahamperrin » Wed Oct 03, 2012 2:38 pm

scasady wrote:… allow attaching a .panic file


Use the full editor. When you begin a new topic, it's the default. When you reply to a message, it's an option.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Kernel Panic (idle state)

Post by scasady » Thu Oct 04, 2012 1:27 pm

did that got this response "extension .panic not allowed"
scasady Offline


 
Posts: 45
Joined: Sat Sep 15, 2012 8:00 am

PreviousNext

Return to Troubleshooting

Who is online

Users browsing this forum: hlxpgxmum, ilovezfs and 0 guests

cron