Kernel Panic (idle state)

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

Re: Kernel Panic (idle state)

Post by BrianDieckman » Tue Oct 09, 2012 1:39 pm

*EDIT* I just realized the source of this backup is a network drive. I'm not sure why the data would flow that way, but it's likely that the NIC can't keep the pipe full. I'm leaving this here for posterities sake! *

I feel so pathetic responding to my own post over and over!

I noticed something odd when doing a backup to the zpool today; the data seemed to be going on the disk in spurts... it would write, pause, write, pause, etc... So I pulled up Activity Monitor and this is what I see:

Image

The write speed is bouncing back and forth between about 80MB/sec and as low as 28KB per sec.

Also, you can see that "kernel_task" running as root is consuming 4.24 GB!!! That must be Zevo CE? There's still over 2 GB free so I'm not concerned about it, just curious.

Does anyone know what might be causing that pause in writes?
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

fibre channel speed and topology, PowerPC Xserve to Xserve R

Post by grahamperrin » Tue Oct 09, 2012 2:41 pm

BrianDieckman wrote:… an obscure knowledge base article …


URL please? Thanks.

BrianDieckman wrote:… "Point to Point" when connected directly to a FC card in the server. (Which I do... no switches involved) …


Here with a PowerPC Xserve, direct connection to a first generation Xserve RAID, the Fibre Channel pane of System Preferences 5.2 (5.2) in Mac OS X Server 10.5.8 is set to automatic for both port speed and port topology. Similarly in the Fibre Channel tab of settings for RAID Admin 1.5.1, both are set to automatic. The underlying Fibre Channel tab (in the main window) confirms what's expected in my case:

  • 2 Gb/s (automatic)
  • arbitrated loop (automatic).

Postscript, possible correction

It's a while since I checked in our server room. Two Xserve RAID units, and there was (long ago) an intention to have the more modern of the two linked to gear elsewhere on campus (not directly connected to an Xserve) but I can't recall how far we got with that plan.
Last edited by grahamperrin on Tue Oct 09, 2012 4:15 pm, edited 2 times in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Activity Monitor

Post by grahamperrin » Tue Oct 09, 2012 2:56 pm

BrianDieckman wrote:… Activity Monitor …


I can't recall whether disk activity measurements by Activity Monitor are (a) between the disk and the file system or (b) between the file system and the app.

For a moment, defocus from ZFS. There's a more generic diagram under item 2. hfsslower.d at Brendan's blog » Top 10 DTrace scripts for Mac OS X (highlights).

With my laptop (MacBookPro5,2) running ZEVO, I reckon the peaks and troughs are normal. Whilst these things are visible in utilities such as Activity Monitor, I sense no corresponding troughs in ZEVO uses of ZFS.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Kernel Panic (idle state)

Post by BrianDieckman » Tue Oct 09, 2012 3:05 pm

The KB article is here: http://support.apple.com/kb/HT1765

"If directly connecting an Xserve RAID to an Apple Fibre Card, the Xserve RAID's topology settings should be set to Point-to-Point. "

That's the first time I've read anything other than "leave it set at Automatic."

Crossing my fingers, but I've not had any panics since setting both controllers to "Point to Point"
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

Re: Kernel Panic (idle state)

Post by grahamperrin » Tue Oct 09, 2012 3:24 pm

BrianDieckman wrote:… Three settings are checked in the RAID configuration; "Use Controller Write Cache", "Allow Host Cache Flushing" and "Use Drive Write Cache". There's a fourth option "Use Steady Streaming Mode" that is unchecked. …


For readers who are unfamiliar with RAID Admin:

2012-10-09 21-19-49 screenshot.png
2012-10-09 21-19-49 screenshot.png (95.24 KiB) Viewed 88 times


– and RAID Admin Help: Xserve RAID Settings: Changing Performance Settings (Drive Cache, Controller Cache)

2012-10-09 22-01-56 screenshot.png
2012-10-09 22-01-56 screenshot.png (113.42 KiB) Viewed 88 times
Last edited by grahamperrin on Tue Oct 09, 2012 4:04 pm, edited 1 time in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Kernel Panic (idle state)

Post by BrianDieckman » Tue Oct 09, 2012 3:36 pm

I hadn't thought about it, but it makes sense to disable "Use Drive Write Cache" in our case. No video or other kind of high throughput IO is going on here; just run-of-the-mill file serving.

Oh, and check out my Network thoughput! (Definitely can't keep the pipe filled :lol: )

Image

I think its time for a dedicated VLAN for storage, don't you? :P
BrianDieckman Offline


 
Posts: 17
Joined: Tue Sep 25, 2012 2:11 pm

Re: Kernel Panic (idle state)

Post by elfpltfn » Fri Oct 12, 2012 10:20 pm

Brian- (and anyone else with this panic)
I can trigger the panic by running the following script (run from your zpool)

#!/bin/bash

while true; do
dd if=/dev/zero bs=1024k of=./zero count=10240
rm ./zero
echo "loop iterated"
done


My machine panicked the first time I ran the script. Then I rebooted- ran the script for a few (<5) loops with no panic. I killed and restarted the script, and got a loop within seconds. Could someone else try it to see if we can reproduce this panic?

Thanks-
elfpltfn Offline


 
Posts: 23
Joined: Fri Sep 14, 2012 9:52 pm

Re: Kernel Panic (idle state)

Post by scasady » Tue Oct 16, 2012 12:08 pm

zpool create internal /dev/disk2

instant panic, on booting the pool was there and works fine but had to use import -f

I get these about once a day if actually writing to a pool, just reading doesn't seem to do it.
Is this snow leopard related ? I may have to go back to openindiana if zevo can't do better on my system.
scasady Offline


 
Posts: 45
Joined: Sat Sep 15, 2012 8:00 am

Re: Kernel Panic (idle state)

Post by grahamperrin » Tue Oct 16, 2012 2:53 pm

scasady wrote:… Is this snow leopard related ? …


Reportedly the zio_notify_parent type of panic only occurs under 10.6.8.

In your case, are all panics of that type?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Kernel Panic (idle state)

Post by scasady » Tue Oct 16, 2012 5:38 pm

about half are the notify_parent type with 4 other kinds
scasady Offline


 
Posts: 45
Joined: Sat Sep 15, 2012 8:00 am

PreviousNext

Return to Troubleshooting

Who is online

Users browsing this forum: bileyqrkq, ilovezfs and 0 guests

cron