A not so friendly ZEVO kernel panic

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

A not so friendly ZEVO kernel panic

Post by CrystalPalace » Tue Nov 20, 2012 12:32 pm

Sun Nov 18 00:06:47 2012
panic(cpu 1 caller 0xffffff7f999461b7): "/staging/zevo/src/uts/common/fs/zfs/arc.c:1333 ZFS assertion failed: hdr->b_state == arc_mru || hdr->b_state == arc_mfu"@/staging/zevo/src/uts/darwin/os/printf.c:43
Backtrace (CPU 1), Frame : Return Address
0xffffff819afcba80 : 0xffffff801741d626
0xffffff819afcbaf0 : 0xffffff7f999461b7
0xffffff819afcbb00 : 0xffffff7f9994eec3
0xffffff819afcbb30 : 0xffffff7f9995e79c
0xffffff819afcbba0 : 0xffffff7f9995ee85
0xffffff819afcbbc0 : 0xffffff7f99963abe
0xffffff819afcbc40 : 0xffffff7f999645a0
0xffffff819afcbc90 : 0xffffff7f999646db
0xffffff819afcbd00 : 0xffffff7f999d1e71
0xffffff819afcbdb0 : 0xffffff8017511d3d
0xffffff819afcbe00 : 0xffffff8017507e89
0xffffff819afcbe50 : 0xffffff8017776abe
0xffffff819afcbef0 : 0xffffff8017776c50
0xffffff819afcbf50 : 0xffffff80177e182a
0xffffff819afcbfb0 : 0xffffff80174ced33
Kernel Extensions in backtrace:
com.getgreenbytes.filesystem.zfs(2012.9.23)[04497DBB-8849-31D8-8496-BE10E5711C53]@0xffffff7f9993b000->0xffffff7f99ad4fff
dependency: com.apple.iokit.IOStorageFamily(1.8)[A3CC4E44-8E10-3D9A-BA8E-95743E79D125]@0xffffff7f97978000

BSD process name corresponding to current thread: Final Cut Pro Tr
Boot args: arch=x86_64 -v

Mac OS version:
12C60

Kernel version:
Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64
Kernel UUID: 69A5853F-375A-3EF4-9247-478FD0247333
Kernel slide: 0x0000000017200000
Kernel text base: 0xffffff8017400000
System model name: MacBookPro5,1 (Mac-F42D86C8)

System uptime in nanoseconds: 30084491975214
last loaded kext at 11613856030725: com.getgreenbytes.driver.zfs 2012.09.14 (addr 0xffffff7f998fb000, size 53248)
last unloaded kext at 11879318408734: com.apple.driver.AppleUSBCDC 4.1.22 (addr 0xffffff7f998e7000, size 12288)
loaded kexts:
com.getgreenbytes.driver.zfs 2012.09.14
com.getgreenbytes.filesystem.zfs 2012.09.23
com.parallels.kext.prl_vnic 7.0 15107.796624
com.parallels.kext.prl_netbridge 7.0 15107.796624
com.parallels.kext.prl_hid_hook 7.0 15107.796624
com.parallels.kext.prl_hypervisor 7.0 15107.796624
com.parallels.kext.prl_usb_connect 7.0 15107.796624
at.obdev.nke.LittleSnitch 3883
com.apple.filesystems.smbfs 1.8
com.apple.driver.AppleBluetoothMultitouch 75.15
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.driver.AudioAUUC 1.60
com.apple.iokit.IOBluetoothSerialManager 4.0.9f33
com.apple.filesystems.autofs 3.0
com.apple.driver.AGPM 100.12.69
com.apple.driver.AppleMikeyHIDDriver 122
com.apple.driver.AppleHDA 2.3.1f2
com.apple.driver.AppleMikeyDriver 2.3.1f2
com.apple.driver.AppleLPC 1.6.0
com.apple.driver.AppleSMCLMU 2.0.2d0
com.apple.driver.AppleMuxControl 3.2.11
com.apple.iokit.IOUserEthernet 1.0.0d1
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.ApplePolicyControl 3.2.11
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.iokit.BroadcomBluetoothHCIControllerUSBTransport 4.0.9f33
com.apple.driver.AppleUpstreamUserClient 3.5.10
com.apple.driver.AppleMCCSControl 1.0.33
com.apple.GeForce 8.0.0
com.apple.driver.SMCMotionSensor 3.0.2d6
com.apple.driver.AppleUSBTCButtons 235.4
com.apple.driver.AppleUSBTCKeyboard 235.4
com.apple.driver.AppleIRController 320.15
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.BootCache 34
com.apple.iokit.SCSITaskUserClient 3.5.1
com.apple.driver.XsanFilter 404
com.apple.iokit.IOAHCIBlockStorage 2.2.2
com.apple.driver.AppleUSBHub 5.2.5
com.apple.driver.AppleFWOHCI 4.9.6
com.apple.driver.AirPort.Brcm4331 602.15.22
com.apple.driver.AppleEFINVRAM 1.6.1
com.apple.driver.AppleAHCIPort 2.4.1
com.apple.nvenet 2.0.19
com.apple.driver.AppleUSBEHCI 5.4.0
com.apple.driver.AppleUSBOHCI 5.2.5
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleRTC 1.5
com.apple.driver.AppleHPET 1.7
com.apple.driver.AppleACPIButtons 1.6
com.apple.driver.AppleSMBIOS 1.9
com.apple.driver.AppleACPIEC 1.6
com.apple.driver.AppleAPIC 1.6
com.apple.driver.AppleIntelCPUPowerManagementClient 196.0.0
com.apple.nke.applicationfirewall 4.0.39
com.apple.security.quarantine 2
com.apple.driver.AppleIntelCPUPowerManagement 196.0.0
com.apple.iokit.IOSCSIBlockCommandsDevice 3.5.1
com.apple.iokit.IOUSBMassStorageClass 3.5.0
com.apple.driver.AppleBluetoothHIDKeyboard 165.5
com.apple.driver.AppleHIDKeyboard 165.5
com.apple.driver.IOBluetoothHIDDriver 4.0.9f33
com.apple.driver.AppleMultitouchDriver 235.28
com.apple.iokit.IOSerialFamily 10.0.6
com.apple.kext.triggers 1.0
com.apple.driver.DspFuncLib 2.3.1f2
com.apple.iokit.IOAudioFamily 1.8.9fc10
com.apple.kext.OSvKernDSPLib 1.6
com.apple.iokit.IOFireWireIP 2.2.5
com.apple.driver.AppleHDAController 2.3.1f2
com.apple.iokit.IOHDAFamily 2.3.1f2
com.apple.driver.AppleBacklightExpert 1.0.4
com.apple.iokit.IOSurface 86.0.3
com.apple.iokit.IOBluetoothFamily 4.0.9f33
com.apple.driver.AppleGraphicsControl 3.2.11
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.driver.IOPlatformPluginFamily 5.2.0d16
com.apple.driver.AppleSMBusPCI 1.0.10d0
com.apple.iokit.AppleBluetoothHCIControllerUSBTransport 4.0.9f33
com.apple.driver.AppleSMBusController 1.0.10d0
com.apple.nvidia.nv50hal 8.0.0
com.apple.NVDAResman 8.0.0
com.apple.iokit.IONDRVSupport 2.3.5
com.apple.iokit.IOGraphicsFamily 2.3.5
com.apple.driver.AppleSMC 3.1.4d2
com.apple.driver.AppleUSBMultitouch 235.7
com.apple.iokit.IOUSBHIDDriver 5.2.5
com.apple.driver.AppleUSBMergeNub 5.2.5
com.apple.driver.AppleUSBComposite 5.2.5
com.apple.iokit.IOSCSIMultimediaCommandsDevice 3.5.1
com.apple.iokit.IOBDStorageFamily 1.7
com.apple.iokit.IODVDStorageFamily 1.7.1
com.apple.iokit.IOCDStorageFamily 1.7.1
com.apple.iokit.IOAHCISerialATAPI 2.5.0
com.apple.iokit.IOSCSIArchitectureModelFamily 3.5.1
com.apple.iokit.IOUSBUserClient 5.2.5
com.apple.iokit.IOFireWireFamily 4.5.5
com.apple.iokit.IO80211Family 500.15
com.apple.iokit.IOAHCIFamily 2.2.1
com.apple.iokit.IONetworkingFamily 3.0
com.apple.iokit.IOUSBFamily 5.4.0
com.apple.driver.NVSMU 2.2.9
com.apple.driver.AppleEFIRuntime 1.6.1
com.apple.iokit.IOHIDFamily 1.8.0
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.sandbox 220
com.apple.kext.AppleMatch 1.0.0d1
com.apple.security.TMSafetyNet 7
com.apple.driver.DiskImages 344
com.apple.iokit.IOStorageFamily 1.8
com.apple.driver.AppleKeyStore 28.21
com.apple.driver.AppleACPIPlatform 1.6
com.apple.iokit.IOPCIFamily 2.7.2
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.corecrypto 1.0
CrystalPalace Offline


 
Posts: 4
Joined: Sat Nov 03, 2012 10:01 am

Re: A not so friendly ZEVO kernel panic

Post by grahamperrin » Tue Nov 20, 2012 4:09 pm

Please, which version of Final Cut Pro?

Can you recall the steps that preceded the kernel panic?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: A not so friendly ZEVO kernel panic

Post by CrystalPalace » Tue Nov 20, 2012 4:48 pm

Final Cut Pro Trial 10.0.6

created project with two large (8GB) movie files on the ZFS volume (external USB HDD). Project was set up not to copy the movie files to internal HDD. Also, no optimized work copies were created.
Panic occured while FCP was rendering preview images in the timeline.
CrystalPalace Offline


 
Posts: 4
Joined: Sat Nov 03, 2012 10:01 am

Re: A not so friendly ZEVO kernel panic

Post by CrystalPalace » Tue Nov 20, 2012 4:58 pm

I had at least one other panic not related to FCP while there was fairly heavy traffic on the drive.
Memory consumption in the kernel grew enormously (approx. 2.5GB total for kernel_task). Repeating the actions after reboot caused similar behavior. Exporting the ZFS volume released 1.8GB kmem.
This machine has 8GB of RAM installed.
CrystalPalace Offline


 
Posts: 4
Joined: Sat Nov 03, 2012 10:01 am

cross reference

Post by grahamperrin » Tue Nov 20, 2012 6:22 pm

CrystalPalace wrote:… Memory consumption in the kernel …


In my experience with OS X 10.8.2 and ZEVO on a MacBookPro5,2 with 8 GB memory: the memory usage that you observe is probably not contributory to kernel panics. Please see memory usage, kernel_task and zstat.

… panic not related to FCP …


Please, what make and model is the drive? How is it connected? What results from scrub?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: A not so friendly ZEVO kernel panic

Post by CrystalPalace » Wed Nov 21, 2012 4:15 pm

zpool scrub output:
Code: Select all
  pool: zfs_500gb
 state: ONLINE
 scan: scrub repaired 0 in 2h58m with 0 errors on Wed Nov 21 21:50:29 2012
config:

   NAME                                         STATE     READ WRITE CKSUM
   zfs_500gb                                    ONLINE       0     0     0
     GPTE_6DD2408C-48C6-4E17-BB99-3F44498FEE5B  ONLINE       0     0     0  at disk1s2

errors: No known data errors

pool never got damaged.
It is a seagate 500gb drive in a external usb 2.0 enclosure. single disk, works without any problems when formatted HFS+ on various machines.
CrystalPalace Offline


 
Posts: 4
Joined: Sat Nov 03, 2012 10:01 am

thoughts and links

Post by grahamperrin » Thu Nov 22, 2012 2:33 pm

I wonder whether this kernel panic with OS X 10.8.2 will be consistently reproducible with the script given at viewtopic.php?p=896#p896 – whilst the topic there is Snow Leopard-oriented, the script is praised and I guess it could help to identify problems in other cases.

If not that script, then maybe the fstorture part of FS Tools (source – trunk). Written to find bugs in Sharity and again I guess it could be useful in other environments.

Your observation about behaviour with HFS Plus might indicate a hardware issue, one that becomes problematic only when the hardware is used more thoroughly (e.g. with checksumming for ZFS).
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Possibly unique

Post by grahamperrin » Mon Apr 15, 2013 10:35 pm

CrystalPalace wrote:Sun Nov 18 00:06:47 2012
panic(cpu 1 caller 0xffffff7f999461b7): "/staging/zevo/src/uts/common/fs/zfs/arc.c:1333 ZFS assertion failed: hdr->b_state == arc_mru || hdr->b_state == arc_mfu"@/staging/zevo/src/uts/darwin/os/printf.c:43


In this context – ARC – mfu and mru are most frequently used and most recently used.

Neither Google [Bot] nor phpBB finds any other mention of mfu in relation to a kernel panic in this forum.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: A not so friendly ZEVO kernel panic

Post by raattgift » Tue Apr 16, 2013 5:00 am

Remember that asserts tell you the source file name and line number. Most of the panics in question are in files that are identical (or very nearly so) across most implementations of ZFS. This one is no different. The problem here is a cache hit on an ARC buffer that has an inconsistent state. It is most likely a generic problem involving large write pressure on an insufficiently large ARC in the face of system memory pressure and a very slow drain of writes because ** usb 2.0 **. Ugh.

This guess does not preclude a ZEVO-specific problem; for example it could be a locking bug (make sure the very latest ZEVO CE 1.1.1 is in use to preclude compiler issues). Additionally, ZEVO constrains the ARC more than ZFS subsystems in other OSes would on the same hardware, and that is likely an exacerbating factor (but not the actual *cause* of the panic). Also, the symptoms described are not fully inconsistent with a hardware fault in the system; I'd look for error messages (particularly ones involving USB) in the logs.

However, the very first diagnostic step should be: Try the same workload pattern using a drive (even the same drive) that does not attach via USB 2 (hell, even FW400 in an ancient enclosure is better), and come back if the problem recurs.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

com.getgreenbytes.filesystem.zfs 2012.09.23

Post by grahamperrin » Tue Apr 16, 2013 5:29 pm

raattgift wrote:… ZEVO CE 1.1.1 …


Confirmed – I treat the following line (in the opening post) as a hallmark:

com.getgreenbytes.filesystem.zfs 2012.09.23
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to Troubleshooting

Who is online

Users browsing this forum: bileyqrkq, ilovezfs and 0 guests

cron