kernel panic when issuing zfs send

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

kernel panic when issuing zfs send

Post by caronni » Fri Jan 25, 2013 8:32 am

Hi there,

I have a zpool attached via firewire, and it seems it got corrupted in the last month -- zpool scrub identifies checksum errors.

status -v lists two files as being corrupt, named x/<0x0> and x/<0x3>. Any advice how to deal with this? I'm happy to just delete the corrupted objects, if I knew which ones they were. Besides the number, no identification is given. Anything I can do with e.g. zdb to help me along?

When issuing a zfs send on the filesystem in question, a kernel panic ensues. That is probably not what should happen. Happy to help you debug this, before cleaning up my ZFS. Here what I have so far, please advise if any more information is useful to you:

Interval Since Last Panic Report: 3804516 sec
Panics Since Last Report: 3
Anonymous UUID: 25A087FB-52ED-431D-AA5C-A6B91EA1E183

Fri Jan 25 13:44:53 2013
panic(cpu 2 caller 0xffffff7f8079c1b7): "/staging/zevo/src/uts/darwin/os/condvar.c:61 ZFS assertion failed: (cp->cv_waiters & CV_WAITERS_MASK) == 0"@/staging/zevo/src/uts/darwin/os/printf.c:43
Backtrace (CPU 2), Frame : Return Address
0xffffff81e0b3bca0 : 0xffffff8000220792
0xffffff81e0b3bd20 : 0xffffff7f8079c1b7
0xffffff81e0b3bd30 : 0xffffff7f807ab06a
0xffffff81e0b3bd50 : 0xffffff7f80794ebf
0xffffff81e0b3bd70 : 0xffffff7f807a7fdb
0xffffff81e0b3bdc0 : 0xffffff7f80831b2e
0xffffff81e0b3be40 : 0xffffff7f8082e369
0xffffff81e0b3be80 : 0xffffff7f80831b8a
0xffffff81e0b3bf00 : 0xffffff7f8082e369
0xffffff81e0b3bf40 : 0xffffff7f8079ec39
0xffffff81e0b3bfb0 : 0xffffff8000820057
Kernel Extensions in backtrace:
com.getgreenbytes.filesystem.zfs(2012.9.23)[04497DBB-8849-31D8-8496-BE10E5711C53]@0xffffff7f80791000->0xffffff7f8092afff
dependency: com.apple.iokit.IOStorageFamily(1.7.2)[40D4AB21-F11E-3C6C-8883-3554F93A251F]@0xffffff7f8076c000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
11G63

Kernel version:
Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64
Kernel UUID: FF3BB088-60A4-349C-92EA-CA649C698CE5
System model name: MacPro4,1 (Mac-F221BEC8)

System uptime in nanoseconds: 1761266360391
last loaded kext at 683785928343: com.apple.driver.AppleUSBCDC 4.1.22 (addr 0xffffff7f80931000, size 16384)
last unloaded kext at 744310699594: com.apple.driver.AppleUSBCDC 4.1.22 (addr 0xffffff7f80931000, size 12288)
loaded kexts:
com.silabs.driver.SiLabsUSBDriver 1.1.2
com.getgreenbytes.driver.zfs 2012.09.14
com.getgreenbytes.filesystem.zfs 2012.09.23
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.filesystems.autofs 3.0
com.apple.filesystems.ntfs 3.10.1
com.apple.driver.AppleMikeyHIDDriver 122
com.apple.driver.AppleUpstreamUserClient 3.5.9
com.apple.driver.AppleMCCSControl 1.0.33
com.apple.driver.AudioAUUC 1.59
com.apple.driver.AppleHDA 2.2.5a5
com.apple.driver.AppleMikeyDriver 2.2.5a5
com.apple.driver.AppleTyMCEDriver 1.0.2d2
com.apple.driver.AGPM 100.12.75
com.apple.iokit.IOUserEthernet 1.0.0d1
com.apple.iokit.IOBluetoothSerialManager 4.0.8f17
com.apple.GeForce 7.3.2
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AudioIPCDriver 1.2.3
com.apple.driver.ApplePolicyControl 3.1.33
com.apple.driver.ACPI_SMC_PlatformPlugin 5.0.0d8
com.apple.driver.AppleLPC 1.6.0
com.apple.driver.BroadcomUSBBluetoothHCIController 4.0.8f17
com.apple.driver.XsanFilter 404
com.apple.iokit.IOAHCIBlockStorage 2.1.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.BootCache 33
com.apple.iokit.SCSITaskUserClient 3.2.1
com.apple.iokit.IOAHCISerialATAPI 2.0.3
com.apple.driver.AppleFWOHCI 4.9.0
com.apple.driver.AppleUSBHub 5.1.0
com.apple.driver.Intel82574L 2.2.0b3
com.apple.driver.AppleAHCIPort 2.3.1
com.apple.driver.AppleUSBEHCI 5.1.0
com.apple.driver.AppleEFINVRAM 1.6.1
com.apple.driver.AppleUSBUHCI 5.1.0
com.apple.driver.AppleRTC 1.5
com.apple.driver.AppleHPET 1.7
com.apple.driver.AppleACPIButtons 1.5
com.apple.driver.AppleSMBIOS 1.9
com.apple.driver.AppleACPIEC 1.5
com.apple.driver.AppleAPIC 1.6
com.apple.driver.AppleIntelCPUPowerManagementClient 195.0.0
com.apple.nke.applicationfirewall 3.2.30
com.apple.security.quarantine 1.4
com.apple.security.TMSafetyNet 8
com.apple.driver.AppleIntelCPUPowerManagement 195.0.0
com.apple.kext.triggers 1.0
com.apple.driver.DspFuncLib 2.2.5a5
com.apple.driver.AppleSMBusController 1.0.10d0
com.apple.iokit.IOSurface 80.0.2
com.apple.iokit.IOSerialFamily 10.0.5
com.apple.iokit.IOAudioFamily 1.8.6fc18
com.apple.kext.OSvKernDSPLib 1.3
com.apple.iokit.IOFireWireIP 2.2.5
com.apple.driver.AppleHDAController 2.2.5a5
com.apple.iokit.IOHDAFamily 2.2.5a5
com.apple.driver.AppleGraphicsControl 3.1.33
com.apple.nvidia.nv50hal 7.3.2
com.apple.driver.AppleSMC 3.1.3d10
com.apple.driver.IOPlatformPluginLegacy 5.0.0d8
com.apple.driver.AppleSMBusPCI 1.0.10d0
com.apple.driver.IOPlatformPluginFamily 5.1.1d6
com.apple.NVDAResman 7.3.2
com.apple.iokit.IONDRVSupport 2.3.4
com.apple.iokit.IOGraphicsFamily 2.3.4
com.apple.driver.AppleUSBBluetoothHCIController 4.0.8f17
com.apple.iokit.IOBluetoothFamily 4.0.8f17
com.apple.iokit.IOUSBMassStorageClass 3.0.3
com.apple.driver.AppleUSBHIDKeyboard 160.7
com.apple.driver.AppleHIDKeyboard 160.7
com.apple.iokit.IOUSBHIDDriver 5.0.0
com.apple.iokit.IOSCSIBlockCommandsDevice 3.2.1
com.apple.iokit.IOFireWireSerialBusProtocolTransport 2.1.0
com.apple.iokit.IOFireWireSBP2 4.2.0
com.apple.driver.AppleUSBMergeNub 5.1.0
com.apple.driver.AppleUSBComposite 5.0.0
com.apple.iokit.IOSCSIMultimediaCommandsDevice 3.2.1
com.apple.iokit.IOBDStorageFamily 1.7
com.apple.iokit.IODVDStorageFamily 1.7.1
com.apple.iokit.IOCDStorageFamily 1.7.1
com.apple.iokit.IOSCSIArchitectureModelFamily 3.2.1
com.apple.iokit.IOFireWireFamily 4.4.8
com.apple.iokit.IOUSBUserClient 5.0.0
com.apple.iokit.IONetworkingFamily 2.1
com.apple.iokit.IOAHCIFamily 2.0.8
com.apple.iokit.IOUSBFamily 5.1.0
com.apple.driver.AppleEFIRuntime 1.6.1
com.apple.iokit.IOHIDFamily 1.7.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.sandbox 177.8
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.DiskImages 331.7
com.apple.iokit.IOStorageFamily 1.7.2
com.apple.driver.AppleKeyStore 28.18
com.apple.driver.AppleACPIPlatform 1.5
com.apple.iokit.IOPCIFamily 2.7
com.apple.iokit.IOACPIFamily 1.4
Model: MacPro4,1, BootROM MP41.0081.B07, 8 processors, Quad-Core Intel Xeon, 2.66 GHz, 16 GB, SMC 1.39f5
Graphics: NVIDIA GeForce GT 120, NVIDIA GeForce GT 120, PCIe, 512 MB
Memory Module: DIMM 1, 2 GB, DDR3 ECC, 1066 MHz, 0x802C, 0x31384A53463235363732415A2D3147314631
Memory Module: DIMM 2, 2 GB, DDR3 ECC, 1066 MHz, 0x802C, 0x31384A53463235363732415A2D3147314631
Memory Module: DIMM 3, 2 GB, DDR3 ECC, 1066 MHz, 0x802C, 0x31384A53463235363732415A2D3147314631
Memory Module: DIMM 4, 2 GB, DDR3 ECC, 1066 MHz, 0x802C, 0x31384A53463235363732415A2D3147314631
Memory Module: DIMM 5, 2 GB, DDR3 ECC, 1066 MHz, 0x802C, 0x31384A53463235363732415A2D3147314631
Memory Module: DIMM 6, 2 GB, DDR3 ECC, 1066 MHz, 0x802C, 0x31384A53463235363732415A2D3147314631
Memory Module: DIMM 7, 2 GB, DDR3 ECC, 1066 MHz, 0x802C, 0x31384A53463235363732415A2D3147314631
Memory Module: DIMM 8, 2 GB, DDR3 ECC, 1066 MHz, 0x802C, 0x31384A53463235363732415A2D3147314631
Bluetooth: Version 4.0.8f17, 2 service, 18 devices, 1 incoming serial ports
Network Service: Ethernet 1, Ethernet, en0
Network Service: Ethernet 2, Ethernet, en1
PCI Card: NVIDIA GeForce GT 120, sppci_displaycontroller, Slot-1
Serial ATA Device: HL-DT-ST DVD-RW GH41N
Serial ATA Device: WDC WD1001FALS-41K1B0, 1 TB
Serial ATA Device: Hitachi HUS724040ALE640, 4 TB
Serial ATA Device: Hitachi HDS724040ALE640, 4 TB
Serial ATA Device: Hitachi HUS724040ALE640, 4 TB
USB Device: hub_device, 0x0424 (SMSC), 0x2502, 0xfa200000 / 2
USB Device: hub_device, 0x0424 (SMSC), 0x2602, 0xfa210000 / 3
USB Device: Fitbit Base Station, 0x10c4 (Silicon Laboratories, Inc.), 0x84c4, 0xfa213000 / 5
USB Device: Flash Card Reader, 0x0424 (SMSC), 0x2228, 0xfa211000 / 4
USB Device: Keyboard Hub, apple_vendor_id, 0x1006, 0xfd300000 / 2
USB Device: USB Gaming Mouse, 0x046d (Logitech Inc.), 0xc041, 0xfd310000 / 4
USB Device: Apple Keyboard, apple_vendor_id, 0x0220, 0xfd320000 / 3
USB Device: BRCM2046 Hub, 0x0a5c (Broadcom Corp.), 0x4500, 0x5a100000 / 2
USB Device: Bluetooth USB Host Controller, apple_vendor_id, 0x8215, 0x5a110000 / 5
FireWire Device: built-in_hub, 800mbit_speed
caronni Offline


 
Posts: 7
Joined: Sat Sep 15, 2012 4:45 am

Link and questions

Post by grahamperrin » Fri Jan 25, 2013 4:21 pm

mistreated hardware, panic, cp->cv_waiters & CV_WAITERS_MASK

In your case, please, what makes and models are the drives in the affected pool?

Was there ever a problem – an interruption of any sort – during destruction of a dataset?
Last edited by grahamperrin on Fri Jan 25, 2013 4:45 pm, edited 2 times in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Link

Post by grahamperrin » Fri Jan 25, 2013 4:42 pm

caronni wrote:… zpool

… status -v lists two files as being corrupt, named x/<0x0> and x/<0x3> …


zpool status -v lists files that are unidentifiable
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Link and questions

Post by caronni » Sat Jan 26, 2013 10:50 am

grahamperrin wrote:mistreated hardware, panic, cp->cv_waiters & CV_WAITERS_MASK

In your case, please, what makes and models are the drives in the affected pool?

Was there ever a problem – an interruption of any sort – during destruction of a dataset?


I have never destroyed a dataset on this device, so ... no. However this being an external disk I can not vouch for connection integrity at all times, anything may have happened.
I am not so much concerned with how the fault occurred or the fact that something is now faulty. (I'm restoring the dataset from backup). What I wanted to point out is that 'zfs send' should not panic the machine in such a case, and I wanted to learn how to pinpoint the faulty data or metadata with zdb.

Drive information follows further below...

If I read zdb correctly, 0x0 and 0x3 are objects in the top level data set, namely:
Dataset x [ZPL], ID 21, cr_txg 1, 243G, 90611 objects
ZIL header: claim_txg 0, claim_blk_seq 0, claim_lr_seq 0 replay_seq 0, flags 0x0
Object lvl iblk dblk dsize lsize %full type
0 7 16K 16K 22.8M 48.0M 92.11 DMU dnode
-1 1 16K 1K 8K 1K 100.00 ZFS user/group used
-2 1 16K 2K 8K 2K 100.00 ZFS user/group used
1 1 16K 512 8K 512 100.00 ZFS master node
2 1 16K 512 8K 512 100.00 SA master node
3 1 16K 3.00K 8K 3.00K 100.00 ZFS delete queue
4 1 16K 2K 8K 2K 100.00 ZFS directory

So the DMU dnode (whatever that is) and the ZFS delete queue.


My Passport 071D:

Manufacturer: WD
Model: 0x71D
GUID: 0x14EE1206ECB7E9
Maximum Speed: Up to 800 Mb/sec
Connection Speed: Up to 800 Mb/sec
Sub-units:
My Passport 071D Unit:
Unit Software Version: 0x10483
Unit Spec ID: 0x609E
Firmware Revision: 0x1041
Product Revision Level: 1041
Sub-units:
My Passport 071D SBP-LUN:
Capacity: 2 TB (2,000,365,289,472 bytes)
Removable Media: Yes
BSD Name: disk4
Partition Map Type: GPT (GUID Partition Table)
S.M.A.R.T. status: Not Supported
Volumes:
disk4s1:
Capacity: 209.7 MB (209,715,200 bytes)
BSD Name: disk4s1
Content: EFI
Lion:
Capacity: 133.75 GB (133,746,188,288 bytes)
Available: 49.67 GB (49,666,580,480 bytes)
Writable: Yes
File System: Journaled HFS+
BSD Name: disk4s2
Mount Point: ...
Content: Apple_HFS
Recovery HD:
Capacity: 650 MB (650,002,432 bytes)
File System: Journaled HFS+
BSD Name: disk4s3
Content: Apple_Boot
disk4s4:
Capacity: 1.87 TB (1,865,625,124,864 bytes)
BSD Name: disk4s4
Content: ZFS
SES Device Unit:
Unit Software Version: 0x10483
Unit Spec ID: 0x609E
Firmware Revision: 0x1041
Product Revision Level: 1041
Sub-units:
SES Device SBP-LUN:
caronni Offline


 
Posts: 7
Joined: Sat Sep 15, 2012 4:45 am


Return to Troubleshooting

Who is online

Users browsing this forum: hlxpgxmum and 0 guests

cron