What causes ZEVO's disks to make FreeNAS kernel panic?

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

What causes ZEVO's disks to make FreeNAS kernel panic?

Post by emory » Sun Dec 30, 2012 10:36 pm

This is very puzzling to me, and I don't know if it's a ZEVO issue, a FreeBSD kernel/FreeNAS issue, or all of the above.

I have a system that I have moved a ZEVO-created mirror into, and subsequently boot that system with various USB flash installs of FreeNAS 8.3.x (tried current RELEASE, two nightly builds and a beta on two different 8GB flash drives) inducing a kernel panic right after it launches all available CPU cores and sees the disk geometry. Anecdotally the "geometry does not match label" message before the fatal trap may or may not be indicative of the root cause, my searching for similar KPs has lead to postings that this isn't a fatal condition or shouldn't be in and of itself.

I get the GEOM error about the label, and then a couple of new lines and then Fatal trap 12 while in kernel mode, a newline, and "Trying to mount root from ufs://dev/ufs/FreeNASs1a".

More recently the current process was `firmware taskq`, but other attempts usually reference `g_event`. It stops in to the debugger and I'm not competent enough to read a trace correctly.

If I pull the disks, the system will boot. If I plug them back in, it KPs. I'm inclined to think that FreeBSD/FreeNAS are not able to handle the GUID -> EFI -> ZFS layout. Honestly I don't even know what FreeBSD's ZFS disks look like or if they have a disklabel like UFS disks or what.

This isn't a show-stopper or anything for me, it's really more of a curiosity at this point. If I want to continue with testing FreeNAS on that hardware I'll just get new disks, right? But it did make me wonder if I would be able to move a ZFS volume back from FreeNAS to OS X and ZEVO if I needed to, or if OS X would balk at the disks FreeBSD's ZFS tools make.

If this isn't an appropriate question, no hard feelings for nuking it or whatever. I'll stick to my core competency of cluttering up grahamperrin's posts like I do elsewhere ;)
emory Offline


 
Posts: 15
Joined: Mon Sep 17, 2012 7:47 pm

Thoughts

Post by grahamperrin » Mon Dec 31, 2012 4:02 am

I don't imagine nuking! LOL and welcome ;)

The round trip story that I see referenced most often is this:


Highlights added by me today (currently blue) draw attention to the partition number used by FreeNAS for ZFS: for three of three slices, p2.

I assume that this use of the GPT is consistent with the slice number s2 that is typically seen when ZEVO is used according to recommendations:


As this topic is about FreeNAS I should draw attention to mentions of FreeBSD under Compatibility with other ZFS systems.

Parallel to all of this is the current guide which states, "ZEVO uses the standard ZFS on-disk format (v28) and is therefore binary compatible with ZFS on other platforms. However, direct interchange with other platforms is not supported in this version." – that's also given under a discussion about safety. Disclaimers aside …

Thoughts

Pay close attention to hardware. I see that you're already doing so –

> tried current RELEASE, two nightly builds and a beta on two different 8GB flash drives

– what makes and models are those flash drives?

Think next of all drives in the pool. Makes and models? How are connections made?

Scrub, completely, whilst the pool is used by ZEVO.

Beyond a scrub, zdb(8) can be used but for now, don't dive into zdb; explanations for panics of FreeNAS may be more mundane.

Attention to cables. Yes, I recently discarded some USB 2 cables with plugs that appeared perfect. Only the paws of cats revealed their true nature. If they weren't firmly named Pickle and Bill I'd rename them Dane and Mun.

If you'd like to share details of one of your test configurations, please paste (as code) outputs from the following commands. It's a standard-ish set of twelve, some of which may be inappropriate at this time, so feel free to cherry pick:

Code: Select all
sw_vers
kextstat | grep zfs
ls -l /Volumes
ls -l /dev/dsk
ls -l /var/zfs/dsk
diskutil list
diskutil coreStorage list
zpool upgrade
zfs upgrade
sudo zpool import
sudo zpool status -vx -T d
sudo zpool status -v


One block of code per output will be ideal. Thanks.

Last but not least, check that your ZEVO is up-to-date. The full installer is currently limited to 1.1.

Oh, and a chat with someone in the FreeNAS area might be revealing. I'm now in irc://irc.freenode.net/#freenas and will draw attention to this topic …
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

A little from the Silver era

Post by grahamperrin » Mon Dec 31, 2012 6:21 am

Again for a moment I'm treating FreeNAS and FreeBSD as under the same umbrella. Whilst saving web archives of topics in the zevo.zendesk.com domain I found a Silver Edition topic, begun by a customer:

Moving disks between OSX and FreeBSD (2012-01-26, with the last substantial post in April)

It's inappropriate for me to quote extensively from that customer area, but the following snippets seem fair:

> We now have a FreeBSD 9 system in our lab

> progress with imports from FreeBSD GPT labeled zfs pools

Without actually using FreeNAS (I plan to do so in 2013), I reckon that ZEVO Community Edition does represent progress in this area.

Refocusing on FreeNAS

Michael Dexter: The FreeNAS Storage Platform: A minute to learn, a lifetime to master – if I understand correctly, findings were presented at the NLUUG conference on 2012-04-11. If findings are available online, expect some reference to ZEVO.

Postscript

The FreeNAS Storage Platform: A minute to learn, a lifetime to master / Open Source Bridge: The conference for open source citizens / June 26-29, 2012 / Portland, OR – refers to http://callfortesting.org/ for Michael Dexter, author of the more recent article linked from my previous post: CFT: ZFS Portability.

From a ZEVO perspective, I assume that the July stuff includes the essence of whatever was in the April and June stuff.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: What causes ZEVO's disks to make FreeNAS kernel panic?

Post by emory » Mon Dec 31, 2012 10:17 am

– what makes and models are those flash drives?


One is a SanDisk and the other is an HP branded drive, I thought it was possible I had (several) bad dd's to the flash when it was panicing on boot, until I tested with no ZEVO disks inserted and it was fine.

The hardware I'm booting FreeNAS on is the same as the ZEVO system. I'm reasonably certain the hardware and cabling is OK, but I can do the commands you listed regardless.
emory Offline


 
Posts: 15
Joined: Mon Sep 17, 2012 7:47 pm

Re: What causes ZEVO's disks to make FreeNAS kernel panic?

Post by emory » Mon Dec 31, 2012 10:25 am

[strike]I should have access to the other old forum still, too. I'll look there for the post you referenced. [/strike]

I was thinking of the old UserVoice forum, not the Zendesk one, nevermind.
emory Offline


 
Posts: 15
Joined: Mon Sep 17, 2012 7:47 pm

ashift property value?

Post by grahamperrin » Sat Sep 28, 2013 11:33 pm

Revisiting this topic following a chat with someone in IRC.

Emory, please, can you recall (or tell from history) the ashift property value at the time of the panics?

9 or 12?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: What causes ZEVO's disks to make FreeNAS kernel panic?

Post by emory » Sun Sep 29, 2013 5:48 am

I'm hesitant to make a declaration one way or the other.

They were 1TB seagate disks and it's more likely that they were 9.
emory Offline


 
Posts: 15
Joined: Mon Sep 17, 2012 7:47 pm

Re: What causes ZEVO's disks to make FreeNAS kernel panic?

Post by grahamperrin » Sun Sep 29, 2013 10:14 am

Thanks for the quick reply.

If it was 9 on a drive where 12 would have been more appropriate (no fault of your own), then the topic here is probably consistent with panics that were discussed in IRC with … by coincidence, the Michael Dexter who is referenced above. Following some discussion of ashift:

> …  I can pass this v28 drive between mac and FreeNAS 9 …
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to Troubleshooting

Who is online

Users browsing this forum: bileyqrkq, ilovezfs, toduefiwu and 0 guests

cron