mount_zfs kernel panic during startup of Mountain Lion

Moderators: jhartley, MSR734, nola

mount_zfs kernel panic during startup of Mountain Lion

Post by grahamperrin » Thu Sep 20, 2012 1:36 am

Since installing ZEVO Community Edition 1.1 over a Ten's Complement private beta: on maybe a handful of occasions the OS has rebooted itself automatically during startup (shortly after EfiLoginUI, almost certainly before loginwindow). Startups are verbose.

I assumed kernel panics, but before today: the text that precedes EfiLoginUI gave no indication that the automatic reboot followed a panic; and there were no .panic files for the incidents.

Today the automatic reboot certainly followed a panic. Key points:

Code: Select all
      Kernel Extensions in backtrace:
         com.getgreenbytes.filesystem.zfs(2012.9.14)[8241F82D-269B-30AD-86AB-AB7C3D520F2B]@0xffffff7f92ba5000->0xffffff7f92d41fff
            dependency: com.apple.iokit.IOStorageFamily


Code: Select all
BSD process name corresponding to current thread: mount_zfs


At a glance this is comparable to Mounting pool via Disk Utility causes KP ( 2012-09-15). However I'd like to keep this topic separate – it's exceptional for me to find this type of panic so early within the system startup routine.

Early observations

The kernel may be more likely to panic in this way if one of my external disks with ZFS is connected before I attempt to start the OS.

For the disk that was attached at the time of this morning's panic, one of the datasets has compression=gzip-9.

Environment

MacBookPro5,2

8 GB memory

Code: Select all
macbookpro08-centrim:~ gjp22$ diskutil list disk0
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *750.2 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS swap                    32.0 GB    disk0s2
   3: FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF               536.9 MB   disk0s3
   4:                  Apple_HFS spare                   671.1 MB   disk0s4
   5:          Apple_CoreStorage                         99.5 GB    disk0s5
   6:                 Apple_Boot Boot OS X               650.0 MB   disk0s6
   7:          Apple_CoreStorage                         616.3 GB   disk0s7
   8:                 Apple_Boot Boot OS X               134.2 MB   disk0s8


disk0s5 provides disk1 at /

disk0s7 provides my home directory at /Volumes/gjp22

(For now, please ignore the type peculiarity at slice 3.)

Code: Select all
macbookpro08-centrim:~ gjp22$ df -lP
Filesystem   512-blocks      Used Available Capacity  Mounted on
/dev/disk1    193682496 166918840  26251656    87%    /
/dev/disk0s4    1310720     56192   1254528     5%    /Volumes/spare
/dev/disk0s2   62500000  17133672  45366328    28%    /Volumes/swap
/dev/disk3    970043331 852761528 117281803    88%    /Volumes/zhandy
/dev/disk3s1  221980016 104698213 117281803    48%    /Volumes/zhandy/Pocket Time Machine
/dev/disk5   1073094480 612942920 460151560    58%    /Volumes/gjp22


Code: Select all
macbookpro08-centrim:~ gjp22$ zfs list -o name,used,available
NAME                          USED   AVAIL
gjp22                        344Gi   219Gi
zhandy                       527Gi  55.9Gi
zhandy/Pocket Time Machine  49.9Gi  55.9Gi
macbookpro08-centrim:~ gjp22$ zpool status
  pool: gjp22
 state: ONLINE
 scan: scrub repaired 0 in 6h17m with 0 errors on Tue Sep 18 01:36:54 2012
config:

   NAME                                         STATE     READ WRITE CKSUM
   gjp22                                        ONLINE       0     0     0
     GPTE_71B8BDA2-3EBA-4B91-9E1C-2AE2B1DAAD06  ONLINE       0     0     0  at disk4s2

errors: No known data errors

  pool: zhandy
 state: ONLINE
 scan: scrub canceled on Wed Sep 19 22:07:47 2012
config:

   NAME                                         STATE     READ WRITE CKSUM
   zhandy                                       ONLINE       0     0     0
     GPTE_1928482A-7FE4-482D-B692-3EC6B03159BA  ONLINE       0     0     0  at disk2s2

errors: No known data errors


Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get compression gjp22
NAME   PROPERTY     VALUE     SOURCE
gjp22  compression  off       local
macbookpro08-centrim:~ gjp22$ zfs get compression zhandy
NAME    PROPERTY     VALUE     SOURCE
zhandy  compression  on        local
macbookpro08-centrim:~ gjp22$ zfs get compression "zhandy/Pocket Time Machine"
NAME                        PROPERTY     VALUE     SOURCE
zhandy/Pocket Time Machine  compression  -         -


Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get all gjp22
NAME   PROPERTY              VALUE                  SOURCE
gjp22  type                  filesystem             -
gjp22  creation              Sat Jul 14 11:13 2012  -
gjp22  used                  344Gi                  -
gjp22  available             219Gi                  -
gjp22  referenced            292Gi                  -
gjp22  compressratio         1.16x                  -
gjp22  mounted               yes                    -
gjp22  quota                 none                   default
gjp22  reservation           none                   default
gjp22  recordsize            128Ki                  default
gjp22  mountpoint            /Volumes/gjp22         default
gjp22  checksum              on                     default
gjp22  compression           off                    local
gjp22  atime                 off                    local
gjp22  devices               on                     default
gjp22  exec                  on                     default
gjp22  setuid                on                     default
gjp22  readonly              off                    default
gjp22  snapdir               visible                local
gjp22  canmount              on                     default
gjp22  copies                1                      default
gjp22  version               5                      -
gjp22  utf8only              on                     -
gjp22  normalization         formD                  -
gjp22  casesensitivity       insensitive            -
gjp22  refquota              none                   default
gjp22  refreservation        none                   default
gjp22  primarycache          all                    default
gjp22  secondarycache        all                    default
gjp22  usedbysnapshots       51.1Gi                 -
gjp22  usedbydataset         292Gi                  -
gjp22  usedbychildren        242Mi                  -
gjp22  usedbyrefreservation  0                      -
gjp22  logbias               latency                default
gjp22  sync                  standard               default
macbookpro08-centrim:~ gjp22$ clear


Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get all zhandy
NAME    PROPERTY              VALUE                  SOURCE
zhandy  type                  filesystem             -
zhandy  creation              Wed Mar 14 17:29 2012  -
zhandy  used                  527Gi                  -
zhandy  available             55.9Gi                 -
zhandy  referenced            407Gi                  -
zhandy  compressratio         1.13x                  -
zhandy  mounted               yes                    -
zhandy  quota                 none                   default
zhandy  reservation           none                   default
zhandy  recordsize            128Ki                  default
zhandy  mountpoint            /Volumes/zhandy        default
zhandy  checksum              on                     default
zhandy  compression           on                     local
zhandy  atime                 off                    local
zhandy  devices               on                     default
zhandy  exec                  on                     default
zhandy  setuid                on                     default
zhandy  readonly              off                    default
zhandy  snapdir               visible                local
zhandy  canmount              on                     default
zhandy  copies                1                      local
zhandy  version               5                      -
zhandy  utf8only              off                    -
zhandy  normalization         none                   -
zhandy  casesensitivity       insensitive            -
zhandy  refquota              none                   default
zhandy  refreservation        none                   default
zhandy  primarycache          all                    default
zhandy  secondarycache        all                    default
zhandy  usedbysnapshots       70.2Gi                 -
zhandy  usedbydataset         407Gi                  -
zhandy  usedbychildren        50.0Gi                 -
zhandy  usedbyrefreservation  0                      -
zhandy  logbias               latency                default
zhandy  sync                  standard               default
macbookpro08-centrim:~ gjp22$ clear


Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get all "zhandy/Pocket Time Machine"
NAME                        PROPERTY              VALUE                                SOURCE
zhandy/Pocket Time Machine  type                  filesystem                           -
zhandy/Pocket Time Machine  creation              Fri Jun 22 15:51 2012                -
zhandy/Pocket Time Machine  used                  49.9Gi                               -
zhandy/Pocket Time Machine  available             55.9Gi                               -
zhandy/Pocket Time Machine  referenced            49.9Gi                               -
zhandy/Pocket Time Machine  compressratio         1.12x                                -
zhandy/Pocket Time Machine  mounted               yes                                  -
zhandy/Pocket Time Machine  quota                 none                                 default
zhandy/Pocket Time Machine  reservation           none                                 default
zhandy/Pocket Time Machine  recordsize            128Ki                                default
zhandy/Pocket Time Machine  mountpoint            /Volumes/zhandy/Pocket Time Machine  default
zhandy/Pocket Time Machine  checksum              on                                   default
zhandy/Pocket Time Machine  atime                 off                                  inherited from zhandy
zhandy/Pocket Time Machine  devices               on                                   default
zhandy/Pocket Time Machine  exec                  on                                   default
zhandy/Pocket Time Machine  setuid                on                                   default
zhandy/Pocket Time Machine  readonly              off                                  default
zhandy/Pocket Time Machine  snapdir               visible                              inherited from zhandy
zhandy/Pocket Time Machine  canmount              on                                   default
zhandy/Pocket Time Machine  copies                1                                    inherited from zhandy
zhandy/Pocket Time Machine  version               5                                    -
zhandy/Pocket Time Machine  utf8only              off                                  -
zhandy/Pocket Time Machine  normalization         none                                 -
zhandy/Pocket Time Machine  casesensitivity       insensitive                          -
zhandy/Pocket Time Machine  refquota              none                                 default
zhandy/Pocket Time Machine  refreservation        none                                 default
zhandy/Pocket Time Machine  primarycache          all                                  default
zhandy/Pocket Time Machine  secondarycache        all                                  default
zhandy/Pocket Time Machine  usedbysnapshots       368Ki                                -
zhandy/Pocket Time Machine  usedbydataset         49.9Gi                               -
zhandy/Pocket Time Machine  usedbychildren        0                                    -
zhandy/Pocket Time Machine  usedbyrefreservation  0                                    -
zhandy/Pocket Time Machine  logbias               latency                              default
zhandy/Pocket Time Machine  sync                  standard                             default
macbookpro08-centrim:~ gjp22$
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: mount_zfs kernel panic during startup of Mountain Lion

Post by dbrady » Thu Sep 20, 2012 3:41 pm

Can you remove the mount_zfs link?

Code: Select all
$ sudo rm /usr/sbin/mount_zfs


It's not need by zfs.
dbrady Offline


 
Posts: 67
Joined: Wed Sep 12, 2012 12:43 am

/usr/sbin/mount_zfs removed

Post by grahamperrin » Fri Sep 21, 2012 8:05 pm

OK, thanks – that symlink to
/System/Library/Filesystems/zfs.fs/Contents/Resources/bin/mount_zfs
is now removed from /usr/sbin

Incidentally:

  • before the removal, I performed a succession of around ten quick and relatively simple attempts to reproduce the panic – probably all involved a restart by a user whose home directory is on the jhfs+ OS volume
  • all those quick attempts were panic-free, which made me wonder whether – on previous occasions – extended usage of some sort, before a restart by me (gjp22), was a factor
  • the laptop now has released Build 12C54 of OS X 10.8.2.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: mount_zfs kernel panic during startup of Mountain Lion

Post by dbrady » Fri Sep 21, 2012 10:20 pm

There is a known panic during a classic remount call under 10.8. We have a fix that we hope to release shortly. Sometimes during a boot, the system will unmount and then remount volumes. It's possible that this mount activity is sensitive to the duration of a boot. The workaround is to avoid using /sbin/usr/mount_zfs until a fix is ready.

You can always use "zpool import" or "zfs mount -a" if an unexpected unmount occurs.
dbrady Offline


 
Posts: 67
Joined: Wed Sep 12, 2012 12:43 am

Re: mount_zfs kernel panic during startup of Mountain Lion

Post by grahamperrin » Sat Sep 22, 2012 3:26 am

dbrady wrote:… It's possible that this mount activity is sensitive to the duration of a boot.


OK.

Pending a fix (thanks), I'll pay close attention to the order of things during verbose boots, but see below re: Mountain Lion and force.

dbrady wrote:… avoid using /sbin/usr/mount_zfs … 


I have no recollection of using mount_zfs – I'll avoid doing so.

dbrady wrote:… if an unexpected unmount occurs.


For what it's worth, I never noticed one.

Back to the opening post:

grahamperrin wrote:… almost certainly before loginwindow. …


Now, I'm a little less certain. I vaguely recall wandering in from another room and seeing the backtrace etc. maybe overlaying the Apple icon, for a split-second. I'll restart shortly to remind myself whether that icon first appears before or after automated login (of the user whose passphrase is entered at EfiLoginUI).

(The force applied by Mountain Lion – no time for a photograph when the kernel panics – can be frustrating.)

----

Another thought. Quoting from Ask Different:

> At unlock time, the system may appear to reject the passphrase (maybe because of the content hint mismatch).

Code: Select all
sh-3.2$ date
Sat 22 Sep 2012 09:07:22 BST
sh-3.2$ sw_vers
ProductName:   Mac OS X
ProductVersion:   10.8.2
BuildVersion:   12C54
sh-3.2$ diskutil coreStorage list
CoreStorage logical volume groups (2 found)
|
+-- Logical Volume Group 039C0D47-F3CF-44D3-A825-B48F01FCF334
|   =========================================================
|   Name:         OS
|   Size:         99484213248 B (99.5 GB)
|   Free Space:   0 B (0 B)
|   |
|   +-< Physical Volume 86D2FA98-8C69-4828-B909-8312AE4A75E2
|   |   ----------------------------------------------------
|   |   Index:    0
|   |   Disk:     disk0s5
|   |   Status:   Online
|   |   Size:     99484213248 B (99.5 GB)
|   |
|   +-> Logical Volume Family 7AA96B8E-0E41-4C3F-9589-5FAE0C956372
|       ----------------------------------------------------------
|       Encryption Status:       Unlocked
|       Encryption Type:         AES-XTS
|       Conversion Status:       Complete
|       Conversion Direction:    -none-
|       Has Encrypted Extents:   Yes
|       Fully Secure:            Yes
|       Passphrase Required:     Yes
|       |
|       +-> Logical Volume B13EE5BF-5D08-49D3-94C2-DF58AFEA1D08
|           ---------------------------------------------------
|           Disk:               disk1
|           Status:             Online
|           Size (Total):       99165437952 B (99.2 GB)
|           Size (Converted):   -none-
|           Revertible:         No
|           LV Name:            OS
|           Volume Name:        OS
|           Content Hint:       Apple_HFS
|
+-- Logical Volume Group 902434C9-0131-4E3A-AE15-2B8B938087AD
    =========================================================
    Name:         gjp22-cs
    Size:         616336003072 B (616.3 GB)
    Free Space:   0 B (0 B)
    |
    +-< Physical Volume 179AADE6-34F1-404C-A994-9FD99C881BA6
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk0s7
    |   Status:   Online
    |   Size:     616336003072 B (616.3 GB)
    |
    +-> Logical Volume Family FFCE2FAF-BE8E-4FEF-9F3E-E221C6CBCA11
        ----------------------------------------------------------
        Encryption Status:       Unlocked
        Encryption Type:         AES-XTS
        Conversion Status:       Complete
        Conversion Direction:    -none-
        Has Encrypted Extents:   Yes
        Fully Secure:            Yes
        Passphrase Required:     Yes
        |
        +-> Logical Volume 0CFAFD38-E79B-40AC-A4BE-63296E6B4331
            ---------------------------------------------------
            Disk:               disk2
            Status:             Online
            Size (Total):       616017227776 B (616.0 GB)
            Size (Converted):   -none-
            Revertible:         No
            LV Name:            gjp22-cs
            Content Hint:       Apple_HFS
sh-3.2$


With my current setup, at EfiLoginUI I login as user 'hello' with a home directory on the startup volume. Think of this account as a dummy, it's typically used for only a few seconds in my startup/login routine. The login keychain for 'hello' includes the password for the UUID of the encrypted CoreStorage logical volume 'gjp22-cs' that underlies the ZFS dataset for my true home directory (user gjp22, Graham Perrin). In other words:

  1. EfiLoginUI unlock of the startup volume as user 'hello'
  2. encrypted ZFS home directory for user 'gjp22' mounts automatically, thanks to the keychain entry
  3. OS X 10.8.2 presents another prompt for the passphrase for LV gjp22-cs – that LV is already mounted, so I ignore the prompt
  4. immediately logout
  5. loginwindow login as user Graham Perrin.

So. Whilst I never notice the following sequence:

  • mount, unmount, mount

– I do always notice:

  • unlock, mount, second prompt to unlock and mount.

I wonder whether that secondary behaviour (an OS response to me using CoreStorage in an unusual way) is a factor in this case. But I never noticed a mount_zfs panic during startup with Ten's Complement ZEVO on 10.8.1 or 10.8.0.
Last edited by grahamperrin on Sat Sep 22, 2012 3:59 am, edited 1 time in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

boot durations: wait4path and dynamic_pager

Post by grahamperrin » Sat Sep 22, 2012 3:51 am

dbrady wrote:… sensitive to the duration of a boot. …


Another peculiarity that I chose for my configuration (sorry!). There's a wait4path here:

Code: Select all
macbookpro08-centrim:~ gjp22$ defaults read /System/Library/LaunchDaemons/com.apple.dynamic_pager
{
    EnableTransactions = 1;
    HopefullyExitsLast = 1;
    Label = "com.apple.dynamic_pager";
    OnDemand = 0;
    ProgramArguments =     (
        "/bin/bash",
        "-c",
        "/bin/wait4path /Volumes/swap && /sbin/dynamic_pager -F /Volumes/swap/swapfile"
    );
}


In practice, in rare cases following a forced restart or forced stop of the OS:
    /Volumes/swap
is a folder (and dynamic_pager writes to that folder);
and the simple JHFS+ volume
    swap
is mounted at an adjacent point.

Whether that sort of thing affects boot duration I don't know, but I thought I should mention it.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 1 guest

cron