Pool import with missing log device

Moderators: jhartley, MSR734, nola

Pool import with missing log device

Post by wonkywonky » Tue Sep 10, 2013 6:16 am

I was under the (apparently mistaken) impression that Pool v19 onwards allowed the both the removal of a ZIL device and the import of a pool with a missing ZIL device. I just tried this but with my cache+log SSD removed, the pool isn't available for import at all.

Which pool versions support pool import with missing log devices?
wonkywonky Offline


 
Posts: 25
Joined: Fri Sep 14, 2012 11:33 pm

Re: Pool import with missing log device

Post by grahamperrin » Tue Sep 10, 2013 1:50 pm

Is the device truly missing, or simply not available for use with ZEVO?

Was ZEVO used to run a zpool remove command?

Are you using the -m option at import time?

If you command
sudo zpool import
then is nothing listed?

ZFS Pool Version 19 (Community Group zfs.19) - XWiki is archived at http://web.archive.org/web/201303160832 ... oup+zfs/19

> ZFS log device removal
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Pool import with missing log device

Post by maxijazz » Fri Sep 27, 2013 4:08 pm

Hi,
I do have "similar" problem. I had configured 8MB ZIL log and 120GB cache on SSD drive. Unfortunately i just lost it, what made pool unavailable.
How do i recover from that?

Code: Select all
$ sudo zpool status
no pools available
$ sudo zpool import -F -m basic
cannot import 'basic': pool may be in use from other system
use '-f' to import anyway
$ sudo zpool import -f -m
  pool: basic
    id: 6542612681790799850
 state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
   devices and try again.
config:

   basic                                          UNAVAIL  missing device
     mirror-0                                     ONLINE
       GPTE_6D8FB03E-C9CA-4D1D-8DBE-C487EF97D0DD  ONLINE
       GPTE_7770868B-4233-4E3A-8783-CD55C68F01B0  ONLINE
   cache
     GPTE_162250FE-5A24-4D1B-94CB-6BDE0A075765

   Additional devices are known to be part of this pool, though their
   exact configuration cannot be determined.
Last edited by maxijazz on Fri Sep 27, 2013 4:14 pm, edited 1 time in total.
maxijazz Offline


 
Posts: 12
Joined: Wed Sep 19, 2012 8:42 pm

Re: Pool import with missing log device

Post by ilovezfs » Fri Sep 27, 2013 4:14 pm

Use OpenIndiana or OmniOS to import & export the pool.
ilovezfs Online


 
Posts: 249
Joined: Sun Feb 10, 2013 9:02 am

Re: Pool import with missing log device

Post by maxijazz » Sat Sep 28, 2013 6:24 am

Thank you for the suggestions.
I did use Live OpenIndiana DVD, imported the "basic" pool, removed "logs" and "cache" from "basic" pool and then cleanly exported. No problems with that, nice experience.
Then upon booting back into OSX i get kernel panic:
Code: Select all
Interval Since Last Panic Report:  148444 sec
Panics Since Last Report:          1
Anonymous UUID:                    DA718344-67B1-48EB-CB2B-6B1DAF44D225

Sat Sep 28 13:02:23 2013
panic(cpu 10 caller 0xffffff7f893b11b7): "/staging/zevo/src/uts/common/fs/zfs/zil.c:367 ZFS assertion failed: !claimed || !(zh->zh_flags & ZIL_CLAIM_LR_SEQ_VALID) || (max_blk_seq == claim_blk_seq && max_lr_seq == claim_lr_seq)"@/staging/zevo/src/uts/darwin/os/printf.c:43
Backtrace (CPU 10), Frame : Return Address
0xffffff82b96636a0 : 0xffffff8008e1d636
0xffffff82b9663710 : 0xffffff7f893b11b7
0xffffff82b9663720 : 0xffffff7f8943e90e
0xffffff82b9663950 : 0xffffff7f8943f087
0xffffff82b9663980 : 0xffffff7f893d3eff
0xffffff82b9663a50 : 0xffffff7f893d291a
0xffffff82b9663a70 : 0xffffff7f89400f13
0xffffff82b9663bb0 : 0xffffff7f893ff92a
0xffffff82b9663c00 : 0xffffff7f893ff46c
0xffffff82b9663c90 : 0xffffff7f894247e3
0xffffff82b9663cd0 : 0xffffff7f894246af
0xffffff82b9663d10 : 0xffffff8008f2774d
0xffffff82b9663d60 : 0xffffff8008f190e4
0xffffff82b9663dd0 : 0xffffff8008f0f509
0xffffff82b9663e20 : 0xffffff8009152f83
0xffffff82b9663e50 : 0xffffff800917fa23
0xffffff82b9663f50 : 0xffffff80091e97ba
0xffffff82b9663fb0 : 0xffffff8008ecf453
      Kernel Extensions in backtrace:
         com.getgreenbytes.filesystem.zfs(2012.9.23)[04497DBB-8849-31D8-8496-BE10E5711C53]@0xffffff7f893a6000->0xffffff7f89540fff
            dependency: com.apple.iokit.IOStorageFamily(1.8)[5BA4CD36-E96D-3A9E-ADFF-A863BBD63BC7]@0xffffff7f89379000

BSD process name corresponding to current thread: zpool

Mac OS version:
12F37

Kernel version:
Darwin Kernel Version 12.5.0: Mon Jul 29 16:33:49 PDT 2013; root:xnu-2050.48.11~1/RELEASE_X86_64
Kernel UUID: B1B58405-A2B2-3C44-B25D-495053D52AB5
Kernel slide:     0x0000000008c00000
Kernel text base: 0xffffff8008e00000
System model name: MacPro5,1 (Mac-F221BEC8)


What should i do now?
maxijazz Offline


 
Posts: 12
Joined: Wed Sep 19, 2012 8:42 pm

Re: Pool import with missing log device

Post by ilovezfs » Sat Sep 28, 2013 6:47 am

Sounds like a ZEVO bug.

I would use OpenIndiana to create send streams of your filesystems, assuming you have the space. Then in a fresh pool, receive the streams.

For example...

On OpenIndiana:
zfs snapshot mypool/mydataset1@restoreme
zfs send mypool/mydataset1@restoreme > mypool-mydataset1-restoreme.sr

On ZEVO:
zpool create mynewpool /dev/.......
zfs receive mynewpool/mydataset1 < mypool-mydataset1-restoreme.sr
ilovezfs Online


 
Posts: 249
Joined: Sun Feb 10, 2013 9:02 am

Re: Pool import with missing log device

Post by grahamperrin » Sat Sep 28, 2013 12:29 pm

Reading part of this topic alongside SPL panic on corrupt ZIL log · Issue #43 · zfsonlinux/spl (closed 2011-07-01) makes me wonder whether in ZEVO Community Edition 1.1.1 (a product in development at the time of its release) there's debug code.

Thoughts?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Pool import with missing log device

Post by ilovezfs » Sat Sep 28, 2013 10:46 pm

grahamperrin wrote:Reading part of this topic alongside SPL panic on corrupt ZIL log · Issue #43 · zfsonlinux/spl (closed 2011-07-01) makes me wonder whether in ZEVO Community Edition 1.1.1 (a product in development at the time of its release) there's debug code.

Thoughts?


Well it does have zdb. The problem is that because of auto import, if there is a pool that results in panic, you are in trouble as soon as it is plugged in. The work around, as you know, is to change the partition type in the gpt using /usr/sbin/gpt or gdisk, which will prevent ZEVO from autoimporting. Then, you can use zdb. Depending on what you want to do with zdb, you will either be running the command against a disk or a pool. If you are running it against a pool, and the pool is exported--as would be the case when you're dealing with a pool that causes panic on import--you tell zdb that the pool is exported with the "-e" option, and you tell zdb the path(s) to the pool's disks with the "-p" option (similar to the "-d" option used with zpool import). So you would zdb -e -p /dev [other options] MyPoolName.

In maxijazz's particular circumstances, the most straightforward path, assuming he has sufficient disk space available, is to use the send/receive method to generate a replacement pool. Alternatively, he could go down the path of using "debug" code (i.e., zdb) and trying some things there. Unfortunately, ZEVO and OS X have no equivalent to mdb, and the aok and zfs_recover options, so there is a limit to how much repair you can accomplish with ZEVO without triggering panic in the process. For an example of the use of mdb, aok, and zfs_recover, see the following:
https://blogs.oracle.com/middleview/ent ... s_disaster

I have not looked into it, but I assume the illumos distributions (e.g., OpenIndiana) have mdb, aok, and zfs_recover, since these are things which date back to the days of OpenSolaris. Since ZEVO has pool version 28, one can of course use Oracle Solaris 11 to do repairs which require mdb, aok, and zfs_recover.
ilovezfs Online


 
Posts: 249
Joined: Sun Feb 10, 2013 9:02 am

Re: Pool import with missing log device

Post by maxijazz » Sun Sep 29, 2013 3:50 am

Thank you ilovezfs for your guidance!

Since i did not have big enough intermediary media (i did try "zfs send basic/mydataset1@restoreme | gzip > basic-mydataset1-restoreme.sr" to external 1TB disk over USB2 with hope the data will fit in), but it was very, very slow, so I did following:
1) on openindiana: split my mirrored pool "basic" (zpool detach basic c1d0s2)
2) on osx: created new zpool "newbasic" on just detached hard drive (c1d0s2)
3) on openindiana: zfs snapshot basic/mydataset1@restoreme; zfs send basic/mydataset1@restoreme | zfs receive -F newbasic/mydataset1; zpool destroy basic
4) on osx: attached the drive from destroyed pool (zfs attach newbasic c0d0s2), it is resilvering right now....


It is good to have so supportive community.
maxijazz Offline


 
Posts: 12
Joined: Wed Sep 19, 2012 8:42 pm

Re: Pool import with missing log device

Post by ilovezfs » Sun Sep 29, 2013 4:10 am

You're welcome. Glad that you got this sorted out. I assume you just had that one dataset?
ilovezfs Online


 
Posts: 249
Joined: Sun Feb 10, 2013 9:02 am

Next

Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 0 guests

cron