recover half mirror

All your general support questions for OpenZFS on OS X.

recover half mirror

Postby patmuk » Wed Apr 10, 2019 1:09 pm

Hi,

I repurposed half my mirrors disk to build up a apfs storage. Unfortunately once I had all data copied over the apfs (JBOD) raid crashed. (I learned: apfs JBOD can not be encrypted).
Luckily all my important data was saved in the cloud.

However, is there a was to get all my data back off my zfs disks?
Unfortunately I forgot to use the split command und just removed the mirrored disks. Additionally I moved the disks to another Mac, though I still have the old one (but nothing is different there).

My disks look like this:
Code: Select all
sudo zpool import
Password:
   pool: datapool2
     id: 11302927174597401557
  state: FAULTED
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
   devices and try again.
   The pool may be active on another system, but can be imported using
   the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-3C
 config:

   datapool2                                       FAULTED  corrupted data
     mirror-0                                      DEGRADED
       17913126321332455576                        UNAVAIL  cannot open
       media-A95BBDDC-8515-D347-8045-29A2E98574A5  ONLINE

   pool: datapool1
     id: 3571308118281011652
  state: FAULTED
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
   devices and try again.
   The pool may be active on another system, but can be imported using
   the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-3C
 config:

   datapool1                                       FAULTED  corrupted data
     mirror-0                                      DEGRADED
       media-5DF040D0-4447-F94F-9F4D-C7C61EF19B7D  ONLINE
       1265025113965553803                         UNAVAIL  cannot open

   pool: secure
     id: 8912466580830241136
  state: DEGRADED
 status: One or more devices are missing from the system.
 action: The pool can be imported despite missing or damaged devices.  The
   fault tolerance of the pool may be compromised if imported.
   see: http://zfsonlinux.org/msg/ZFS-8000-2Q
 config:

   secure                                          DEGRADED
     mirror-0                                      DEGRADED
       media-02113D4C-4BCB-EC46-86A9-5FF955DAF611  ONLINE
       2241914880731699939                         UNAVAIL  cannot open

Code: Select all
sudo zpool import datapool1
Password:
cannot import 'datapool1': one or more devices is currently unavailable

The status is FAULTED - but then again the disk is shown as online. Is there a way to access the data?

Code: Select all
sudo zpool import secure

Runs indefinitely. I had it running for a week, until I switched it off. I guess the process hangs.


I am running o3x v1.8.1 on Mac OS Mojave 10.14.3
Code: Select all
brew cask info openzfs
openzfs: 1.8.1,25
https://openzfsonosx.org/
/usr/local/Caskroom/openzfs/1.8.1,25 (19 files, 56.2MB)


Any ideas? Thanks a lot for reading and thinking about my issue!!!
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby lundman » Wed Apr 10, 2019 9:15 pm

AH yes, you should be able to import those half mirrors, so I'm surprised you are having troubles doing so.

Perhaps if you run import on a pool, and it "hangs" can you run "spindump" so we can see what it is doing?
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: recover half mirror

Postby patmuk » Thu Apr 11, 2019 3:34 am

Sure - I started
Code: Select all
sudo import secure

today at 10.30h, now it is still running (since 3 hours).

Activity Monitor does not show the process as hanging (not sure if it did before), but it should not take that long :)
Attached is the spindump for 3 hour after start (I leave it running).

I have a spindump for right after starting the process, if needed.

Thanks!
Attachments
spindump_zpool_import_secure_3h.txt
spindump 3h after start
(1.72 MiB) Downloaded 632 times
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby patmuk » Fri Apr 12, 2019 2:08 am

Here is another spindump, now running for 25.5h.
Attachments
spindump_zpool_import_secure_25.5h.txt
(1.74 MiB) Downloaded 562 times
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby patmuk » Sun Apr 14, 2019 2:18 pm

It is still running, without any visible difference ... attached is a spindump after 61.5h.

Any ideas what I could do?
Well, the important data on this pool "secure" (my pictures, etc.) I could backup, only my backups I could not save in time (no big loss).
But how about the "datapool" pools which are reported as faulty but have the remaining drive as online?

Thanks again.
Attachments
spindump_zpool_import_secure_61.5h.txt
(1.7 MiB) Downloaded 561 times
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby lundman » Sun Apr 14, 2019 4:39 pm

Sorry, crazy weekend - you didn't have to leave it running after the first spin dump.. taking a look now.

Ok, import thread:
Code: Select all
                                 *1000  spa_import + 1336 (zfs + 395638) [0xffffff7f853eb976]
                                   *1000  spa_config_update + 193 (zfs + 430236) [0xffffff7f853f409c]
                                     *1000  txg_wait_synced + 212 (zfs + 460197) [0xffffff7f853fb5a5]


So it is waiting on txg to sync, and txg syncing is...

Code: Select all
   *1000  txg_sync_thread + 632 (zfs + 459078) [0xffffff7f853fb146]
     *1000  spa_sync + 3184 (zfs + 415526) [0xffffff7f853f0726]
       *1000  bpobj_iterate_impl + 638 (zfs + 53442) [0xffffff7f853980c2]
         *1000  dmu_free_range + 57 (zfs + 93508) [0xffffff7f853a1d44]
           *1000  dnode_free_range + 441 (zfs + 166817) [0xffffff7f853b3ba1]
             *1000  dmu_buf_will_dirty_impl + 138 (zfs + 66717) [0xffffff7f8539b49d]
               *1000  dbuf_read + 1642 (zfs + 61911) [0xffffff7f8539a1d7]
                 *1000  zio_wait + 610 (zfs + 820007) [0xffffff7f85453327]
                   *1000  spl_cv_wait + 51 (spl + 6465) [0xffffff7f84197941]
                     *1000  msleep + 98 (kernel + 7178386) [0xffffff80008d8892]


Is waiting on IO to be read, and never getting it.

You could consider trying some import flags, for example zpool import -o readonly=on poolname which could skip the free_range call that locks.
But if that fails, zpool import -F poolname, or zpool import -V poolname

You only have to wait about a minute on the import to know it has hung.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: recover half mirror

Postby patmuk » Tue Apr 16, 2019 12:50 am

Hi Lundman,

no problem at all - you have a real life as well! Your help here in the forum and contribution to the project in general is highly appreciated and by no way expected!!
Thanks a lot for this!!!

Your suggestion helped, I was able to import my pool secure with
Code: Select all
sudo zpool import -o readonly=on secure


I had to disable the auto-import first. This was most probably stocked in the first place, so that all subsequent calls to pool import where blocked by this.
To anybody with a similar problem, you can do this with
Code: Select all
sudo mkdir /Library/LaunchDaemons/deactivated
sudo mv /Library/LaunchDaemons/org.openzfsonosx.zpool-import-all.plist /Library/LaunchDaemons/deactivated/org.openzfsonosx.zpool-import-all.plist


Importing my other pools look strange. It looks like they are imported, but not mounted. Is that what the -V flag does (I could not figure this.) How can I mount it to my file system?
Here is my command trail:
Code: Select all
$ sudo zpool import -o readonly=on datapool1
cannot import 'datapool1': one or more devices is currently unavailable
$ sudo zpool import -V -o readonly=on datapool1

$ sudo zpool import -V -o readonly=on datapool1
cannot import 'datapool1': a pool with that name already exists
use the form 'zpool import <pool | id> <newpool>' to give it a new name

$ sudo zfs list
no datasets available

$ sudo zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
datapool1      -      -      -        -         -      -      -      -  FAULTED  -

$ sudo zpool status
  pool: datapool1
 state: FAULTED
status: One or more devices could not be opened.  There are insufficient
   replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
   see: http://zfsonlinux.org/msg/ZFS-8000-3C
  scan: none requested
config:

   NAME                                            STATE     READ WRITE CKSUM
   datapool1                                       FAULTED      0     0     0  corrupted data
     mirror-0                                      DEGRADED     0     0     0
       media-5DF040D0-4447-F94F-9F4D-C7C61EF19B7D  ONLINE       0     0     0
       1265025113965553803                         UNAVAIL      0     0     0  was /private/var/run/disk/by-id/media-47DBA685-EEE4-7842-B437-1347665B384F


It looks somehow imported but not mounted?

Thanks a lot!
Patrick
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby patmuk » Tue Apr 16, 2019 1:48 pm

something more: if I mount it with:
Code: Select all
zfs mount datapool1

I get
Code: Select all
cannot open 'datapool1': I/O error


I have snapshots on that drive, though I never made a checkpoint manually.
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby lundman » Tue Apr 16, 2019 7:28 pm

So perhaps it really thinks it needs the mirror for that dataset. Most peculiar indeed.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: recover half mirror

Postby patmuk » Tue Apr 23, 2019 2:24 am

I have been off some days (happy eastern!).

Strange for a mirror - shouldn't all data be replicated on both disks?
Is there anything coming to your mind to recover the dataset, or at least to be very sure that it is gone?
What could be missing from one disk in a mirror?


Thanks!
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Next

Return to General Help

Who is online

Users browsing this forum: No registered users and 22 guests