recover half mirror

All your general support questions for OpenZFS on OS X.

Re: recover half mirror

Postby lundman » Tue Apr 23, 2019 9:13 pm

All writes happen to all vdevs in a mirror, at the same time, so it is hard to imagine this situation to happen. If it was a failing HDD maybe, but in this case, you just removed a mirror. Did you try the -T txg import? Since you can import txg-1, and therefore step one txg back into the past, maybe up to -16, and see if it was a recent write. You could also send the data, and turn on the zfs send ignore-errors tunable.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: recover half mirror

Postby patmuk » Wed Apr 24, 2019 3:37 pm

Yes, I just removed the drive. Should have used
Code: Select all
split
, but didn't thought about that.

Thanks a bunch for your suggestions. They didn't work, but maybe I understood them wrong?

Code: Select all
sanpat@SanPat-Mac-mini ~> sudo zpool import -o readonly=on -T 16 datapool1 d16
cannot import 'datapool1' as 'd16': one or more devices is currently unavailable

sanpat@SanPat-Mac-mini ~> sudo zpool import -o readonly=on -T 16 -V datapool1 d16
sanpat@SanPat-Mac-mini ~> sudo zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
d16       -      -      -        -         -      -      -      -  FAULTED  -
sanpat@SanPat-Mac-mini ~> sudo zfs mount d16
cannot open 'd16': I/O error


And send
Code: Select all
sanpat@SanPat-Mac-mini ~> sudo zfs send -ignore-errors d16 >> t.txt
cannot open 'd16': I/O error


Did I misunderstood you?
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby lundman » Wed Apr 24, 2019 4:51 pm

Ah so the rolling back of txg, you first need to lookup the current txg, which you can do with zdb $pool | grep txg. Say it is 123456. Then you try import -T 123455 - ie, one less, to go back one write, in case that last write was what made it wrong.
You can go back as many as there are uberblocks, so something like 16.

The send, you need to set
sysctl kstat.zfs.darwin.tunable.zfs_send_corrupt_data=1

then issue the send.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: recover half mirror

Postby patmuk » Thu Apr 25, 2019 11:12 am

First of all: Thanks a lot for your continuing support and not giving up yet :))

I still could not make it work, here are the responese:

Code: Select all
sanpat@SanPat-Mac-mini ~> sudo zdb -e -d datapool1
zdb: can't open 'datapool1': Device not configured

It works for the pool "secure"

Code: Select all
sanpat@SanPat-Mac-mini ~> sudo sysctl kstat.zfs.darwin.tunable.zfs_send_corrupt_data=1
kstat.zfs.darwin.tunable.zfs_send_corrupt_data: 0 -> 1
sanpat@SanPat-Mac-mini ~>  sudo zfs send -ignore-errors datapool1 >> t.txt
cannot open 'datapool1': I/O error


I can´t export the pool as well
Code: Select all
sudo zpool export -f datapool1
cannot export 'datapool1': one or more devices is currently unavailable


Any other options coming to your mind? Can I force zdb in some way?
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby lundman » Thu Apr 25, 2019 3:36 pm

Yes, you can use "-l" with zdb and feed it a /dev/disk entry instead, and it reads the label off the disk.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: recover half mirror

Postby patmuk » Sun Apr 28, 2019 1:48 pm

Thanks! Looks like I am getting closer - but I still get the I/O error.
It says that no trusted uberblck was found ...
Code: Select all
sanpat@SanPat-Mac-mini ~> sudo zdb -e datapool1 -l distk2s1

Configuration for import:
        vdev_children: 1
        version: 5000
        pool_guid: 3571308118281011652
        name: 'datapool1'
        txg: 18742113
        state: 0
        hostid: 2779865448
        hostname: 'Patricks-Mac-mini-2.local'
        vdev_tree:
            type: 'root'
            id: 0
            guid: 3571308118281011652
            children[0]:
                type: 'mirror'
                id: 0
                guid: 6299982245232111674
                whole_disk: 0
                metaslab_array: 34
                metaslab_shift: 35
                ashift: 12
                asize: 4000771997696
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 6980676228928256395
                    whole_disk: 1
                    DTL: 229
                    create_txg: 4
                    path: '/private/var/run/disk/by-id/media-5DF040D0-4447-F94F-9F4D-C7C61EF19B7D'
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 1265025113965553803
                    path: '/private/var/run/disk/by-id/media-47DBA685-EEE4-7842-B437-1347665B384F'
                    whole_disk: 1
                    not_present: 1
                    DTL: 76
                    create_txg: 4
zdb: can't open 'datapool1': Device not configured

ZFS_DBGMSG(zdb):
spa_import: importing datapool1
spa_load(datapool1, config trusted): LOADING
spa_load(datapool1, config untrusted): FAILED: no valid uberblock found
spa_load(datapool1, config untrusted): UNLOADING


I tried importing it with txg 18742113 - 18742107, 18742100, 18742000 and 10742000, but always got
Code: Select all
sanpat@SanPat-Mac-mini ~> sudo zpool import -o readonly=on -T 18742109 -V datapool1 d1
sanpat@SanPat-Mac-mini ~> zfs mount d1
cannot open 'd1': I/O error
sanpat@SanPat-Mac-mini ~> zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
d1        -      -      -        -         -      -      -      -  FAULTED  -


Admittedly, I imported it always using a new label, without exporting it, as the export (even with -f) does not work as it claims that one or more device is busy.

can I do something about the spa_laoding failure?
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby lundman » Mon May 06, 2019 4:25 pm

No uber blocks? that is peculiar. I honestly don't know what to suggest at this point - is it possible to see if it imports on other ZFS platforms?
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: recover half mirror

Postby patmuk » Wed May 08, 2019 11:20 pm

I can connect the disks to another Mac. I could install a linux platform and try it with another hfs implementation, if that is what you mean?
Or do you mean importing another dataset?
patmuk
 
Posts: 19
Joined: Fri May 13, 2016 2:41 pm

Re: recover half mirror

Postby lundman » Thu May 09, 2019 4:51 pm

Right yeah, boot one of the other platform CDs, like Linux or Illumos - no need to necessarily install the OS, then try importing. I heard some people had success with Klennet ZFS Recovery as well, but I've not tried it.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Previous

Return to General Help

Who is online

Users browsing this forum: Google [Bot] and 32 guests