Unable to import pool

All your general support questions for OpenZFS on OS X.

Unable to import pool

Postby talios » Mon May 25, 2020 4:52 pm

Hi all,

Long time user but first time poster, things have been good and I've never had to post - anyway (and I can already hear cries of anguish and horror as I type this out)....

I've been running a 5 bay USB enclosure on and now, very very aging (2010) macbook pro with a ZFS pool of 4 drives (and sadly when I set it up it was striped, so no redundancy ) - and now, either that enclosure or the macbook has degraded its self so far that I started getting the occasional "insufficient replicas" on the pool when too many things start working on it - and normally shutting down and rebooting the laptop works gets things up and running again (due to this I've been investigating proper NAS units and/or alternatives for storage needs) but....

as of last night the pool doesn't appear to import anymore, I see all the drives/partitions appearing as USB attached drives - just nothing imports - and no sign of logging, or errors I could find either ( I'm not on that machine at the moment to pull out exact details just now ).

After shutting down again, I tried plugging the enclosure into this machine and tried importing, but was greeted with a message saying it was still in use with another hardware id, and I should import with -f.

This however, also just greeted me with no errors, no visible logging, and no import. I'm beginning to suspect the enclosure is toast, but I was wondering if anyone could point me at where I should be looking for import related logging, or any tools I could try?

Whilst most of the content is backed up in the cloud, and I have a 4tb mirror of my RAW photos on another drive - restoring 2tb of music and audio recordings over the internet is less than ideal so any pointers/assistance in helping me resurrect the drives ( if possible, now I'm really wishing I'd fixed my stripe'd setup early on ).
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby JasonBelec » Tue May 26, 2020 8:07 am

Enclosure is toast. Should have been replaced when you started getting errors. Replace the Connecting cable as well. I do hope you were scrubbing regularly, especially after each issue. Stop. Breathe. Get a new enclosure. Then try importing. Report back.
JasonBelec
 
Posts: 32
Joined: Mon Oct 26, 2015 1:07 pm

Re: Unable to import pool

Postby talios » Fri May 29, 2020 5:55 pm

JasonBelec wrote:Enclosure is toast. Should have been replaced when you started getting errors. Replace the Connecting cable as well. I do hope you were scrubbing regularly, especially after each issue. Stop. Breathe. Get a new enclosure. Then try importing. Report back.


So I've finally managed to get myself a new pair of enclosures (ended up having pick up two 2-bays for now), after plugging in the drives and powering up running
Code: Select all
diskutil list
gives me:

Code: Select all
/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk2
   1:                        ZFS orico                   4.0 TB     disk2s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk2s9

/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *6.0 TB     disk3
   1:                        ZFS orico                   6.0 TB     disk3s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk3s9

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk4
   1:                        ZFS orico                   4.0 TB     disk4s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk4s9

/dev/disk5 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk5
   1:                        ZFS orico                   4.0 TB     disk5s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk5s9


Running
Code: Select all
sudo zpool import orico
just sits there idle/hung, nothing happening and no audible activity on the drives - altho the red 'activity' light for all drives seems to be constantly on. Running
Code: Select all
zpool status
from another terminal just hung as well.

There doesn't appear to be anything useful/informative in any of the
Code: Select all
/var/log/org.openzfsforosx.*
logs ( and that which is in there, doesn't seem to include timestamps - at least in some of them ).

Is there a way to enable verbose logging/debug information at all? I'll power cycle and try importing again with -F and see if that gives a better response, but so far it's looking to be similar behaviour with the other enclosure.
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby talios » Fri May 29, 2020 6:16 pm

After rebooting and attempting to import with -F - whilst tailing all /var/log/org.openzfs* all I see is:

Code: Select all
Invoking "all-syslog.sh" eid=8 pid=584
Finished "all-syslog.sh" eid=8 pid=584 exit=0
Invoking "all-syslog.sh" eid=9 pid=798
Finished "all-syslog.sh" eid=9 pid=798 exit=0
Invoking "all-syslog.sh" eid=10 pid=800
Finished "all-syslog.sh" eid=10 pid=800 exit=0
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby talios » Fri May 29, 2020 10:42 pm

Reading thru https://github.com/openzfs/zfs/issues/6244 of similar hangs on imports (tho that's Linux, this is High Siera) I wonder if I'm just.... impatient and not having ANY output in logs, or console is just causing me to reboot and stop any metadata validation or something?

zbd gives me a whole bunch of information of which I have 0 clue about, but the fact it's given me such information makes me think on some level the zpool/disks are 'ok' (for some measure of ok):

Code: Select all
orico:
    version: 5000
    name: 'orico'
    state: 0
    txg: 21854772
    pool_guid: 13366791899605306046
    errata: 0
    hostid: 4130486334
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 4
    vdev_tree:
        type: 'root'
        id: 0
        guid: 13366791899605306046
        children[0]:
            type: 'disk'
            id: 0
            guid: 11354069279616341825
            path: '/private/var/run/disk/by-id/media-B4A7A707-023F-9D44-B31B-8BECDC1B1CAA'
            whole_disk: 1
            metaslab_array: 38
            metaslab_shift: 35
            ashift: 12
            asize: 4000771997696
            is_log: 0
            DTL: 222
            create_txg: 4
            com.delphix:vdev_zap_leaf: 457
            com.delphix:vdev_zap_top: 458
        children[1]:
            type: 'disk'
            id: 1
            guid: 6339990289877466168
            path: '/private/var/run/disk/by-id/media-B3892C7B-DF73-4041-99FF-44514FAA3807'
            whole_disk: 1
            metaslab_array: 35
            metaslab_shift: 35
            ashift: 12
            asize: 4000771997696
            is_log: 0
            DTL: 224
            create_txg: 4
            com.delphix:vdev_zap_leaf: 459
            com.delphix:vdev_zap_top: 460
        children[2]:
            type: 'disk'
            id: 2
            guid: 10554904334305637959
            path: '/private/var/run/disk/by-id/media-40E96DD8-6737-C44F-BB34-23BEE12505A7'
            whole_disk: 1
            metaslab_array: 172
            metaslab_shift: 35
            ashift: 9
            asize: 4000771997696
            is_log: 0
            DTL: 223
            create_txg: 269686
            com.delphix:vdev_zap_leaf: 461
            com.delphix:vdev_zap_top: 462
        children[3]:
            type: 'disk'
            id: 3
            guid: 17672770556559364431
            path: '/private/var/run/disk/by-id/media-52003BB9-E6CE-9949-9A0A-16E077A709D3'
            whole_disk: 1
            metaslab_array: 402
            metaslab_shift: 35
            ashift: 9
            asize: 6001160355840
            is_log: 0
            DTL: 404
            create_txg: 5149503
            com.delphix:vdev_zap_leaf: 463
            com.delphix:vdev_zap_top: 464
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data



I have noticed I can feel slight activity on the drives pressing my hand against the top of the enclosures, so even tho I don't see any visual indication of activity anywhere, it seems _something_ is happening.

Any pointers or additional suggestions?
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby nodarkthings » Sat May 30, 2020 1:01 am

I wish I could help, but I can only bring my two cents: when I discovered ZFS, years ago, I started experimenting with USB enclosures or keys and fried a few ones... This probably wasn't due to ZFS but the intense activity the drives/keys had to endure, but I read other people, then, having issues with USB: it was enough to keep me away from ZFS on USB. :mrgreen:
Since then, I've read contrasted experiences about it but I still think it's not the best interface for ZFS.
If you search for USB in the forum, you'll soon be convinced, and I can only quote lundman:
https://openzfsonosx.org/forum/viewtopic.php?f=11&t=3207&p=9379&hilit=usb#p9080
Perhaps things are better with USB3 but I can't tell...
nodarkthings
 
Posts: 174
Joined: Mon Jan 26, 2015 10:32 am

Re: Unable to import pool

Postby Persecutor » Sat May 30, 2020 4:59 am

Hey,

here is what you can do, not harming anything. I don't know your OS, so...

1.) Install OpenZFS from the provided installer .dmg, https://openzfsonosx.org/w/images/6/6f/ ... _1.9.4.dmg, according to your OS. Update brew with "brew update && brew upgrade" !!!
2.) Restart without any USB Hardware attached!
3.) Run ONYX.app with everything enabled to clear out or delete. Download according to your OS! https://www.titanium-software.fr/en/onyx.html - Restart!
3.a) Run: kextstat | grep lundman in Terminal. If both Extensions are loaded s. §4, If not, please report back here!
4.) After restart, attach the original Case for the HDD`s, with the original Slot-order!!!
5.) Run "sudo zpool import <poolname>", give your password and see what happens.

Important is the Slot-Order for the HDD`s and their naming in discutil !!!

If you want, I can give it a look with Teamviewer.
Persecutor
 
Posts: 19
Joined: Wed Apr 15, 2020 1:33 am

Re: Unable to import pool

Postby talios » Sat May 30, 2020 11:49 pm

1) OpenZFS was already installed on the (same/original) - High Sierra) machine - maybe I should try it on this machine Catalina one as well.... Updated home brew - wow, didn't realize I'd not updated on that machine for along time :)
2) done.
3) Ran ONYX and cleaned up all the things it seemed to want to clean up
3a) kextstat gave me
Code: Select all
   75    1 0xffffff7f80c6f000 0x4fc      0x4fc      net.lundman.kernel.dependencies.33 (12.5.0) EC2CE887-355D-4B75-8824-08165B5860BC
   76    1 0xffffff7f80c70000 0x11f5000  0x11f5000  net.lundman.spl (1.9.4) 3FF65C3D-2197-371C-8888-0CA18513E410 <75 7 5 4 3 1>
   77    0 0xffffff7f81ea5000 0x3bb000   0x3bb000   net.lundman.zfs (1.9.4) 1967216C-271B-3B09-BEAB-F884ACB0F245 <76 25 7 5 4 3 1>

4) Moved the HDDs back into the original enclosure (correct order).
5) Ran sudo zpool import -f - initially things actually looked like they were doing something, first drive was being accessed, then the other drives were being touch based, then nothing. The zpool import call has been running/idle for about 1.5 hours now - tailing all the log files shows no activity at all. At least point, I'd be super happy seeing an I/O error at least.

..... ok - WEIRD. I decided to reboot, and was planning on running the import via strace and seeing what low level info I could get, but low and behold - DISK ACTIVITY!

and now I'm greeting with:

Code: Select all
  pool: orico
 state: ONLINE
status: One or more devices has experienced an error resulting in data
   corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
   entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: resilvered 0 in 0 days 00:00:05 with 0 errors on Mon Apr 27 21:47:01 2020
config:

   NAME                                          STATE     READ WRITE CKSUM
   orico                                         ONLINE       0     0     0
     media-B4A7A707-023F-9D44-B31B-8BECDC1B1CAA  ONLINE       0     0     0
     media-B3892C7B-DF73-4041-99FF-44514FAA3807  ONLINE       0     0     0
     media-40E96DD8-6737-C44F-BB34-23BEE12505A7  ONLINE       0     0     0
     media-52003BB9-E6CE-9949-9A0A-16E077A709D3  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <metadata>:<0x0>
        <metadata>:<0x20e>
        <metadata>:<0x1f>
        <metadata>:<0x21f>
        <metadata>:<0x4f>
        <metadata>:<0x8f>
        <metadata>:<0x1a7>
        <metadata>:<0xcc>
        <metadata>:<0x1d5>
        <metadata>:<0x1f5>
        <metadata>:<0x1f7>
        /Volumes/orico/Movies Unwatched/The.Art.Of.Self-Defense.2019.720p.BluRay.x264-[YTS.LT].mp4/<xattrdir>/com.apple.FinderInfo


and

Code: Select all
NAME           USED  AVAIL  REFER  MOUNTPOINT
orico         12.9T  2.88T  3.61T  /Volumes/orico
orico/Music   2.65T  2.88T  2.65T  /Volumes/audiowork
orico/Photos  6.61T  2.88T  6.61T  /Volumes/orico/Photos
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
orico                                         16.3T  12.9T  3.39T        -         -    32%    79%  1.00x  ONLINE  -
  media-B4A7A707-023F-9D44-B31B-8BECDC1B1CAA  3.62T  3.59T  37.1G        -         -    53%  99.0%
  media-B3892C7B-DF73-4041-99FF-44514FAA3807  3.62T  3.59T  37.0G        -         -    48%  99.0%
  media-40E96DD8-6737-C44F-BB34-23BEE12505A7  3.62T  3.59T  36.4G        -         -    43%  99.0%
  media-52003BB9-E6CE-9949-9A0A-16E077A709D3  5.44T  2.16T  3.28T        -         -     1%  39.7%


OMG - best birthday present ever seeing those metadata errors!

Right - I just started a scrub, with a plan to move the data off those drives elsewhere ASAP. I do note the fan on that old enclosure is just not doing anything, so I'm thinking I should more than likely stop the scrub, export the zpool, move the drives to the two new enclosures I got and continue the scrub.

Is there anything specific one needs to know about moving zpool drives beween enclosures?

PS: You guys rock, and here's hoping QNAP release that new TS653D soon (and I hope QuTS Hero support is available - which moves QNAP NASs to ZFS).
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby talios » Sun May 31, 2020 12:59 am

Ok... may have spoken too soon. After awhile that turned up with "insufficient replicas", and once again won't import - trying again with -fF ( I noticed I wasn't using the capital F one before ) but no joy yet.
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby Persecutor » Sun May 31, 2020 1:52 am

talios wrote:Ok... may have spoken too soon. After awhile that turned up with "insufficient replicas", and once again won't import - trying again with -fF ( I noticed I wasn't using the capital F one before ) but no joy yet.


Try the new enclosure. Remember the Slot-order! The next steps won't work on Catalina!!!
After successful mounting, do this.

1.) Finder-Info for the Pool. Change it to your Username as owner rw! and wheel r and everyone r. Apply for everything on Disk! Could take a while, so be patient.
2.) Run touch /Volumes/<poolname>/.metadata_never_index
3.) Move the Spotlight Index to the Trash and delete it.
4.) Run sudo xattr -rd com.apple.quarantine / - Could take some time, and you needn't worry about the output.
5.) Run sudo xattr -rd com.apple.quarantine /<poolname>
6.) Run exit before closing the Terminal!
7.) Restart
Persecutor
 
Posts: 19
Joined: Wed Apr 15, 2020 1:33 am

Next

Return to General Help

Who is online

Users browsing this forum: No registered users and 37 guests