Unable to import pool

All your general support questions for OpenZFS on OS X.

Re: Unable to import pool

Postby Persecutor » Mon Jun 08, 2020 3:21 am

talios wrote:Ok... so reran Onyx and cleaned up caches and everything - rebooted - imported the pool (no -fF or anything) and.... all 3 pools came up.

Immediately ran export, which seemed to fail on the first sub pool it tried, "falling back on umount".

Looking at zpool status I see the pool is being resilvered - and then once again log stream mentioned the uncorrectable I/O failure :(


run diskutil list

Do you have a tool to check smart status of Hard drives?
Persecutor
 
Posts: 14
Joined: Wed Apr 15, 2020 1:33 am

Re: Unable to import pool

Postby talios » Mon Jun 08, 2020 1:33 pm

(thanks for being mega patient) :)

Tried it again and ran disk-util list afterwards - all drives still seem to be listed there fine. Doesn't look like I can do any SMART checks - either the enclosure doesn't support it or the drives dont ( Seagate 4tbs, and one WD Red 6tb ).

One interesting thing I notice however, is that the 6tb drive seems to have moved from disk3 to disk2 - even tho physically it's still in the same bay on the enclosure.

Code: Select all
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *512.1 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         511.9 GB   disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +511.9 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Oldmac                  408.4 GB   disk1s1
   2:                APFS Volume Preboot                 22.1 MB    disk1s2
   3:                APFS Volume Recovery                512.1 MB   disk1s3
   4:                APFS Volume VM                      5.4 GB     disk1s4

/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *6.0 TB     disk2
   1:                        ZFS orico                   6.0 TB     disk2s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk2s9

/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk3
   1:                        ZFS orico                   4.0 TB     disk3s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk3s9

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk4
   1:                        ZFS orico                   4.0 TB     disk4s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk4s9

/dev/disk5 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk5
   1:                        ZFS orico                   4.0 TB     disk5s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk5s9

/dev/disk6 (internal, virtual):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        +17.9 TB    disk6
   1:                ZFS Dataset                         17.9 TB    disk6s1



I'm also curious about the resilvering - since the zpool was striped, I didn't think resilvering would be possible? Or is that more copying data from one part of say disk2, to another part of disk2?
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby tangles » Mon Jun 08, 2020 4:21 pm

Having read these posts… wow…

This is a whole bag of hurt… :shock:

Clearly there's no backup given your efforts to resurrect…

I haven't found on the web how to stop a resilver, only how to stop a scrub and mounting as readonly doesn't seem to help you here either.

you could to commands back to back.
Try:
Code: Select all
zpool mount <pool> ; zfs scrub -s <pool>
in an effect to stop but I don't think it will work. The only way I know is to yank the disk, which is not what you want to do…

Hoping things will just magically fix themselves by repeatedly trying to mount the pool isn't (hasn't?) going to get you anywhere.
You need to put USB to one side (ideally forever for ZFS) and use SATA so that you can read the health of your disks.

To verify:
I'd go and buy a Thunderbolt to eSATA adapter, then buy a SATA drive dock of some sorts and test each disk via Smart Utility. This will, once and for all, tell you if any disks are having physical hardware issues. However if all report healthy, then you know to never use USB ever again! :geek:

Other options are spending considerable money with a professional data-recovery company if you do identify a faulty disk. Otherwise I think you're data are cactus and I think it's time to move on.

Even if some ZFS guru can get your pool mounted sans the problematic disk(s), every file you've ever written to the pool has been spread across all the disks, and so you're still unlikely to recover anything usable.

Moving forward:

If your 6TB drive is dead, create a new raidz with the remaining 4TB disks, and purchase a single 10TB to back everything up…

Don't use USB for ZFS-hosted data you can't afford to lose. (USB chipsets although cheap, can be nasty (i.e. they lie to the OS when flushing cache to disk and just not worth the risk)
Don't stripe your pools if you don't have redundancy. (if you wan't a faster pool using rotational disks, use more disks that are smaller/cheaper in size and more vdevs in your pool)
Don't use any computer if you don't have a backup of data you cannot afford to lose.

I don't mean to stomp and I know it's a painful lesson.
tangles
 
Posts: 180
Joined: Tue Jun 17, 2014 6:54 am

Re: Unable to import pool

Postby talios » Mon Jun 08, 2020 6:42 pm

Clearly there's no backup given your efforts to resurrect…


Thankfully - for the truly important bits of information ( 5tb of RAW photos and 1.5tb of music (itunes, plus podcast production/field recordings) all backed up/replicated off site (in the cloud, and for the photos also on another HDD locally ). That leaves about 1.5tb of movies/tv episodes, and rendered recordings that whilst would be good to recover, are not essential.

I guess a big part now however - is in part, just wanting to recover SOME thing -just because-, but also - at least being able to generate a file listing so I know what I've lost so I don't keep coming back down the line going "where was that content..." - most of that would be useful to have, but not essential.

With regards to chaining the zpool mount <pool ; scrub -s calls - the call to 'mount' never actually returns - no I/O error, nothing.

Re "professional data-recovery company" - had already spoken to a company locally who quoted (after a chat about the scenario) a starting point of around N$1000 - and given I'm looking at replacing it with a $1.5k NAS box plus new drives, not worth it.

eSATA dock/adapter is an option - thankfully this other machine I have here has Thunderbolt.
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby talios » Mon Jun 08, 2020 6:48 pm

I don't mean to stomp and I know it's a painful lesson.


...and one I knew was coming - the data, and the near dead machine it was attached to were both slated for retirement. Lockdown/pandemic didn't help with delays in hardware releases, or availability.

(and god - the time it takes to restore 1.5tb over the internet is hideous )
talios
 
Posts: 15
Joined: Mon Feb 06, 2017 1:21 am

Re: Unable to import pool

Postby Persecutor » Tue Jun 09, 2020 12:10 am

talios wrote:(thanks for being mega patient) :)

Tried it again and ran disk-util list afterwards - all drives still seem to be listed there fine. Doesn't look like I can do any SMART checks - either the enclosure doesn't support it or the drives dont ( Seagate 4tbs, and one WD Red 6tb ).

One interesting thing I notice however, is that the 6tb drive seems to have moved from disk3 to disk2 - even tho physically it's still in the same bay on the enclosure.

Code: Select all
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *512.1 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         511.9 GB   disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +511.9 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Oldmac                  408.4 GB   disk1s1
   2:                APFS Volume Preboot                 22.1 MB    disk1s2
   3:                APFS Volume Recovery                512.1 MB   disk1s3
   4:                APFS Volume VM                      5.4 GB     disk1s4

/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *6.0 TB     disk2
   1:                        ZFS orico                   6.0 TB     disk2s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk2s9

/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk3
   1:                        ZFS orico                   4.0 TB     disk3s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk3s9

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk4
   1:                        ZFS orico                   4.0 TB     disk4s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk4s9

/dev/disk5 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk5
   1:                        ZFS orico                   4.0 TB     disk5s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk5s9

/dev/disk6 (internal, virtual):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        +17.9 TB    disk6
   1:                ZFS Dataset                         17.9 TB    disk6s1



I'm also curious about the resilvering - since the zpool was striped, I didn't think resilvering would be possible? Or is that more copying data from one part of say disk2, to another part of disk2?



Do the pools unmount them selves after a wile?

After the initial import run this.

4.) Run sudo xattr -rd com.apple.quarantine / - Could take some time, and you needn't worry about the output.
5.) Run sudo xattr -rd com.apple.quarantine /<poolname>
6.) Run exit before closing the Terminal!
7.) Restart
Persecutor
 
Posts: 14
Joined: Wed Apr 15, 2020 1:33 am

Re: Unable to import pool

Postby Persecutor » Tue Jun 09, 2020 12:46 am

After the initial import force-quit the Finder and see if the re-silvering continues properly
Persecutor
 
Posts: 14
Joined: Wed Apr 15, 2020 1:33 am

Re: Unable to import pool

Postby tangles » Sun Jun 14, 2020 10:28 pm

Seeing though this is just for kicks…

How bout booting from Ubuntu Live 20.04 which has ZFS built in?

Nothing in /etc/zfs/zpool.cache to hinder your attempts

if the pool mounts, but not the datasets, try zfs mount -a as well.

It might be able to get you a bit further and then see if you also progress further when booting back to macOS…

:?
tangles
 
Posts: 180
Joined: Tue Jun 17, 2014 6:54 am

Re: Unable to import pool

Postby lundman » Mon Jun 15, 2020 10:20 pm

Sorry, busy with new port, I forget to take breaks and check in here. Come on irc if you want my attention faster.

zpool scrub -s will stop a scrub of course (not resilver tho). But there's the option of letting it finish too.
There are tunables to make scrub do no-IO, and it just goes through the motions.
User avatar
lundman
 
Posts: 813
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Previous

Return to General Help

Who is online

Users browsing this forum: No registered users and 5 guests