corrupted metadata / mixed up HDDs?

[pollux23]

Hey everyone,

I signed up for this forum because a couple of hours of research, testing and trying didn't get me anywhere.

I have the following problem:
- Had a ZFS pool running on my Mac Pro 2012, consisting of 4x 4TB HDDs
- moved the drives to another machine (TrueNAS Scale VM with dedicated HBA), but didn't export the pool before that
- couldn't import my pool on the TrueNAS VM, so moved the drives back to my Mac Pro
- now zpool import won't let me import the pool

Depending on which parameters I use for the import, I get different clues about the errors:

simple zpool import (-f and -F give the same output as well)

Code: Select all

sudo zpool import                                                 
   pool: tank
     id: 7522410235045551686
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
   devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
 config:

   tank                                            UNAVAIL  insufficient replicas
     raidz1-0                                      UNAVAIL  insufficient replicas
       disk4                                       ONLINE
       media-5A484847-B333-3E44-A0B3-632CF3EC20A6  UNAVAIL  cannot open
       media-9CEF4C13-418D-3F41-804B-02355E699FED  ONLINE
       media-7F264D47-8A0E-3242-A971-1D0BD7D755F4  UNAVAIL  cannot open


Specifying a device:

Code: Select all

sudo zpool import -d /dev/disk4s1
   pool: tank
     id: 7522410235045551686
  state: FAULTED
status: One or more devices contains corrupted data.
 action: The pool cannot be imported due to damaged devices or data.
   The pool may be active on another system, but can be imported using
   the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
 config:

   tank                                            FAULTED  corrupted data
     raidz1-0                                      DEGRADED
       media-026CF59D-BEBE-F043-B0A3-95F3FC1D4EDF  ONLINE
       disk4                                       ONLINE
       media-9CEF4C13-418D-3F41-804B-02355E699FED  ONLINE
       disk6                                       FAULTED  corrupted data


Specifying disk6s1 even returns all drives as ONLINE:

Code: Select all

sudo zpool import -d /dev/disk6s1
   pool: tank
     id: 7522410235045551686
  state: FAULTED
status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
   The pool may be active on another system, but can be imported using
   the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72
 config:

   tank                                            FAULTED  corrupted data
     raidz1-0                                      ONLINE
       media-026CF59D-BEBE-F043-B0A3-95F3FC1D4EDF  ONLINE
       media-17A0A5DF-B586-114C-8606-E1FB316FA23D  ONLINE
       media-9CEF4C13-418D-3F41-804B-02355E699FED  ONLINE
       disk6                                       ONLINE


What I've tried so far:
- looked a zdb -l for all the relevant partitions
- found that no all symlinks have been created, for example media-5A484847-B333-3E44-A0B3-632CF3EC20A6 is missing in /private/var/run/disk/by-id and /var/run/disk/by-id. Creating these manually didn't help.

I was thinking about somehow modifying the metadata that is shown with zdb -l, as it's different for each drive (when referencing the other drives), but not sure if that is even possible. What led me to think about that was when specifying disk6s1, all drives show as online and also have different IDs.

Does anyone have ideas about how to solve this? Help is greatly appreciated!

[jawbroken]

might help to specify what version(s) of OpenZFS you are running and what commands you tried on TrueNAS, etc.

edit: what does "zpool status -x" show, as documented at the link provided

[pollux23]

Good point.
I've updated to 2.2.2 on my Mac Pro as a measure trying to figure out to get things working again.
To my recollection, on TrueNAS I didn't even really try importing the pool. If I did and just don't remember, I'm sure I didn't do anything else than a zpool import -f.
"zpool status -x" doesn't show anything, as the pool can't be imported in the first place.

Closest I can get to is this:

Code: Select all

sudo zpool import -fd /dev/disk7s1
   pool: tank
     id: 7522410235045551686
  state: FAULTED
status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
   The pool may be active on another system, but can be imported using
   the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72
 config:

   tank                                            FAULTED  corrupted data
     raidz1-0                                      ONLINE
       media-026CF59D-BEBE-F043-B0A3-95F3FC1D4EDF  ONLINE
       media-17A0A5DF-B586-114C-8606-E1FB316FA23D  ONLINE
       media-9CEF4C13-418D-3F41-804B-02355E699FED  ONLINE
       disk7                                       ONLINE


Note how after a reboot suddenly all devices show as online, when trying to import the pool with disk7s1. This was the case (with all devices showing as online) before when trying to import the pool via /dev/disk6s1.

[jawbroken]

it's weird that it says that the pool cannot be imported but then can be imported using "-f", which you are already using. i presume you've tried "sudo zpool import -fd /dev/disk7s1 tank" and that doesn't work?

[pollux23]

This is what I get in that case (different messages for each diskXsY).

Code: Select all

sudo zpool import -fd /dev/disk4s1 tank
cannot import 'tank': I/O error
   Destroy and re-create the pool from
   a backup source.

Code: Select all

sudo zpool import -fd /dev/disk5s1 tank
cannot import 'tank': one or more devices is currently unavailable

Code: Select all

sudo zpool import -fd /dev/disk6s1 tank
cannot import 'tank': no such device in pool
   Destroy and re-create the pool from
   a backup source.

Code: Select all

sudo zpool import -fd /dev/disk7s1 tank
cannot import 'tank': I/O error
   Destroy and re-create the pool from
   a backup source.

Usually at the point of "I/O error" I'd give up on the issue. But given that all the IDs are mixed up and the pool shows up differently depending on how the import command is called etc., I was hoping that it can be somehow fixed.

[jawbroken]

hopefully someone with more recovery knowledge comes across this. what does "zdb -C -e -p /dev/diskXs1 tank" look like for all of these drives?

edit: and "diskutil list"

edit2: this might be helpful

[jawbroken]

did you have any luck? might help someone else in future, either way