Strange ZFS errors

All your general support questions for OpenZFS on OS X.

Strange ZFS errors

Postby ryjorny » Thu Nov 15, 2018 5:19 pm

I've had a great experience with ZFS up until something strange happened. I downloaded the most recent version of this menu bar application https://github.com/cbreak-black/ZetaWatch to help me manage my zpools at boot. I have 4 pools of mirrored vdevs and they each have an encrypted dataset except for Fentanyl where I store data directly on the pool (Fentanyl 2x1.5TB SATA motherboard controller, Ketamina 2x3TB SATA motherboard controller, Diprivan 2x3TB LSI PCIe SAS HBA, Midazolam 2x3TB LSI PCIe SAS HBA). I ran ZetaWatch and it failed to import the pools that had an encrypted dataset on it but it did report that the plain unencrypted pool Fentanyl was OK.

After this I tried to zpool import -a and got an error "zfs: cannot import Diprivan: I/O error Destroy and re-create the pool from a backup" and the same error for Midazolam. I exported all pools and rebooted. This time Fentanyl still worked fine and Midazolam was able to be imported zpool import Midazolam and the encrypted dataset within mounted, but Diprivan had the same I/O error. Ketamina no longer existed at all. It wasn't in disk utility and when I ran diskutil list the two disks that used to be called Ketamina are now called Midazolam also. Four disks in diskutil were reporting their names as Midazolam. I exported everything, shutdown and removed the disks that make up my two working pools, Fentanyl and Midazolam. I rebooted and tried to import Ketamina, no pool exists and when I tried to import Midazolam (falsely named) I got "cannot import 'Midazolam': one or more devices is currently unavailable". I have not tried to fix Diprivan yet fearing I could make this situation worse so only work with one at a time.

Reading up on the FreeNAS and zfsonlinux boards haven't given me any leads. Thank you for your time and any ideas you can give me how to fix this.

macos high sierra 10.13.6
OpenZFS on OS X 1.8.2
ryjorny
 
Posts: 13
Joined: Thu Nov 15, 2018 4:54 pm

Re: Strange ZFS errors

Postby lundman » Thu Nov 15, 2018 6:50 pm

Can you provide more output, for example

"diskutil list" to make sure disks are seen by OSX.
"zpool import -d /dev" and see if any pools show up at all? (just list pools).

Is the encrypted datasets using old crypt version, so will only import read/only ?
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Strange ZFS errors

Postby ryjorny » Thu Nov 15, 2018 7:05 pm

Absolutely, thank you for replying so quickly. Right now I only have my nvme drive with various OSs installed, an msata scratch drive, two working zfs vdevs and two of the corrupted vdevs
Code: Select all
elodie$ sudo -i zpool import -d /dev
Password:
   pool: Fentanyl
     id: 7217856158381481435
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

   Fentanyl    ONLINE
     mirror-0  ONLINE
       disk6   ONLINE
       disk7   ONLINE

   pool: Midazolam
     id: 1397141627867242428
  state: UNAVAIL
 status: One or more devices contains corrupted data.
 action: The pool cannot be imported due to damaged devices or data.
   see: http://zfsonlinux.org/msg/ZFS-8000-5E
 config:

   Midazolam                                       UNAVAIL  insufficient replicas
     mirror-0                                      UNAVAIL  insufficient replicas
       media-B6B475DE-7D70-734E-AA12-27CBAF295727  FAULTED  corrupted data
       media-95A2D0FF-123D-D246-BF8B-EFD2A465E91F  FAULTED  corrupted data



elodie$ diskutil list
/dev/disk0 (internal):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                         1.0 TB     disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         500.1 GB   disk0s2
   3:       Microsoft Basic Data Sugammadex              119.9 GB   disk0s3
   4:       Microsoft Basic Data FEDORA                  130.0 GB   disk0s4
   5:       Microsoft Basic Data WENDY                   249.6 GB   disk0s5

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.1 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Elodie                  18.4 GB    disk1s1
   2:                APFS Volume Preboot                 21.6 MB    disk1s2
   3:                APFS Volume Recovery                519.0 MB   disk1s3
   4:                APFS Volume VM                      20.5 KB    disk1s4

/dev/disk2 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *256.1 GB   disk2
   1:                        EFI EFI                     209.7 MB   disk2s1
   2:       Microsoft Basic Data TULSA                   255.8 GB   disk2s2

/dev/disk3 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *7.8 GB     disk3
   1:       Microsoft Basic Data WINDOWS 10              7.8 GB     disk3s1

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk4
   1:                        ZFS Midazolam               3.0 TB     disk4s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk4s9

/dev/disk5 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk5
   1:                        ZFS Midazolam               3.0 TB     disk5s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk5s9

/dev/disk6 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.5 TB     disk6
   1:                        ZFS Fentanyl                1.5 TB     disk6s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk6s9

/dev/disk7 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.5 TB     disk7
   1:                        ZFS Fentanyl                1.5 TB     disk7s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk7s9





I have been in the /private/var/run/disk/by-id folder and trying to account for each entry. I have recorded the Volume UUID and Disk UUID by doing diskutil info diskXsY for each item above. The APFS boot drive "Elodie" is a little confusing to me but I recorded all of the info I could gather

Code: Select all
NVME - 960 EVO disk0
Vol UUID                      EFI = 0E239BC6-F960-3107-89CF-1C97F78BB46B
DISK UUID                                   = 1CD99689-4836-4BD3-A42D-10C7E26A1EA0
               Sug = 9B4134A6-4403-36E0-A36A-59AD2E7225F2
                  = 6F6B114F-5227-4B82-A10E-1B2EF747DD01
               Fed = 46E2E3BC-E3F5-3438-A55C-DD974B0C160A
                  = 222EDA0C-7509-49FE-8B44-28E209E3522C
               Wen = 6BCC7FEA-834B-34ED-AD26-32950D91ECF7
                  = 5BB46F98-1724-47C7-8C66-9C363E681A93
DUUID     APFS 0s2 Elod? = 3BAC37C6-5267-4680-A10D-A5AC39ABBDA5

      APFS disk1
              IOContent = EF57347C-0000-11AA-AA11-00306543ECAC
DUUID               = 38571C9A-48A8-4DEB-9ACA-FAB0785DA8B1


Vol UUID   disk1s1 Elod = D7A8D3AE-929A-3660-BC11-785B518657DE
DUUID             = D7A8D3AE-929A-3660-BC11-785B518657DE      
Partition Type                = 41504653-0000-11AA-AA11-00306543ECAC

Vol UUID   disk1s2   Pre = 0DD4D386-DC06-4930-8BA5-A7465738ED2B
DUUID             = 0DD4D386-DC06-4930-8BA5-A7465738ED2B      
Partition Type                = 41504653-0000-11AA-AA11-00306543ECAC

Vol UUID   disk1s3   Rec = BBBB0050-FA92-4000-B965-0EAD7ABE39EE
DUUID             = BBBB0050-FA92-4000-B965-0EAD7ABE39EE      
Partition Type                = 41504653-0000-11AA-AA11-00306543ECAC

Vol UUID   disk1s4   VM = A6367284-16D2-46C1-B741-F8194A06895C
DUUID            = A6367284-16D2-46C1-B741-F8194A06895C      
Partition Type               = 41504653-0000-11AA-AA11-00306543ECAC

##############

mSATA - PM841 currently disk2 the scratch drive one partition ExFAT named "Tulsa"
Vol UUID         EFI = 0E239BC6-F960-3107-89CF-1C97F78BB46B
DISK UUID                   =
            Tul = D707C975-264D-3802-8320-87CAE11E370F

##############

SATA - Samsung HD154 “Fentanyl” The working zpool w/o encrypted datasets currently disk6 and disk7

DUUID               = 0838DB4B-0C77-FD48-9E41-C6EBEEBA762E

DUUID               = B9E18975-57FC-1740-9779-C47B3066C314
   

###############

SATA - WD30 “*Midazolam*” I suspect this is actually Ketamina

DUUID               = B6B475DE-7D70-734E-AA12-27CBAF295727

DUUID               = 95A2D0FF-123D-D246-BF8B-EFD2A465E91F
   


Would you like for me to reboot with the other problematic zpool "Diprivan" installed and gather the information for that one as well or tackle one at a time?

I created these encrypted data sets about two weeks ago using v 1.8.0 I believe, maybe 1.8.1. Whichever was from the main download page not the forum only release using the following commands.
Code: Select all
zpool create -o ashift=12 -o failmode=continue -O atime=off -O compression=lz4 -O casesensitivity=insensitive -O normalization=formD -O checksum=edonr -O mountpoint=/Volumes or Tmp/[$ZPOOL] -O canmount=noauto [$ZPOOL] [$VDEVTYPE] [$DEVICE(S)]
zpool set feature@encryption=enabled
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase -o keylocation=prompt -o pbkdf2iters=100000 [$ZPOOL]/[$DATASET]
zfs set com.apple.ignoreowner=on [$ZPOOL or $DATASET]
zfs set mountpoint=/Volumes/[$DATASET] [$ZPOOL/$DATASET]
Last edited by ryjorny on Fri Nov 16, 2018 4:27 pm, edited 1 time in total.
ryjorny
 
Posts: 13
Joined: Thu Nov 15, 2018 4:54 pm

Re: Strange ZFS errors

Postby lundman » Thu Nov 15, 2018 7:25 pm

ok Fentanyl looks to me OK but Midazolam is unhappy.

Should be disk4 and disk5 at a guess. We can expand symlinks, ie, /private/var/run/disk into /dev/diskX with, what is it, "-L" but that is for "zpool status" and not "zpool import".
Can you confirm InvariantDisk is running? ie "ps -axww|grep Invariant"
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Strange ZFS errors

Postby ryjorny » Thu Nov 15, 2018 7:47 pm

We can expand symlinks, ie, /private/var/run/disk into /dev/diskX with, what is it, "-L" but that is for "zpool status" and not "zpool import".


I don't understand what you mean by this. Are you saying when I created my pools? When I made all of these pools I used the media-####### identifier not the BSD diskX identifier if that makes a difference at all

Code: Select all
ELODIE:~ elodie$ ps -axww|grep Invariant
    48 ??             0:00.03 /usr/local/bin/InvariantDisks
 1063 ttys000    0:00.00 grep Invariant


The disk calling itself Midazolam right now isn't actually Midazolam. I am able to import the real Midazolam and mount its encrypted dataset Midazolam/Versed and confirm data intact
Last edited by ryjorny on Fri Nov 16, 2018 4:28 pm, edited 2 times in total.
ryjorny
 
Posts: 13
Joined: Thu Nov 15, 2018 4:54 pm

Re: Strange ZFS errors

Postby ryjorny » Thu Nov 15, 2018 8:12 pm

I removed Fentanyl and attached the real, working Midazolam in addition to the problematic Diprivan (or Ketamina). I kept Ketamina (or Diprivan), the drive incorrectly reporting itself as Midazolam that we were previously working on, installed in the system. I will list just relevant ZFS info below leaving off the multiboot drive and the scratch drive to make it easier on the eyes. macOS is reporting these disks as external but they are in fact internal I just haven't edited the kext plist yet.

Code: Select all
ELODIE:~ root# diskutil list
/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk3
   1:                        ZFS Midazolam               3.0 TB     disk3s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk3s9

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk4
   1:                        ZFS Midazolam               3.0 TB     disk4s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk4s9

/dev/disk5 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk5
   1:                        ZFS Midazolam               3.0 TB     disk5s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk5s9

/dev/disk6 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk6
   1:                        ZFS Midazolam               3.0 TB     disk6s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk6s9

/dev/disk7 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk7
   1:                        ZFS Diprivan                3.0 TB     disk7s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk7s9

/dev/disk8 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk8
   1:                        ZFS Diprivan                3.0 TB     disk8s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk8s9


and
Code: Select all
ELODIE:~ root# zpool import -d /dev
   pool: Diprivan
     id: 1397141627867242428
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:

   Diprivan    FAULTED  corrupted data
     mirror-0  ONLINE
       disk7   ONLINE
       disk8   ONLINE

   pool: Midazolam
     id: 10584724269050003278
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

   Midazolam   ONLINE
     mirror-0  ONLINE
       disk5   ONLINE
       disk6   ONLINE

ELODIE:~ root# zpool import Midazolam
ELODIE:~ root# zpool status
  pool: Midazolam
 state: ONLINE
  scan: scrub repaired 0 in 3h17m with 0 errors on Mon Nov  5 06:53:28 2018
config:

   NAME                                            STATE     READ WRITE CKSUM
   Midazolam                                       ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-25DAF45E-2E70-3941-9D44-E8D664CF19C1  ONLINE       0     0     0
       media-3F577885-EF09-024D-B3E8-BF0432C03058  ONLINE       0     0     0

errors: No known data errors
ELODIE:~ root# zfs mount -a -l
Enter passphrase for 'Midazolam/Versed':

Code: Select all
ELODIE:~ root# zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
Midazolam         1.17T  1.46T   520K  /tmp/Midazolam
Midazolam/Versed  1.17T  1.46T  1.17T  /Volumes/Versed
Last edited by ryjorny on Sat Nov 17, 2018 2:41 am, edited 2 times in total.
ryjorny
 
Posts: 13
Joined: Thu Nov 15, 2018 4:54 pm

Re: Strange ZFS errors

Postby ryjorny » Thu Nov 15, 2018 8:26 pm

current drive arrangement

Code: Select all
ELODIE:~ root# diskutil info disk3s1
   Device Identifier:        disk3s1
   Device Node:              /dev/disk3s1
   Whole:                    No
   Part of Whole:            disk3

   Volume Name:              Midazolam
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    B6B475DE-7D70-734E-AA12-27CBAF295727
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

Code: Select all
ELODIE:~ root# diskutil info disk4s1
   Device Identifier:        disk4s1
   Device Node:              /dev/disk4s1
   Whole:                    No
   Part of Whole:            disk4

   Volume Name:              Midazolam
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    95A2D0FF-123D-D246-BF8B-EFD2A465E91F
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

Code: Select all
ELODIE:~ root# diskutil info disk5s1
   Device Identifier:        disk5s1
   Device Node:              /dev/disk5s1
   Whole:                    No
   Part of Whole:            disk5

   Volume Name:              Midazolam
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    25DAF45E-2E70-3941-9D44-E8D664CF19C1
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

Code: Select all
ELODIE:~ root# diskutil info disk6s1
   Device Identifier:        disk6s1
   Device Node:              /dev/disk6s1
   Whole:                    No
   Part of Whole:            disk6

   Volume Name:              Midazolam
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    3F577885-EF09-024D-B3E8-BF0432C03058
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

Code: Select all
ELODIE:~ root# diskutil info disk7s1
   Device Identifier:        disk7s1
   Device Node:              /dev/disk7s1
   Whole:                    No
   Part of Whole:            disk7

   Volume Name:              Diprivan
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    C4ED39CC-0689-B94E-9BCF-A3058196D8CB
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

Code: Select all
ELODIE:~ root# diskutil info disk8s1
   Device Identifier:        disk8s1
   Device Node:              /dev/disk8s1
   Whole:                    No
   Part of Whole:            disk8

   Volume Name:              Diprivan
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    60A51406-0625-DD41-874C-E67CC253D180
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No
ryjorny
 
Posts: 13
Joined: Thu Nov 15, 2018 4:54 pm

Re: Strange ZFS errors

Postby ryjorny » Sat Nov 17, 2018 2:06 am

The ZetaWatch developer responded to my issue on github https://github.com/cbreak-black/ZetaWatch/issues/2

From your description it looks like your issues are unrelated to ZetaWatch. I import and mount datasets with ZetaWatch all the time without such problems. You should try to use the command line, zpool status, zpool import and maybe zdb -l. If you have the option, trying to import the pool with other systems might also give some insight. But as of now, my hypothesis is that your pools died to something hardware or zfs driver related.


Is it possible to further damage my pools by trying to import on linux with with the latest zfsonlinux release? A few weeks back I was attempting to make my data sets accessible to both Mac OS and Linux but after trying multiple different configurations/zpool creation options I was never able to get them to mount RW on Linux. I can't read the errors now obviously but I remember it saying datto and something else were not supported. I have all but ruled out hardware error because the good pools were using the same hardware as the two that were damaged at the same time. None of my other SATA or SAS drives that I have connected since have had hardware problems either.

From your description it looks like your issues are unrelated to ZetaWatch. I import and mount datasets with ZetaWatch all the time without such problems. You should try to use the command line, zpool status, zpool import and maybe zdb -l. If you have the option, trying to import the pool with other systems might also give some insight. But as of now, my hypothesis is that your pools died to something hardware or zfs driver related.


I get what he's saying but I can't shake the idea that for a month I don't have any problems and the first time I do something different and use this program, it fails to work and my pools are corrupted. Literally nothing else was different and logic dictates the two are related somehow.
ryjorny
 
Posts: 13
Joined: Thu Nov 15, 2018 4:54 pm

Re: Strange ZFS errors

Postby ryjorny » Sun Nov 18, 2018 10:52 am

I tried mounting on Manjaro with latest ZFS on Linux and the cross platform problems I was having were only further complicated by my new issues. I booted up the system I created the pools on and ran "zdb" as seen below. I have backed up this /etc/zfs/zpool.cache file. Does anyone know how I could restore the devices' UUIDs? I think the problem is that somehow UUIDs were reassigned thus making ZFS rename them incorrectly. The metadata doesn't match up now that two pools are misnamed and that's why they are faulting.

Below is zdb from the system I created them out which shows the correct UUID.

Code: Select all

Ekaterina:~ root# zdb
Diprivan:
    version: 5000
    name: 'Diprivan'
    state: 0
    txg: 3700944
    pool_guid: 1397141627867242428
    errata: 0
    hostid: 211863825
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 1397141627867242428
        create_txg: 4
        children[0]:
            type: 'mirror'
            id: 0
            guid: 18330183062559595606
            metaslab_array: 39
            metaslab_shift: 34
            ashift: 12
            asize: 3000578342912
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 36
            children[0]:
                type: 'disk'
                id: 0
                guid: 14642587019985537927
                path: '/private/var/run/disk/by-id/media-B6B475DE-7D70-734E-AA12-27CBAF295727'
                whole_disk: 1
                DTL: 230
                create_txg: 4
                com.delphix:vdev_zap_leaf: 37
            children[1]:
                type: 'disk'
                id: 1
                guid: 15507805968823826666
                path: '/private/var/run/disk/by-id/media-95A2D0FF-123D-D246-BF8B-EFD2A465E91F'
                whole_disk: 1
                DTL: 229
                create_txg: 4
                com.delphix:vdev_zap_leaf: 38
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Fentanyl:
    version: 5000
    name: 'Fentanyl'
    state: 0
    txg: 578260
    pool_guid: 7217856158381481435
    errata: 0
    hostid: 211863825
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 7217856158381481435
        create_txg: 4
        children[0]:
            type: 'mirror'
            id: 0
            guid: 14778680404874167870
            metaslab_array: 39
            metaslab_shift: 33
            ashift: 12
            asize: 1500287336448
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 36
            children[0]:
                type: 'disk'
                id: 0
                guid: 1366341001950380000
                path: '/private/var/run/disk/by-id/media-0838DB4B-0C77-FD48-9E41-C6EBEEBA762E'
                whole_disk: 1
                DTL: 221
                create_txg: 4
                com.delphix:vdev_zap_leaf: 37
            children[1]:
                type: 'disk'
                id: 1
                guid: 15038258882216102655
                path: '/private/var/run/disk/by-id/media-B9E18975-57FC-1740-9779-C47B3066C314'
                whole_disk: 1
                DTL: 224
                create_txg: 4
                com.delphix:vdev_zap_leaf: 38
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Ketamina:
    version: 5000
    name: 'Ketamina'
    state: 0
    txg: 876541
    pool_guid: 4565409986572198625
    errata: 0
    hostid: 211863825
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 4565409986572198625
        create_txg: 4
        children[0]:
            type: 'mirror'
            id: 0
            guid: 14703789332219001749
            metaslab_array: 39
            metaslab_shift: 34
            ashift: 12
            asize: 3000578342912
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 36
            children[0]:
                type: 'disk'
                id: 0
                guid: 8321884040462525887
                path: '/private/var/run/disk/by-id/media-60A51406-0625-DD41-874C-E67CC253D180'
                whole_disk: 1
                DTL: 231
                create_txg: 4
                com.delphix:vdev_zap_leaf: 37
            children[1]:
                type: 'disk'
                id: 1
                guid: 11453163972271216550
                path: '/private/var/run/disk/by-id/media-C4ED39CC-0689-B94E-9BCF-A3058196D8CB'
                whole_disk: 1
                DTL: 230
                create_txg: 4
                com.delphix:vdev_zap_leaf: 38
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Midazolam:
    version: 5000
    name: 'Midazolam'
    state: 0
    txg: 562660
    pool_guid: 10584724269050003278
    errata: 0
    hostid: 211863825
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 10584724269050003278
        create_txg: 4
        children[0]:
            type: 'mirror'
            id: 0
            guid: 904590903429353085
            metaslab_array: 39
            metaslab_shift: 34
            ashift: 12
            asize: 3000578342912
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 36
            children[0]:
                type: 'disk'
                id: 0
                guid: 8841273357318706858
                path: '/private/var/run/disk/by-id/media-25DAF45E-2E70-3941-9D44-E8D664CF19C1'
                whole_disk: 1
                DTL: 232
                create_txg: 4
                com.delphix:vdev_zap_leaf: 231
            children[1]:
                type: 'disk'
                id: 1
                guid: 10205511252349193051
                path: '/private/var/run/disk/by-id/media-3F577885-EF09-024D-B3E8-BF0432C03058'
                whole_disk: 1
                DTL: 235
                create_txg: 4
                com.delphix:vdev_zap_leaf: 234
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data



And here I have the output of diskutil info -all showing the wrong UUIDs and the wrong pool names. The working mirrored zpools "Midazolam" and "Fentanyl" are sitting next to me on the desk.

Code: Select all
   Device Identifier:        disk4
   Device Node:              /dev/disk4
   Whole:                    Yes
   Part of Whole:            disk4
   Device / Media Name:      WDC WD30EFRX-68A

   Volume Name:              Not applicable (no file system)
   Mounted:                  Not applicable (no file system)
   File System:              None

   Content (IOContent):      GUID_partition_scheme
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported

   Disk Size:                3.0 TB (3000592982016 Bytes) (exactly 5860533168 512-Byte-Units)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          Fixed

   Virtual:                  No
   Hardware AES Support:     No

**********

   Device Identifier:        disk4s1
   Device Node:              /dev/disk4s1
   Whole:                    No
   Part of Whole:            disk4

   Volume Name:              Midazolam
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    B6B475DE-7D70-734E-AA12-27CBAF295727
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

**********

   Device Identifier:        disk4s9
   Device Node:              /dev/disk4s9
   Whole:                    No
   Part of Whole:            disk4

   Volume Name:              Not applicable (no file system)
   Mounted:                  Not applicable (no file system)
   File System:              None

   Partition Type:           6A945A3B-1DD2-11B2-99A6-080020736631
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    510FB29C-C280-D443-B1FC-EF5D9948B15C
   Partition Offset:         3000584110080 Bytes (5860515840 512-Byte-Device-Blocks)

   Disk Size:                8.4 MB (8388608 Bytes) (exactly 16384 512-Byte-Units)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

**********

   Device Identifier:        disk5
   Device Node:              /dev/disk5
   Whole:                    Yes
   Part of Whole:            disk5
   Device / Media Name:      WDC WD30EFRX-68A

   Volume Name:              Not applicable (no file system)
   Mounted:                  Not applicable (no file system)
   File System:              None

   Content (IOContent):      GUID_partition_scheme
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported

   Disk Size:                3.0 TB (3000592982016 Bytes) (exactly 5860533168 512-Byte-Units)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          Fixed

   Virtual:                  No
   Hardware AES Support:     No

**********

   Device Identifier:        disk5s1
   Device Node:              /dev/disk5s1
   Whole:                    No
   Part of Whole:            disk5

   Volume Name:              Midazolam
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    95A2D0FF-123D-D246-BF8B-EFD2A465E91F
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

**********

   Device Identifier:        disk5s9
   Device Node:              /dev/disk5s9
   Whole:                    No
   Part of Whole:            disk5

   Volume Name:              Not applicable (no file system)
   Mounted:                  Not applicable (no file system)
   File System:              None

   Partition Type:           6A945A3B-1DD2-11B2-99A6-080020736631
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    2E256F2C-32A9-1E41-809B-D876510937FB
   Partition Offset:         3000584110080 Bytes (5860515840 512-Byte-Device-Blocks)

   Disk Size:                8.4 MB (8388608 Bytes) (exactly 16384 512-Byte-Units)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

**********

   Device Identifier:        disk6
   Device Node:              /dev/disk6
   Whole:                    Yes
   Part of Whole:            disk6
   Device / Media Name:      HUS72303CLAR3000

   Volume Name:              Not applicable (no file system)
   Mounted:                  Not applicable (no file system)
   File System:              None

   Content (IOContent):      GUID_partition_scheme
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported

   Disk Size:                3.0 TB (3000592982016 Bytes) (exactly 5860533168 512-Byte-Units)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          Fixed

   Virtual:                  No
   Hardware AES Support:     No

**********

   Device Identifier:        disk6s1
   Device Node:              /dev/disk6s1
   Whole:                    No
   Part of Whole:            disk6

   Volume Name:              Diprivan
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    C4ED39CC-0689-B94E-9BCF-A3058196D8CB
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

**********

   Device Identifier:        disk6s9
   Device Node:              /dev/disk6s9
   Whole:                    No
   Part of Whole:            disk6

   Volume Name:              Not applicable (no file system)
   Mounted:                  Not applicable (no file system)
   File System:              None

   Partition Type:           6A945A3B-1DD2-11B2-99A6-080020736631
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    FAF1A3BA-02E1-6A4E-89A8-684E00BC4471
   Partition Offset:         3000584110080 Bytes (5860515840 512-Byte-Device-Blocks)

   Disk Size:                8.4 MB (8388608 Bytes) (exactly 16384 512-Byte-Units)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

**********

   Device Identifier:        disk7
   Device Node:              /dev/disk7
   Whole:                    Yes
   Part of Whole:            disk7
   Device / Media Name:      HUS72303CLAR3000

   Volume Name:              Not applicable (no file system)
   Mounted:                  Not applicable (no file system)
   File System:              None

   Content (IOContent):      GUID_partition_scheme
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported

   Disk Size:                3.0 TB (3000592982016 Bytes) (exactly 5860533168 512-Byte-Units)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          Fixed

   Virtual:                  No
   Hardware AES Support:     No

**********

   Device Identifier:        disk7s1
   Device Node:              /dev/disk7s1
   Whole:                    No
   Part of Whole:            disk7

   Volume Name:              Diprivan
   Mounted:                  No

   Partition Type:           ZFS
   File System Personality:  ZFS
   Type (Bundle):            zfs
   Name (User Visible):      ZFS Dataset

   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    60A51406-0625-DD41-874C-E67CC253D180
   Partition Offset:         1048576 Bytes (2048 512-Byte-Device-Blocks)

   Disk Size:                3.0 TB (3000583061504 Bytes) (exactly 5860513792 512-Byte-Units)
   Device Block Size:        512 Bytes

   Volume Total Space:       0 B (0 Bytes) (exactly 0 512-Byte-Units)
   Volume Free Space:        0 B (0 Bytes) (exactly 0 512-Byte-Units)

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (not mounted)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

**********

   Device Identifier:        disk7s9
   Device Node:              /dev/disk7s9
   Whole:                    No
   Part of Whole:            disk7

   Volume Name:              Not applicable (no file system)
   Mounted:                  Not applicable (no file system)
   File System:              None

   Partition Type:           6A945A3B-1DD2-11B2-99A6-080020736631
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SAS
   SMART Status:             Not Supported
   Disk / Partition UUID:    CEB3B42D-AEAE-4F44-81EE-932D8BF222C6
   Partition Offset:         3000584110080 Bytes (5860515840 512-Byte-Device-Blocks)

   Disk Size:                8.4 MB (8388608 Bytes) (exactly 16384 512-Byte-Units)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          Fixed

   Hardware AES Support:     No

**********


Apparently "Diprivan" is confused and thinks it is "Midazolam" and "Ketamina" is confused calling itself "Diprivan".

Please help folks :shock:
ryjorny
 
Posts: 13
Joined: Thu Nov 15, 2018 4:54 pm


Return to General Help

Who is online

Users browsing this forum: No registered users and 16 guests