Trouble adding encrypted disk to create a mirror

New to OpenZFS on OS X (Or ZFS in general)? Ask your questions here!

Trouble adding encrypted disk to create a mirror

Postby Sharko » Sat Jun 02, 2018 10:36 am

Hi, I've been trying to convert my single disk backup to a two disk mirror. I'm still running ZFS 1.5.2 under El Capitan. I bought an OWC two disk enclosure, and put the original 4TB backup disk and the new 4TB backup disk in the same enclosure, which is connected through ESATA. My original single disk backup was encrypted with the Filevault procedure, so I went through the procedure to encrypt the new disk with Filevault using the GUI Disk Utility. You can see what happens when :

Code: Select all
sh-3.2# diskutil cs list | grep Conversion
|       Conversion Status:       Complete
|       Conversion Status:       Complete
|       Conversion Status:       Complete
|       Conversion Status:       Complete
|       Conversion Status:       Complete
|       Conversion Status:       Complete
        Conversion Status:       Complete
sh-3.2# diskutil cs list
CoreStorage logical volume groups (7 found)
|

********* snipped data from first six groups ***************

+-- Logical Volume Group 600666BA-2B23-456F-803F-8CF1D006347E
    =========================================================
    Name:         EliteProDual-2_HFS
    Status:       Online
    Size:         4000443056128 B (4.0 TB)
    Free Space:   16560128 B (16.6 MB)
    |
    +-< Physical Volume E79F2AB9-62E0-466A-B323-AC80C7F08396
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk13s2
    |   Status:   Online
    |   Size:     4000443056128 B (4.0 TB)
    |
    +-> Logical Volume Family CB1951D4-F3CE-4DBF-82A5-D472D2239A16
        ----------------------------------------------------------
        Encryption Type:         AES-XTS
        Encryption Status:       Unlocked
        Conversion Status:       Complete
        High Level Queries:      Fully Secure
        |                        Passphrase Required
        |                        Accepts New Users
        |                        Has Visible Users
        |                        Has Volume Key
        |
        +-> Logical Volume B3A51B3A-FB94-4DF0-826B-9843D0F244BC
            ---------------------------------------------------
            Disk:                  disk15
            Status:                Online
            Size (Total):          4000074170368 B (4.0 TB)
            Revertible:            No
            LV Name:               EliteProDual-2_HFS
            Volume Name:           EliteProDual-2_HFS
            Content Hint:          Apple_HFS
sh-3.2# zpool status -L ELITE
  pool: ELITE
 state: ONLINE
  scan: scrub repaired 0 in 10h10m with 0 errors on Sun Mar 11 03:31:41 2018
config:

   NAME        STATE     READ WRITE CKSUM
   ELITE       ONLINE       0     0     0
     disk14    ONLINE       0     0     0

errors: No known data errors
sh-3.2# diskutil unmount "/Volumes/EliteProDual-2_HFS/"
Volume EliteProDual-2_HFS on disk15 unmounted
sh-3.2# zpool attach                  ******** just to double check the argument order *************
missing pool name argument
usage:
   attach [-f] [-o property=value] <pool> <device> <new-device>
sh-3.2#
sh-3.2# zpool attach ELITE disk14 disk15
cannot attach disk15 to disk14: no such device in pool
sh-3.2#
 


So, the zpool status command gave me disk14 as the <device>. The diskutil command gave me disk15 as the <new-device>. I'm not sure how to interpret the error message. Is it possible that "zpool attach" is unable to make use of an encrypted unmounted HFS logical volume in the way that "zpool create" is able to?
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Trouble adding encrypted disk to create a mirror

Postby Sharko » Sat Jun 02, 2018 10:50 am

Interesting... I exported the existing ELITE pool, and shut down the backup enclosure. When I powered it on again, OS X told me a disk was unreadable and it offered to take me to Disk Utility to initialize it. I chose Ignore, and then ran the terminal command "diskutil cs list" and got this for the new disk:

Code: Select all
+-- Logical Volume Group 600666BA-2B23-456F-803F-8CF1D006347E
    =========================================================
    Name:         EliteProDual-2_HFS
    Status:       Online
    Size:         4000443056128 B (4.0 TB)
    Free Space:   16560128 B (16.6 MB)
    |
    +-< Physical Volume E79F2AB9-62E0-466A-B323-AC80C7F08396
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk12s2
    |   Status:   Online
    |   Size:     4000443056128 B (4.0 TB)
    |
    +-> Logical Volume Family CB1951D4-F3CE-4DBF-82A5-D472D2239A16
        ----------------------------------------------------------
        Encryption Type:         AES-XTS
        Encryption Status:       Unlocked
        Conversion Status:       Complete
        High Level Queries:      Fully Secure
        |                        Passphrase Required
        |                        Accepts New Users
        |                        Has Visible Users
        |                        Has Volume Key
        |
        +-> Logical Volume B3A51B3A-FB94-4DF0-826B-9843D0F244BC
            ---------------------------------------------------
            Disk:                  disk15
            Status:                Online
            Size (Total):          4000074170368 B (4.0 TB)
            Revertible:            No
            LV Name:               EliteProDual-2_HFS
            Content Hint:          Apple_HFS


So it says it is online, but it has no Volume name any longer and it doesn't show up on the desktop. It appears that the zpool attach command did something to it, but failed to take it over completely. Trying to mount the logical volume in Disk Utility fails.
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Trouble adding encrypted disk to create a mirror

Postby Sharko » Sat Jun 30, 2018 7:45 pm

Well, I thought that perhaps this was a problem caused by running such an old version (1.5.2) of ZFS, so I installed the latest stable version (1.7.2) on my laptop, and tried working with that. Sadly, same result.

Code: Select all
Admins-MacBook-Pro:~ adminkurt$ diskutil cs list | grep Conversion
|       Conversion Status:       Complete
|       Conversion Status:       Complete
|       Conversion Status:       Complete
        Conversion Status:       Complete
Admins-MacBook-Pro:~ adminkurt$ sudo sh
Password:
sh-3.2# zpool status -L ELITE
  pool: ELITE
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
   still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
   the pool may no longer be accessible by software that does not support
   the features. See zpool-features(5) for details.
  scan: scrub repaired 0 in 10h10m with 0 errors on Sun Mar 11 03:31:41 2018
config:

   NAME                                          STATE     READ WRITE CKSUM
   ELITE                                         ONLINE       0     0     0
     disk4                                       ONLINE       0     0     0

errors: No known data errors
sh-3.2# diskutil unmount "/Volumes/EliteProDual-2_HFS/"
Volume EliteProDual-2_HFS on disk6 unmounted
sh-3.2# zpool attach -f ELITE /dev/disk4 /dev/disk6
cannot attach /dev/disk6 to /dev/disk4: no such device in pool
sh-3.2#


So, it looks like I will have to go the long, slow route: destroy the current backups in pool ELITE, then create a mirror using the native ZFS encryption, then replicate all the snapshots over to the new mirrored pool. Oh, well. Seems like this should have been possible.
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Trouble adding encrypted disk to create a mirror

Postby lundman » Sun Jul 01, 2018 4:48 pm

Do "zpool iostat -g" to get the guids, and use the GUID number instead of /dev/disk4
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Trouble adding encrypted disk to create a mirror

Postby Sharko » Sun Jul 01, 2018 9:04 pm

Thank you, lundman! That worked (although it was actually "zpool status -g ELITE" that was needed to get the GUID). Wonder how many days the resilver is going to take... 3.2 TB on the disk, transfer appears to be moving at between 5MB/s and 10MB/s :roll: even with the ESata connection. Ha! Starting from scratch may well have been quicker, but this is the path now.
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Trouble adding encrypted disk to create a mirror

Postby Sharko » Mon Jul 02, 2018 9:30 am

Just wanted to post this observation: the resilver did indeed start off VERY slowly, at between 5 and 10 MB/sec. At that rate zpool status was predicting that the operation would take 642 hours (roughly four weeks!). However, after about 45 minutes or so the rate picked up, and proceeded at something more in line with an ESATA connection: 90 to 160 MB/sec. I believe that there have been some other posts in the forum about speed and cache size, with speed being slow until the cache is filled; I have 8 GB allocated to zfs, so perhaps the slow speed I'm seeing is due to waiting for that large cache to fill.

Another little datapoint: zpool status was reporting that an error had been encountered in one of my files. I thought that was interesting, so I decided to fire up DriveDX (which checks SMART data). Big mistake! Although I have run DriveDX while other things have been going on in the system and it has worked, this time it launched and then the computer mouse became unresponsive, and a moment later the computer decided to reboot! Apparently it is not a good idea to check SMART data while doing something as I/O-intensive as a resilver. Here's the interesting part, though: after powering off the external drive and importing the ELITE pool again it began to resilver as expected, but the speed was back down at 5-10 MB/sec. And once again it stayed at that low speed for roughly 45 minutes, at which point it kicked back up to normal ESATA speeds.
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Trouble adding encrypted disk to create a mirror

Postby lundman » Mon Jul 02, 2018 4:21 pm

The resilver/scrub process starts heavily in metadata (filenames, acls, xattrs) which is all tiny reads, hence slow due to lots of seeking. Once that has passed, it branches out into actual file data, which are large (often sequential) reads, and speed can pick up.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Trouble adding encrypted disk to create a mirror

Postby Sharko » Fri Jul 06, 2018 9:03 am

This pool is still problematic, it seems. I've let it resilver several times, but each time I start the computer up it insists on resilvering the whole darn thing again from scratch. I don't know if this is relevant, but I'm currently connecting to the dual disk enclosure via USB 3, while the original disk was in an enclosure that used ESATA. I'm attempting to do this resilver on my MacBook Pro under the 1.7.2 release on an un-upgraded pool. The terminal capture below shows the result of a status command before and after a power off/power on event.

Code: Select all
Admins-MacBook-Pro:~ adminkurt$ sudo zpool status ELITE
Password:
  pool: ELITE
 state: ONLINE
status: One or more devices has experienced an error resulting in data
   corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
   entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: resilvered 3.21T in 17h46m with 1 errors on Thu Jul  5 04:54:34 2018
config:

   NAME                                            STATE     READ WRITE CKSUM
   ELITE                                           ONLINE       0     0     1
     mirror-0                                      ONLINE       0     0     2
       media-C40F97A1-3105-4673-BA93-0F515599592B  ONLINE       0     0     2
       media-ED91733C-404C-4881-AD1E-DBFD914A2771  ONLINE       0     0     2

errors: 1 data errors, use '-v' for a list
Admins-MacBook-Pro:~ adminkurt$ sudo zpool export ELITE
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/ELITE/QRECALL_BACKUP'
Unmount successful for /Volumes/ELITE/QRECALL_BACKUP
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/ELITE/MOM_BACKUP'
Unmount successful for /Volumes/ELITE/MOM_BACKUP
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/ELITE/MEDIA_BACKUP'
Unmount successful for /Volumes/ELITE/MEDIA_BACKUP
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/ELITE/JONP_BACKUP'
Unmount successful for /Volumes/ELITE/JONP_BACKUP
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/ELITE/HOME_BACKUP'
Unmount successful for /Volumes/ELITE/HOME_BACKUP
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/ELITE'
Unmount successful for /Volumes/ELITE
Admins-MacBook-Pro:~ adminkurt$
  [Restored Jul 6, 2018, 9:57:37 AM]
Last login: Fri Jul  6 09:57:37 on ttys000
Restored session: Fri Jul 6 00:07:36 PDT 2018
Admins-MacBook-Pro:~ adminkurt$ sudo zpool import ELITE
Password:
Admins-MacBook-Pro:~ adminkurt$ sudo sh
sh-3.2# zpool status ELITE
  pool: ELITE
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
   continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Jul  6 09:58:24 2018
   1.59M scanned out of 3.22T at 60.4K/s, (scan is slow, no estimated time)
    1.58M resilvered, 0.00% done
config:

   NAME                                            STATE     READ WRITE CKSUM
   ELITE                                           ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-C40F97A1-3105-4673-BA93-0F515599592B  ONLINE       0     0     0
       media-ED91733C-404C-4881-AD1E-DBFD914A2771  ONLINE       0     0     0

errors: 1 data errors, use '-v' for a list
sh-3.2# exit
exit
Admins-MacBook-Pro:~ adminkurt$


Any ideas? Or should I just destroy the pool and try to recreate it with both disks from the start? Follow-up question if I have to start over: can I destroy the pool without waiting the 17 hours for the resilver to complete?
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Trouble adding encrypted disk to create a mirror

Postby lundman » Sun Jul 08, 2018 5:21 pm

So you have some checksum errors, you could do zpool status -v, and see what files they are. The checksum errors could also be stuck in snapshots.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Trouble adding encrypted disk to create a mirror

Postby Sharko » Mon Jul 09, 2018 3:18 pm

I have run zpool status with the -v, and it told me that the error was in a particular file called repository.data. Unfortunately, this is a very large file (> 1 TB) that is the main reason for that dataset - it is the file that holds the quanta (chunks) of a backup made using QRecall. I didn't bother with snapshots on that dataset since it was only a backup, and at a minimum it seems like I need to zfs destroy that dataset. Maybe then it will resilver one last time and be stable?
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm


Return to Absolute Beginners Section

Who is online

Users browsing this forum: No registered users and 17 guests