Files not visible in native encrypted dataset

All your general support questions for OpenZFS on OS X.

Files not visible in native encrypted dataset

Postby Sharko » Tue Apr 02, 2019 9:21 pm

Sorry, not enough room in the title box for the full title, so I'll have to explain more here. I have an OWC dual disk enclosure which I access through eSATA. The enclosure has two 4 TB disks in it (duh!), set up as a zpool mirror. The name of the pool is ELITE, and there is a child dataset within it called ELITE/ENCRYPTED, which is encrypted with native ZFS encryption. ENCRYPTED is read/write. Then within ELITE/ENCRYPTED there are various read-only backup datasets, all with multiple snapshots within them. Until today, everything was working more or less expected: I would plug in the eSATA cable, turn on the drive, issue 'zpool import ELITE', followed by 'zfs mount -l ELITE/ENCRYPTED'. This would only mount two drives on the desktop, ELITE and ENCRYPTED, but that was OK because I could still access the other backups as sub-directories of ENCRYPTED.

Last night I realized that it had been a while since I had run a scrub on ELITE (months). So I ran a scrub, which took 15 hours. Since then, there has been a change: if I do the normal zpool import and zfs mount I still get appear to get access to the backup directories within ENCRYPTED, in the sense that they are listed within ENCRYPTED, but when I click on one or try to list its contents in Terminal they show up as empty! When I run zfs list -r -t all ELITE I see that they all still have their snapshots and are using space as before, but if I rely on the Finder it shows them as empty. Note that these child datasets are not shown as volumes on the desktop; instead I'm opening the ENCRYPTED volume, and seeing them there. If I try to drill down further to look into one of these child datasets, that's when everything comes up blank.

Thinking that this might be transient weirdness, I re-booted. Same behavior.

On a hunch, I tried mounting one of the child datasets of ENCRYPTED, and lo now everything is visible. So there is a work-around, but this is definitely weirdness.

Some facts about my system: currently running Mojave 10.14.3. It was a clean install, but I let migration assistant transfer zfs 1.8.1 over from a temporary clean install of High Sierra 10.13.6. I haven't upgraded my pool yet from (I think) 1.7.2; it was one of the early versions that supported native encryption.

Here is the zpool status of the ELITE enclosure:

Code: Select all
sh-3.2# zpool status ELITE
  pool: ELITE
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
   still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
   the pool may no longer be accessible by software that does not support
   the features. See zpool-features(5) for details.
  scan: scrub repaired 0 in 14h47m with 0 errors on Mon Apr  1 23:09:11 2019

   NAME                                        STATE     READ WRITE CKSUM
   ELITE                                       ONLINE       0     0     0
     mirror-0                                  ONLINE       0     0     0
       PCI0@0-IOU1@7-PXS2@0-PRT1@1-PMP@0-@0:1  ONLINE       0     0     0
       PCI0@0-IOU1@7-PXS2@0-PRT1@1-PMP@1-@1:1  ONLINE       0     0     0

errors: No known data errors

Here is a sample property list from one of those encrypted child datasets:

Code: Select all
NAME                          PROPERTY               VALUE                                  SOURCE
ELITE/ENCRYPTED/SHOME_BACKUP  type                   filesystem                             -
ELITE/ENCRYPTED/SHOME_BACKUP  creation               Fri Aug 17  9:06 2018                  -
ELITE/ENCRYPTED/SHOME_BACKUP  used                   1.26T                                  -
ELITE/ENCRYPTED/SHOME_BACKUP  available              860G                                   -
ELITE/ENCRYPTED/SHOME_BACKUP  referenced             429G                                   -
ELITE/ENCRYPTED/SHOME_BACKUP  compressratio          1.05x                                  -
ELITE/ENCRYPTED/SHOME_BACKUP  mounted                no                                     -
ELITE/ENCRYPTED/SHOME_BACKUP  quota                  none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  reservation            none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  recordsize             128K                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  sharenfs               off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  checksum               on                                     default
ELITE/ENCRYPTED/SHOME_BACKUP  compression            lz4                                    inherited from ELITE/ENCRYPTED
ELITE/ENCRYPTED/SHOME_BACKUP  atime                  off                                    inherited from ELITE/ENCRYPTED
ELITE/ENCRYPTED/SHOME_BACKUP  devices                on                                     default
ELITE/ENCRYPTED/SHOME_BACKUP  exec                   on                                     default
ELITE/ENCRYPTED/SHOME_BACKUP  setuid                 on                                     default
ELITE/ENCRYPTED/SHOME_BACKUP  readonly               on                                     local
ELITE/ENCRYPTED/SHOME_BACKUP  zoned                  off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  snapdir                hidden                                 default
ELITE/ENCRYPTED/SHOME_BACKUP  aclmode                passthrough                            default
ELITE/ENCRYPTED/SHOME_BACKUP  aclinherit             restricted                             default
ELITE/ENCRYPTED/SHOME_BACKUP  canmount               on                                     default
ELITE/ENCRYPTED/SHOME_BACKUP  xattr                  on                                     default
ELITE/ENCRYPTED/SHOME_BACKUP  copies                 1                                      default
ELITE/ENCRYPTED/SHOME_BACKUP  version                5                                      -
ELITE/ENCRYPTED/SHOME_BACKUP  utf8only               on                                     -
ELITE/ENCRYPTED/SHOME_BACKUP  normalization          formD                                  -
ELITE/ENCRYPTED/SHOME_BACKUP  casesensitivity        insensitive                            -
ELITE/ENCRYPTED/SHOME_BACKUP  vscan                  off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  nbmand                 off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  sharesmb               off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  refquota               none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  refreservation         none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  primarycache           all                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  secondarycache         all                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  usedbysnapshots        862G                                   -
ELITE/ENCRYPTED/SHOME_BACKUP  usedbydataset          429G                                   -
ELITE/ENCRYPTED/SHOME_BACKUP  usedbychildren         0                                      -
ELITE/ENCRYPTED/SHOME_BACKUP  usedbyrefreservation   0                                      -
ELITE/ENCRYPTED/SHOME_BACKUP  logbias                latency                                default
ELITE/ENCRYPTED/SHOME_BACKUP  dedup                  off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  mlslabel               none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  sync                   standard                               default
ELITE/ENCRYPTED/SHOME_BACKUP  refcompressratio       1.05x                                  -
ELITE/ENCRYPTED/SHOME_BACKUP  written                0                                      -
ELITE/ENCRYPTED/SHOME_BACKUP  logicalused            1.31T                                  -
ELITE/ENCRYPTED/SHOME_BACKUP  logicalreferenced      449G                                   -
ELITE/ENCRYPTED/SHOME_BACKUP  filesystem_limit       none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  snapshot_limit         none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  filesystem_count       none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  snapshot_count         none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  snapdev                hidden                                 default
ELITE/ENCRYPTED/SHOME_BACKUP       on                                     default
ELITE/ENCRYPTED/SHOME_BACKUP  off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP    on                                     inherited from ELITE
ELITE/ENCRYPTED/SHOME_BACKUP      poolonly                               default
ELITE/ENCRYPTED/SHOME_BACKUP  shareafp               off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  redundant_metadata     all                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  overlay                off                                    default
ELITE/ENCRYPTED/SHOME_BACKUP  encryption             aes-256-ccm                            -
ELITE/ENCRYPTED/SHOME_BACKUP  keylocation            none                                   default
ELITE/ENCRYPTED/SHOME_BACKUP  keyformat              passphrase                             -
ELITE/ENCRYPTED/SHOME_BACKUP  pbkdf2iters            342K                                   -
ELITE/ENCRYPTED/SHOME_BACKUP  encryptionroot         ELITE/ENCRYPTED                        -
ELITE/ENCRYPTED/SHOME_BACKUP  keystatus              available                              -

Here is a before explicit mounting and after explicit mounting example of what the 'ls -a' command gives:

Code: Select all
sh-3.2# ls -a /Volumes/ELITE/ENCRYPTED/MOM_BACKUP/
.   ..
sh-3.2# ls -a /Volumes/ELITE/ENCRYPTED/MOM_BACKUP/
.            .Spotlight-V100         .VolumeIcon.icns      Mom's Beige G3.sparseimage   Mom's iMac.sparseimage
..            .Trashes         .fseventsd         Mom's MacBook.sparseimage

Weird, huh? Any suggestions?
Posts: 79
Joined: Thu May 12, 2016 12:19 pm

Re: Files not visible in native encrypted dataset

Postby lundman » Wed Apr 03, 2019 9:03 pm

Hmm there was some changes to encryption and hierarchy but not sure if it applies here. Normally I would have thought you would go "zpool import -l" to import and mount everything, but perhaps you leave the encrypted to be mounted only as needed. But then you could use "zfs mount -la" which would ask for key and "mount all".
User avatar
Posts: 530
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Files not visible in native encrypted dataset

Postby Sharko » Thu Apr 04, 2019 9:28 am

Hi Lundman, thank you for the suggestions, the 'zfs mount -la' looks especially useful going forward.

It's still weird though, isn't it, that with no code changes, only doing a scrub changed the way that things worked? The scrub supposedly did not repair anything, yet something about the pool seems to have changed: subdirectory (or child) datasets under ENCRYPTED appear, yet are non-functional (i.e. there appears to be nothing in them). They're in this in-between state of being visible but not mounted. If you would like me to supply any additional data I would be happy to do so.

The ironic thing is that I just edited the wiki page on encryption the other day to reflect how to use encryption with child datasets, and now one of the statements I wrote is invalid! I'll go back and change it.

Posts: 79
Joined: Thu May 12, 2016 12:19 pm

Re: Files not visible in native encrypted dataset

Postby Sharko » Sat Apr 06, 2019 9:40 am

OK, a little more poking around... turns out this was not the result of the scrub. It is probably because I recently upgraded the operating system (duh!) from El Capitan to Mojave, since I see the same behavior on a pool residing on an external disk that hasn't been scrubbed ever.
Posts: 79
Joined: Thu May 12, 2016 12:19 pm

Return to General Help

Who is online

Users browsing this forum: No registered users and 1 guest