Moving home directory over to a ZFS dataset 2.1.6 Intel

All your general support questions for OpenZFS on OS X.

Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby Sharko » Thu Dec 01, 2022 10:45 pm

So, I'm getting errors when I try to run my non-admin user ('userkurt') with the home directory living in a folder on a ZFS dataset under the newly released 2.1.6 installed on Intel Monterey. I used to be able to do this with Mojave/O3X 1.9.4, but it's been a long time since I set it up - maybe I'm missing something. Here are the properties for the dataset I'm trying to use; note that I've set com.apple.mimic to apfs in the hope that will work, but so far no joy:

Code: Select all
sh-3.2# zfs get all MICRON/HOME
NAME         PROPERTY               VALUE                     SOURCE
MICRON/HOME  type                   filesystem                -
MICRON/HOME  creation               Fri Nov 11 22:21 2022     -
MICRON/HOME  used                   693G                      -
MICRON/HOME  available              167G                      -
MICRON/HOME  referenced             640G                      -
MICRON/HOME  compressratio          1.06x                     -
MICRON/HOME  mounted                yes                       -
MICRON/HOME  quota                  none                      default
MICRON/HOME  reservation            none                      default
MICRON/HOME  recordsize             128K                      default
MICRON/HOME  mountpoint             /Volumes/MICRON/HOME      default
MICRON/HOME  sharenfs               off                       default
MICRON/HOME  checksum               on                        default
MICRON/HOME  compression            lz4                       inherited from MICRON
MICRON/HOME  atime                  off                       inherited from MICRON
MICRON/HOME  devices                on                        default
MICRON/HOME  exec                   on                        default
MICRON/HOME  setuid                 on                        default
MICRON/HOME  readonly               off                       default
MICRON/HOME  zoned                  off                       default
MICRON/HOME  snapdir                hidden                    default
MICRON/HOME  aclmode                discard                   default
MICRON/HOME  aclinherit             restricted                default
MICRON/HOME  createtxg              17305                     -
MICRON/HOME  canmount               on                        default
MICRON/HOME  xattr                  on                        default
MICRON/HOME  copies                 1                         default
MICRON/HOME  version                5                         -
MICRON/HOME  utf8only               on                        -
MICRON/HOME  normalization          formD                     -
MICRON/HOME  casesensitivity        insensitive               -
MICRON/HOME  vscan                  off                       default
MICRON/HOME  nbmand                 off                       default
MICRON/HOME  sharesmb               off                       default
MICRON/HOME  refquota               none                      default
MICRON/HOME  refreservation         none                      default
MICRON/HOME  guid                   13257738383987624662      -
MICRON/HOME  primarycache           all                       default
MICRON/HOME  secondarycache         all                       default
MICRON/HOME  usedbysnapshots        53.1G                     -
MICRON/HOME  usedbydataset          640G                      -
MICRON/HOME  usedbychildren         0B                        -
MICRON/HOME  usedbyrefreservation   0B                        -
MICRON/HOME  logbias                latency                   default
MICRON/HOME  objsetid               7650                      -
MICRON/HOME  dedup                  off                       default
MICRON/HOME  mlslabel               none                      default
MICRON/HOME  sync                   standard                  default
MICRON/HOME  dnodesize              legacy                    default
MICRON/HOME  refcompressratio       1.06x                     -
MICRON/HOME  written                19.0G                     -
MICRON/HOME  logicalused            719G                      -
MICRON/HOME  logicalreferenced      666G                      -
MICRON/HOME  volmode                default                   default
MICRON/HOME  filesystem_limit       none                      default
MICRON/HOME  snapshot_limit         none                      default
MICRON/HOME  filesystem_count       none                      default
MICRON/HOME  snapshot_count         none                      default
MICRON/HOME  snapdev                hidden                    default
MICRON/HOME  acltype                nfsv4                     default
MICRON/HOME  context                none                      default
MICRON/HOME  fscontext              none                      default
MICRON/HOME  defcontext             none                      default
MICRON/HOME  rootcontext            none                      default
MICRON/HOME  relatime               on                        default
MICRON/HOME  redundant_metadata     all                       default
MICRON/HOME  overlay                on                        default
MICRON/HOME  encryption             aes-256-gcm               -
MICRON/HOME  keylocation            prompt                    local
MICRON/HOME  keyformat              passphrase                -
MICRON/HOME  pbkdf2iters            350000                    -
MICRON/HOME  encryptionroot         MICRON/HOME               -
MICRON/HOME  keystatus              available                 -
MICRON/HOME  special_small_blocks   0                         default
MICRON/HOME  snapshots_changed      Thu Dec  1 21:01:59 2022  -
MICRON/HOME  com.apple.browse       on                        default
MICRON/HOME  com.apple.ignoreowner  off                       default
MICRON/HOME  com.apple.mimic        apfs                      local
MICRON/HOME  com.apple.devdisk      poolonly                  default
sh-3.2#


Has anyone succeeded in running a user this way in recent MacOS releases? Any tips?
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby lundman » Fri Dec 02, 2022 12:29 am

What goes wrong?
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby Sharko » Fri Dec 02, 2022 8:48 am

When I try to switch users from admin account to the new Frankenstein'ed non-admin account via the menu bar it accepts the password just fine, then the password box goes away and is replaced by the small "rotating matchsticks" icon for about 15 seconds, then it puts up a small MacOS notification window saying essentially that "there is an error, you can't log into this user account." The only option is an OK button. When I press that it takes me to the login screen where all accounts are visible; I can select the admin account (which is on true APFS), and that works.

Just for completeness, here is the overall picture of how I'm trying to do this upgrade:

Starting point was a working arrangement of ZFS 1.9.4 under Mojave classic Mac Pro 2010 six processor Westmere. On that computer (which is still untouched, functional, and completely separate) I have a SATA boot drive running standard Mojave on the standard Apple APFS, and a separate SATA vdev formatted as O3X. There are two accounts on that computer: adminkurt with home directory on the apfs boot drive, and my daily driver non-admin account 'userkurt' on an encrypted dataset. The standard procedure for booting up that computer requires me to boot up, login to adminkurt, decrypt the zfs dataset containing the home directory for the non-admin user, then I switch or login to the non-admin account 'userkurt'. The zfs pool was actually a mirrored vdev, so I zfs-detached one of the mirror drives and then wiped it to apfs in preparation for using it on the new computer.

I wanted to re-create that arrangement on a new 12 core westmere 2010 classic Mac Pro. I mastered the OpenCore process from the macrumors forum posting (the so-called "purist" or "vanilla" method authored by @cdf, not OCLP). I got Monterey 12.6.1 up and running on an NVME drive, and loaded "OpenZFSonOsX-2.1.99-Catalina-10.15.pkg" on it in mid-October. I put the empty SATA SSD, temporarily formatted as apfs, in the new computer, and created a new pool named MICRON with the usual standard properties of

Code: Select all
ashift=13 -O compression=lz4 -O casesensitivity=insensitive -O atime=off -O normalization=formD


I didn't want to try and move the encrypted dataset directly from 1.9.4 to 2.1+; it seemed like the upgrade process on an encrypted dataset might come back to bite me later in some way. So I took what I thought was lowest risk method to move my data: I did a non-raw send of the original Mojave+1.9.4 dataset to non-encrypted dataset (created under 1.9.4) on a external drive with zpool NOCRYPT while still on the Mojave+1.9.4 computer. Then I brought that external drive over to the new computer running Monterey, where I had upgraded to 2.1.6rc4 at that point, and did a zfs send of the dataset into a non-encrypted dataset on a completely different internal drive where the zpool had been created by O3X 2.1.6rc4, named EXO/HOME_BACKUP_UNENCRYPTED (it's a Seagate exos drive). The point of all this was to not have to trust that a zpool upgrade was going to work - ultimately the dataset crossed the barrier of "zpool created under 1.9.4" to "zpool created under 2.1.6rc4" via the mechanism of a non-encrypted send and receive, which I think is lowest risk, since the replication process and format has been stable for a very long time.

OK, so now I have a hopefully clean dataset on a zpool created under 2.1.6rc4 named EXO/HOME_BACKUP_UNENCRYPTED. Everything that I describe from this point onward occurs on the new Monterey system, by the way. I then created an encrypted empty dataset EXO/ENCRYPTED on that same external disk (it's a big disk, 10 TB). Then I attempted to do a zfs send from the unencrypted dataset EXO/HOME_BACKUP_UNENCRYPTED to the newly created dataset EXO/ENCRYPTED/HOME_BACKUP. This is the point where I ran into the problem (described in another post in the Development section of the forum) where it appeared that AES-NI hardware instructions were not being utilized in encrypted datasets. You very kindly fixed that problem with a custom version of rc4 that fixed that problem. I think that I killed the zfs send - receive that was doing software encryption because it was slow originally, so there was no EXO/ENCRYPTED/HOME_BACKUP until I did a successful send - receive with your custom rc4.

I think that at this point I switched from rc4 to rc6. Recall that I had the SSD drive that I split off from the original Mojave vdev? I installed that in the new computer, and created a pool with the usual parameters (shown above in the code window). Then I did a raw send of a portion of the encrypted dataset EXO/ENCRYPTED/HOME_BACKUP ; it was only a portion in the sense that I didn't need every snapshot going back to the dawn of Mojave on the new working dataset MICRON/HOME (note that I reused the old zpool/dataset name from Mojave, but it is a different zpool/dataset). I did it as a raw send so that MICRON/HOME and EXO/ENCRYPTED/HOME_BACKUP shared the same IV; in the future I want to be able to do raw sends in the other direction, from MICRON/HOME to EXO/ENCRYPTED/HOME_BACKUP.

I was continuing to use my Mojave computer for computing tasks while all this was going on, so I had to transfer two snapshots via incremental send receive process. These were done over ssh: zfs send running under 1.9.4 to zfs receive running 2.1.6rc6 to an unencrypted dataset, then unencrypted to EXO/ENCRYPTED/HOME_BACKUP, then local raw send-receive from EXO/ENCRYPTED/HOME_BACKUP to MICRON/HOME (latter in non-decrypted state to preserve IV).

OK, so I've finally got what was my original Mojave dataset on MICRON/HOME. This is when I switched to the O3X 2.1.6 zfs release. The idea at this point is to create a non-admin user on the Monterey apfs boot drive, connect it up to iCloud, and get it completely functional but without any real data in the user home directory. I do this, and of course it works perfectly in this arranagement - this is standard Apple practice of creating a new user in Users and Groups, signing into iCloud, etc. Then I log out of the non-admin userkurt account, and log in to the admin account. I decrypt MICRON/HOME.

In the adminkurt account I go to Users and Groups of System Preferences, right click on userkurt and select advanced options. In the screen that opens I set the home directory for userkurt to be the image of the old Mojave home directory of userkurt on MICRON/HOME. Then I open terminal, "sudo sh" to root and cd to the new userkurt home directory on MICRON/HOME. I mv Library (the old Mojave Library) to OldLibrary to make room for the new Library I'm going to copy over from the boot drive. I use ditto with this command to move the Library over:

Code: Select all
ditto /Users/userkurt/Library/ /Volumes/MICRON/HOME/Users/userkurt/Library


This does create a Library folder in the expected place (top level of userkurt home directory), though it isn't visible in Finder (perhaps because by default Library is never visible in Finder). This ditto command executed silently without complaint.

I then tried to copy my Firefox and Thunderbird profiles from OldLibrary into the new Library, and ran into those phantom file problems using ditto (I saw your reply to my report in the other branch of the forum). I ignored this particular problem for the moment, rebooted the computer (which must be done when you relocate the home directory). Logged into the adminkurt account, then I unlocked MICRON/HOME dataset. Then tried to switch into the userkurt account, and got the error described at the very top of this posting.

I remembered something that I forgot to do originally: set the mimic property. So I set "com.apple.mimic=apfs" on MICRON/HOME and rebooted. Decrypted MICRON/HOME, tried to switch into userkurt. Same error.

I thought it could be a user:group mismatch, but this looks OK:

Code: Select all
sh-3.2# ls -l -F -O -s /Volumes/MICRON/HOME/Users/userkurt/
total 14571
   17 -rw-r--r--   1 userkurt  staff  -             3 Feb 25  2008 .CFUserTextEncoding
   17 -rw-r--r--@  1 userkurt  staff  hidden    26628 Aug 25  2016 .DS_Store
   97 drwx------   5 userkurt  staff  -            79 Nov 27 20:23 .Trash/
   33 drwxr-xr-x   2 userkurt  staff  -             2 Jan 22  2010 .Xcode/
   17 -rw-------   1 userkurt  staff  -          9987 Nov 12 09:44 .bash_history
  129 drwx------   2 userkurt  staff  -           186 Nov 12 09:44 .bash_sessions/
   33 drwxr-xr-x   7 userkurt  staff  -             7 Apr 20  2015 .cache/
   33 drwxr-xr-x   7 userkurt  staff  -             7 Sep 23  2017 .config/
   33 drwx------   2 userkurt  staff  -             3 Jan 15  2010 .cups/
   33 drwx------   4 userkurt  staff  -            20 May  7  2014 .dropbox/
   33 drwxr-xr-x   6 userkurt  staff  -             7 Jun 20  2019 .dvdcss/
   97 drwxr-xr-x   2 userkurt  staff  -            29 Jun  6  2011 .fontconfig/
   33 drwxr-xr-x   2 userkurt  staff  -            14 Jun  6  2011 .fonts/
   33 drwx------   3 userkurt  staff  -             9 May  7  2014 .gnupg/
   33 drwxr-xr-x   2 userkurt  staff  -             3 Jun 30  2012 .keepassx/
   17 -rw-------   1 userkurt  staff  -            43 Feb  4  2011 .lesshst
   33 drwx------   4 userkurt  staff  -             4 Sep 23  2017 .local/
   33 drwxr-xr-x   2 userkurt  staff  -             3 Apr 26  2022 .matplotlib/
   33 drwx------   2 userkurt  staff  -             2 Jun 21  2010 .mozilla/
   33 drwxr-xr-x   2 userkurt  staff  -             3 Apr  9  2019 .oracle_jre_usage/
   17 -rw-------   1 root      staff  -          1297 Sep 23  2013 .sh_history
   17 -rw-r--r--   1 userkurt  staff  -          8192 Oct 31  2021 .sheepshaver_nvram
   17 -rw-r--r--   1 userkurt  staff  -           847 Oct 31  2021 .sheepshaver_prefs
   33 drwxr-xr-x   4 root      staff  -            20 Jan  4  2014 .shsh/
   33 drwx------   2 userkurt  staff  -             5 Apr 12  2020 .ssh/
   17 -rw-------   1 userkurt  staff  -          6907 Nov  3  2015 .viminfo
   17 -rw-r--r--   1 userkurt  staff  -           142 Dec 26  2021 .vuescanrc
   33 drwxr-xr-x   3 userkurt  staff  -             4 Jul  8  2008 Applications/
   97 drwx------  13 userkurt  staff  -            48 Nov 29 08:28 Desktop/
   33 drwx------  50 userkurt  staff  -            68 Nov 17 06:45 Documents/
  321 drwx------  33 userkurt  staff  -           714 Nov 26 09:38 Downloads/
12513 -rw-r--r--   1 userkurt  staff  -      29519193 Sep  8  2014 FileTypeList
   17 -rwxr-xr-x@  1 userkurt  staff  -           112 Sep  8  2014 GetFileTypeListing.sh*
   33 drwx------@ 78 userkurt  staff  -            79 Dec  1 21:31 Library/
   97 drwx------  14 userkurt  staff  -           153 Mar 21  2017 Movies/
   33 drwx------@ 13 userkurt  staff  -            16 Nov 17  2016 Music/
   33 drwx------@ 71 userkurt  staff  -            72 Mar  7  2021 OldLibrary/
   33 drwx------  32 userkurt  staff  -            41 Nov 30 22:07 Pictures/
   97 drwxr-xr-x   3 userkurt  staff  -            17 May 10  2019 Public/
   97 drwxr-xr-x  17 userkurt  staff  -            28 Mar  7  2017 Sites/
   33 drwxr-xr-x   6 userkurt  staff  -             8 Mar  7  2021 VirtualBox VMs/
   33 -rw-r--r--@  1 userkurt  staff  -         49219 Sep  8  2014 Word5_Files
   97 drwxr-xr-x   2 userkurt  staff  -            24 Oct  7  2015 dwhelper/
sh-3.2#


Some thoughts on how I might go forward:

- I could create a test user on the boot disk, and rsync that users' whole home directory to MICRON/HOME/Users/testuser. Then see if I can point Monterey to use that relocated home directory.
- I could delete the folder /Volumes/MICRON/HOME/Users/userkurt/Library, and then try to recreate it with rsync (as opposed to ditto). Forget about the Firefox and Thunderbird profiles to start with, also.
- Worst case I could manually transfer data (or migration assistant) from MICRON/HOME into the standard userkurt account on the boot drive and then just run from that going forward. I could do daily rsyncs to a mirror of that account on a zfs dataset on another disk.

Any other ideas? Thanks for reading this... surely I can't be the only one in our community to have relocated their user account to a ZFS dataset, can I?
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby Sharko » Fri Dec 02, 2022 9:21 am

Took a shower, had a bit of a think: I might have erred in where I set the mimic property. I set the mimic property at MICRON/HOME, whereas the operating system was told that the home directory was at /Volumes/MICRON/HOME/yaddayadda. So I think that I need to have set mimic at the top level, on MICRON, since that is what the OS will see as the volume. I will try this approach later today, right now I need to get off to work.
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby lundman » Sat Dec 03, 2022 6:00 pm

I've tried to replicate this, and going on your ZFS created a dataset like

# zfs create -ocompression=lz4
-o normalization=formD -o casesensitivity=insensitive -o encryption=aes-256-gcm
-o keyformat=passphrase -o com.apple.mimic=apfs BOOM/test

Then it looked like you had errors in FontCollection;

# ditto /Users/lundman/Library/FontCollections/ /Volumes/BOOM/test/Font/

But no issues there.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby lundman » Sat Dec 03, 2022 6:04 pm

aha, Developer/CoreSimulator/Devices/37DF4C13-2EA0-4171-A70C-039B40B39C04/data/Media/iTunes_Control/iTunes/MediaLibrary.sqlitedb

chflags("/Volumes/BOOM/test/Font/MediaLibrary.sqlitedb\0", 0x20, 0x0) = -1 Err#22
write_nocancel(0x2, "ditto: \0", 0x7) = 7 0
getrlimit(0x1008, 0x7FFEE059DA10, 0x0) = 0 0
write_nocancel(0x2, "/Volumes/BOOM/test/Font/MediaLibrary.sqlitedb: Invalid argument\0", 0x3F) = 63 0
write_nocancel(0x2, "\n\0", 0x1) = 1 0

So, chflags eh hmm
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby lundman » Sat Dec 03, 2022 8:30 pm

#define UF_COMPRESSED 0x00000020

Guess we reply error to that, but maybe we shouldn't.

ditto --nopreserveHFSCompression

removes error
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby Sharko » Sat Dec 03, 2022 10:52 pm

Haha, I didn't get much out of that other than try the ditto flag! But at least it looks like you got something to chew on. Thank you for the hint, will report back tomorrow.

Cheers,

Kurt
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby lundman » Sun Dec 04, 2022 2:13 am

It is unlikely we will fix it. Normally:

* write the file data
* compress data, and write out compressed xattr.
* set UF_COMPRESSED
* truncate file data to 0 bytes.

leaving only compressed xattr data and "fake" file data. We intercept this, so that
when it tries to truncate the file to 0 bytes, we say "sure!" and delete the compressed xattr instead.
With ZFS, you want to use the compression of ZFS and not the outdated decmpfs that goes into xattr.
Normally, this is done transparently.

ditty, what happens here is:
* create 0 len file
* write compressed xattr
* set UF_COMPRESS

So we can't transparently fake/undo the compression, as we never get the uncompressed data.
ditto should skip hfsCompression automatically when we claim to be "zfs", so it might only be
an issue with mimic on. It is unfortunate, but there is no way to work around it.
Only apple and xnu knows the decmpfs (especially if they change it) and we aren't allowed to
call it.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Moving home directory over to a ZFS dataset 2.1.6 Intel

Postby Sharko » Sun Dec 04, 2022 8:40 am

Thanks that makes sense. By the way, when I saw 'decmpfs' in your response I recognized it as one of the xattr attributes that I saw on files transferred: com.apple.decmpfs. 'Ditto' is certainly a weird little utility; I've always used it in the past because I trusted it to copy and preserve any weird little Apple-specific flags on files, but I think rsync is going to be my new friend. Thanks much, Mr. Lundman.

Kurt
Sharko
 
Posts: 230
Joined: Thu May 12, 2016 12:19 pm

Next

Return to General Help

Who is online

Users browsing this forum: No registered users and 21 guests