Perplexing problem: ZFS home directory can't be written to

[Sharko]

So, I've used a ZFS dataset for a home directory for years. Even used a natively encrypted home directory for years. But, after a couple of recent upgrades (described here shortly), I have run into a problem that I just can't figure out.

The last known good working state for my Mac Pro 5,1 was Monterey 12.6.3-ish, ZFS 2.1.6, OpenCore 0.8.7-ish with a 2TB ZFS home directory that was unencrypted. With that combination I was able to boot directly into the user account that was on ZFS, and everything seemed to work for months without issue. I had been wanting to upgrade to latest Monterey (12.7.3, and so I had my boot rom rebuilt by tsialex from the MacRumors forum. Then I undertook an upgrade process that went like this:

- I copied a working version of OpenCore 0.9.7 from my backup computer to a fresh SATA SSD with its EFI partition.
- I booted a Monterey USB installer through that OpenCore ESP, and installed Monterey 12.7.3 on that fresh SATA SSD.
- On first boot of 12.7.3 I used Migration assistant to copy over all the programs, all the user data, and all the settings from the old 12.6.3 SSD (which was NVMe, if it matters) to the new SATA SSD.
- Migration assistant didn't copy over the zfs kexts, or if it did they were not properly activated. So, I thought that I might as well upgrade ZFS to 2.2.2. I did the install, approved the kext in System Preferences, rebooted, and now ZFS seemed to be working.

Everything seemed to work, though I mostly interacted with the 12.7.3 in my admin user, which is NOT on ZFS - it's under /Users/admin on the boot disk. I do recall that I specifically booted into the non-admin user briefly to make sure that iMessage was working after the rom rebuild - tsialex had warned me that iMessage is kind of the bellwether for whether a boot rom is OK. iMessage was working fine, and I don't recall any other funny clues at that time. About a week after the upgrade I booted the computer, logged into the admin account, and noticed that the zpool that houses the user dataset was not automatically mounting. I thought that was odd, but I imported the pool without incident. When I switched to the non-admin account on the ZFS dataset it took a very long time to populate the desktop, and there were some iCloud warnings. I went to System Preferences>iCloud, but I don't think it even asked me for credentials.

Anyway, I started doing some audio work in the non-admin account. At certain point I noticed: hey, I can't save my work! I would hit Cmd-S, the system would beachball for 15 seconds, and then nothing: no file dialog to choose a name to save under. I switched to TextEdit, created a new file, typed in some text, and tried to save: same thing, no file save dialog box and lots of beachballing.

I thought that perhaps the NVMe SSD that housed the non-admin user's home directory had failed, so I did zfs send-receive to transfer a recent snapshot to another 2TB NVMe I had laying around. I changed the home directory location to point to the new NVMe SSD and rebooted... same exact behavior: I can't write to that non-admin user's home directory on the replicated ZFS dataset.

I double-checked, and com.apple.mimic is set apfs.

Other SATA SSD's and hard disks with zpools on them mount automatically at boot. I think that for a short period I had the two NVMe zpools in the system, and neither of them import and mounted automatically, despite the fact that encryption is NOT turned on.

I don't know where to go from here. Prepare a fresh 12.7.3 disk, and run migration assistant again? I tried looking in Console, but I nothing jumps out at me, even when the system is beachballing.

I would appreciate your suggestions. Thanks.

[Sharko]

Just for completeness, I should add that the ZFS dataset for the home folder is NOT readonly.

[nodarkthings]

Hi Sharko!
It probably has nothing to do with your issue, and my system has nothing in common with yours, but FWIW: shortly after latest XProtect update (2186), a few strange things happened, 2 apps disappeared and another became very sluggish at launch... I couldn't trace for sure that XProtect was the culprit, but I haven't found anything else and after a few days and reinstalling the missing apps, all came back to normal.
Have you tried creating a new user to see if it behaves the same?
I guess you've tried uninstalling ZFS, rebooting and reinstalling it again?

[nodarkthings]

Also: stepping from OpenCore 0.8.7 to OpenCore 0.9.7 is a pretty big jump... I would try booting with your original 0.8.7 EFI if it's compatible with your OS.
I'd check the SIP settings between both versions too — and SecureBootModel.
Sorry if it doesn't help, but that's all that came to my mind.

[Sharko]

Thank you for your suggestions! I'll try them out one at a time. I honestly think the problem is something that Migration Assistant brought in; in the past I've always done my own migration as follows: set up the computer with OpenCore and OS, install ZFS, install apps, create users, connect to iCloud and get all the Apple services working with a minimal setup so that all the Apple stuff is consolidated and working within ~/Library. Then, and only then, rsync the ~/Library on the boot disk over to the external home directory, change the home directory to the external ZFS file system, and reboot.

I did discover what the problem with failure to automount was about: the fresh install of the operating system erased my permissions for zpool and Terminal to access the full disk. Once I granted those permissions again the SSD with the external home directory would automount on boot again, but I still can't write while logged into that user.

Anyway, this is kind of recreational for me at this point - I had bought a used 2019 Mac Pro last fall, and had been dinking along getting it set up. The problems with the Mac Pro 5,1 pushed me to finally move my files over and commit to the 2019 Mac Pro. MacOS is getting so fussy about the underlying hardware and software/filesystem that I've taken a different, more conservative approach: now all my home directories are on the boot disk under APFS, and I Carbon Copy Cloner the boot disk to a ZFS dataset on a disk in the chassis. Then I will snapshot the data and replicate it around to various destinations: TrueNAS, the backup cMP, etx.

Cheers,

Kurt

[nodarkthings]

Yes, the simpler the better.
Cheers!