March 14th build fails to mount encrypted filesystem

[glessard]

Unfortunately the April 5th RC2 build did not fix the issue for me. Has it fixed it for everyone/anyone else?
Unconundrum and FadingIntoBlue seemed to have the same issue as me...

[FadingIntoBlue]

Unfortunately the April 5th RC2 build did not fix the issue for me. Has it fixed it for everyone/anyone else?
Unconundrum and FadingIntoBlue seemed to have the same issue as me...

I am in the same boat, it hasn't fixed it for me either. The error message shows a problem at the root filesystem of the pool (if it is encrypted) and also at any child encrypted filesystem.
The one bit of good news for me is that the same pools and child filesystems are still mountable with no errors on 1.9.4, so the error does not appear to be making a lasting change to the filesystem.
Happy to explore further, but I am about at the limit of my knowledge of how to test this further.

[lundman]

Wonder if the new 2.0 debug code can help us out here. "zfs mount -l" will fail right? So could
you try:

Code: Select all

# sysctl kstat.zfs.darwin.tunable.zfs_flags=513
# zfs mount -l datasetthatfails
# sysctl kstat.zfs.misc.dbgmsg.dbgmsg


The dbgmsg can get quite large, so you might want to do when the system is fairly quiet (ZFS wise). Set flags back to 0 when done.
1 is to show debug prints, and 512 is to print SET_ERROR() - so maybe it will show us where it fails.

(edit: fix typos)

[FadingIntoBlue]

# ./scripts/cmd-macos.sh zfs mount -l datasetthatfails

Not sure where the /scripts directory is, and locate doesn't show cmd-macos.sh on my Macs, sorry

Below a pool status -v of one of the pools/filesystems I can't mount - since I haven't seen the actual error listed yet. This pool opens fine under 1.9.4

Code: Select all

  pool: P2021-01H
 state: ONLINE
status: One or more devices has experienced an error resulting in data
   corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
   entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
config:

   NAME                                          STATE     READ WRITE CKSUM
   P2021-01H                                     ONLINE       0     0     0
     media-14117272-6A32-7F43-954E-81112CDCE122  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        P2021-01H:<0x0>
        P2021-01H/Arc2021:<0x0>[quote][/quote]


P2021-01H is encrypted, Arc2021 inherits the encryption. I have also been able to mount an unencrypted root, and had the encrypted child filesystem be unmountable

Can confirm that zfs mount -l fails with the error

Code: Select all

cannot mount 'P2021-01H': Unknown error: -1

Hope that helps

[lundman]

Sorry, the script/cmd-macos is just a wrapper to run "zfs" without installing it for developers. Just run the command itself.

Now if you are getting metadata errors, that would suggest the crypto routines are getting different results, either MAC or Auth. But
that's also weird, since crypto works itself. We should also check if it is the assembler routines, by setting it to use generic. (sysctl),

[FadingIntoBlue]

Sorry, the script/cmd-macos is just a wrapper to run "zfs" without installing it for developers. Just run the command itself.

Cool, did that, with the following error

Code: Select all

% sudo zfs mount -l P2021-01H
Enter passphrase for 'P2021-01H':
cannot mount 'P2021-01H': Unknown error: -1
% sysctl kstat.zfs.misc.dbgmsg.dbgms
sysctl: unknown oid 'kstat.zfs.misc.dbgmsg.dbgms'

Tried with sudo as well, same error

[lundman]

Should be a "g" on that last command:

sysctl kstat.zfs.misc.dbgmsg.dbgmsg

[FadingIntoBlue]

Should be a "g" on that last command:

That did the trick - you were right about a lot of output. About 3 times as much as I can fit into a message - see attached file

kstat.zfs.misc.dbgmsg.dbgmsg-output.txt

output from sudo sysctl kstat.zfs.misc.dbgmsg.dbgmsg

(184.14 KiB) Downloaded 204 times

[lundman]

zfs_vfs_mount: zfs_domount returned 5

Probably the only hint we'll get there.

I was going to suggest we disable the asm versions with:

sysctl kstat.zfs.darwin.tunable.icp_aes_impl=generic

BUT - it turns out setting any string kstat will PANIC - so I'll need some time
to fix that and push an rc3.

[lundman]

OK, posted OpenZFSonOsX-2.0.0-Big.Sur-11.0.pkg with kstat fix, so we can try:

Code: Select all

kstat.zfs.darwin.tunable.icp_gcm_impl: cycle [fastest] generic
kstat.zfs.darwin.tunable.icp_aes_impl: cycle [fastest] generic
kstat.zfs.darwin.tunable.zfs_vdev_raidz_impl: cycle [fastest] original scalar

Assuming you use AES:

sysctl kstat.zfs.darwin.tunable.icp_aes_impl=generic


Then attempt the zfs mount -l again.