BUG: ZFS 2.1.6 Ventura 13 amd auto import fails

All your general support questions for OpenZFS on OS X.

BUG: ZFS 2.1.6 Ventura 13 amd auto import fails

Postby mike0810 » Mon Jan 02, 2023 6:26 am

I run openzfs 2.1.6 on a M1 Mac Mini with external thunderbolt enclosure

Since the upgrade to Ventura 13.1 the auto import does not work anymore.
I removed and reinstalled zfs, no changes.

So I observed some odds:
1. /var/log/org.openzfsonosx.zconfigd.err is populated with following messages despite I think these should not go into err, but log:
Code: Select all
Looking for service matching osx_openzfsonosx_zfs_zvol
Starting the run loop


2. /var/log/org.openzfsonosx.zed.err is populated with following messages (inline comments from me):
Code: Select all
Ignoring "zed.rc": not executable by user
##Its there chmod like in the package-no exec bit: -rw-r--r--   1 root  wheel   4.1K Dec  1 12:23 zed.rc
Failed to stat "/etc/zfs/zed.d/zvol.remove.sh": No such file or directory
## It is existing: lrwxr-xr-x   1 root  wheel    47B Jan  2 15:05 zvol.remove.sh -> /usr/local/zfs/libexec/zfs/zed.d/zvol.remove.sh
Registered zedlet "vdev_clear-led.sh"
Registered zedlet "history_event-zfs-list-cacher.sh"
Registered zedlet "data-notify.sh"
Registered zedlet "all-syslog.sh"
Failed to stat "/etc/zfs/zed.d/zvol.create.sh": No such file or directory
##It is existing: lrwxr-xr-x   1 root  wheel    47B Jan  2 15:05 zvol.create.sh -> /usr/local/zfs/libexec/zfs/zed.d/zvol.create.sh
Registered zedlet "pool_import-led.sh"
Registered zedlet "scrub_finish-notify.sh"
Registered zedlet "resilver_finish-notify.sh"
Registered zedlet "vdev_attach-led.sh"
Registered zedlet "snapshot_unmount.sh"
Registered zedlet "statechange-notify.sh"
Registered zedlet "resilver_finish-start-scrub.sh"
Registered zedlet "statechange-led.sh"
Registered zedlet "snapshot_mount.sh"
Ignoring "zed-functions.sh": not executable by user
## Same issue with zed.rc, no exec bit: -rw-r--r--   1 root  wheel    16K Dec  1 12:23 zed-functions.sh
ZFS Event Daemon 2.1.6-1 (PID 414)
Add Agent: init
Diagnosis Engine: register module
Retire Agent: register module
Processing events since eid=0


3. /var/log/org.openzfsonosx.zpool-import-all.err is populated with following messages:
Code: Select all
/Library/LaunchDaemons/org.openzfsonosx.zpool-import.plist: service already loaded
Load failed: 37: Operation already in progress


4. /var/log/org.openzfsonosx.zpool-import-all.log is populated with following messages:
Code: Select all
+zpool-import-all.sh
Mon Jan  2 15:08:36 CET 2023
Waiting up to 60 seconds for the InvariantDisks idle file /var/run/disk/invariant.idle to exist
Found /var/run/disk/invariant.idle after 0 iterations of sleeping 0.1 seconds
Mon Jan  2 15:08:51 CET 2023
Mon Jan  2 15:08:51 CET 2023
-zpool-import-all.sh
Loading and starting org.openzfsonosx.zpool-import
Mon Jan  2 15:09:01 CET 2023
Status: 0
Touching the file /var/run/org.openzfsonosx.zpool-import-all.didRun
Mon Jan  2 15:09:01 CET 2023
-zpool-import-all.sh


No auto import happens. Is there anything more to submit?

Code: Select all
kextstat | grep zfs
Executing: /usr/bin/kmutil showloaded
No variant specified, falling back to release
  242    0 0xfffffe0006c15400 0x8d568    0x8d568    org.openzfsonosx.zfs (2.1.6) A31057B6-7886-336C-A417-56FC977D9210 <21 7 5 4 3 1>


/var/run/disk is populated, zpool status looks like this (if i run manually sudo zpool import -a):

Code: Select all
zpool status
  pool: data
 state: ONLINE
  scan: scrub repaired 0B in 12:51:09 with 0 errors on Sat Dec 31 11:42:21 2022
remove: Removal of vdev 7 copied 560K in 0h0m, completed on Fri Sep 17 15:28:11 2021
   2.46K memory used for removed device mappings
config:

   NAME                                            STATE     READ WRITE CKSUM
   data                                            ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-35C2F2A3-3D74-E843-AFF9-ECAB451966BE  ONLINE       0     0     0
       media-7DBE657E-24CD-4A4E-9A62-8398AE6D2109  ONLINE       0     0     0
     mirror-2                                      ONLINE       0     0     0
       media-A26F4CEC-166E-9F40-9AEA-0DBD408B9B37  ONLINE       0     0     0
       media-8F11C8FB-EFD5-184E-A2E9-4E950D075F9E  ONLINE       0     0     0
     mirror-9                                      ONLINE       0     0     0
       media-173CEBF9-E228-B94E-90E6-40FF3C9FD77E  ONLINE       0     0     0
       media-EC173497-E8A0-3E4F-8F0A-3962812EF6A4  ONLINE       0     0     0
   logs
     media-DFE06C1D-C4DC-074D-95C7-491016AAEB07    ONLINE       0     0     0
   cache
     media-10B7F028-01B3-3E40-ACFE-15D09224387C    ONLINE       0     0     0

errors: No known data errors


To add some remarks: bash and zpool are marked as "Full Disk Access" in the Privacy & Security Pane
I do not know exactly, what purpose the org.openzfsonosx.xxxx.plist.in files have

I had a temporary success by using sudo launchctl unload -w xxx.zpool-import-all.plist and the xxx.zpool-import.plist, and then load -w again both. Then the import.plist did the import. But next reboot did not work again.
Last edited by mike0810 on Tue Jan 03, 2023 12:48 pm, edited 2 times in total.
mike0810
 
Posts: 51
Joined: Fri Jan 16, 2015 5:17 pm

Re: ZFS 2.1.6 Ventura 13 amd auto import fails

Postby mike0810 » Mon Jan 02, 2023 7:01 am

At least I found a bug causing the log errors described above in the zed.err log:

The aliases point to the wrong file names:
ls /etc/zfs/zed.d/
lrwxr-xr-x 1 root wheel 47B Jan 2 15:05 zvol.create.sh -> /usr/local/zfs/libexec/zfs/zed.d/zvol.create.sh
lrwxr-xr-x 1 root wheel 47B Jan 2 15:05 zvol.remove.sh -> /usr/local/zfs/libexec/zfs/zed.d/zvol.remove.sh

ls /usr/local/zfs/libexec/zfs/zed.d/
-rwxr-xr-x 1 root wheel 1.0K Dec 1 12:23 zvol_create.sh
-rwxr-xr-x 1 root wheel 784B Dec 1 12:23 zvol_remove.sh
mike0810
 
Posts: 51
Joined: Fri Jan 16, 2015 5:17 pm

Re: ZFS 2.1.6 Ventura 13 amd auto import fails

Postby nadigo » Mon Jan 02, 2023 5:29 pm

Have the same issue / errors on openzfs 2.1.6 on a M1 Mac Mini + external thunderbolt GD
In my case, I can not import manually despite seeing that the pools are online 'sudo zpool import -d /dev'

Any luck?
nadigo
 
Posts: 5
Joined: Mon Jan 02, 2023 1:11 pm

Re: ZFS 2.1.6 Ventura 13 amd auto import fails

Postby mike0810 » Tue Jan 03, 2023 8:56 am

I recognized, if i after the mac mini started, i do a:
Code: Select all
sudo launchctl unload -w /Library/LaunchDaemons/org.openzfsonosx.zpool-import-all.plist
sudo launchctl load -w /Library/LaunchDaemons/org.openzfsonosx.zpool-import-all.plist


then the pools get auto imported. But not if I only reboot. Somehow the Invariant Disk exits to early and creates the invariant.idle file to quick, after 0 iterations.
Code: Select all
Waiting up to 60 seconds for the InvariantDisks idle file /var/run/disk/invariant.idle to exist
Found /var/run/disk/invariant.idle after 0 iterations of sleeping 0.1 seconds

The zpool-import-all script runs then without the disks already there, thats my assumption. Somehow this is a regression from earlier versions afaik.
Last edited by mike0810 on Tue Jan 03, 2023 12:43 pm, edited 1 time in total.
mike0810
 
Posts: 51
Joined: Fri Jan 16, 2015 5:17 pm

Re: ZFS 2.1.6 Ventura 13 amd auto import fails

Postby mike0810 » Tue Jan 03, 2023 12:41 pm

So the solution is as follows:

The org.openzfsonosx.zpool-import.plist daemon gets loaded once at boot but exits with -9, you see when you run
Code: Select all
sudo launchctl list | grep org.openzfsonosx


Below flag in the plist is set to true, which means it can only launched once per boot. But it already "RunAtLoad" and did exit -9
Code: Select all
   <key>LaunchOnlyOnce</key>
   <true/>


So I set the key to false in the org.openzfsonosx.zpool-import.plist.
Code: Select all
   <key>LaunchOnlyOnce</key>
   <false/>


Secondly, in the /usr/local/zfs/libexec/zfs/launchd.d/zpool-import-all.sh, I added the flags "-kp" to the line, which means it kills the process if it runs (which is interpreted as already loaded because it exited -9) and additionally it prints out the process id to stout in the log:

Code: Select all
/bin/launchctl kickstart -kp system/org.openzfsonosx.zpool-import


Now auto import works...
mike0810
 
Posts: 51
Joined: Fri Jan 16, 2015 5:17 pm

Re: BUG: ZFS 2.1.6 Ventura 13 amd auto import fails

Postby xenophon » Thu Jan 19, 2023 1:04 am

Thanks for the writeup. Will give it a try.
xenophon
 
Posts: 17
Joined: Tue Jul 28, 2015 11:58 pm


Return to General Help

Who is online

Users browsing this forum: No registered users and 2 guests

cron