2.1.6 hung on rollback, other ditto weirdness

Developer discussions.

2.1.6 hung on rollback, other ditto weirdness

Postby Sharko » Sat Dec 03, 2022 9:20 am

I'm apparently exercising the code quite a bit as I try to make this transition to running a user in a O3X 2.1.6 encrypted dataset. As I try different things, and see some oddities, I've been doing a rollback to the snapshot that represented the last full snapshot of the user data that originated on the original Mojave system (migrated through various send/receive processes as outlined in one of my other posts https://openzfsonosx.org/forum/viewtopic.php?f=26&t=3745.

I did the first rollback using pretty standard command:

Code: Select all
Admins-cMP-DP1:userkurt adminkurt$ sudo sh
Password:
sh-3.2# pwd
/Volumes/MICRON/HOME/Users/userkurt
sh-3.2# zfs rollback MICRON/HOME@2022_11_30_user_data_Mojave_final
sh-3.2# mv Library OldLibrary
sh-3.2# ditto /Users/userkurt/Library/ /Volumes/MICRON/HOME/Users/userkurt/Library
ditto: /Volumes/MICRON/HOME/Users/userkurt/Library/FontCollections/Fixed Width.collection: Invalid argument
ditto: /Volumes/MICRON/HOME/Users/userkurt/Library/FontCollections/Fun.collection: Invalid argument
ditto: /Volumes/MICRON/HOME/Users/userkurt/Library/FontCollections/Traditional.collection: Invalid argument
ditto: /Volumes/MICRON/HOME/Users/userkurt/Library/FontCollections/Modern.collection: Invalid argument
ditto: /Volumes/MICRON/HOME/Users/userkurt/Library/FontCollections/Web.collection: Invalid argument
ditto: /Volumes/MICRON/HOME/Users/userkurt/Library/FontCollections/PDF.collection: Invalid argument


I did some searching Ask Different, and found someone who had discovered that the "Invalid argument" error was related to metadata (https://apple.stackexchange.com/questions/130389/invalid-argument-using-ditto). So, I looked at metadata for these font files in both the source location and the destination location using xattr. What I found was that there was no metadata in the source location files, but for some reason ditto added two bits of metadata to the destination; one was a resource fork, and the other was something "com.apple.<something>". I can't see it right now because I used the 'less' filter in case it was going to be large. So that's weird.

I thought I would try moving the Library files via rsync instead of ditto, but when I ran the same rollback command a second time it hung completely - no disk activity, no CPU activity, but no command completion, either. I sampled it via Activity Monitor, and here is what it says it is doing (just top of the file):

Code: Select all
OS Version:      macOS 12.6.1 (21G217)
Report Version:  7
Analysis Tool:   /usr/bin/sample

Physical footprint:         1372K
Physical footprint (peak):  1372K
----

Call graph:
    2568 Thread_44146   DispatchQueue_1: com.apple.main-thread  (serial)
      2568 start  (in dyld) + 462  [0x11070c52e]
        2568 main  (in zfs) + 530  [0x102e5e992]
          2568 zfs_do_rollback  (in zfs) + 496  [0x102e60b50]
            2568 zfs_rollback  (in libzfs.4.dylib) + 480  [0x1030268e0]
              2568 lzc_rollback_to  (in libzfs_core.3.dylib) + 62  [0x102faa8be]
                2568 lzc_ioctl  (in libzfs_core.3.dylib) + 305  [0x102fa94d1]
                  2568 lzc_ioctl_fd  (in libzfs_core.3.dylib) + 71  [0x102fab617]
                    2568 ioctl  (in libsystem_kernel.dylib) + 150  [0x7ff807310eb3]
                      2568 __ioctl  (in libsystem_kernel.dylib) + 10  [0x7ff807310ec6]

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        __ioctl  (in libsystem_kernel.dylib)        2568

Binary Images:
       0x102e5c000 -        0x102e73fff +zfs (0) <14C4CD0E-698C-3E0E-9820-D32010165A72> /usr/local/zfs/bin/zfs
       0x102fa5000 -        0x102facff3 +libzfs_core.3.dylib (0) <7D7223F6-76CD-3FB5-A561-862254A3D3CA> /usr/local/zfs/lib/libzfs_core.3.dylib
       0x102fc5000 -        0x102fcefff +libintl.8.dylib (0) <FA921CC0-395B-3155-8259-EA61DE25C5D2> /usr/local/zfs/lib/libintl.8.dylib
       0x102fe3000 -        0x102feaffb +libuutil.3.dylib (0) <ECE725DF-44AC-36CD-9DCA-E8BE71638769> /usr/local/zfs/lib/libuutil.3.dylib
       0x103019000 -        0x103074fcf +libzfs.4.dylib (0)


Curious what you think, Mr. Lundman?

Edit: forgot to mention, but I did make sure that I had done a cd back to home directory of adminkurt before initiating the rollback that resulted in the hang.
Sharko
 
Posts: 228
Joined: Thu May 12, 2016 12:19 pm

Re: 2.1.6 hung on rollback, other ditto weirdness

Postby lundman » Sat Dec 03, 2022 4:49 pm

OK so it sounds like some issues with `ditto` possibly xattr related. When it comes to hangs, you need to run spindump to get the kernel side stacks, all yours show is it calling ioctl() to enter the kernel.

There is a new xattr tunable that changes some behavior with namespace as well
User avatar
lundman
 
Posts: 1334
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: 2.1.6 hung on rollback, other ditto weirdness

Postby Sharko » Sat Dec 03, 2022 10:49 pm

Darn, sorry I didn't get anything useful for the problem. I could try re-playing the same commands, but something tells me this might be a tricky intermittent problem.

Kurt
Sharko
 
Posts: 228
Joined: Thu May 12, 2016 12:19 pm

Re: 2.1.6 hung on rollback, other ditto weirdness

Postby Sharko » Wed Dec 07, 2022 8:57 pm

For those that stumble on this thread, and have problems with ditto, the solution was outlined in another thread:

viewtopic.php?f=26&t=3745

Briefly, you use the '--nopreserveHFSCompression' flag with ditto to make it behave.
Sharko
 
Posts: 228
Joined: Thu May 12, 2016 12:19 pm


Return to OpenZFS on OS X Development

Who is online

Users browsing this forum: No registered users and 1 guest