Problems with xattr=sa and on (but now different ones)

All your general support questions for OpenZFS on OS X.

Problems with xattr=sa and on (but now different ones)

Postby Arne » Thu Jan 13, 2022 10:20 am

I’m using zfs 1.9.4 on my Mac-Mini(2009).

As I mentioned in a former post viewtopic.php?f=26&t=3642 I had problems with xattr=on.
Cause of the vast amount of „on_delete_queue“ entries I decided to transform my datasets from xattr=on (dir) to xattr=sa.

Created new datasets with xattr=sa and copied the data with „rsync -XAa“.

Everything was fine, deleting, snapshotting, creating, copying from files without „on_delete_queue“ entries.

Out of curiosity, I looked at a pdf that I read with Skim.app. Skim is a nice tool to highlight text in a pdf and conveniently having all those parts together in a sidebar and I can easily print them or save them in a file. I wondered if skim saves the comments in the pdf or xattr?
It does so in xattr, so there are a lot of them (about 150) cause I highlighted so many parts.

Code: Select all
-rw-r--r--@  1 arne  staff    16745387 21 Jul  2017 Re-Awaken_the_Giant_Within_Highres.pdf
   net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-7         2048
   net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-61         2048
   net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-16         2048
   net_sourceforge_skim-app_96C1FFDB-1746-4974-A391-6CB3B3AC3012-5         2048
   net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-85         2048
   net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-24         2048
   net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-53         2048
   net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-101         2048



Nice. But…

This file lies on an external pool which has xattr=on.
I created a test dataset „zfs-extern/arne/testsa“ with an additional „zfs snapshot zfs-extern/arne/testsa@test“.
When I tried to copy the file per drag and drop with Finder onto the dataset with xattr=sa the copy process starts but stops half way and the file is not copied at all.
A „zfs diff“ shows a lot of „on_delete_queue“ entries.

Per cp or rsync, it will copy the xattr in „<xattr>“ directory although the dataset has xattr=sa.(?)

Code: Select all
M   /Users/arne/testsa/
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf/<xattrdir>/net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-82
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf/<xattrdir>
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf/<xattrdir>/net_sourceforge_skim-app_96C1FFDB-1746-4974-A391-6CB3B3AC3012-8
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf/<xattrdir>/net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-0
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf/<xattrdir>/net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-86
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf/<xattrdir>/net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-68
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf/<xattrdir>/net_sourceforge_skim-app_96C1FFDB-1746-4974-A391-6CB3B3AC3012-6
+   /Users/arne/testsa/Re-Awaken_the_Giant_Within_Highres.pdf/<xattrdir>/net_sourceforge_skim-app_926CF362-9D34-4BE0-8009-8F8183834F0D-15


Loading it in skim there are all highlights present.

Deleting per Finder or rm, there are a lot of „on_delete_queue“ entries. The „<xattr>/…“ I think. I guess that’s also the reason for those entries caused by the unsuccessful drag and drop copy with Finder.

I thought that 2.1.0 might handle it correctly.

So I changed to 2.1.0:
But it is worse than with 1.9.4.
With finder the file is copied, but not with all xattr, so that there are no notes visible in the pdf opened with skim.

Copying with rsync shows a lot of errors, stating on the attributes from skim: „… the current attribute … is not found (error 93)“.

I worried if there are errors in the xattr of the files in my production datasets which I copied from the old datasets with xattr=on(dir) to ones with xattr=sa before.

I tested it by copying a directory with rsync and it shocked me. rsync says that there are a lot of files where the attribute „com.apple.finderinfo 32“ is empty (0). And by looking into those files with „xattr -l“ they are indeed empty.

And shocking again that in the original datasets with xattr=on there are a lot of files with empty com.apple.finderinfo.

So I turned back to 1.9.4, again.

There are no files with empty „com.apple.finderinfo“.
Copying with rsync shows no problems with the files of the changed xattr property (in contrast to 2.1.0).
rsync and cp are copying the skim file, and the notes are again viewable in skim (as mentioned above).
And when I copy it with finder, the file is not copied. So it is as before with 1.9.4. Fine.

Strange though, when I copy the file first to my home directory (hfs+) per drag&drop it is copied and I than can copy it per drag&drop to the test directory with xattr=sa.

I tried changing some properties (com.apple.mimikhfs, dnodesize, com.apple.browse, ...) but nothing changes the problem with copying a skim file from a dataset with xattr=on to one with sa.

I almost loosed the faith in zfs, cause I was scared that all xattr where screwed with 2.1.0.
But it seems that only 2.1.0 has this severe problems.
With 1.9.4 I only have the problem with the one file from skim. I tested with other (normal, non Skim) files but those worked well. And I didn’t find one which has so many xattr like the one above.
And there is this strange writing of the xattr to directories instead of system-attributes although „xattr=sa“ is set. Or is it cause there are so many xattr and they are all 2048 byte big?

It now feels a little spooky to me using zfs. I make regular scrubs to be sure that there are no errors.
But there seems something wrong with this xattr property. I am not really certain anymore that zfs handles my files correctly, especially those with xattr.

Has anybody experienced this, too? Or am I the only one?

Should I avoid xattr at all and set xattr=off? But what about those programs that need ext. attributes?
My system: Mini 2009 (early) with El-Capitan 10.11.6
Arne
 
Posts: 30
Joined: Mon Oct 29, 2018 7:59 am

Re: Problems with xattr=sa and on (but now different ones)

Postby lundman » Thu Jan 13, 2022 3:42 pm

Hello!

So 1.9.4 uses the old xattr code, where we guessed as best as we could and I think we got it close. But xattr=sa was somewhat new to it and not as heavily tested.

With 2.x branch, ZOL had cleaned up xattrs, and made it its own nice file. So I threw out our code, and just theirs. That way we know it works the same as
OpenZFS - especially when it comes to mixing (changing the value of xattr=on and xattr=sa). Naturally, macOS does do a bunch of secret things, like that damned finderinfo xattr. So there
are some special case macOS code naturally.

So if you have found xattr issues with 2.x, I'd like to fix them. Especially if you know a sequence of commands when copying something from hfs/apfs and zfs does the wrong thing.
That would help me a great deal. There shouldn't be any difference between "on" and "sa" now, but pick one at a time, and let's get to fixing it.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Problems with xattr=sa and on (but now different ones)

Postby lundman » Mon Jan 24, 2022 5:16 pm

Quick peek at this, you can't really take a 1.9.4 pool and use with 2.1.0 - as the xattrs are handled differently. This does suck - I wonder if there is a way around it. 1.9.4 was just done differently.
If you start with a dataset on 2.1.0 can you find where it does things wrong?

Code: Select all
# ditto ~/Downloads/OpenZFS_on_OS_X_1.9.4.dmg /Volumes/BOOM/
# xattr -l  ~/Downloads/OpenZFS_on_OS_X_1.9.4.dmg > 1     
# xattr -l  /Volumes/BOOM/OpenZFS_on_OS_X_1.9.4.dmg > 2
(sort)
# diff --side-by-side 1 2
com.apple.FinderInfo:                  com.apple.FinderInfo:
00000000  00 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00  |.   00000000  00 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00  |.
00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |.   00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |.
00000020                     00000020
com.apple.diskimages.fsck:               com.apple.diskimages.fsck:
00000000  7E BC 3B 93 2E 6B 3E 8F 59 04 4A 22 FB E5 D2 6A  |~   00000000  7E BC 3B 93 2E 6B 3E 8F 59 04 4A 22 FB E5 D2 6A  |~
00000010  44 1D CD 58                                      |D   00000010  44 1D CD 58                                      |D
00000014                     00000014
com.apple.lastuseddate#PS:               com.apple.lastuseddate#PS:
00000000  69 90 6A 5F 00 00 00 00 DD 4B BF 18 00 00 00 00  |i   00000000  69 90 6A 5F 00 00 00 00 DD 4B BF 18 00 00 00 00  |i
00000010                     00000010
com.apple.metadata:_kMDItemUserTags:            com.apple.metadata:_kMDItemUserTags:
00000000  62 70 6C 69 73 74 30 30 A1 01 57 47 72 65 65 6E  |b   00000000  62 70 6C 69 73 74 30 30 A1 01 57 47 72 65 65 6E  |b
00000010  0A 32 08 0A 00 00 00 00 00 00 01 01 00 00 00 00  |.   00000010  0A 32 08 0A 00 00 00 00 00 00 01 01 00 00 00 00  |.
00000020  00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  |.   00000020  00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  |.
00000030  00 00 00 12                                      |.   00000030  00 00 00 12                                      |.
00000034                     00000034
com.apple.metadata:kMDItemDownloadedDate:         com.apple.metadata:kMDItemDownloadedDate:
00000000  62 70 6C 69 73 74 30 30 A1 01 33 41 C2 8D 63 F4  |b   00000000  62 70 6C 69 73 74 30 30 A1 01 33 41 C2 8D 63 F4  |b
00000010  B1 5C 18 08 0A 00 00 00 00 00 00 01 01 00 00 00  |.   00000010  B1 5C 18 08 0A 00 00 00 00 00 00 01 01 00 00 00  |.
00000020  00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00  |.   00000020  00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00  |.
00000030  00 00 00 00 13                                   |.   00000030  00 00 00 00 13                                   |.
00000035                     00000035
com.apple.metadata:kMDItemWhereFroms:            com.apple.metadata:kMDItemWhereFroms:
00000000  62 70 6C 69 73 74 30 30 A2 01 02 5F 10 40 68 74  |b   00000000  62 70 6C 69 73 74 30 30 A2 01 02 5F 10 40 68 74  |b
00000010  74 70 73 3A 2F 2F 6F 70 65 6E 7A 66 73 6F 6E 6F  |t   00000010  74 70 73 3A 2F 2F 6F 70 65 6E 7A 66 73 6F 6E 6F  |t
00000020  73 78 2E 6F 72 67 2F 77 2F 69 6D 61 67 65 73 2F  |s   00000020  73 78 2E 6F 72 67 2F 77 2F 69 6D 61 67 65 73 2F  |s
00000030  36 2F 36 66 2F 4F 70 65 6E 5A 46 53 5F 6F 6E 5F  |6   00000030  36 2F 36 66 2F 4F 70 65 6E 5A 46 53 5F 6F 6E 5F  |6
00000040  4F 53 5F 58 5F 31 2E 39 2E 34 2E 64 6D 67 5F 10  |O   00000040  4F 53 5F 58 5F 31 2E 39 2E 34 2E 64 6D 67 5F 10  |O
00000050  27 68 74 74 70 73 3A 2F 2F 6F 70 65 6E 7A 66 73  |'   00000050  27 68 74 74 70 73 3A 2F 2F 6F 70 65 6E 7A 66 73  |'
00000060  6F 6E 6F 73 78 2E 6F 72 67 2F 77 69 6B 69 2F 44  |o   00000060  6F 6E 6F 73 78 2E 6F 72 67 2F 77 69 6B 69 2F 44  |o
00000070  6F 77 6E 6C 6F 61 64 73 08 0B 4E 00 00 00 00 00  |o   00000070  6F 77 6E 6C 6F 61 64 73 08 0B 4E 00 00 00 00 00  |o
00000080  00 01 01 00 00 00 00 00 00 00 03 00 00 00 00 00  |.   00000080  00 01 01 00 00 00 00 00 00 00 03 00 00 00 00 00  |.
00000090  00 00 00 00 00 00 00 00 00 00 78                 |.   00000090  00 00 00 00 00 00 00 00 00 00 78                 |.
0000009b                     0000009b
com.apple.quarantine: 0083;5f6a9069;Safari;0BAF3E5E-EE74-43B9   com.apple.quarantine: 0083;5f6a9069;Safari;0BAF3E5E-EE74-43B9
                           >


And if you set com.apple.FinderInfo to all zeros, it is supposed to be the same as not having "com.apple.FinderInfo". Like
Code: Select all
# xattr -wx com.apple.FinderInfo "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00" /Volumes/BOOM/OpenZFS_on_OS_X_1.9.4.dmg
# xattr -px com.apple.FinderInfo /Volumes/BOOM/OpenZFS_on_OS_X_1.9.4.dmg
xattr: /Volumes/BOOM/OpenZFS_on_OS_X_1.9.4.dmg: No such xattr: com.apple.FinderInfo


2.1.0 handles that in the xattr setting, when it detects all zeros, it will remove the com.apple.FinderInfo. Unfortunately, 1.9.4 allows it to set
com.apple.FinderInfo to all zeros, and would skip it in list, and get, requests - but it still lived on disk. So a 1.9.4 pool in 2.1.0 would
have zerod com.apple.FinderInfo. We could add compat code to 2.1.0 to handle this case though.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Problems with xattr=sa and on (but now different ones)

Postby Arne » Wed Jan 26, 2022 8:21 am

Thanks that you are still working on it.

I decided to stay on 1.9.4.
But you gave me a way to compare xattrs which I couldn't find out by myself. So, I will take another look at it. But not this week.

Some days ago I tried a few things with 1.9.4 and 2.1.0, but got irritating results, and a quiet strange behaviour of 1.9.4, too. Compared to 2.1.0 it showed only a few EA, and out of a sudden (or cause I copied something? I don't know what it was) it shows a lot more EA, real EA that are not empty or zero.

And under 2.1.0 the way of copying the skim-pdf from a 1.9.4 pool with xattr=on(dir) to hfs to a dataset with xattr=sa didn't work (any more?). Under 1.9.4. it was a workaround to get the skim-pdf with all EA to a dataset with xattr=sa. And I swear that it was possible with 2.1.0 before, too. Don't know. I can't remember.

If I get to it I will test it again and maybe I will find something tangible, something that you can try by yourself.
My system: Mini 2009 (early) with El-Capitan 10.11.6
Arne
 
Posts: 30
Joined: Mon Oct 29, 2018 7:59 am

Re: Problems with xattr=sa and on (but now different ones)

Postby lundman » Wed Jan 26, 2022 8:11 pm

I guess in a way it would be good to know that

A) is there a xattr problem with 2.1.x and copying files from/to hfs/apfs (no 1.9.4 involved)
B) should we add patch to handle existing 1.9.4 pools better

where A) concerns me the most, and B) concerns anyone with existing pools :)
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Problems with xattr=sa and on (but now different ones)

Postby Arne » Sat Feb 12, 2022 11:37 am

First "B)"
Right now I am still on 1.9.4 and only testing 2.1.0 for "A)". When I decide to use 2.1.0 for production then I will create a whole new pool and copy all data from my external backup hfs partition to the new 2.1.0 pool.
If that is possible for others, too, then there is no need for patching 1.9.4-2.1.0.

Now "A)": 2.1.0 on its own
I installed 2.1.0, created a new pool and 2 datasets with xattr=sa and xattr=on.

Normal files:
I tested a whole directory. Even wrote a script to compare all xattr separately of any file in this directory with the ones where they were copied to (source, destination).
It seems that 2.1.0 works good in regard of xattr.

Tested copying a directory from -> to:
hfs -> sa
hfs -> on
sa -> hfs
on -> hfs
on -> sa
sa -> on

I used ditto, rsync -XAa, cp -a and the Finder to copy the directory.

All went well for normal files. All xattr were copied.
Except when copying files from zfs (sa or on) to hfs per Finder then existing "com.apple.quarantine" xattr on zfs will not be copied to hfs.

Now to a "not normal" file. The PDF that I worked on with the skim.app:
hfs -> on
on -> hfs
no problems, all xattr are copied and the file has all highlights

hfs -> sa
Code: Select all
ditto: /Volumes/zfswd/sa/skim/ditto/.BC.T_uZ9Vs4: Attribute not found

rsync: [receiver] rsync_xal_set: lsetxattr("/Volumes/zfswd/sa/skim/rsync/.Re-Awaken_the_Giant_Within_Highres.pdf.sWVeFC","net_sourceforge_skim-app_F25CD215-CC46-4012-8AA9-A55A599404FF-6") failed: Attribute not found (93)

cp: /Volumes/WD-Mac/Personal Development/Tony Robbins/Re-Awaken_the_Giant_Within_Highres.pdf: could not copy extended attributes to /Volumes/zfswd/sa/skim/cp/Re-Awaken_the_Giant_Within_Highres.pdf: Attribute not found

Finder: just a few xattr are copied and opening the file in skim.app there a no highlights.


sa -> hfs, on -> sa, sa -> on didn't work, too.

So there seems a problem with this PDF when working with xattr=sa.

I set dnodesize=auto but problem stays.

Maybe too many highlights?
- Did a binary search like approach (saving with only the half of the hightlights, then the half of the half that is not working, and so on). You know.

And it shows that it is indeed the amount of xattr of the skim-PDF (I could scale it down to one letter more or less that made it work or not.):

Cause skim saves the xattr different every time, this is only one example of the right amount of hightlights that worked.

Code: Select all
# ls -l@ Re-Awaken_the_Giant_Within_Highres-shortened.pdf

net_sourceforge_skim-app_notes#S        245
net_sourceforge_skim-app_1AF07861-479F-49B3-8E00-E7A4F17C2FD5-0#S  2048
net_sourceforge_skim-app_1AF07861-479F-49B3-8E00-E7A4F17C2FD5-1#S  2048
[…]                                    
net_sourceforge_skim-app_1AF07861-479F-49B3-8E00-E7A4F17C2FD5-30#S  958
com.apple.FinderInfo  32


I calculated ...

Count of xattr:
33 (32 skim + 1 FinderInfo)

Size of xattr:
skim-app: 245 + 30 * 2048 + 958 = 62643
FinderInfo: 32
Total: 62643 + 32 = 62675

Lenght of xattr (count of letters of the names): 33 + 10 * 66 + 21 * 67 + 21 = 2121
Size + Length: 62675 + 2121 = 64796

64796 reminds me of 2^16 = 65536.

I can save the shortened version of the PDF and copy it on the dataset with xattr=sa and copy from/to (hfs<->sa, on<->sa). And the highlights are present.

Strange: Saving with skim is different than copying the PDF to a dataset with xattr=sa. When I want to copy the file from hfs or from a dataset with xattr=on to one with xattr=sa the amount of highlights need be even a littel shorter (75% of the size of the saveable amount), although the calculated size+length is only a bit smaller (about 63000).

Enough testing and scripting and thinking and ...
Back to 1.9.4. Back to my life 3 days ago.

Hope you can get something out of it.
My system: Mini 2009 (early) with El-Capitan 10.11.6
Arne
 
Posts: 30
Joined: Mon Oct 29, 2018 7:59 am

Re: Problems with xattr=sa and on (but now different ones)

Postby Bingo » Thu Apr 06, 2023 6:04 am

Hey :)

Sorry, I realize this is an old post, but I'm having trouble with xattr, too, after returning to macOS after a couple of years.

I'm copying data off a dying zpool onto a new one, and on this dying pool I had previously moved a massive amount of data (from a third dying pool :lol: ) with a script I made that hashed the files while copying using md5sum and crc32, and then stored the checksums in "crc" and "md5" xattr properties.

Odd thing is that I cannot find a single file with any xattrs on it. I know for absolute certain that these files were augmented on this pool with xattr at some point to add these checksums. Heck, I even made a script to extract them for a directory tree and create .sfv or .md5 files on the basis of them, and I've tested it and used it.

I don't remember which xattr binary I used to create these, or if it matters, but I think it could have been either native Apple, Homebrew, or MacPorts. I remember discovering that the native Apple `mv` binary would preserve these xattrs while the GNU one from coreutils would not. So I'm confused about what is causing my xattrs to not show.

Could it be this change in ZFS for macOS? I'm currently zfs send|zfs receiving a filesystem, so I can't immediately try downgrading, and I'm wondering if I may have upgraded the pool out of habit so that I cannot even downgrade? Which version should I try in this case? Or is there a zfs xattr property value I could try? I have xattr=on across the board, and don't remember having modified anything :)

Thanks!
Bingo
 
Posts: 16
Joined: Thu Mar 04, 2021 11:18 pm


Return to General Help

Who is online

Users browsing this forum: beska, Google [Bot] and 37 guests