Duplicated files become hidden in Finder

All your general support questions for OpenZFS on OS X.

Re: Duplicated files become hidden in Finder

Postby evereasy » Thu Apr 08, 2021 7:04 am

Could it be a big endian vs little endian problem?
evereasy
 
Posts: 30
Joined: Sat Feb 27, 2021 6:38 am

Re: Duplicated files become hidden in Finder

Postby evereasy » Thu Apr 08, 2021 9:09 am

Used the following shell script to perform tests. All the tests on ZFS volumes are failed - no extended attributes get updated. The output files are attached.

Code: Select all
:
data=`cat << END_OF_LIST
x000000000000000000000000000000000000000000000000000000000000000
0x00000000000000000000000000000000000000000000000000000000000000
00x0000000000000000000000000000000000000000000000000000000000000
000x000000000000000000000000000000000000000000000000000000000000
0000x00000000000000000000000000000000000000000000000000000000000
00000x0000000000000000000000000000000000000000000000000000000000
000000x000000000000000000000000000000000000000000000000000000000
0000000x00000000000000000000000000000000000000000000000000000000
00000000x0000000000000000000000000000000000000000000000000000000
000000000x000000000000000000000000000000000000000000000000000000
0000000000x00000000000000000000000000000000000000000000000000000
00000000000x0000000000000000000000000000000000000000000000000000
000000000000x000000000000000000000000000000000000000000000000000
0000000000000x00000000000000000000000000000000000000000000000000
00000000000000x0000000000000000000000000000000000000000000000000
000000000000000x000000000000000000000000000000000000000000000000
0000000000000000x00000000000000000000000000000000000000000000000
00000000000000000x0000000000000000000000000000000000000000000000
000000000000000000x000000000000000000000000000000000000000000000
0000000000000000000x00000000000000000000000000000000000000000000
00000000000000000000x0000000000000000000000000000000000000000000
000000000000000000000x000000000000000000000000000000000000000000
0000000000000000000000x00000000000000000000000000000000000000000
00000000000000000000000x0000000000000000000000000000000000000000
000000000000000000000000x000000000000000000000000000000000000000
0000000000000000000000000x00000000000000000000000000000000000000
00000000000000000000000000x0000000000000000000000000000000000000
000000000000000000000000000x000000000000000000000000000000000000
0000000000000000000000000000x00000000000000000000000000000000000
00000000000000000000000000000x0000000000000000000000000000000000
000000000000000000000000000000x000000000000000000000000000000000
0000000000000000000000000000000x00000000000000000000000000000000
00000000000000000000000000000000x0000000000000000000000000000000
000000000000000000000000000000000x000000000000000000000000000000
0000000000000000000000000000000000x00000000000000000000000000000
00000000000000000000000000000000000x0000000000000000000000000000
000000000000000000000000000000000000x000000000000000000000000000
0000000000000000000000000000000000000x00000000000000000000000000
00000000000000000000000000000000000000x0000000000000000000000000
000000000000000000000000000000000000000x000000000000000000000000
0000000000000000000000000000000000000000x00000000000000000000000
00000000000000000000000000000000000000000x0000000000000000000000
000000000000000000000000000000000000000000x000000000000000000000
0000000000000000000000000000000000000000000x00000000000000000000
00000000000000000000000000000000000000000000x0000000000000000000
000000000000000000000000000000000000000000000x000000000000000000
0000000000000000000000000000000000000000000000x00000000000000000
00000000000000000000000000000000000000000000000x0000000000000000
000000000000000000000000000000000000000000000000x000000000000000
0000000000000000000000000000000000000000000000000x00000000000000
00000000000000000000000000000000000000000000000000x0000000000000
000000000000000000000000000000000000000000000000000x000000000000
0000000000000000000000000000000000000000000000000000x00000000000
00000000000000000000000000000000000000000000000000000x0000000000
000000000000000000000000000000000000000000000000000000x000000000
0000000000000000000000000000000000000000000000000000000x00000000
00000000000000000000000000000000000000000000000000000000x0000000
000000000000000000000000000000000000000000000000000000000x000000
0000000000000000000000000000000000000000000000000000000000x00000
00000000000000000000000000000000000000000000000000000000000x0000
000000000000000000000000000000000000000000000000000000000000x000
0000000000000000000000000000000000000000000000000000000000000x00
00000000000000000000000000000000000000000000000000000000000000x0
000000000000000000000000000000000000000000000000000000000000000x
END_OF_LIST`

exec 3> /tmp/APFS.log
exec 4> /tmp/ZFS.log

for i in 1 2 3 4 5 6 7 8 9 A B C D E F
do
   echo "$data" | while read x
   do
      y=`echo $x | sed s/x/$i/g`
      (
      echo "================================================================================"
      echo $y
      echo "================================================================================"
      set -x
      xattr -wx com.apple.FinderInfo $y $a/Presentation1.pptx
      xattr -l $a/Presentation1.pptx
      ) >&3 2>&1
      (
      echo "================================================================================"
      echo $y
      echo "================================================================================"
      set -x
      xattr -wx com.apple.FinderInfo $y $z/Presentation1.pptx
      xattr -l $z/Presentation1.pptx
      ) >&4 2>&1
   done
done
Attachments
ZFS.log
xattr tests performed on a ZFS volume
(358.13 KiB) Downloaded 259 times
APFS.log
xattr tests performed on a APFS volume
(533.44 KiB) Downloaded 242 times
evereasy
 
Posts: 30
Joined: Sat Feb 27, 2021 6:38 am

Re: Duplicated files become hidden in Finder

Postby evereasy » Fri Apr 09, 2021 7:45 am

Performed some additional tests using finderinfo-rust. Because this is the first time that I use rust and cargo, what I can do now is just to run and get the results.

lib.rs - source
https://docs.rs/finder_info/0.2.2/src/f ... ml#134-199
Code: Select all
//! Structs and functions to manipulate MacOS/HFS+ structs, e.g. com.apple.FinderInfo.
//!
//! Note that HFS+ is big-endian, and so all serialization/deserialization has to be byteswapped
//! appropriately. APFS isn't big-endian, but it pretends pretty hard internally (and does so
//! here).

0. Prepare the tool - finderinfo

0.1 Modify Cargo.toml to enable feature "xattr" <= This is what I missed in previous attempt
Code: Select all
[features]
default = ["xattr"]

0.2 Use cargo to build and install ~/.cargo/bin/finderinfo
Code: Select all
$ cargo build
$ cargo install --path .

1. Perform tests on APFS volume

1.1 Clear extended attributes
Code: Select all
$ xattr -c $a/Presentation1.pptx
$ xattr -l $a/Presentation1.pptx
$ ./finderinfo read $a/Presentation1.pptx
Attempting to read FinderInfo from "$a/Presentation1.pptx"
Err(
    Os {
        code: 93,
        kind: Other,
        message: "Attribute not found",
    },
)

1.2 Use Finder add "Red" tag

1.3 Examine results
Code: Select all
$ xattr -l $a/Presentation1.pptx
com.apple.FinderInfo:
00000000  00 00 00 00 00 00 00 00 00 0C 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000020
com.apple.metadata:_kMDItemUserTags:
00000000  62 70 6C 69 73 74 30 30 A1 01 55 52 65 64 0A 36  |bplist00..URed.6|
00000010  08 0A 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
00000020  00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 10                                            |..|
00000032
$ ./finderinfo read $a/Presentation1.pptx
Attempting to read FinderInfo from "$a/Presentation1.pptx"
Ok(
    File(
        FinderInfoFile {
            file_info: FileInfo {
                fileType: Ok("\u{0}\u{0}\u{0}\u{0}"),
                fileCreator: Ok("\u{0}\u{0}\u{0}\u{0}"),
                finderFlags: FinderFlags {
                    raw: 12,
                    flags: [
                        "Red",
                    ],
                },
                location: Point {
                    v: 0,
                    h: 0,
                },
                reservedField: 0,
            },
            extended_file_info: ExtendedFileInfo {
                extendedFinderFlags: ExtendedFinderFlags {
                    raw: 0,
                    flags: [],
                },
                putAwayFolderID: 0,
            },
        },
    ),
)

2. Perform tests on ZFS volume

2.1 Clear extended attributes
Code: Select all
$ xattr -c $z/Presentation1.pptx
$ xattr -l $z/Presentation1.pptx
$ ./finderinfo read $z/Presentation1.pptx
Attempting to read FinderInfo from "$z/Presentation1.pptx"
Err(
    Os {
        code: 93,
        kind: Other,
        message: "Attribute not found",
    },
)


2.2 Use Finder add "Red" tag

2.3 Examine results
Code: Select all
$ ./finderinfo read $z/Presentation1.pptx
Attempting to read FinderInfo from "$z/Presentation1.pptx"
Ok(
    File(
        FinderInfoFile {
            file_info: FileInfo {
                fileType: Ok("\u{0}\u{0}\u{0}\u{0}"),
                fileCreator: Err(FromUtf8Error { bytes: [134, 255, 255, 255], error: Utf8Error { valid_up_to: 0, error_len: Some(1) } }),
                finderFlags: FinderFlags {
                    raw: 14848,
                    flags: [
                        "kIsStationery",
                        "kNameLocked",
                        "kHasBundle",
                    ],
                },
                location: Point {
                    v: 0,
                    h: 0,
                },
                reservedField: 0,
            },
            extended_file_info: ExtendedFileInfo {
                reserved1: [
                    0,
                    0,
                    0,
                    0,
                ],
                extendedFinderFlags: ExtendedFinderFlags {
                    raw: 32823,
                    flags: [
                        "kExtendedFlagsAreInvalid",
                        "kExtendedFlagHasCustomBadge",
                    ],
                },
                reserved2: -1216,
                putAwayFolderID: 0,
            },
        },
    ),
)
$ xattr -l $z/Presentation1.pptx
com.apple.metadata:_kMDItemUserTags:
00000000  62 70 6C 69 73 74 30 30 A1 01 55 52 65 64 0A 36  |bplist00..URed.6|
00000010  08 0A 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
00000020  00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 10                                            |..|
00000032
com.apple.FinderInfo:
00000000  00 00 00 00 86 FF FF FF 3A 00 00 00 00 00 00 00  |........:.......|
00000010  00 00 00 00 00 00 00 00 80 37 FB 40 00 00 00 00  |.........7.@....|
00000020
evereasy
 
Posts: 30
Joined: Sat Feb 27, 2021 6:38 am

Re: Duplicated files become hidden in Finder

Postby evereasy » Fri Apr 09, 2021 6:18 pm

Let me summarize what I have so far:

1. Tools or software that will have this problem:
1.1 Finder - duplicate, or change change color tag
1.2 xattred - can edit com.apple.FinderInfo using ex. 00000000 00000000 000C0000 00000000 00000000 00000000 00000000 00000000 and reproduce the problem. However, it doesn't have command mode. (xattred https://eclecticlight.co/downloads/)
1.3 xattr - can be used to set com.apple.FinderInfo. It can not reproduce the same problem as xattred or Finder. However, the test results on APFS and ZFS are different. It might be worth to investigate it ...

2. Use xattr to fill a byte and observe the results. Hope this can provide some useful information.
2.1 on APFS, works fine, as in APFS-2021-04-10-091100.log
2.2 on ZFS, have problems, as in ZFS-2021-04-10-091100.log
Code: Select all
:
data=`cat << END_OF_LIST
000000000000000000xy00000000000000000000000000000000000000000000
END_OF_LIST`

s=`date +%Y-%m-%d-%H%M%S`
exec 3> $HOME/tmp/APFS-$s.log
exec 4> $HOME/tmp/ZFS-$s.log

for i in 0 1 2 3 4 5 6 7 8 9 A B C D E F
do
   for j in 0 1 2 3 4 5 6 7 8 9 A B C D E F
   do
        echo "$data" | while read p
       do
          p=`echo $p | sed s/x/$i/g`
          p=`echo $p | sed s/y/$j/g`
          (
          echo "================================================================================"
          echo $p
          echo "================================================================================"
          set -x
          xattr -wx com.apple.FinderInfo $p $a/Presentation1.pptx
          xattr -vpx com.apple.FinderInfo $a/Presentation1.pptx
          finderinfo read $a/Presentation1.pptx
          ) >&3 2>&1
          (
          echo "================================================================================"
          echo $p
          echo "================================================================================"
          set -x
          xattr -wx com.apple.FinderInfo $p $z/Presentation1.pptx
          xattr -vpx com.apple.FinderInfo $z/Presentation1.pptx
          finderinfo read $z/Presentation1.pptx
          ) >&4 2>&1
       done
      done
  done
Attachments
APFS-2021-04-10-091100.log
Test Results on APFS
(362.37 KiB) Downloaded 253 times
ZFS-2021-04-10-091100.log
Test results on ZFS
(174 KiB) Downloaded 234 times
evereasy
 
Posts: 30
Joined: Sat Feb 27, 2021 6:38 am

Re: Duplicated files become hidden in Finder

Postby lundman » Fri Apr 09, 2021 8:07 pm

So, have you found any place where we differ? Summarise more! :)
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Duplicated files become hidden in Finder

Postby evereasy » Fri Apr 09, 2021 9:27 pm

Is it possible for you to dump what ZFS received when running the following xattr command on ZFS volume? $z is the path to a ZFS volume.
xattr -ws com.apple.FinderInfo 0000000000000000000200000000000000000000000000000000000000000000 $z/Presentation1.pptx

Here is one of the example -

1. on APFS volume, xattr can set the 3 bits color tag correctly
xattr -ws com.apple.FinderInfo 0000000000000000000200000000000000000000000000000000000000000000 $a/Presentation1.pptx
Code: Select all
================================================================================
0000000000000000000200000000000000000000000000000000000000000000
================================================================================
++ xattr -wx com.apple.FinderInfo 0000000000000000000200000000000000000000000000000000000000000000 $a/Presentation1.pptx
++ xattr -vpx com.apple.FinderInfo $a/Presentation1.pptx
$a/Presentation1.pptx:
00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
++ finderinfo read $a/Presentation1.pptx
Attempting to read FinderInfo from "$a/Presentation1.pptx"
Ok(
    File(
        FinderInfoFile {
            file_info: FileInfo {
                fileType: Ok("\u{0}\u{0}\u{0}\u{0}"),
                fileCreator: Ok("\u{0}\u{0}\u{0}\u{0}"),
                finderFlags: FinderFlags {
                    raw: 2,
                    flags: [
                        "Gray",
                    ],
                },
                location: Point {
                    v: 0,
                    h: 0,
                },
                reservedField: 0,
            },
            extended_file_info: ExtendedFileInfo {
                extendedFinderFlags: ExtendedFinderFlags {
                    raw: 0,
                    flags: [],
                },
                putAwayFolderID: 0,
            },
        },
    ),
)

2. on ZFS volume, xattr can not set the 3 bits color tag correctly
xattr -ws com.apple.FinderInfo 0000000000000000000200000000000000000000000000000000000000000000 $z/Presentation1.pptx
Code: Select all
================================================================================
0000000000000000000200000000000000000000000000000000000000000000
================================================================================
++ xattr -wx com.apple.FinderInfo 0000000000000000000200000000000000000000000000000000000000000000 $z/Presentation1.pptx
++ xattr -vpx com.apple.FinderInfo $z/Presentation1.pptx
xattr: $z/Presentation1.pptx: No such xattr: com.apple.FinderInfo
$z/Presentation1.pptx:
++ finderinfo read $z/Presentation1.pptx
Attempting to read FinderInfo from "$z/Presentation1.pptx"
Err(
    Os {
        code: 93,
        kind: Other,
        message: "Attribute not found",
    },
)
evereasy
 
Posts: 30
Joined: Sat Feb 27, 2021 6:38 am

Re: Duplicated files become hidden in Finder

Postby lundman » Sat Apr 10, 2021 2:02 am

# xattr -ws com.apple.FinderInfo 0000000000000000000200000000000000000000000000000000000000000000 /Volumes/BOOM/z

0xffffff80249c9f00 'com.apple.FinderInfo' (enabled: 1) resid 64 0x0
2021-04-10 03:00:15.334 Df kernel.development[0:16a2] (zfs) <zfs`zfs_vnop_setxattr (zfs_vnops_osx.c:0)> xottr given:
2021-04-10 03:00:15.334 Df kernel.development[0:16a2] (zfs) <zfs`zfs_vnop_setxattr (zfs_vnops_osx.c:3746)> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Given all zeros by the looks of it. This is how it is given to ZFS
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Duplicated files become hidden in Finder

Postby lundman » Sat Apr 10, 2021 2:21 am

hmm no I must be doing something quite wrong:

xattr -ws com.apple.FinderInfo
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
/Volumes/BOOM/z

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

checking debug code
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Duplicated files become hidden in Finder

Postby lundman » Sat Apr 10, 2021 2:31 am

Ah ok, we must set UIO direction now, and the xattr code didn't. Also, xattr needs -x for setting hex, rather than literal "0" (0x30).

Next build will:

xattr -xws com.apple.FinderInfo ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff /Volumes/BOOM/z
(zfs_vnops_osx.c:0)> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
xattr after procesing
(zfs_vnops_osx.c:3752)> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 00 00


xattr -xws com.apple.FinderInfo 0000000000000000000200000000000000000000000000000000000000000000 /Volumes/BOOM/z
(zfs_vnops_osx.c:0)> 00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Duplicated files become hidden in Finder

Postby evereasy » Sat Apr 10, 2021 2:38 am

Thank you so much for performing additional test.

Sorry, it was a typo. Yes. It should be -x as in the script.

I will test the new build once it is available.
evereasy
 
Posts: 30
Joined: Sat Feb 27, 2021 6:38 am

PreviousNext

Return to General Help

Who is online

Users browsing this forum: Google [Bot] and 25 guests