OpenZFS on OS X

by **roemer** » Fri Mar 21, 2014 11:15 pm

I have run some copy tests between a jHFS+ and an OpenZFS volume to check how well OSX's meta-data is support.
The result was a bit puzzling: ZFS shows quite some problems to correctly handle extended attributes, bsd flags and access control lists.
The main issue (for me) is that there are apparently com.apple.FinderInfo extended attributes are added to each file that is copied onto the ZFS volume!
Why is that?
And why does the ACL test fail too?

*EDIT 23 March*
I have further experimented with this and it seems that the additional FinderInfo attribute, which I see on the zfs volume, is only transient:
Once I copy the files back from zfs to another HFS+ volume, only those FinderInfos are copied which were already there before.
So is there a mechanism in OpenZFS which returns a 'default' FinderInfo object to certain API calls even if there is no FinderInfo stored with the underlying file?
The ACL and bsd flags are quite mungled though once one goes through the HFS+ -> ZFS -> HFS+ copy chain...

The details are as follows:

Test Environment
- Apple iMac8,1 under OS X 10.9.2 (13C64)
- external 2.5'' Seagate HDD via Firewire-400 connection
- OpenZFS for OSX 1.2
- ChronoSync 4.4.4
- Backup Bouncer 0.2.0
(http://www.n8gray.org/code/backup-bouncer/)
(http://www.n8gray.org/blog/2009/09/15/b ... uncer-020/)
all software run with admin / sudo rights

ZFS volume created on external HDD as follows (full flag list at end of post):

Code: Select all: sudo zpool create -f -o ashift=12 -O atime=off -O casesensitivity=insensitive -O normalization=formD ztank /dev/disk1s3 sudo chown -R admin:staff /ztank sudo mdutil -i off /ztank

Copy-test between a journaled HFS+ volume (diskimage) using ChronoSync to the external harddisk which was formed as shown above with zfs resulted into the following verification result (note the FAILs on extended attributes, bsd flags and ACL tests):

Code: Select all: admin$ sudo ./bbouncer verify -d /Volumes/Src/ /ztank/ Password: Verifying: basic-permissions ... ok (Critical) Verifying: timestamps ... ok (Critical) Verifying: symlinks ... ok (Critical) Verifying: symlink-ownership ... ok Verifying: hardlinks ... ok (Important) Verifying: resource-forks ... Sub-test: on files ... ok (Critical) Sub-test: on hardlinked files ... ok (Important) Verifying: finder-flags ... ok (Critical) Verifying: finder-locks ... ok Verifying: creation-date ... FAIL [due to an additional com.apple.FinderInfo] Verifying: bsd-flags ... FAIL Verifying: extended-attrs ... Sub-test: on files ... FAIL (Important) Sub-test: on directories ... FAIL (Important) Sub-test: on symlinks ... FAIL Verifying: access-control-lists ... Sub-test: on files ... FAIL (Important) Sub-test: on dirs ... FAIL (Important) Verifying: fifo ... FAIL Verifying: devices ... FAIL Verifying: combo-tests ... Sub-test: xattrs + rsrc forks ... FAIL Sub-test: lots of metadata ... FAIL

Cross-check with very same software, but using a jHFS+ disk image as destination. Verification result looks as follows:

Code: Select all: admin$ sudo ./bbouncer verify -d /Volumes/Src/ /Volumes/Dst/ Verifying: basic-permissions ... ok (Critical) Verifying: timestamps ... ok (Critical) Verifying: symlinks ... ok (Critical) Verifying: symlink-ownership ... ok Verifying: hardlinks ... ok (Important) Verifying: resource-forks ... Sub-test: on files ... ok (Critical) Sub-test: on hardlinked files ... ok (Important) Verifying: finder-flags ... ok (Critical) Verifying: finder-locks ... ok Verifying: creation-date ... ok Verifying: bsd-flags ... ok Verifying: extended-attrs ... Sub-test: on files ... ok (Important) Sub-test: on directories ... ok (Important) Sub-test: on symlinks ... FAIL [due to an additional com.apple.FinderInfo] Verifying: access-control-lists ... Sub-test: on files ... ok (Important) Sub-test: on dirs ... ok (Important) Verifying: fifo ... FAIL Verifying: devices ... FAIL Verifying: combo-tests ... Sub-test: xattrs + rsrc forks ... ok Sub-test: lots of metadata ... ok

Here are the details of the failed copy tests between HFS+ volume and the ZFS tank:

Code: Select all: admin$ ls -al@ /Volumes/Src/50-creation-date/ total 0 drwxr-xr-x 3 root staff 102 22 Mar 17:23 . drwxr-xr-x 22 admin staff 816 22 Mar 17:23 .. -rw-r--r-- 1 root staff 0 10 Oct 2000 creation-date-test admin$ ls -al@ /ztank/50-creation-date/ total 35 drwxr-xr-x@ 2 root staff 3 22 Mar 17:23 . com.apple.FinderInfo 32 drwxr-xr-x@ 20 admin staff 23 22 Mar 17:31 .. com.apple.FinderInfo 32 -rw-r--r--@ 1 root staff 0 10 Oct 2000 creation-date-test com.apple.FinderInfo 32 admin$ ls -al@ /Volumes/Src/60-bsd-flags/ total 0 drwxr-xr-x 4 root staff 136 22 Mar 17:23 . drwxr-xr-x 22 admin staff 816 22 Mar 17:23 .. drwxr-xr-x 2 root staff 68 22 Mar 17:23 dir-with-flags -rw-r--r-- 1 root staff 0 22 Mar 17:23 file-with-flags admin$ /usr/bin/stat -f '%f' /Volumes/Src/60-bsd-flags/dir-with-flags 15 admin$ /usr/bin/stat -f '%f' /Volumes/Src/60-bsd-flags/file-with-flags 15 admin$ ls -al@ /ztank/60-bsd-flags/ total 52 drwxr-xr-x@ 3 root staff 4 22 Mar 17:23 . com.apple.FinderInfo 32 drwxr-xr-x@ 20 admin staff 23 22 Mar 17:31 .. com.apple.FinderInfo 32 drwxr-xr-x@ 2 admin staff 2 22 Mar 17:23 dir-with-flags com.apple.FinderInfo 32 -rw-r--r--@ 1 root staff 0 22 Mar 17:23 file-with-flags com.apple.FinderInfo 32 admin$ /usr/bin/stat -f '%f' /ztank/60-bsd-flags/dir-with-flags 13 admin$ /usr/bin/stat -f '%f' /ztank/60-bsd-flags/file-with-flags 15 admin$ ls -al@ /Volumes/Src/70-extended-attrs/ total 8 drwxr-xr-x 5 root staff 170 22 Mar 17:23 . drwxr-xr-x 22 admin staff 816 22 Mar 17:23 .. drwxr-xr-x@ 2 root staff 68 22 Mar 17:23 dir-with-xattrs mamma.mia 28 lrwxr-xr-x@ 1 root staff 12 22 Mar 17:23 symlink-with-xattrs -> ./xattr-test good.grief 39 -rw-r--r--@ 1 root staff 0 22 Mar 17:23 xattr-test message 12 this.that 10 admin$ ls -al@ /ztank/70-extended-attrs/ total 53 drwxr-xr-x@ 3 root staff 5 22 Mar 17:23 . com.apple.FinderInfo 32 drwxr-xr-x@ 20 admin staff 23 22 Mar 17:31 .. com.apple.FinderInfo 32 drwxr-xr-x@ 2 root staff 2 22 Mar 17:23 dir-with-xattrs mamma.mia 28 com.apple.FinderInfo 32 lrwxr-xr-x@ 1 root staff 12 22 Mar 17:23 symlink-with-xattrs -> ./xattr-test com.apple.FinderInfo 32 -rw-r--r--@ 1 root staff 0 22 Mar 17:23 xattr-test message 12 com.apple.FinderInfo 32 this.that 10 admin$ ls -al@ /Volumes/Src/80-access-control-lists/ total 0 drwxr-xr-x 4 root staff 136 22 Mar 17:23 . drwxr-xr-x 22 admin staff 816 22 Mar 17:23 .. -rw-r--r--+ 1 root staff 0 22 Mar 17:23 acl-test drwxr-xr-x+ 2 root staff 68 22 Mar 17:23 acl-test-dir admin$ ls -Plde /Volumes/Src/80-access-control-lists/* -rw-r--r--+ 1 root staff 0 22 Mar 17:23 /Volumes/Src/80-access-control-lists/acl-test 0: user:nobody allow read 1: user:_www allow read drwxr-xr-x+ 2 root staff 68 22 Mar 17:23 /Volumes/Src/80-access-control-lists/acl-test-dir 0: user:nobody allow list 1: user:_www allow list admin$ ls -al@ /ztank/80-access-control-lists/ total 52 drwxr-xr-x@ 3 root staff 4 22 Mar 17:23 . com.apple.FinderInfo 32 drwxr-xr-x@ 20 admin staff 23 22 Mar 17:31 .. com.apple.FinderInfo 32 -rw-r--r--@ 1 root staff 0 22 Mar 17:23 acl-test com.apple.FinderInfo 32 d---------@ 2 root staff 2 22 Mar 17:23 acl-test-dir com.apple.FinderInfo 32 admin$ ls -Plde /ztank/80-access-control-lists/* -rw-r--r--@ 1 root staff 0 22 Mar 17:23 /ztank/80-access-control-lists/acl-test 0: group:owner allow read,write,append,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity,chown 1: group:group allow read,readattr,readextattr,readsecurity 2: group:everyone allow read,readattr,readextattr,readsecurity d---------@ 2 root staff 2 22 Mar 17:23 /ztank/80-access-control-lists/acl-test-dir 0: group:owner allow readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity,chown 1: group:group allow readattr,readextattr,readsecurity 2: group:everyone allow readattr,readextattr,readsecurity admin$ ls -al@ /Volumes/Src/99-combo-tests/ total 24 drwxr-xr-x 4 root staff 136 22 Mar 17:23 . drwxr-xr-x 22 admin staff 816 22 Mar 17:23 .. -rw----r--@ 1 _www _www 13 22 Mar 17:23 many-metadata com.apple.FinderInfo 32 com.apple.ResourceFork 23 big.time 25 big.time2 25 big.time3 25 -rw-r--r--@ 1 root staff 0 22 Mar 17:23 xattr-with-rfork com.apple.ResourceFork 24 fairy.tale 55 admin$ ls -al@ /ztank/99-combo-tests/ total 44 drwxr-xr-x@ 2 root staff 4 22 Mar 17:23 . com.apple.FinderInfo 32 drwxr-xr-x@ 20 admin staff 23 22 Mar 17:31 .. com.apple.FinderInfo 32 -rw----r--@ 1 _www _www 13 22 Mar 17:23 many-metadata big.time2 25 big.time 25 com.apple.ResourceFork 23 com.apple.FinderInfo 32 big.time3 25 -rw-r--r--@ 1 root staff 0 22 Mar 17:23 xattr-with-rfork fairy.tale 55 com.apple.ResourceFork 24 com.apple.FinderInfo 32

And finally, the zfs filesystem in question has the following flags set:

Code: Select all: bash-3.2$ sudo zfs get all /ztank NAME PROPERTY VALUE SOURCE ztank type filesystem - ztank creation Sat Mar 22 16:35 2014 - ztank used 1.59G - ztank available 137G - ztank referenced 1.59G - ztank compressratio 1.00x - ztank mounted yes - ztank quota none default ztank reservation none default ztank recordsize 128K default ztank mountpoint /ztank default ztank sharenfs off default ztank checksum on default ztank compression off default ztank atime off local ztank devices on default ztank exec on default ztank setuid on default ztank readonly off default ztank zoned off default ztank snapdir hidden default ztank aclmode discard default ztank aclinherit restricted default ztank canmount on default ztank xattr on default ztank copies 1 default ztank version 5 - ztank utf8only on - ztank normalization formD - ztank casesensitivity insensitive - ztank vscan off default ztank nbmand off default ztank sharesmb off default ztank refquota none default ztank refreservation none default ztank primarycache all default ztank secondarycache all default ztank usedbysnapshots 0 - ztank usedbydataset 1.59G - ztank usedbychildren 420K - ztank usedbyrefreservation 0 - ztank logbias latency default ztank dedup off default ztank mlslabel none default ztank sync standard default ztank refcompressratio 1.00x - ztank written 1.59G - ztank logicalused 1.58G - ztank logicalreferenced 1.58G - ztank snapdev hidden default ztank com.apple.browse on default ztank com.apple.ignoreowner off default

by **ilovezfs** » Sun Mar 23, 2014 2:13 pm

ACLs function differently on ZFS vs. HFS+ and you should not attempt to preserve them.

You should use Homebrew's version of rsync if you're trying to round-trip data HFS+ -> ZFS -> HFS+.

Apple's rsync command lumps ACLs and extended attributes together (the -E option), so it is unsuitable for this purpose.

Homebrew's version of rsync splits these apart (-A for ACLs and -X for extended attributes). You should only use the -X.

I use
alias sur='sudo /usr/local/bin/rsync -aHXvhP --itemize-changes --fileflags'

And verify with
alias surfake='sudo /usr/local/bin/rsync -HXvhP --itemize-changes -rlD -nc'

by **glessard** » Fri Mar 28, 2014 8:19 am

ACLs function differently on ZFS vs. HFS+ and you should not attempt to preserve them.

Is there a cogent explanation of this other than that it's hard?

As it is, attempting to use the Finder edit permissions for a file that lives on OpenZFS is pretty much doomed to failure, due to the redundant nature of the so-called "trivial ACLs". They're not trivial: they get in the way. I'm not sure they do anything good anywhere outside of OpenSolaris, and I doubt many people are using OpenZFS on OSX or Linux in order to get a taste of OpenSolaris. We want to use ZFS for the promise of data integrity! However when copying files between OpenZFS and HFS+ volumes mangles the metadata (including ACLs), the confidence in data integrity is badly diminished.

Cheers,
Guillaume Lessard

by **roemer** » Sat Mar 29, 2014 12:05 am

ilovezfs wrote:ACLs function differently on ZFS vs. HFS+ and you should not attempt to preserve them.

Ok, fine. But there are at least two issues here:
Firstly, I indeed can use rsync to copy files without ACL properly between ZFS and HFS+, as well as my beloved ChronoSync tool allows to ignore ACLs.
But the moment I copy a file simply via Drag&Drop using Finder from ZFS, these 'default' ACLs pop up which weren't there before.
I consider this a bug.

Secondly, there are files where I simply do not have control over ACLs.
Try to copy a application bundle, say Xcode or Server from Apple, directly to ZFS. ACLs mangled. Hmmm.
Not nice. Workaround is to create disk images (which are internally HFS+ formatted) - but that's not really user-friendly I would say.

ilovezfs wrote:I use
alias sur='sudo /usr/local/bin/rsync -aHXvhP --itemize-changes --fileflags'

And verify with
alias surfake='sudo /usr/local/bin/rsync -HXvhP --itemize-changes -rlD -nc'

Thanks, I will keep this in mind. Good tip -> worth for the wiki?
But the Finder issue worries me. As Guillaume in the previous post has pointed out correctly: ZFS is all about data integrity and the file meta-data is on the Mac as important as normal file contents in many cases.

by **ilovezfs** » Sat Mar 29, 2014 12:53 am

This has been discussed ad nauseam. It is a design choice not a bug.

by **glessard** » Sat Mar 29, 2014 2:57 pm

ilovezfs wrote:This has been discussed ad nauseam. It is a design choice not a bug.

Citation needed. I'd love to see the reasoning. I can't figure out the logic. (Other than it's hard.)

by **ilovezfs** » Sat Mar 29, 2014 7:17 pm

glessard wrote:
ilovezfs wrote:This has been discussed ad nauseam. It is a design choice not a bug.

Citation needed. I'd love to see the reasoning. I can't figure out the logic. (Other than it's hard.)

Citation: http://en.wikipedia.org/wiki/Ad_nauseam

by **roemer** » Sat Mar 29, 2014 8:52 pm

ilovezfs wrote:This has been discussed ad nauseam. It is a design choice not a bug.

Please forgive my ignorance, but I am not on the developer email list:
Which design choice do you mean:
To not support Mac ACLs as in 'they can't be stored'?
Or to return 'trivial' ACLs to OS X once it asks about the ACLs of some ZFS files?

by **ilovezfs** » Sat Mar 29, 2014 9:07 pm

roemer wrote:
ilovezfs wrote:This has been discussed ad nauseam. It is a design choice not a bug.

Please forgive my ignorance, but I am not on the developer email list:
Which design choice do you mean:
To not support Mac ACLs as in 'they can't be stored'?
Or to return 'trivial' ACLs to OS X once it asks about the ACLs of some ZFS files?

The choice is to keep the mode in sync with the ACLs.

https://github.com/openzfsonosx/zfs/issues/86

by **roemer** » Tue Apr 01, 2014 4:35 am

Thanks for the link. So the rationale is pool-compatibility with Linux/Solaris.
In that discussion thread on Github, you pointed to the 'xattr_darwinacls' branch as potential alternative.
That branch was last modified about 4 months ago.
How is that branch kept up-to-date with bugfixes in the master branch?

Secondly, if I compile my own kernel extension from that branch, it is not signed and would hence create a warning at each boot from OS X 10.9?
Anyway to avoid this (other than paying 99 bucks/anno to Apple

OpenZFS on OS X

extended attributes and ACLs not handled correctly?

extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Re: extended attributes and ACLs not handled correctly?

Who is online