OpenZFS on OS X - User contributions [en]

Stopping Spotlight etc. from changing ZFS without permission

2017-01-23T06:07:12Z

Foobaron: revised to focus on a simpler use-case

== The Issue ==

ZFS is a great solution for backing up data, and moving data between platforms (e.g. Linux and Mac OS X). However, if you connect an external ZFS drive to Mac OS X, you may be unpleasantly surprised to find that Spotlight, Finder etc. will modify the filesystems on the drive without asking (and even in violation of the user privileges granted by those filesystems). This goes against several basic principles that you probably expect:

* the filesystem should not change unless you change data on the filesystem. E.g. simply plugging in the drive should not change the filesystem.

* any attempted changes to the filesystem should obey the privilege rules set on that filesystem. E.g. if the current user plugging in the drive lacks write privileges to the filesystem, no changes should be allowed.

However, Spotlight, Finder etc. do not obey these expectations. This can lead to serious problems for standard ZFS activities such as snapshot based backup. For example, say you backup data by sending ZFS snapshots from one ZFS filesystem to another ZFS drive. When you try to push incremental snapshots to an existing filesystem, it may fail with messages like "data has been changed since snapshot XYZ" -- even if you never wrote to that backup filesystem since your last snapshot. Now you are in a quandary. You will have to manually investigate the backup filesystem(s) to see what all the changes are, and whether might be any changes that need to be preserved (rather than just rolled back). If you are used to working with ZFS on Linux this may initially baffle you, because on Linux a ZFS filesystem will not change in content unless you deliberately write data to it!

The following Mac OS X services will modify your ZFS filesystems silently:

* Spotlight
* Finder (.DS_Store, Trash)

== Solutions ==

* in general, if you are connecting an external ZFS drive to a Mac solely to READ data from it, '''just import it readonly'''. This is the one trivially easy way to avoid the whole problem, e.g.:

'''zpool import -o readonly=on tank'''

If that is not an option, then you are going to have clean up your ZFS filesystems (every time) after OS X has "dirtied" them:

* '''always take a snapshot of ZFS filesystems BEFORE plugging them in to a Mac'''. This makes it relatively easy to know which changes are "OS X dirt" vs. previous data changes that you made before plugging into the Mac, and to rollback the dirt. For example: '''zfs diff -H tank@lastsnap tank''' will give you a list of new, modified and deleted files since lastsnap. Use Google to learn what Spotlight and Finder's hidden file names are.

* if you did not deliberately make changes to a given ZFS filesystem while connected to the Mac, then '''just rollback to the latest snapshot''' (as your first action once you import that drive to a non-Mac host).

* add "flag files" to each file system to prevent various OS X services from dirtying them. E.g. '''touch /tank/.metadata_never_index''' should [http://superuser.com/questions/1000280/disable-ds-store-spotlight-and-other-metadata-on-mac-disk-image prevent Spotlight indexing]. We should collect a list here of all the rules for preventing such "dirty" services.

Stopping Spotlight etc. from changing ZFS without permission

2017-01-23T05:30:56Z

Foobaron: first version of this page

== The Issue ==

ZFS is a great solution for moving data between platforms (e.g. Linux and Mac OS X). However, if you connect an external ZFS drive to Mac OS X, you may be unpleasantly surprised to find that Spotlight, Finder etc. will modify the filesystems on the drive without asking (and even in violation of the user privileges granted by those filesystems). This goes against several basic principles that you probably expect:

* the filesystem should not change unless you change data on the filesystem. E.g. simply plugging in the drive should not change the filesystem.

* any attempted changes to the filesystem should obey the privilege rules set on that filesystem. E.g. if the current user plugging in the drive lacks write privileges to the filesystem, no changes should be allowed.

However, Spotlight, Finder etc. do not obey these expectations. This can lead to serious problems for standard ZFS activities such as snapshot based backup. For example, say you backup data from Computer A by sending ZFS snapshots to ZFS filesystems on an external drive. At some point you connect this drive to a Mac (Computer B), e.g. to copy some of its data to the Mac. However, the next time you try to backup some new snapshots from Computer A to this drive, it will fail with messages like "data has been changed since snapshot XYZ". Note that this will happen even if the user login on Computer B lacks write privileges to the external drive's ZFS filesystems. Now you are in a quandary. You will have to manually investigate all the changed filesystems to see what all the changes are, and whether there are any changes that need to be preserved (rather than just rolled back).

The following Mac OS X services will modify your ZFS filesystems silently:

* Spotlight
* Finder (.DS_Store, Trash)

== Solutions ==

* in general, if you are connecting an external ZFS drive to a Mac solely to READ data from it, '''just import it readonly'''. This is the one trivially easy way to avoid the whole problem, e.g.:

'''zpool import -o readonly=on tank'''

If that is not an option, then you are going to have clean up your ZFS filesystems (every time) after OS X has "dirtied" them:

* '''always take a snapshot of ZFS filesystems BEFORE plugging them in to a Mac'''. This makes it relatively easy to know which changes are "OS X dirt" vs. previous data changes that you made before plugging into the Mac, and to rollback the dirt.

* if you did not deliberately make changes to a given ZFS filesystem while connected to the Mac, then '''just rollback to the latest snapshot''' (as your first action once you import that drive to a non-Mac host).

* add "flag files" to each file system to prevent various OS X services from dirtying them. E.g. '''touch /tank/.metadata_never_index''' should [http://superuser.com/questions/1000280/disable-ds-store-spotlight-and-other-metadata-on-mac-disk-image prevent Spotlight indexing]. We should collect a list here of all the rules for preventing such "dirty" services.

Documentation

2017-01-23T04:12:20Z

Foobaron: add page for Spotlight/Finder behavior issues

[[Category:About O3X]]
[[Category:Getting and installing O3X]]

== Documentation ==

General OpenZFS usage can be found on the [http://open-zfs.org OpenZFS wiki]. The main focus of this documentation is on OS X platform specifics.

* [[zpool#Creating_a_pool|Creating a pool]]
* [[zpool#Unplugging_a_pool|Unplugging a pool]]
* [[zpool#Feature_flags|Feature flags]]
* [[Device names]]
* [[Pool upgrade]]
* [[Memory utilization]]
* [[Creating user privileges]]
* [[Stopping Spotlight etc. from changing ZFS without permission]]
* [[Encryption]]
* [[Bash completion]]
* [[DegradedPool|Degraded pool]]
* [[panic|Kernel panic]]
* [[ZFS on Boot]]

Suppressing the annoying pop-up

2017-01-23T03:32:38Z

Foobaron:

[[Category:Integration]]

As of version 1.3.0, you should not see the message "The disk you inserted was not readable by this computer" unless one or more devices in your pool is a partition with a type code specifying a file system other than ZFS. The correct [http://en.wikipedia.org/wiki/GUID_Partition_Table#Partition_type_GUIDs partition type GUID] for OpenZFS on OS X is "6A898CC3-1DD2-11B2-99A6-080020736631," which may display as any of the following, depending on what tool you're using:

* bf01
* ZFS
* EFI_USR
* Solaris /usr & Mac ZFS
* 6A898CC3-1DD2-11B2-99A6-080020736631

The easiest way to correct the partition type is to use [http://www.rodsbooks.com/gdisk/ gdisk] which is distributed via [http://brew.sh Homebrew] as the formula "gptfdisk" or via [http://sourceforge.net/projects/gptfdisk/files/gptfdisk/ SourceForge] as a pkg installer. The gdisk function for changing partition type is "t," as explained [http://www.rodsbooks.com/gdisk/gdisk.html here].

'''Warning''': as of El Capitan (10.11) and later, [http://www.rodsbooks.com/gdisk/ Apple has blocked normal methods of modifying a partition type] (such as those mentioned in the preceding paragraph), through a mechanism called "System Integrity Protection" (SIP). You will have to use one of the following methods (starting with the easiest):

* Boot an earlier version of Mac OS X (e.g. 10.6), and use it to run gdisk as outlined above on the disk containing your target partition.

* boot using a Linux emergency disk and use it to run gdisk on the disk containing your target partition.

* figure out how to disable Apple's "System Integrity Protection", then run gdisk.

For further information, see the gdisk home page link above.

Please '''note''' that you don't need to worry about fs type in the case you use coreStorage based encryption as explained in the [[encryption]] wiki page.

Suppressing the annoying pop-up

2017-01-23T03:16:06Z

Foobaron: added note about circumventing new SIP problem

[[Category:Integration]]

As of version 1.3.0, you should not see the message "The disk you inserted was not readable by this computer" unless one or more devices in your pool is a partition with a type code specifying a file system other than ZFS. The correct [http://en.wikipedia.org/wiki/GUID_Partition_Table#Partition_type_GUIDs partition type GUID] for OpenZFS on OS X is "6A898CC3-1DD2-11B2-99A6-080020736631," which may display as any of the following, depending on what tool you're using:

* bf01
* ZFS
* EFI_USR
* Solaris /usr & Mac ZFS
* 6A898CC3-1DD2-11B2-99A6-080020736631

The easiest way to correct the partition type is to use [http://www.rodsbooks.com/gdisk/ gdisk] which is distributed via [http://brew.sh Homebrew] as the formula "gptfdisk" or via [http://sourceforge.net/projects/gptfdisk/files/gptfdisk/ SourceForge] as a pkg installer. The gdisk function for changing partition type is "t," as explained [http://www.rodsbooks.com/gdisk/gdisk.html here].

'''Warning''': as of El Capitan (10.11) and later, '''Apple has blocked all normal methods of modifying a partition type''' (such as those mentioned in the preceding paragraph), through a mechanism called "System Integrity Protection" (SIP). You will have to use one of the following methods (starting with the easiest):

* Boot an earlier version of Mac OS X (e.g. 10.6), and use it to run gdisk as outlined above on the disk containing your target partition.

* boot using a Linux emergency disk and use it to run gdisk on the disk containing your target partition.

* figure out how to disable Apple's "System Integrity Protection", then run gdisk.

For further information, see the gdisk home page link above.

Please '''note''' that you don't need to worry about fs type in the case you use coreStorage based encryption as explained in the [[encryption]] wiki page.