ZFS on Boot

From OpenZFS on OS X
Revision as of 03:36, 15 October 2019 by Lundman (Talk | contribs)

Jump to: navigation, search

ZFS as Boot

  Status: Beta
  Most of the commands listed below must be run as root or with sudo.

Updated for macOS Catalina

This guide assumes you have installed the latest Open ZFS on OS X on the system, both to create and populate the rpool with data, but that the data copied will have the kexts and prelinked kernels containing OpenZFS kexts.

Install OpenZFS

Download latest OpenZFSonOsX pkg and install.

Partitioning disk

You need generally a 200MB EFI partition, your ZFS partition, then finally a 200MB Apple boot partition.

In this example, $NEWDISK is the blank disk to use; set it now

(Change /dev/disk1 to your blank disk!)
# export NEWDISK=/dev/disk1            
# echo $NEWDISK

Default diskutil partitions appear to work:

# diskutil partitionDisk $NEWDISK GPT ZFS rpool 0b
(You can ignore this error: )
Could not mount disk1s2 after erase
Error: -69832: File system formatter failed

/dev/disk1 (internal, physical):
  #:                       TYPE NAME                    SIZE       IDENTIFIER
  0:      GUID_partition_scheme                        *64.4 GB    disk1
  1:                        EFI EFI                     209.7 MB   disk1s1
  2: FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF               64.1 GB    disk1s2
  3:                 Apple_Boot Boot OS X               134.2 MB   disk1s3

Setting up ZFS slice 2

Create the actual pool:

# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
  -O normalization=formD -O atime=off -O compression=lz4 \
  -O mountpoint=none -O canmount=off rpool ${NEWDISK}s2

Create minimal datasets for ROOT

# zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# zfs create -o mountpoint=legacy -o com.apple.mimic_hfs=on rpool/ROOT/Catalina
# zpool set bootfs=rpool/ROOT/Catalina rpool
Optional, create further datasets, perhaps for /Users, and/or, individual users...
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Populate the rpool from the running system:

# zfs set mountpoint=/Volumes/ZFSBoot rpool/ROOT/Catalina
# zfs mount rpool/ROOT/Catalina
# rsync -axH --progress --exclude=".Spotlight-V100" --exclude=".fseventsd" --exclude=".vol" / /Volumes/ZFSBoot/

.. and wait.

(It would be neat if we could (ab)use the installer for the OS: ie some version of installer -pkg [path to]/Install macOS Catalina.app/Contents/SharedSupport/InstallInfo.plist -target /Volumes/ZFSBoot -verbose -dumplog but possibly it always insists on partitioning.)

Update the Boot.plist:

# vim /Volumes/ZFSBoot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

        <key>Kernel Flags</key>
        <string>-v keepsyms=y zfs_boot=rpool</string>
        <key>Root UUID</key>

Unmount the ZFS dataset.

# zfs umount rpool/ROOT/Catalina
# zfs set mountpoint=legacy rpool/ROOT/Catalina

Setting up boot slice 3

These steps require slice 2 copy to have finished.

# newfs_hfs -J -v "boot" ${NEWDISK}s3
# diskutil mount ${NEWDISK}s3
# mkdir -p /Volumes/boot/System/Library/CoreServices
# mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

Populate boot:

# rsync -a /Volumes/ZFSBoot/System/Library/CoreServices/PlatformSupport.plist \
# rsync -a /Volumes/ZFSBoot/System/Library/CoreServices/SystemVersion.plist \
# rsync -a /Volumes/ZFSBoot/System/Library/PrelinkedKernels/prelinkedkernel \
# rsync -a /Volumes/ZFSBoot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
# rsync -a /Volumes/ZFSBoot/usr/standalone/i386/ \

Bless the volume:

# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
 --file /Volumes/boot/System/Library/CoreServices/boot.efi \
 --bootefi /Volumes/ZFSBoot/System/Library/CoreServices/boot.efi \
 --label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See blessing the boot device to change the startup device. Another useful argument is --nextonly to just affect one boot, which is great for testing. To get out of trouble, remember that you can hold `Alt` during boot to select disk to boot from.

Booting ZFS

Finally, tell your Mac to use the boot helper on boot. You may want to try the --nextonly option to avoid permanent changes.

# sudo bless --device ${NEWDISK}s3 --setBoot
# reboot

The remaining text is the old guide.

Because you have to compile your own KEXTs, they will not be signed. System Integrity Protection (SIP) needs to be disabled for unsigned KEXTs to work. A future version of OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

Disk Layout

Play disk: disk1

I used gptfdisk, but gpt and diskutil can be used also. Create the following partitions on the test boot disk:

  Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
  Number  Start (sector)    End (sector)  Size       Code  Name
  1            2048          411647   200.0 MiB   EF00  EFI System
  2          411648        82200575   39.0 GiB    BF01  rpool
  3        82200576        83886046   823.0 MiB   AB00  Apple boot
  • s1 is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
  • s2 is the ZFS pool used for booting. Note: You are not limited to using the name rpool.
  • s3 is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

  • The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
  • The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

Populating the ZFS Root s2

Root pool

Any pool name may be used - rpool is used here for example. Additional research is needed on atime, casesensitivity, and normalization (and probably more).

# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
 -O normalization=formD -O atime=off -O compression=lz4 \
 -O mountpoint=none -O canmount=off rpool disk1s2

Root dataset

Create ROOT and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).

# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.

# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:

# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

Install macOS

Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)

# sudo rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
 --exclude=".vol" / /Volumes/Capitan/

Install ZFS

Boot support is now included with the binary install versions, so you don't have to compile it yourself.

In you haven't done so already, install homebrew (or macports, but in the example we will stick to homebrew as it's quicker to set-up):

# /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install automake and libtool (required to compile spl and zfs):

# brew install automake libtool

Create target directory for installation:

# sudo mkdir -p /Volumes/Capitan/usr/local

Use your own repository:

# cd $your-zfs-source-area

Or clone it one from github:

# git clone https://github.com/openzfsonosx/spl.git
# git clone https://github.com/openzfsonosx/zfs.git

Build and install spl:

# cd spl
# git fetch --all
# ./autogen.sh
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Build and install zfs:

# cd ../zfs
# git fetch --all
# ./autogen.sh
# ./configure --enable-boot 
# make
# sudo make install DESTDIR=/Volumes/Capitan

Edit Boot.plist

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at com.apple.Boot.plist

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: Kernel Flags and Root UUID:

Look for

  <key>Kernel Flags</key>

Replace with:

  <key>Kernel Flags</key>
  <string>-v keepsyms=y zfs_boot=rpool</string>
  <key>Root UUID</key>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See Boot.plist for more examples.

ZVOL boot from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

Setting up boot slice s3


Journaled HFS+ filesystem, and populated thusly; (Volume named boot here) These can be rsynced from the (normal) booted OS, or from the Capitan mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot


# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

Required Files

# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \

Bless the volume

The boot helper needs to know where to load boot.efi from.

You may use any label in place of ZFS Boot here, it will appear at the option-boot screen.

# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
  --file /Volumes/boot/System/Library/CoreServices/boot.efi \
  --bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
  --label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See blessing the boot device to change the startup device.

Generate prelinkedkernel

Generate caches for kernel

# sudo kextcache -arch x86_64 -local-root -volume-root /Volumes/Capitan \
  -kernel /Volumes/Capitan/System/Library/Kernels/kernel \
  -prelinked-kernel /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
  -- /Volumes/Capitan/System/Library/Extensions /Volumes/Capitan/Library/Extensions /Volumes/boot/Library/Extensions

If you get a symlink error here like;

symlink("/Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel", "/Volumes/Capitan/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache") failed 17 (File exists) <createPrelinkedKernel 2795>

Just delete the symlink, and run again, ie;

rm /Volumes/Capitan/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache

For developers who also have development, or debug kernels, do the same for each one. For example, caches for kernel.development

# kextcache -arch x86_64 -local-root -volume-root /Volumes/Capitan \
  -kernel /Volumes/Capitan/System/Library/Kernels/kernel.development \
  -prelinked-kernel /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel.development \
  -- /Volumes/Capitan/System/Library/Extensions /Volumes/boot/Library/Extensions

Copy prelinkedkernel to boot helper

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \

Set the boot device

Finally, tell your Mac to use the boot helper on boot. You may want to try the --nextonly option to avoid permanent changes.

# sudo bless --device /dev/disk1s3 --setBoot


Booting on ESX

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

fs0 disk0 EFI partition
fs1 disk0 OSX boot
fs2 disk0 OSX Recovery
fs3 disk1 EFI
fs4 disk1 boot HFS
fs5 disk1 recovery
fs6 disk1 ZFS

So in my case

EFI> fs4:
fs4: ls
usr System Library
fs4: cd System 
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel. You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool rpool (using bootfs= property) into single-user mode, with verbose output.

Booting on VirtualBox

SIP can be disabled in VirtualBox, but must be disabled before every boot. One way to do this is to install rEFInd into the EFI partition. It is possible to write a startup.nsh script to do so.

[Please provide details on how to disable SIP in startup.nsh]

VirtualBox does not like the boot helper as created above. There are two options to get a bootable setup:

Minimal boot partition for VirtualBox (tested with macOS Sierra)
# sudo mkdir -p /Volumes/boot//Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices/com.apple.recovery.boot
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/autodiskmount.plist /Volumes/boot/Library/Preferences/SystemConfiguration/autodiskmount.plist
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist /Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist /Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/boot.efi /Volumes/boot/System/Library/CoreServices/boot.efi
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/bootbase.efi /Volumes/boot/System/Library/CoreServices/bootbase.efi
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/com.apple.recovery.boot/PlatformSupport.plist /Volumes/boot/System/Library/CoreServices/com.apple.recovery.boot/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel /Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
Alternative solution using recovery boot base system image
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"


# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"

Or if you prefer asr:

# sudo asr restore -source "/Volumes/OS X Base System/" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with no com.apple.boot.R:

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

(See Boot.plist)

Booting on real Macs

  Always make extra backups!
  Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or asr and edit the [com.apple.Boot.plist] to put a different pool name in zfs_boot in the Kernel flags.

I was unable to boot Sierra on 2016 MacBookPro with "com.apple.boot.R" setup, but the minimal VirtualBox boot partition solution above has worked just fine.

Hold option while powering on or restarting your Mac.

Look for ZFS Boot as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in zfs_boot above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the Capitan dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the option key. Select the partition with standard MacOS to reboot normally. Once booted, check and set System Preferences->Startup Disk.


Mounting other boot environments

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote

# sudo zfs promote rpool/ROOT/Capitan2

Then if you want

# sudo zfs destroy -r rpool/ROOT/Capitan

Blessing the boot device

Need only be run once, to setup the boot volume:

# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
  --file /Volumes/boot/System/Library/CoreServices/boot.efi \
  --bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:

# sudo bless --device /dev/disk1s3 --setBoot --verbose

This will also work, as it will locate the boot helper following the ZFS partition:

# sudo bless --device /dev/disk1s2 --setBoot --verbose

Currently --folder and --mount options will work (partially) but may select the wrong boot helper if there are multiple bootable ZFS pools. (For example, if you have ZFS boot setup on both the internal disk and an external disk.)

# sudo bless --folder /Volumes/Capitan --setBoot --verbose
# sudo bless --mount /Volumes/Capitan --setBoot --verbose