OpenZFS on OS X - User contributions [en]

ZFS on Boot

2016-07-27T06:51:15Z

Evansus: /* Install ZFS */

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local

With own source repository:
# cd $your-zfs-source-area
Or by getting sources:
# git clone https://github.com/openzfsonosx/spl.git
# git clone https://github.com/openzfsonosx/zfs.git

Build and install both spl and zfs:
# cd spl
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Copy prelinkedkernel to boot helper ====

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device /dev/disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device /dev/disk1s3 --setBoot --verbose

This will also work, as it will locate the boot helper following the ZFS partition:
# sudo bless --device /dev/disk1s2 --setBoot --verbose

Currently --folder and --mount options will work (partially) but may select the wrong boot helper if there are multiple bootable ZFS pools.
(For example, if you have ZFS boot setup on both the internal disk and an external disk.)
# sudo bless --folder /Volumes/Capitan --setBoot --verbose
# sudo bless --mount /Volumes/Capitan --setBoot --verbose

ZFS on Boot

2016-07-27T06:46:49Z

Evansus: /* Root pool */

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# git clone https://github.com/openzfsonosx/spl.git
# git clone https://github.com/openzfsonosx/zfs.git

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Copy prelinkedkernel to boot helper ====

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device /dev/disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device /dev/disk1s3 --setBoot --verbose

This will also work, as it will locate the boot helper following the ZFS partition:
# sudo bless --device /dev/disk1s2 --setBoot --verbose

Currently --folder and --mount options will work (partially) but may select the wrong boot helper if there are multiple bootable ZFS pools.
(For example, if you have ZFS boot setup on both the internal disk and an external disk.)
# sudo bless --folder /Volumes/Capitan --setBoot --verbose
# sudo bless --mount /Volumes/Capitan --setBoot --verbose

ZFS on Boot

2016-07-27T06:44:53Z

Evansus: Add missing /dev/ prefix

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# git clone https://github.com/openzfsonosx/spl.git
# git clone https://github.com/openzfsonosx/zfs.git

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Copy prelinkedkernel to boot helper ====

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device /dev/disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device /dev/disk1s3 --setBoot --verbose

This will also work, as it will locate the boot helper following the ZFS partition:
# sudo bless --device /dev/disk1s2 --setBoot --verbose

Currently --folder and --mount options will work (partially) but may select the wrong boot helper if there are multiple bootable ZFS pools.
(For example, if you have ZFS boot setup on both the internal disk and an external disk.)
# sudo bless --folder /Volumes/Capitan --setBoot --verbose
# sudo bless --mount /Volumes/Capitan --setBoot --verbose

Bless

2016-07-27T06:43:54Z

Evansus: Created page with "Using a layout as specified in ZFS on Boot, using disk1: The simplest version of this command will locate the boot helper following the ZFS partition: # sudo bless --dev..."

Using a layout as specified in [[ZFS on Boot]], using disk1:

The simplest version of this command will locate the boot helper following the ZFS partition:
# sudo bless --device /dev/disk1s2 --setBoot --verbose

Currently --folder and --mount options will work (partially) but may select the wrong boot helper if there are multiple bootable ZFS pools.
(For example, if you have ZFS boot setup on both the internal disk and an external disk.)
# sudo bless --folder /Volumes/Capitan --setBoot --verbose
# sudo bless --mount /Volumes/Capitan --setBoot --verbose

There is an undocumented environment variable that can resolve the conflict. For example, if the ZFS pool is located on disk3s2, and the boot helper is disk3s3:
# sudo bless --mount /Volumes/Capitan --setBoot --verbose
{ ... Truncated output ... }
Returning booter information dictionary:
<CFBasicHash 0x7fd7c2400a30 [0x7fff75c10440]>{type = mutable dict, count = 3, entries =>
0 : <CFString 0x10bd97990 [0x7fff75c10440]>{contents = "System Partitions"} = (
disk1s1,
disk0s1,
disk2s1,
disk3s1
)
1 : <CFString 0x10bd98170 [0x7fff75c10440]>{contents = "Data Partitions"} = (
disk1s2,
disk2s2,
disk3s2
)
2 : <CFString 0x10bd98190 [0x7fff75c10440]>{contents = "Auxiliary Partitions"} = (
'''disk1s3,'''
disk2s3,
'''disk3s3'''
)
}
Substituting booter '''disk1s3'''
{ ... Truncated output ... }

Adding an environment variable 'BL_PRIMARY_BOOTER_INDEX' can instruct bless to use the correct partition. The index will start from 0 and count upwards in the list of "Auxiliary Partitions"

In this case, the pool is located on disk3s2 and the boot helper is disk3s3, so adding an index of '''2''' will select the third boot device:
# sudo env BL_PRIMARY_BOOTER_INDEX=2 bless --mount /Volumes/Capitan --setBoot --verbose
{ ... Truncated output ... }
Returning booter information dictionary:
<CFBasicHash 0x7fd7c2400a30 [0x7fff75c10440]>{type = mutable dict, count = 3, entries =>
0 : <CFString 0x10bd97990 [0x7fff75c10440]>{contents = "System Partitions"} = (
disk3s1,
disk1s1,
disk0s1,
disk2s1
)
1 : <CFString 0x10bd98170 [0x7fff75c10440]>{contents = "Data Partitions"} = (
disk3s2,
disk1s2,
disk2s2
)
2 : <CFString 0x10bd98190 [0x7fff75c10440]>{contents = "Auxiliary Partitions"} = (
'''disk3s3,'''
disk1s3,
disk2s3
)
}
{ ... Truncated output ... }
Substituting booter '''disk3s3'''
IOMedia '''disk3s3''' has UUID 4C5E73A9-05D0-4B7F-A6A8-050715D620A0
Setting EFI NVRAM:
efi-boot-device='<array><dict><key>IOMatch</key><dict><key>IOProviderClass</key>
<string>IOMedia</string><key>IOPropertyMatch</key>
<dict><key>UUID</key><string>4C5E73A9-05D0-4B7F-A6A8-050715D620A0</string></dict></dict>
<key>BLLastBSDName</key><string>'''disk3s3'''</string></dict></array>'
{ ... Truncated output ... }

ZVOL boot

2016-07-11T07:40:02Z

Evansus: /* Create zvol */

=== ZVOL Boot ===

Using the same steps to create the boot helper partition as [[ZFS on Boot]], you may install Mac OS X onto a ZVOL device and use it as your boot volume.

==== Disable bootfs ====

zvol boot does not currently co-exist with zfs boot (planned to be fixed). The presence of the '''bootfs''' property will take precedence at boot time.

You will need to clear bootfs when using zvol boot:

# <nowiki>sudo zpool set bootfs='' poolname</nowiki>

Using two single quotes will remove the previous bootfs.

==== Create zvol ====
# zfs create -sV 80g rpool/ZVOL/MacOS

Get the diskN for that device
# zvol=`ioreg -trn "ZVOL rpool/ZVOL/MacOS Media" | grep "BSD Name" | awk -F'"' '{print $4}'`
(it would be nice if we could just # zfs get zvol-disk rpool/ZVOL/MacOS)

Erase and create a volume

With partition map:
# diskutil eraseDisk JHFS+ MacOS ${zvol}
Note: direct OS install may only be allowed with a partition map, testing needed

Without partitions:
# diskutil eraseVolume JHFS+ MacOS ${zvol}

Alternate method, also without partitions and using zvol symlink:
# newfs_hfs -J -v "MacOS" /var/run/zfs/zvol/rpool/ZVOL/MacOS

==== Clone root ====
Clone an existing Mac OS
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" "/Volumes/Macintosh HD/" "/Volumes/MacOS/"

Or use Apple System Restore
# asr --source "/Volumes/Macintosh HD" --target "/Volumes/MacOS" --erase

==== New install ====
It is possible, but requires a few steps, to install Mac OS onto a ZVOL with the standard Apple installer.
Download the installer from the App Store, and create an installer flash drive.
Boot into the Installer, with a standard Mac OS disk available '''Macintosh HD''' (that has ZFS installed).

From '''Utilities->Terminal''' you can:
# kextload "/Volumes/Macintosh HD/Library/Extensions/spl.kext"
# kextload "/Volumes/Macintosh HD/Library/Extensions/zfs.kext"

At that point you need access to the zpool and zfs commands, which require the libzfs and libzpool libraries. To work around the install environment, simply:
# chroot "/Volumes/Macintosh HD"
# zpool import
Or if necessary
# /usr/local/sbin/zpool import

This worked with El Capitan Install 10.11 and the signed O3x from [[Downloads]].
I don't remember which version of O3x, but I believe it was 1.4.5 or so.

==== Setup boot helper ====

Boot helper can be setup similar to the boot helper for [[ZFS on Boot]] with some differences.

Root UUID must be set to the Volume UUID of the zvol. You can retrieve this by:

# diskutil info diskN | grep "Volume UUID"

For a volume directly within a zvol, or

# diskutil info diskNsN | grep "Volume UUID"

For a zvol with a partition map.

Kernel Flags should contain "zfs_boot=poolname"

See [[Boot.plist]] for sample plist files.

At this time zfs_boot=poolname/zvol or poolname/dataset/zvol will only import the zpool "poolname" and does not check or mount the zvol specified. Correctly setting Root UUID to match will inform the OS to locate the root volume.

In the future, it will be possible to specify a zvol by name, which will then be inspected for matching volume UUID and presented for mount. This would prevent any duplicate UUID clashes due to `zfs clone` copies. To do so will require an additional property zfs_root_uuid=<uuid>, or perhaps zfs_root_partition=N, so that the OS does not try to automatically load the first matching volume.

ZFS on Boot

2016-07-11T05:50:45Z

Evansus: Add git clone commands

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# git clone https://github.com/openzfsonosx/spl.git
# git clone https://github.com/openzfsonosx/zfs.git

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Copy prelinkedkernel to boot helper ====

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-11T05:48:44Z

Evansus: /* Install ZFS continued ... */

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Copy prelinkedkernel to boot helper ====

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-11T05:48:05Z

Evansus: /* Generate prelinkedkernel */

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Copy prelinkedkernel to boot helper ====

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-11T05:45:46Z

Evansus: /* Edit Boot.plist */

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-11T05:44:38Z

Evansus: /* Edit Boot.plist */

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/MacOS/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-11T05:42:18Z

Evansus: /* Disk Layout */

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-11T05:39:03Z

Evansus: Revise boot helper notes about kextd updates

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

'''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).

'''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.

'''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

The EFI partition should always exist at the beginning of the disk. It is usually 200 MB.

The Apple Boot (ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

The boot helper partition is updated automatically at boot time or when software is installed/updated. Development is ongoing to fully support this. A potential issue with atime=on has been identified where kextd may try to run multiple workers at one time.

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

You cannot use '''Startup Disk''' to select the boot disk, see [[#Blessing the boot device|blessing the boot device]].

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-11T05:27:21Z

Evansus: /* Root dataset */

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

'''s1''' is not really used, but added for EFI.

'''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.

'''s3''' is the small HFS boot which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

The EFI partition should always exist at the beginning of the disk.

The Apple Boot (ab00) partition should directly follow the ZFS partition, and while it can be as small as 50 MB, it is better to use 650MB or more (to imitate a Recovery HD).

Note: at this time, the boot helper partition is not updated by the OS
and must be manually updated. It may be possible to script this in a
launchd service, or even 'bless' the device so that macOS updates
it automatically. Additional development is ongoing.

You cannot use '''Startup Disk''' to select the boot disk, see [[#Blessing the boot device|blessing the boot device]].

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-11T05:20:09Z

Evansus: Revise /dev/disk note, no longer an issue

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

'''s1''' is not really used, but added for EFI.

'''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.

'''s3''' is the small HFS boot which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

The EFI partition should always exist at the beginning of the disk.

The Apple Boot (ab00) partition should directly follow the ZFS partition, and while it can be as small as 50 MB, it is better to use 650MB or more (to imitate a Recovery HD).

Note: at this time, the boot helper partition is not updated by the OS
and must be manually updated. It may be possible to script this in a
launchd service, or even 'bless' the device so that macOS updates
it automatically. Additional development is ongoing.

You cannot use '''Startup Disk''' to select the boot disk, see [[#Blessing the boot device|blessing the boot device]].

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.

==== Root dataset ====
Note: com.apple.mimic_hfs is known to '''NOT''' be compatible at this time. (It will panic!)

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZVOL boot

2016-07-11T05:13:50Z

Evansus: /* Create zvol */

=== ZVOL Boot ===

Using the same steps to create the boot helper partition as [[ZFS on Boot]], you may install Mac OS X onto a ZVOL device and use it as your boot volume.

==== Disable bootfs ====

zvol boot does not currently co-exist with zfs boot (planned to be fixed). The presence of the '''bootfs''' property will take precedence at boot time.

You will need to clear bootfs when using zvol boot:

# <nowiki>sudo zpool set bootfs='' poolname</nowiki>

Using two single quotes will remove the previous bootfs.

==== Create zvol ====
# zfs create -sV 80g rpool/ZVOL/MacOS

Get the diskN for that device
# zvol=`ioreg -trn "ZVOL rpool/ZVOL/MacOS Media" | grep "BSD Name" | awk -F'"' '{print $4}'`
(it would be nice if we could just # zfs get zvol-disk rpool/ZVOL/MacOS)

Erase and create a volume

With partition map:
# diskutil eraseDisk JHFS+ MacOS ${zvol}

Without partitions:
# diskutil eraseVolume JHFS+ MacOS ${zvol}

Alternate method, also without partitions and using zvol symlink:
# newfs_hfs -J -v "MacOS" /var/run/zfs/zvol/rpool/ZVOL/MacOS

==== Clone root ====
Clone an existing Mac OS
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" "/Volumes/Macintosh HD/" "/Volumes/MacOS/"

Or use Apple System Restore
# asr --source "/Volumes/Macintosh HD" --target "/Volumes/MacOS" --erase

==== New install ====
It is possible, but requires a few steps, to install Mac OS onto a ZVOL with the standard Apple installer.
Download the installer from the App Store, and create an installer flash drive.
Boot into the Installer, with a standard Mac OS disk available '''Macintosh HD''' (that has ZFS installed).

From '''Utilities->Terminal''' you can:
# kextload "/Volumes/Macintosh HD/Library/Extensions/spl.kext"
# kextload "/Volumes/Macintosh HD/Library/Extensions/zfs.kext"

At that point you need access to the zpool and zfs commands, which require the libzfs and libzpool libraries. To work around the install environment, simply:
# chroot "/Volumes/Macintosh HD"
# zpool import
Or if necessary
# /usr/local/sbin/zpool import

This worked with El Capitan Install 10.11 and the signed O3x from [[Downloads]].
I don't remember which version of O3x, but I believe it was 1.4.5 or so.

==== Setup boot helper ====

Boot helper can be setup similar to the boot helper for [[ZFS on Boot]] with some differences.

Root UUID must be set to the Volume UUID of the zvol. You can retrieve this by:

# diskutil info diskN | grep "Volume UUID"

For a volume directly within a zvol, or

# diskutil info diskNsN | grep "Volume UUID"

For a zvol with a partition map.

Kernel Flags should contain "zfs_boot=poolname"

See [[Boot.plist]] for sample plist files.

At this time zfs_boot=poolname/zvol or poolname/dataset/zvol will only import the zpool "poolname" and does not check or mount the zvol specified. Correctly setting Root UUID to match will inform the OS to locate the root volume.

In the future, it will be possible to specify a zvol by name, which will then be inspected for matching volume UUID and presented for mount. This would prevent any duplicate UUID clashes due to `zfs clone` copies. To do so will require an additional property zfs_root_uuid=<uuid>, or perhaps zfs_root_partition=N, so that the OS does not try to automatically load the first matching volume.

ZVOL boot

2016-07-11T05:10:54Z

Evansus: /* ZVOL Boot */

=== ZVOL Boot ===

Using the same steps to create the boot helper partition as [[ZFS on Boot]], you may install Mac OS X onto a ZVOL device and use it as your boot volume.

==== Disable bootfs ====

zvol boot does not currently co-exist with zfs boot (planned to be fixed). The presence of the '''bootfs''' property will take precedence at boot time.

You will need to clear bootfs when using zvol boot:

# <nowiki>sudo zpool set bootfs='' poolname</nowiki>

Using two single quotes will remove the previous bootfs.

==== Create zvol ====
# zfs create -sV 80g rpool/ZVOL/MacOS

Get the diskN for that device
# zvol=`ioreg -trn "ZVOL rpool/ZVOL/MacOS Media" | grep "BSD Name" | awk -F'"' '{print $4}'`
(it would be nice if we could just # zfs get zvol-disk rpool/ZVOL/MacOS)

Erase and create a volume
# diskutil eraseDisk JHFS+ MacOS ${zvol}

==== Clone root ====
Clone an existing Mac OS
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" "/Volumes/Macintosh HD/" "/Volumes/MacOS/"

Or use Apple System Restore
# asr --source "/Volumes/Macintosh HD" --target "/Volumes/MacOS" --erase

==== New install ====
It is possible, but requires a few steps, to install Mac OS onto a ZVOL with the standard Apple installer.
Download the installer from the App Store, and create an installer flash drive.
Boot into the Installer, with a standard Mac OS disk available '''Macintosh HD''' (that has ZFS installed).

From '''Utilities->Terminal''' you can:
# kextload "/Volumes/Macintosh HD/Library/Extensions/spl.kext"
# kextload "/Volumes/Macintosh HD/Library/Extensions/zfs.kext"

At that point you need access to the zpool and zfs commands, which require the libzfs and libzpool libraries. To work around the install environment, simply:
# chroot "/Volumes/Macintosh HD"
# zpool import
Or if necessary
# /usr/local/sbin/zpool import

This worked with El Capitan Install 10.11 and the signed O3x from [[Downloads]].
I don't remember which version of O3x, but I believe it was 1.4.5 or so.

==== Setup boot helper ====

Boot helper can be setup similar to the boot helper for [[ZFS on Boot]] with some differences.

Root UUID must be set to the Volume UUID of the zvol. You can retrieve this by:

# diskutil info diskN | grep "Volume UUID"

For a volume directly within a zvol, or

# diskutil info diskNsN | grep "Volume UUID"

For a zvol with a partition map.

Kernel Flags should contain "zfs_boot=poolname"

See [[Boot.plist]] for sample plist files.

At this time zfs_boot=poolname/zvol or poolname/dataset/zvol will only import the zpool "poolname" and does not check or mount the zvol specified. Correctly setting Root UUID to match will inform the OS to locate the root volume.

In the future, it will be possible to specify a zvol by name, which will then be inspected for matching volume UUID and presented for mount. This would prevent any duplicate UUID clashes due to `zfs clone` copies. To do so will require an additional property zfs_root_uuid=<uuid>, or perhaps zfs_root_partition=N, so that the OS does not try to automatically load the first matching volume.

ZVOL boot

2016-07-11T05:02:17Z

Evansus: Add boot.plist UUID requirement

=== ZVOL Boot ===

Using the same steps to create the boot helper partition as [[ZFS on Boot]], you may install Mac OS X onto a ZVOL device and use it as your boot volume.

==== Create zvol ====
# zfs create -sV 80g rpool/ZVOL/MacOS

Get the diskN for that device
# zvol=`ioreg -trn "ZVOL rpool/ZVOL/MacOS Media" | grep "BSD Name" | awk -F'"' '{print $4}'`
(it would be nice if we could just # zfs get zvol-disk rpool/ZVOL/MacOS)

Erase and create a volume
# diskutil eraseDisk JHFS+ MacOS ${zvol}

==== Clone root ====
Clone an existing Mac OS
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" "/Volumes/Macintosh HD/" "/Volumes/MacOS/"

Or use Apple System Restore
# asr --source "/Volumes/Macintosh HD" --target "/Volumes/MacOS" --erase

==== New install ====
It is possible, but requires a few steps, to install Mac OS onto a ZVOL with the standard Apple installer.
Download the installer from the App Store, and create an installer flash drive.
Boot into the Installer, with a standard Mac OS disk available '''Macintosh HD''' (that has ZFS installed).

From '''Utilities->Terminal''' you can:
# kextload "/Volumes/Macintosh HD/Library/Extensions/spl.kext"
# kextload "/Volumes/Macintosh HD/Library/Extensions/zfs.kext"

At that point you need access to the zpool and zfs commands, which require the libzfs and libzpool libraries. To work around the install environment, simply:
# chroot "/Volumes/Macintosh HD"
# zpool import
Or if necessary
# /usr/local/sbin/zpool import

This worked with El Capitan Install 10.11 and the signed O3x from [[Downloads]].
I don't remember which version of O3x, but I believe it was 1.4.5 or so.

==== Setup boot helper ====

Boot helper can be setup similar to the boot helper for [[ZFS on Boot]] with some differences.

Root UUID must be set to the Volume UUID of the zvol. You can retrieve this by:

# diskutil info diskN | grep "Volume UUID"

For a volume directly within a zvol, or

# diskutil info diskNsN | grep "Volume UUID"

For a zvol with a partition map.

Kernel Flags should contain "zfs_boot=poolname"

See [[Boot.plist]] for sample plist files.

At this time zfs_boot=poolname/zvol or poolname/dataset/zvol will only import the zpool "poolname" and does not check or mount the zvol specified. Correctly setting Root UUID to match will inform the OS to locate the root volume.

In the future, it will be possible to specify a zvol by name, which will then be inspected for matching volume UUID and presented for mount. This would prevent any duplicate UUID clashes due to `zfs clone` copies. To do so will require an additional property zfs_root_uuid=<uuid>, or perhaps zfs_root_partition=N, so that the OS does not try to automatically load the first matching volume.

ZFS on Boot

2016-07-11T03:40:56Z

Evansus: Move edit boot.plist above creating boot slice, so boot.plist is configured before rsync

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

'''s1''' is not really used, but added for EFI.

'''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.

'''s3''' is the small HFS boot which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

The EFI partition should always exist at the beginning of the disk.

The Apple Boot (ab00) partition should directly follow the ZFS partition, and while it can be as small as 50 MB, it is better to use 650MB or more (to imitate a Recovery HD).

Note: at this time, the boot helper partition is not updated by the OS
and must be manually updated. It may be possible to script this in a
launchd service, or even 'bless' the device so that macOS updates
it automatically. Additional development is ongoing.

You cannot use '''Startup Disk''' to select the boot disk, see [[#Blessing the boot device|blessing the boot device]].

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

We strongly recommend '''NOT''' using /dev/disk paths due to disk renumbering issues!

==== Root dataset ====
Note: com.apple.mimic_hfs is known to '''NOT''' be compatible at this time. (It will panic!)

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

===== Install ZFS with own source repository =====

# cd $your-zfs-source-area

===== Install ZFS by getting sources =====

# Insert lines here, git clone, etc

===== Install ZFS continued ... =====

# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds, that can also be done.

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

==== Edit Boot.plist ====

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-06T13:19:03Z

Evansus: Reword plist comment

== ZFS as Boot ==

Status: Not Ready

Most of the commands listed below must be run as root or with '''sudo'''.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

'''s1''' is not really used, but added for EFI.

'''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.

'''s3''' is the small HFS boot which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

The EFI partition should always exist at the beginning of the disk.

The Apple Boot (ab00) partition should directly follow the ZFS partition, and while it can be as small as 50 MB, it is better to use 650MB or more (to imitate a Recovery HD).

Note: at this time, the boot helper partition is not updated by the OS
and must be manually updated. It may be possible to script this in a
launchd service, or even 'bless' the device so that macOS updates
it automatically. Additional development is ongoing.

You cannot use '''Startup Disk''' to select the boot disk, see [[#Blessing the boot device|blessing the boot device]].

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

Using /dev/disk paths will most likely break!

==== Root dataset ====
Note: com.apple.mimic_hfs is known to '''NOT''' be compatible at this time. (It will panic!)

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

= Note: Missing branches =

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

# cd $your-zfs-source-area
# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot --with-spl=../spl
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds
# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Edit Boot.plist ====

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

And if needed, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly’’’ option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

Booting on ESX, I hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH /Volumes/OS\ X\ Recovery/ /Volumes/boot/
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Recovery 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with no com.apple.boot.R:
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

ZFS on Boot

2016-07-06T13:18:03Z

Evansus:

== ZFS as Boot ==

Status: Not Ready

Most of the commands listed below must be run as root or with '''sudo'''.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

'''s1''' is not really used, but added for EFI.

'''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.

'''s3''' is the small HFS boot which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

The EFI partition should always exist at the beginning of the disk.

The Apple Boot (ab00) partition should directly follow the ZFS partition, and while it can be as small as 50 MB, it is better to use 650MB or more (to imitate a Recovery HD).

Note: at this time, the boot helper partition is not updated by the OS
and must be manually updated. It may be possible to script this in a
launchd service, or even 'bless' the device so that macOS updates
it automatically. Additional development is ongoing.

You cannot use '''Startup Disk''' to select the boot disk, see [[#Blessing the boot device|blessing the boot device]].

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

This step will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

Using /dev/disk paths will most likely break!

==== Root dataset ====
Note: com.apple.mimic_hfs is known to '''NOT''' be compatible at this time. (It will panic!)

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

= Note: Missing branches =

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

# cd $your-zfs-source-area
# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot --with-spl=../spl
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds
# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Edit Boot.plist ====

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two changes are needed:

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

And if needed, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
-kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly’’’ option to avoid permanent changes.
# sudo bless --device disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

Booting on ESX, I hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH /Volumes/OS\ X\ Recovery/ /Volumes/boot/
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Recovery 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with no com.apple.boot.R:
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device disk1s3 --setBoot

Boot.plist

2016-07-06T13:01:16Z

Evansus: hmm, meant to use capital B

Boot.plist

==== com.apple.Boot.plist ====

Use these sample Boot.plist files to enable [[ZFS on Boot]] or [[ZVOL boot]]

===== Verbose boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Single-user boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v -s debug=0x144 zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Debug boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v io=0xffffffff debug=0x144 keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Standard boot with kext symbols =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Standard boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

ZVOL boot

2016-07-06T12:57:55Z

Evansus:

ZVOL boot

2016-07-06T12:51:01Z

Evansus: Created page with "=== ZVOL boot === Using the same steps to create the boot helper partition as ZFS on Boot, you may install Mac OS X onto a ZVOL device and use it as your boot volume. ==..."

=== ZVOL boot ===

Using the same steps to create the boot helper partition as [[ZFS on Boot]], you may install Mac OS X onto a ZVOL device and use it as your boot volume.

==== Create zvol ====
# zfs create -sV 80g rpool/ZVOL/MacOS

Get the diskN for that device
# zvol=`ioreg -trn "ZVOL rpool/ZVOL/MacOS Media" | grep "BSD Name" | awk -F'"' '{print $4}'`
(it would be nice if we could just # zfs get zvol-disk rpool/ZVOL/MacOS)

Erase and create a volume
# diskutil eraseDisk JHFS+ MacOS ${zvol}

==== Clone root ====
Clone an existing Mac OS
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" "/Volumes/Macintosh HD/" "/Volumes/MacOS/"

Or use Apple System Restore
# asr --source "/Volumes/Macintosh HD" --target "/Volumes/MacOS" --erase

==== New install ====
It is possible, but requires a few steps, to install Mac OS onto a ZVOL with the standard Apple installer.
Download the installer from the App Store, and create an installer flash drive.
Boot into the Installer, with a standard Mac OS disk available '''Macintosh HD''' (that has ZFS installed).

From '''Utilities->Terminal''' you can:
# kextload "/Volumes/Macintosh HD/Library/Extensions/spl.kext"
# kextload "/Volumes/Macintosh HD/Library/Extensions/zfs.kext"

At that point you need access to the zpool and zfs commands, which require the libzfs and libzpool libraries. To work around the install environment, simply:
# chroot "/Volumes/Macintosh HD"
# zpool import
Or if necessary
# /usr/local/sbin/zpool import

This worked with El Capitan Install 10.11 and the signed O3x from [[Downloads]].
I don't remember which version of O3x, but I believe it was 1.4.5 or so.

==== Setup boot helper ====
See [[Boot.plist]] for sample plist files.

Boot.plist

2016-07-06T12:23:21Z

Evansus: Add links to ZFS on Boot and ZVOL Boot

Boot.plist

==== com.apple.Boot.plist ====

Use these sample Boot.plist files to enable [[ZFS on Boot]] or [[ZVOL Boot]]

===== Verbose boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Single-user boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v -s debug=0x144 zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Debug boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v io=0xffffffff debug=0x144 keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Standard boot with kext symbols =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Standard boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

Boot.plist

2016-07-06T12:17:27Z

Evansus: Provide several sample Boot.plist files

Boot.plist

==== com.apple.Boot.plist ====

===== Verbose boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Single-user boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v -s debug=0x144 zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Debug boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>-v io=0xffffffff debug=0x144 keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Standard boot with kext symbols =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>

===== Standard boot =====
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Kernel Flags</key>
<string>zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>
</plist>