Difference between revisions of "ZFS on Boot"

From OpenZFS on OS X
Jump to: navigation, search
m
Line 2: Line 2:
 
== ZFS as Boot ==
 
== ZFS as Boot ==
  
−
Status: Not Ready
+
  Status: Not Ready
−
 
+
 
 +
  Most of the commands listed below must be run as root or with '''sudo'''.
  
 
=== Disk Layout ===
 
=== Disk Layout ===
Line 9: Line 10:
 
Play disk: '''disk1'''
 
Play disk: '''disk1'''
  
 +
I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
 +
Create the following partitions on the test boot disk:
  
−
Created the following partition on the test boot disk:
+
  Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
 +
  Number  Start (sector)    End (sector)  Size      Code  Name
 +
  1            2048          411647  200.0 MiB  EF00  EFI System
 +
  2          411648        82200575  39.0 GiB    BF01  rpool
 +
  3        82200576        83886046  823.0 MiB  AB00  Apple boot
  
−
Number  Start (sector)    End (sector)  Size      Code  Name
+
'''s1''' is not really used, but added for EFI.
−
    1            2048          411647  200.0 MiB  EF00  EFI System
+
−
    2          411648        2508799  1024.0 MiB  AF00  Apple HFS/HFS+
+
−
    3        2508800        3737599  600.0 MiB  AB00  Recovery HD
+
−
    4        3737600        83886046  38.2 GiB    BF01  Solaris /usr & Mac ZFS
+
  
−
'''s1''' is not really used, but added for EFI.
+
'''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
−
'''s2''' is the small HFS boot, which will carry the kernel (and the kext caches) to load the kernel with ZFS.
+
−
'''s3''' is currently not used, but created in case we want it in future
+
−
'''s4''' is the ZFS pool used for booting.
+
  
−
=== Setting up boot slice '''s2''' ===
+
'''s3''' is the small HFS boot which will carry the prelinkedkernel to load the kernel with ZFS.
  
−
Regular HFS filesystem, and populated thusly; (Volume named '''boot''' here)
+
The layout need not be exactly this, but:
−
In this case, we have both the '''kernel''' and '''kernel.development'''. But normal users will only have '''kernel'''.
+
−
These are rsynced from the (normal) booted OS.
+
  
−
# diskutil mount disk1s2
+
The EFI partition should always exist at the beginning of the disk.
−
# mdutil -i off /Volumes/boot
+
−
# cd /Volumes boot
+
−
./usr/standalone/i386/boot.efi
+
−
./System/Library/Kernels/kernel
+
−
./System/Library/Kernels/kernel.development
+
−
./System/Library/PrelinkedKernels/prelinkedkernel
+
−
./System/Library/PrelinkedKernels/prelinkedkernel.development
+
−
./System/Library/Caches/com.apple.kext.caches/Startup/IOKitPersonalities_x86_64.ioplist.gz
+
−
./System/Library/Caches/com.apple.kext.caches/Startup/kernelcache
+
−
./System/Library/Caches/com.apple.kext.caches/Startup/kernelcache.development
+
−
./System/Library/Caches/com.apple.kext.caches/Startup/KextPropertyValues_OSBundleHelper_x86_64.plist.gz
+
−
./System/Library/Caches/com.apple.kext.caches/Startup/KextPropertyValues_PGO_x86_64.plist.gz
+
−
./System/Library/CoreServices/boot.efi
+
−
./System/Library/CoreServices/SystemVersion.plist
+
  
−
# cat SystemVersion.plist
+
The Apple Boot (ab00) partition should directly follow the ZFS partition, and while it can be as small as 50 MB, it is better to use 650MB or more (to imitate a Recovery HD).
−
<?xml version="1.0" encoding="UTF-8"?>
+
−
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+
−
<plist version="1.0">
+
−
<dict>
+
−
        <key>ProductBuildVersion</key>
+
−
        <string>15F34</string>
+
−
        <key>ProductCopyright</key>
+
−
        <string>1983-2016 Apple Inc.</string>
+
−
        <key>ProductName</key>
+
−
        <string>Mac OS X</string>
+
−
        <key>ProductUserVisibleVersion</key>
+
−
        <string>10.11.5</string>
+
−
        <key>ProductVersion</key>
+
−
        <string>10.11.5</string>
+
−
</dict>
+
−
</plist>
+
  
−
Now we install the ZFS kexts and rebuild the kernel caches
+
  Note: at this time, the boot helper partition is not updated by the OS
 +
  and must be manually updated. It may be possible to script this in a
 +
  launchd service, or even 'bless' the device so that macOS updates
 +
  it automatically. Additional development is ongoing.
  
−
# mkdir -p /Volumes/boo/Library/Extensions
+
  You cannot use '''Startup Disk''' to select the boot disk, see [[#Blessing the boot device|blessing the boot device]].
−
# cd your-source-directory-for-zfs
+
−
# cd spl/module/spl
+
−
# make install DESTDIR=/Volumes/boot/
+
−
# cd ../../../zfs/module/zfs
+
−
# make install DESTDIR=/Volumes/boot/
+
  
−
Technically, that will also install some extra stuff which isn't needed into the boot slice but that fine.
+
=== Populating the ZFS Root '''s2''' ===
  
−
Generate caches for '''kernel'''
+
==== Root pool ====
 +
Any pool name may be used - '''rpool''' is used here for example.
 +
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
 +
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
 +
  -O normalization=formD -O atime=off -O compression=lz4 \
 +
  -O mountpoint=none -O canmount=off rpool disk1s2
  
−
kextcache -arch x86_64 -local-root -all-loaded -kernel /Volumes/boot/System/Library/Kernels/kernel \
+
This step will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
−
  -prelinked-kernel /Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel \
+
# sudo zpool export rpool
−
  -mkext /Volumes/boot/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache \
+
# sudo zpool import -d /var/run/disk/by-id rpool
−
  -volume-root / /System/Library/Extensions /Library/Extensions /Volumes/boot/Library/Extensions/
+
  
−
And if needed, caches for '''kernel.development'''
+
  Using /dev/disk paths will most likely break!
  
−
kextcache -arch x86_64 -local-root -all-loaded -kernel /Volumes/boot/System/Library/Kernels/kernel.development \
+
==== Root dataset ====
−
   -prelinked-kernel /Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel.development \
+
   Note: com.apple.mimic_hfs is known to '''NOT''' be compatible at this time. (It will panic!)
−
  -mkext /Volumes/boot/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache.development \
+
−
  -volume-root / /System/Library/Extensions /Library/Extensions /Volumes/boot/Library/Extensions/
+
  
−
Here there will be steps for blessing the boot, and setting up EFI etc. But we will use EFI Shell to boot for now.
+
Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
 +
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
 +
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
 +
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool
  
 +
You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
 +
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
 +
# sudo zfs create rpool/HOME/username
  
−
=== Populating the ZFS Boot ===
+
Mount it:
 +
# sudo mkdir /Volumes/Capitan
 +
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan
  
−
Create the pool and datasets
+
==== Install macOS ====
 +
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
 +
# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
 +
  --exclude=".vol" / /Volumes/Capitan/
  
−
# zpool create -f -o ashift=12 -O casesensitivity=insensitive -O normalization=formD -O atime=off -O compression=lz4 rpool disk1s4
+
==== Install ZFS ====
−
# zfs create rpool/ROOT
+
−
# zfs create rpool/ROOT/10.11
+
  
−
Create ROOT and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else
+
  = Note: Missing branches =
  
−
Copy over the OS as you see fit, rsync will generally do. Remember that etc, tmp and var are symlinks into /private 
+
  # sudo mkdir -p /Volumes/Capitan/usr/local
−
 
+
  # cd /Volumes/Capitan/usr/local
−
  # rsync -ar --exclude="/Volumes/rpool/*" / /Volumes/rpool/ROOT/10.11/
+
  # sudo ln -s ../../sbin sbin
−
 
+
−
Install ZFS
+
−
 
+
−
  # cd /Volumes/rpool/ROOT/10.11/
+
−
# mkdir -p usr/local
+
−
# cd usr/local
+
−
  # ln -s ../../sbin sbin
+
 
   
 
   
 
  # cd $your-zfs-source-area
 
  # cd $your-zfs-source-area
 
  # cd spl
 
  # cd spl
−
  # make install DESTDIR=/Volumes/rpool/ROOT/10.11/
+
  # git fetch --all
 +
# git checkout spl-boot
 +
# ./autogen
 +
# ./configure --enable-boot
 +
# make
 +
# sudo make install DESTDIR=/Volumes/Capitan
 
  # cd ../zfs
 
  # cd ../zfs
−
  # make install DESTDIR=/Volumes/rpool/ROOT/10.11/
+
  # git fetch --all
−
  # cd /Volumes/rpool/ROOT/10.11/
+
# git checkout ldi-boot
−
  # cd sbin
+
# ./autogen
−
  # mv fsck fsck_real
+
  # ./configure --enable-boot --with-spl=../spl
−
# cp ../usr/bin/true fsck
+
  # make
 +
  # sudo make install DESTDIR=/Volumes/Capitan
  
−
When ready, the bootfs needs to be changed to legacy
+
Or use a stage directory for zfs builds
 +
# make install DESTDIR=/stage_dir
 +
# rsync -acH /stage_dir/ /Volumes/Capitan/
  
−
# zfs set mountpoint=legacy rpool/ROOT/10.11
+
=== Setting up boot slice '''s3''' ===
  
−
If you want to change it in future, use inherit to mount it back.
+
==== Format ====
 +
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
 +
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.
  
−
  # zfs inherit mountpoint rpool/ROOT/10.11
+
  # newfs_hfs -J -v "boot" /dev/disk1s3
 +
# diskutil mount disk1s3
 +
# sudo mdutil -i off /Volumes/boot
  
−
Once done, remember to change it back to legacy
+
==== Directories ====
 +
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
 +
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
 +
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
 +
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386
  
−
=== Hardcoded magic ===
+
==== Required Files ====
 +
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
 +
  /Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
 +
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
 +
  /Volumes/boot/System/Library/CoreServices/SystemVersion.plist
 +
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
 +
  /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
 +
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
 +
  /Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
 +
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
 +
  /Volumes/boot/com.apple.boot.R/usr/standalone/i386/
  
−
  # git checkout zfs_boot
+
==== Bless the volume ====
−
  # vi module/zfs/zfs_vfsops.c
+
The boot helper needs to know where to load boot.efi from.
−
+      if (error = zfs_domount(mp, rdev, "rpool/ROOT/10.11", ctx)) {
+
 
−
  +      vfs_mountedfrom(mp, "/dev/disk1s4");
+
You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
 +
  # sudo bless --folder /Volumes/boot/System/Library/CoreServices \
 +
  --file /Volumes/boot/System/Library/CoreServices \
 +
  --bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
 +
  --label "ZFS Boot"
 +
 
 +
This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.
 +
 
 +
==== Edit Boot.plist ====
 +
 
 +
More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]
 +
 
 +
  # sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
 +
 
 +
Two changes are needed:
 +
 
 +
Look for
 +
  <key>Kernel Flags</key>
 +
  <string></string>
 +
 
 +
Replace with:
 +
  <key>Kernel Flags</key>
 +
  <string>-v keepsyms=y zfs_boot=rpool</string>
 +
  <key>Root UUID</key>
 +
  <string>00000000-0000-0000-0000-000000000000</string>
 +
 
 +
Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").
 +
 
 +
See [[Boot.plist]] for more examples.
 +
 
 +
[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.
 +
 
 +
==== Generate prelinkedkernel ====
 +
Generate caches for '''kernel'''
 +
 
 +
# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
 +
  -kernel /Volumes/rpool/System/Library/Kernels/kernel \
 +
  -prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
 +
  -- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions
 +
 
 +
And if needed, caches for '''kernel.development'''
 +
 
 +
  # kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
 +
  -kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
 +
  -prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
 +
  -- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions
 +
 
 +
==== Set the boot device ====
 +
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly’’’ option to avoid permanent changes.
 +
# sudo bless --device disk1s3 --setBoot
  
 
=== Booting ===
 
=== Booting ===
 +
==== Booting on ESX ====
  
 
Booting on ESX, I hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered
 
Booting on ESX, I hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered
Line 162: Line 208:
 
  fs4: cd Library
 
  fs4: cd Library
 
  fs4: cd CoreServices
 
  fs4: cd CoreServices
−
  fs4: boot.efi rd=disk1s4 -s
+
  fs4: boot.efi
 +
 
 +
Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
 +
 
 +
==== Booting on VirtualBox ====
 +
 
 +
SIP can be disabled in VirtualBox, but must be disabled before every boot.
 +
One way to do this is to install '''rEFInd''' into the EFI partition.
 +
It is possible to write a startup.nsh script to do so.
 +
 
 +
VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
 +
# diskutil mount disk0s3
 +
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"
 +
 
 +
Then
 +
# sudo rsync -axH /Volumes/OS\ X\ Recovery/ /Volumes/boot/
 +
Or if you prefer '''asr''':
 +
# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
 +
# diskutil rename "/Volumes/OS X Recovery 1" "boot"
 +
 
 +
And the paths used above will change, as we are using a “real” disk layout with no com.apple.boot.R:
 +
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
 +
  /Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
 +
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
 +
(See [[#com.apple.Boot.plist|Boot.plist]])
 +
 
 +
==== Booting on real Macs ====
 +
 
 +
  Always make extra backups!
 +
  Test this out using flash drives first!
 +
 
 +
Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.
 +
 
 +
Hold '''option''' while powering on or restarting your Mac.
 +
 
 +
Look for '''ZFS Boot''' as set by bless --label above, and double click.
 +
 
 +
ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.
 +
 
 +
If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.
 +
 
 +
After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.
 +
 
 +
If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.
 +
 
 +
=== Additional ===
 +
 
 +
==== Mounting other boot environments ====
  
−
Which runs (boots) boot.efi, using rootdev (rd=) as disk1s4 (ZFS Boot). We use '''-s''' to go into single user, needed for now.
+
# sudo zfs snapshot rpool/ROOT/Capitan1@updating
 +
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
 +
# mkdir /Volumes/Capitan2
 +
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2
  
−
SingleUser Mode Entered
+
Later you can promote
−
  Issue
+
  # sudo zfs promote rpool/ROOT/Capitan2
−
fsck
+
Then if you want
−
  mount -uw /
+
  # sudo zfs destroy -r rpool/ROOT/Capitan
−
then exit to boot
+
  
−
We need to remount "/" to be read-write.
+
==== Blessing the boot device ====
  
−
  /sbin/mount_zfs -u /dev/disk1s4 /
+
Need only be run once, to setup the '''boot''' volume:
−
exit
+
  # sudo bless --folder /Volumes/boot/System/Library/CoreServices \
 +
  --file /Volumes/boot/System/Library/CoreServices/boot.efi \
 +
  --bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi
  
−
Boot resumes, maybe.
+
This command actually sets the boot helper to be used on reboot:
 +
# sudo bless --device disk1s3 --setBoot

Revision as of 13:18, 6 July 2016

ZFS as Boot

  Status: Not Ready
  
  Most of the commands listed below must be run as root or with sudo.

Disk Layout

Play disk: disk1

I used gptfdisk, but gpt and diskutil can be used also. Create the following partitions on the test boot disk: 

  Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
  Number  Start (sector)    End (sector)  Size       Code  Name
  1            2048          411647   200.0 MiB   EF00  EFI System
  2          411648        82200575   39.0 GiB    BF01  rpool
  3        82200576        83886046   823.0 MiB   AB00  Apple boot

s1 is not really used, but added for EFI.

s2 is the ZFS pool used for booting. Note: You are not limited to using the name rpool.

s3 is the small HFS boot which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

The EFI partition should always exist at the beginning of the disk.

The Apple Boot (ab00) partition should directly follow the ZFS partition, and while it can be as small as 50 MB, it is better to use 650MB or more (to imitate a Recovery HD). 

  Note: at this time, the boot helper partition is not updated by the OS
  and must be manually updated. It may be possible to script this in a
  launchd service, or even 'bless' the device so that macOS updates
  it automatically. Additional development is ongoing.

  You cannot use Startup Disk to select the boot disk, see blessing the boot device.

Populating the ZFS Root s2

Root pool

Any pool name may be used - rpool is used here for example. Additional research is needed on atime, casesensitivity, and normalization (and probably more). 

# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
 -O normalization=formD -O atime=off -O compression=lz4 \
 -O mountpoint=none -O canmount=off rpool disk1s2

This step will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time. 

# sudo zpool export rpool
# sudo zpool import -d /var/run/disk/by-id rpool

  Using /dev/disk paths will most likely break!

Root dataset

  Note: com.apple.mimic_hfs is known to NOT be compatible at this time. (It will panic!)

Create ROOT and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate). 

# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility. 

# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it: 

# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

Install macOS

Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private) 

# rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
 --exclude=".vol" / /Volumes/Capitan/

Install ZFS

  = Note: Missing branches =

# sudo mkdir -p /Volumes/Capitan/usr/local
# cd /Volumes/Capitan/usr/local
# sudo ln -s ../../sbin sbin

# cd $your-zfs-source-area
# cd spl
# git fetch --all
# git checkout spl-boot
# ./autogen
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan
# cd ../zfs
# git fetch --all
# git checkout ldi-boot
# ./autogen
# ./configure --enable-boot --with-spl=../spl
# make
# sudo make install DESTDIR=/Volumes/Capitan

Or use a stage directory for zfs builds 

# make install DESTDIR=/stage_dir
# rsync -acH /stage_dir/ /Volumes/Capitan/

Setting up boot slice s3

Format

Journaled HFS+ filesystem, and populated thusly; (Volume named boot here) These can be rsynced from the (normal) booted OS, or from the Capitan mount. 

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

Directories

# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

Required Files

# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
 /Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
 /Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
 /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
 /Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
 /Volumes/boot/com.apple.boot.R/usr/standalone/i386/

Bless the volume

The boot helper needs to know where to load boot.efi from.

You may use any label in place of ZFS Boot here, it will appear at the option-boot screen. 

# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
  --file /Volumes/boot/System/Library/CoreServices \
  --bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
  --label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See blessing the boot device to change the startup device.

Edit Boot.plist

More examples of the Boot.plist can be found at com.apple.Boot.plist 

# sudo vim /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two changes are needed:

Look for 

  <key>Kernel Flags</key>
  <string></string>

Replace with: 

  <key>Kernel Flags</key>
  <string>-v keepsyms=y zfs_boot=rpool</string>
  <key>Root UUID</key>
  <string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See Boot.plist for more examples.

ZVOL boot from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

Generate prelinkedkernel

Generate caches for kernel 

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
  -kernel /Volumes/rpool/System/Library/Kernels/kernel \
  -prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
  -- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

And if needed, caches for kernel.development 

# kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
  -kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
  -prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
  -- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions

Set the boot device

Finally, tell your Mac to use the boot helper on boot. You may want to try the --nextonly’’’ option to avoid permanent changes. 

# sudo bless --device disk1s3 --setBoot

Booting

Booting on ESX

Booting on ESX, I hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered 

# map

will list all available maps. For me, 

fs0 disk0 EFI partition
fs1 disk0 OSX boot
fs2 disk0 OSX Recovery
fs3 disk1 EFI
fs4 disk1 boot HFS
fs5 disk1 recovery
fs6 disk1 ZFS

So in my case 

EFI> fs4:
fs4: ls
usr System Library
fs4: cd System 
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.

Booting on VirtualBox

SIP can be disabled in VirtualBox, but must be disabled before every boot. One way to do this is to install rEFInd into the EFI partition. It is possible to write a startup.nsh script to do so.

VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition. 

# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then 

# sudo rsync -axH /Volumes/OS\ X\ Recovery/ /Volumes/boot/

Or if you prefer asr: 

# sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Recovery 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with no com.apple.boot.R: 

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
 /Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

(See Boot.plist)

Booting on real Macs

  Always make extra backups!
  Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or asr and edit the [com.apple.Boot.plist] to put a different pool name in zfs_boot in the Kernel flags.

Hold option while powering on or restarting your Mac.

Look for ZFS Boot as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in zfs_boot above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the Capitan dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the option key. Select the partition with standard MacOS to reboot normally. Once booted, check and set System Preferences->Startup Disk.

Additional

Mounting other boot environments

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote 

# sudo zfs promote rpool/ROOT/Capitan2

Then if you want 

# sudo zfs destroy -r rpool/ROOT/Capitan

Blessing the boot device

Need only be run once, to setup the boot volume: 

# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
  --file /Volumes/boot/System/Library/CoreServices/boot.efi \
  --bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot: 

# sudo bless --device disk1s3 --setBoot
Retrieved from "https://openzfsonosx.org/w/index.php?title=ZFS_on_Boot&oldid=1078"