Editing ZFS on Boot

Jump to: navigation, search

Warning: You are not logged in.

Your IP address will be recorded in this page's edit history.
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
Latest revision Your text
Line 5: Line 5:
 
   Most of the commands listed below must be run as root or with '''sudo'''.
 
   Most of the commands listed below must be run as root or with '''sudo'''.
  
=== SIP ===
+
Because you have to compile own kexts, they will not be signed. SIP needs to be disabled for Kext signing to work.
before running the bless command you may have to disable System Integrity Protection [SIP]:
+
The next OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.
boot into recovery mode by holding '''Cmd+R''', open Utilities > Terminal and run
+
    csrutil disable
+
then reboot; to re-enable SIP boot into recovery mode by holding '''Cmd+R''', open Utilities > Terminal and run
+
    csrutil enable
+
 
+
+
'''Updated for macOS Catalina '''
+
 
+
This guide assumes you have installed the latest Open ZFS on OS X on the system, both
+
to create and populate the '''rpool''' with data, but that the data copied will have the
+
kexts and prelinked kernels containing OpenZFS kexts.
+
 
+
=== Install OpenZFS ===
+
 
+
Download latest OpenZFSonOsX pkg and install.
+
 
+
=== Partitioning disk ===
+
 
+
You need generally a 200MB EFI partition, your ZFS partition, then finally a 200MB Apple boot partition.
+
 
+
''In this example, $NEWDISK is the blank disk to use; set it now''
+
 
+
(Change /dev/disk1 to your blank disk!)
+
# export NEWDISK=/dev/disk1           
+
# echo $NEWDISK
+
/dev/disk1
+
 
+
Default '''diskutil''' partitions appear to work:
+
 
+
# diskutil partitionDisk $NEWDISK GPT ZFS rpool 0b
+
(You can ignore this error: )
+
Could not mount disk1s2 after erase
+
Error: -69832: File system formatter failed
+
+
/dev/disk1 (internal, physical):
+
  #:                      TYPE NAME                    SIZE      IDENTIFIER
+
  0:      GUID_partition_scheme                        *64.4 GB    disk1
+
  1:                        EFI EFI                    209.7 MB  disk1s1
+
  2: FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF              64.1 GB    disk1s2
+
  3:                Apple_Boot Boot OS X              134.2 MB  disk1s3
+
 
+
=== Setting up ZFS slice 2 ===
+
 
+
Create the actual pool:
+
 
+
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
+
  -O normalization=formD -O atime=off -O compression=zstd \
+
  -O mountpoint=none -O canmount=off rpool ${NEWDISK}s2
+
 
+
Create minimal datasets for ROOT
+
 
+
# zfs create -o mountpoint=none -o canmount=off rpool/ROOT
+
# zfs create -o mountpoint=legacy -o com.apple.mimic=hfs rpool/ROOT/Catalina
+
# zpool set bootfs=rpool/ROOT/Catalina rpool
+
 
+
Optional, create further datasets, perhaps for /Users, and/or, individual users...
+
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
+
# sudo zfs create rpool/HOME/username
+
 
+
Populate the rpool from the running system:
+
 
+
# mkdir /Volumes/ZFSBoot
+
# /Library/Filesystems/zfs.fs/Contents/Resources/mount_zfs rpool/ROOT/Catalina /Volumes/ZFSBoot
+
# rsync -axH --progress --exclude=".Spotlight-V100" --exclude=".fseventsd" --exclude=".vol" / /Volumes/ZFSBoot/
+
 
+
.. and wait.
+
 
+
(It would be neat if we could (ab)use the installer for the OS: ie some version of
+
installer -pkg [path to]/Install macOS Catalina.app/Contents/SharedSupport/InstallInfo.plist -target /Volumes/ZFSBoot -verbose -dumplog
+
but possibly it always insists on partitioning.)
+
 
+
Update the Boot.plist:
+
 
+
# vim /Volumes/ZFSBoot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
+
+
<dict>
+
        <key>Kernel Flags</key>
+
        <string>-v keepsyms=y zfs_boot=rpool</string>
+
        <key>Root UUID</key>
+
        <string>5A465320-626F-6F74-2064-657669636500</string>
+
</dict>
+
 
+
(The uuid doesnt seem to matter, it just needs to be formatted valid, you can copy the one in the docs, it has nothing to do with your devices.)
+
 
+
=== Setting up boot slice 3 ===
+
 
+
These steps require slice 2 copy to have finished.
+
 
+
# newfs_hfs -J -v "boot" ${NEWDISK}s3
+
# diskutil mount ${NEWDISK}s3
+
# mkdir -p /Volumes/boot/System/Library/CoreServices
+
# mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
+
# mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
+
# mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386
+
 
+
Populate boot:
+
 
+
# rsync -a /Volumes/ZFSBoot/System/Library/CoreServices/PlatformSupport.plist \
+
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
+
# rsync -a /Volumes/ZFSBoot/System/Library/CoreServices/SystemVersion.plist \
+
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
+
# rsync -a /Volumes/ZFSBoot/System/Library/PrelinkedKernels/prelinkedkernel \
+
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
+
# rsync -a /Volumes/ZFSBoot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
+
/Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
+
# rsync -a /Volumes/ZFSBoot/usr/standalone/i386/ \
+
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/
+
 
+
Bless the volume:
+
 
+
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
+
  --file /Volumes/boot/System/Library/CoreServices/boot.efi \
+
  --bootefi /Volumes/ZFSBoot/System/Library/CoreServices/boot.efi \
+
  --label "ZFS Boot"
+
 
+
This step does not change the current boot device, unless you add ''--setBoot''. See blessing the boot device to change the startup device.
+
Another useful argument is ''--nextonly'' to just affect one boot, which is great for testing.
+
To get out of trouble, remember that you can
+
hold `Alt` during boot to select disk to boot from.
+
 
+
 
+
=== Booting ZFS ===
+
 
+
Finally, tell your Mac to use the boot helper on boot. You may want to try the ''--nextonly'' option to avoid permanent changes.
+
 
+
# sudo bless --device ${NEWDISK}s3 --setBoot
+
 
+
( if you get an error like '''Could not set boot device property: 0xe00002bc''' you should disable SIP, see [[#SIP|SIP]] for instructions)
+
 
+
# reboot
+
 
+
( you can re-enable SIP now, see [[#SIP|SIP]] for instructions)
+
 
+
Additional tricks. If updating the kexts, and you want to update the boot volume as well:
+
 
+
Mount and update ZFSBoot
+
# mkdir /Volumes/ZFSBoot
+
# /Library/Filesystems/zfs.fs/Contents/Resources/mount_zfs rpool/ROOT/Catalina /Volumes/ZFSBoot
+
+
If you prefer to update the kexts in /Volumes/ZFSBoot/Library/Extensions/
+
# kextcache -c "/Volumes/ZFSBoot/System/Library/PrelinkedKernels/prelinkedkernel" -K "/Volumes/ZFSBoot/System/Library/Kernels/kernel" -l -- "/Volumes/ZFSBoot/System/Library/Extensions"  "/Volumes/ZFSBoot/Library/Extensions/"
+
+
Or if you install new kexts on machine, and then update /Volumes/ZFSBoot/
+
# rsync -ar /Library/Extensions/ /Volumes/ZFSBoot/Library/Extensions/
+
# kextcache -c "/Volumes/ZFSBoot/System/Library/PrelinkedKernels/prelinkedkernel" -K "/System/Library/Kernels/kernel" -l -- "/System/Library/Extensions"  "/Library/Extensions/"
+
 
+
Also update the tiny boot volume - find the disk it is on, in example, disk1s3. Pick kernel either on "/" or "/Volumes/ZFSBoot/"
+
# diskutil mount disk1s3
+
# mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/Caches/com.apple.kext.caches/Startup
+
# kextcache -c "/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel" -K "/System/Library/Kernels/kernel" -l -- "/System/Library/Extensions" "/Library/Extensions/"
+
 
+
----
+
 
+
''' The remaining text is the old guide. '''
+
 
+
----
+
 
+
Because you have to compile your own KEXTs, they will not be signed. [https://support.apple.com/en-us/HT204899 System Integrity Protection] (SIP) needs to be disabled for unsigned KEXTs to work.
+
A future version of OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.
+
  
 
=== Disk Layout ===
 
=== Disk Layout ===
Line 195: Line 36:
 
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
 
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
 
  # sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
 
  # sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
   -O normalization=formD -O atime=off -O compression=zstd \
+
   -O normalization=formD -O atime=off -O compression=lz4 \
 
   -O mountpoint=none -O canmount=off rpool disk1s2
 
   -O mountpoint=none -O canmount=off rpool disk1s2
 +
 +
This step (optional) will relabel the disks using InvariantDisks paths (by UUID), so they can be located at boot time.
 +
# sudo zpool export rpool
 +
# sudo zpool import -d /var/run/disk/by-id rpool
 +
 +
If /dev/disk paths are used create the pool, it will import at boot time using /var/run/disk/by-id/media-UUID InvariantDisk paths.
  
 
==== Root dataset ====
 
==== Root dataset ====
Line 205: Line 52:
 
  # sudo zpool set bootfs=rpool/ROOT/Capitan rpool
 
  # sudo zpool set bootfs=rpool/ROOT/Capitan rpool
  
Note: com.apple.mimic=hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.
+
Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.
  
 
You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
 
You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
Line 217: Line 64:
 
==== Install macOS ====
 
==== Install macOS ====
 
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
 
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
  # sudo rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
+
  # rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
 
   --exclude=".vol" / /Volumes/Capitan/
 
   --exclude=".vol" / /Volumes/Capitan/
  
 
==== Install ZFS ====
 
==== Install ZFS ====
  
Boot support is now included with the binary install versions, so you don't have to compile it yourself.  
+
# sudo mkdir -p /Volumes/Capitan/usr/local
 +
# cd /Volumes/Capitan/usr/local
 +
# sudo ln -s ../../sbin sbin
 +
 +
===== Install ZFS with own source repository =====
  
In you haven't done so already, install homebrew (or macports, but in the example we will stick to homebrew as it's quicker to set-up):
+
  # cd $your-zfs-source-area
  # /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
+
  
Install automake and libtool (required to compile spl and zfs):
+
===== Install ZFS by getting sources =====
# brew install automake libtool
+
  
Create target directory for installation:
+
  # Insert lines here, git clone, etc
  # sudo mkdir -p /Volumes/Capitan/usr/local
+
  
Use your own repository:
+
===== Install ZFS continued ... =====
# cd $your-zfs-source-area
+
Or clone it one from github:
+
# git clone https://github.com/openzfsonosx/spl.git
+
# git clone https://github.com/openzfsonosx/zfs.git
+
  
Build and install spl:
 
 
  # cd spl
 
  # cd spl
 
  # git fetch --all
 
  # git fetch --all
  # ./autogen.sh
+
# git checkout spl-boot
 +
  # ./autogen
 
  # ./configure --enable-boot
 
  # ./configure --enable-boot
 
  # make
 
  # make
 
  # sudo make install DESTDIR=/Volumes/Capitan
 
  # sudo make install DESTDIR=/Volumes/Capitan
 
Build and install zfs:
 
 
 
  # cd ../zfs
 
  # cd ../zfs
 
  # git fetch --all
 
  # git fetch --all
  # ./autogen.sh
+
# git checkout ldi-boot
 +
  # ./autogen
 
  # ./configure --enable-boot  
 
  # ./configure --enable-boot  
 
  # make
 
  # make
 
  # sudo make install DESTDIR=/Volumes/Capitan
 
  # sudo make install DESTDIR=/Volumes/Capitan
 +
 +
 +
Or use a stage directory for zfs builds, that can also be done.
 +
 +
# make install DESTDIR=/stage_dir
 +
# rsync -acH /stage_dir/ /Volumes/Capitan/
 +
  
 
==== Edit Boot.plist ====
 
==== Edit Boot.plist ====
Line 306: Line 155:
 
   /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
 
   /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
 
  # sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
 
  # sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
   /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
+
   /Volumes/boot/com.apple.boot.R/Library/Preferences/com.apple.Boot.plist
 
  # sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
 
  # sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
 
   /Volumes/boot/com.apple.boot.R/usr/standalone/i386/
 
   /Volumes/boot/com.apple.boot.R/usr/standalone/i386/
Line 315: Line 164:
 
You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
 
You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
 
  # sudo bless --folder /Volumes/boot/System/Library/CoreServices \
 
  # sudo bless --folder /Volumes/boot/System/Library/CoreServices \
   --file /Volumes/boot/System/Library/CoreServices/boot.efi \
+
   --file /Volumes/boot/System/Library/CoreServices \
 
   --bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
 
   --bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
 
   --label "ZFS Boot"
 
   --label "ZFS Boot"
Line 324: Line 173:
 
Generate caches for '''kernel'''
 
Generate caches for '''kernel'''
  
  # sudo kextcache -arch x86_64 -local-root -volume-root /Volumes/Capitan \
+
  # kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
   -kernel /Volumes/Capitan/System/Library/Kernels/kernel \
+
   -kernel /Volumes/rpool/System/Library/Kernels/kernel \
   -prelinked-kernel /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
+
   -prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel \
   -- /Volumes/Capitan/System/Library/Extensions /Volumes/Capitan/Library/Extensions /Volumes/boot/Library/Extensions
+
   -- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions
 
+
If you get a symlink error here like;
+
symlink("/Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel", "/Volumes/Capitan/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache") failed 17 (File exists) <createPrelinkedKernel 2795>
+
 
+
Just delete the symlink, and run again, ie;
+
rm /Volumes/Capitan/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache
+
 
+
  
 
For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''
 
For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''
  
  # kextcache -arch x86_64 -local-root -volume-root /Volumes/Capitan \
+
  # kextcache -arch x86_64 -local-root -volume-root /Volumes/rpool \
   -kernel /Volumes/Capitan/System/Library/Kernels/kernel.development \
+
   -kernel /Volumes/rpool/System/Library/Kernels/kernel.development \
   -prelinked-kernel /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel.development \
+
   -prelinked-kernel /Volumes/rpool/System/Library/PrelinkedKernels/prelinkedkernel.development \
   -- /Volumes/Capitan/System/Library/Extensions /Volumes/boot/Library/Extensions
+
   -- /Volumes/rpool/System/Library/Extensions /Volumes/boot/Library/Extensions
 
+
==== Copy prelinkedkernel to boot helper ====
+
 
+
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
+
  /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
+
  
 
==== Set the boot device ====
 
==== Set the boot device ====
 
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
 
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
  # sudo bless --device /dev/disk1s3 --setBoot
+
  # sudo bless --device disk1s3 --setBoot
  
 
=== Booting ===
 
=== Booting ===
Line 395: Line 232:
 
It is possible to write a startup.nsh script to do so.
 
It is possible to write a startup.nsh script to do so.
  
[Please provide details on how to disable SIP in startup.nsh]
+
VirtualBox does not like the boot helper as created above. It used to function in a previous version (4.3.x?) but at this time, the easiest option I have found is to clone the contents of BaseSystem.dmg from any MacOS recovery partition.
 
+
VirtualBox does not like the boot helper as created above. There are two options to get a bootable setup:
+
 
+
===== Minimal boot partition for VirtualBox (tested with macOS Sierra) =====
+
# sudo mkdir -p /Volumes/boot//Library/Preferences/SystemConfiguration
+
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices/com.apple.recovery.boot
+
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/autodiskmount.plist /Volumes/boot/Library/Preferences/SystemConfiguration/autodiskmount.plist
+
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
+
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist /Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
+
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist /Volumes/boot/System/Library/CoreServices/SystemVersion.plist
+
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/boot.efi /Volumes/boot/System/Library/CoreServices/boot.efi
+
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/bootbase.efi /Volumes/boot/System/Library/CoreServices/bootbase.efi
+
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/com.apple.recovery.boot/PlatformSupport.plist /Volumes/boot/System/Library/CoreServices/com.apple.recovery.boot/PlatformSupport.plist
+
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel /Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
+
 
+
===== Alternative solution using recovery boot base system image =====
+
 
  # diskutil mount disk0s3
 
  # diskutil mount disk0s3
 
  # hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"
 
  # hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"
Line 418: Line 239:
 
  # sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
 
  # sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
 
Or if you prefer '''asr''':
 
Or if you prefer '''asr''':
  # sudo asr restore -source "/Volumes/OS X Base System/" -target "/Volumes/boot" -erase
+
  # sudo asr restore -source "/Volumes/OS X Recovery" -target "/Volumes/boot" -erase
 
  # diskutil rename "/Volumes/OS X Base System 1" "boot"
 
  # diskutil rename "/Volumes/OS X Base System 1" "boot"
  
Line 433: Line 254:
  
 
Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.
 
Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.
 
''I was unable to boot Sierra on 2016 MacBookPro with "com.apple.boot.R" setup, but the minimal VirtualBox boot partition solution above has worked just fine.''
 
  
 
Hold '''option''' while powering on or restarting your Mac.
 
Hold '''option''' while powering on or restarting your Mac.
Line 470: Line 289:
  
 
This command actually sets the boot helper to be used on reboot:
 
This command actually sets the boot helper to be used on reboot:
  # sudo bless --device /dev/disk1s3 --setBoot --verbose
+
  # sudo bless --device disk1s3 --setBoot
 
+
This will also work, as it will locate the boot helper following the ZFS partition:
+
# sudo bless --device /dev/disk1s2 --setBoot --verbose
+
 
+
Currently --folder and --mount options will work (partially) but may select the wrong boot helper if there are multiple bootable ZFS pools.
+
(For example, if you have ZFS boot setup on both the internal disk and an external disk.)
+
# sudo bless --folder /Volumes/Capitan --setBoot --verbose
+
# sudo bless --mount /Volumes/Capitan --setBoot --verbose
+

Please note that all contributions to OpenZFS on OS X may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see OpenZFS on OS X:Copyrights for details). Do not submit copyrighted work without permission!

Cancel | Editing help (opens in new window)