So I've experimented with ZFS on Mac off and on, but never actually had a completely ideal storage setup for seriously using it. However I'm now looking at upgrading either to one of the new Mac Minis, or the new Mac Pro; obviously the Mac Pro is preferred for ECC, but I'm wary that Apple's going to punch me in the guts on price here in the UK, especially when entry level Mac Pros have never been good value for money (not that the Mac Minis are great either, the non-upgradeable storage costs are insane!)
Anyway, now that I'm looking at ZFS seriously again, I've a few questions, and am hoping for some recommendations.
Encryption
First question is; what is the current state of native encryption? I was very excited when I heard that this feature was getting close; while I'm not looking to upgrade for a few months yet, I'm curious how production ready the feature is considered to be, as it would be a huge boon to be able to eliminate the need to layer ZFS onto Core Storage, with all the headaches that entails!
Secondly on encryption, but what is the best way to automatically decrypt volumes? Since my system volume will be encrypted (and probably only bootable using my admin account), I'm happy for my ZFS devices to be unlocked automatically; does OpenZFS support doing this using the system keychain? If not, what are the best options for auto-mounting encrypted ZFS volumes on a Mac?
Setup
My intention is to use the new machine's fast internal storage for my system volume, which I'm probably just going to leave as APFS to take advantage of hardware encryption, and because bootable ZFS sounds like a lot of added complexity. I'm looking to get around a 1tb internal drive, but may only allocate around half of that at most for the system volume (probably less), leaving the rest of the drive available as a cache for ZFS.
I will then have an external array for main storage, and am currently planning to use four 4tb HDDs, probably WD Reds or similar. I'll be moving all my main data onto this volume, which includes a variety of content including raw HD video (possibly 4k in future); while performance isn't strictly a concern (I'm not really a professional video editor, I just work with it sometimes) more is always better.
I'll also have a second external array for backup, likely also four disks for now but with higher capacity and performance less of a concern (WD Greens or similar maybe?). My intention is to create an HFS+ formatted zvol so that Time Machine can backup the system volume, and then I'll use snapshots and zfs send to backup my main storage (my plan is to use a script so that I can ensure that both don't run at the same time, but instead one after the other).
Does this seem reasonable/sensible?
I'm interested in recommendations for how I should setup the drives for each array. For example, I was thinking of having my main storage array be four drives in two mirrored pairs, since this should keep performance high while giving me a minimum of one disk's worth of redundancy, but I'm a little wary about this, as I'm given to understand that with drive capacities being what they are, it may not be possible to guarantee an error doesn't occur while replacing a failed drive, so should I be aiming for more redundancy? In what configuration? Would five drives be significantly better?
For the backup drive I've been considering whether to use RAID-Z2 as, while this will still cut the storage in half, it should give a full two disks of redundancy, is this worth doing? For my backup volume I'm not so concerned about performance. My intention is to setup an HFS+ formatted zvol for Time Machine to backup my system volume, and then use snapshots and ZFS send to backup my main storage array. In the off chance that Time Machine is updated to run on APFS, how is this likely to impact a zvol (i.e- is APFS on a zvol counter-productive since both support snapshots and copy-on-write)?
In terms of upgrading, can I just swap drives out for larger ones (i.e- swap one, wait for array to rebuild, then swap another and so-on till all are the same)? I'm not really looking to add disks to the arrays, they'll probably be four or five bays, six max.
Lastly, I'm interested in recommendations on how to setup caching; like I say I'm aiming for 1tb for the internal SSD, and shouldn't need loads of space for the system volume (I think 256gb should be plenty, as a I rule I tend not to install apps into /Applications, but instead into per user ~/Applications folders instead, wherever possible). Also I'm curious what the best way to handle caching will be; for example, if my ZFS devices are encrypted, is it safe to just use an internal, unencrypted partition, or could/should I use APFS volumes somehow for this? Is it worth using write caching (zlog device I think) for either of my ZFS devices? In the off chance that Apple doesn't price the Mac Pro completely outside my budget, I'll likely get it with two internal SSDs (assuming Apple enables bootable APFS RAID-1), how would this change how I should configure caching?
Sorry if some of these questions seem obvious, also there's a lot of them and some are very open-ended; I'm very, very far from an expert, I'd consider myself more of an enthusiast in that I know various things out of interest and understand them in theory, but since a lot of things presumably won't be easy to change, it's important I have as clear a plan as I can make