ZFS boot questions?

[znubee]

I saw from a thread that you would prefer us to carry discussions over to the forum here for booting to ZFS. I also see that it may not be the best idea yet. I should be able to handle problems that arise, but I would like some advice here.

I have a system with some drives that I can split up. For example, two are SSD drives and the third set of drives I want to put together into a ZRAID. I hate to do it, but I can put the OS on HFS+ and I can put everything else onto ZFS volumes.

Is there a preferred way of mapping User folders and such in this scenario that will minimize conflicts?

Or should I just go balls to the wall and do ZFS on boot? I am a developer so I can work through issues, but I do use this for work and want to minimize headaches.

[Brendon]

@znubee

The process for setting up ZFS boot is documented at https://openzfsonosx.org/wiki/ZFS_on_Boot. To my knowledge there has only been 2-3 people that have tried this, including the developer. There are definitely risks and limitations, most of them probably unknown at this time.

ZFS boot is not generally considered to be mature.

- Brendon

[lundman]

Pretty much yep.

If you want to play, then play. Run a VM, see how far you can get.

But putting it as boot on hardware that you intend to use? Not quite yet. Too many missing features for it to actually handle all the little finesses in OSX.

[znubee]

Thanks guys. I'll steer clear of bootable for the moment on a production system.

Let me ask you for your input here then given my use case:

I have 1 very fast 512 GB SSD (M.2), 1 2TB SSD, 1 200GB SSD (will be free after I dismantle a CoreStorage drive), and 4x6 TB traditional drives with 48 GB RAM. I would like to use RAIDZ(5) for the 4 drives, but I'm thinking that I might want to do something with ZIL and LARC with the other drives, BUT I don't want to consume too much RAM with ZFS so I guess I need to be careful with LARC.

I really want to not worry about HFS+ corruption anymore when things go south after kext fiddling or such, but I also want max performance.

What would be your recommendations with such a configuration for a video production and development workstation?

[tangles]

stick with HFS for boot, u can't afford the down time if this video production rig is earning u revenue.

Personally, I'd switch to a mirror for the rotationals instead of raidz, (especially with 4 disks).

U haven't mentioned what codec u author in, ProRes?

I'm not sure that any ZIL/Cache would be all that advantageous with large video files, as you'd think you'd be issuing large contiguous read/writes versus small (random) read writes that love ZIL/cache.

Mind you, get a project off the spindles and into cache ( or better ARC) would be ideal, so I'd still be going with copious RAM over ZIL/Cache options for video workflows...

[znubee]

Why mirror instead of RAIDZ?

Just m4v for archival and upload of scraped imagery turned to footage and mixed with other footage. I also will be storing a lot of smaller files on this that are from scraping for database entry and so forth (probably about 1/4 of drive for that; another 1/4 for largish images) so it has a bit of a mixed usage.

[tangles]

Why mirror instead of RAIDZ?

You mentioned a heap of SSDs and flash storage at the start of your post, which I (may've mistakenly) interpreted as you liking your rigs speedy... You've since mentioned archival purposes, (instead of video production) so I'm not really sure what you're after now...

Here's what I do know:

With ZFS, the more vdevs you have within a pool, the more speed you'll get, especially for reads...
So a RAIDZ(5) as you put it, which in ZFS terms is defined as RAIDZ1 (not advisable with even numbered physical disks from memory) will only give you one vdev, where as a mirror X 2 (or striped-mirror known as a RAID10 using conventional RAID notation) will give you 2 vdevs.

You'll have n-2 as usable storage instead of n-1 (i.e. 12TB v 18TB) so you will no doubt weigh up what direction you want to go.

I have a 10 disk pool at the moment, 4TB drives mirrored so that gives me 5 vdevs. No ZIL or cache as I too only use it for glorified archival purposes at home. It's fast! and fortunately I have Chelseo 10GB-Ethernet cards to take advantage of it between the server and my main workstation. Everything else on the network is just Gigabit which is plenty.

have a read of https://calomel.org/zfs_raid_speed_capacity.html, it's most interesting!

Once you digest all that, you can then move on to what HBA you should go with...

Like anything, I find: "... The more you learn about something, the less you actually know ..."

Cheers,

[znubee]

So I thought I'd experiment a little here. I went ahead and set up a raidz array and found that the performance wasn't bad when I benchmarked it against the other options. I figure that this will be okay until I get another drive or set up another storage array when the landscape changes in storage in the next year as it appears it will with devices.

I have noticed a problem though. After letting it do some copying over several hours, I found that kernel_task is now using 47GB (64GB configuration) and the copying has slowed down dramatically from the SSD that is sending the data. I am using zfs from master on github. I am also noticing a lot of small intermittent "hangs" that last only a couple to few seconds on cursor movement that coincide with copy pauses. Is there something that could be going wrong to consume so much memory that is currently known?

[znubee]

So I did a little more poking and found that kstat.spl.misc.spl_misc.os_mem_alloc: 44621889536 (about 44GB?). That worries me. Should I set kstat.zfs.darwin.tunable.zfs_arc_max to something more reasonable to address this high consumption? Might that have effect on performance too?

I've gone ahead and set the limit to something that seems more sane to me (kstat.spl.misc.spl_misc.simulate_pressure=8589934592) and hopefully that will provide some relief. The memory usage has already dropped and the system seems perhaps a little better already even without a reboot.

[znubee]

So things have been better after limiting the memory usage. Speed has improved and overall system responsiveness has improved. My cursor is still a bit jerky on bluetooth for heavy read/write situations, so that is strange.