High CPU Usage (encryption or compression)?

[Haravikk]

I've begun copying a large amount of data into a ZFS dataset and I've noticed that my CPU usage spiked while doing so, kernel task was running as high at 1,000% load (on a 2018 hexacore i7 Mac Mini so 16 hardware threads total).

The confusing part is that I've been using ZFS for a while now with similar settings; I replaced my plain HFS+ Time Machine backup drive some time ago with an HFS+ formatted encrypted ZVOL with ZSTD compression, running on a two disk mirror, and I've never noticed big CPU spikes like this during backups, but then maybe Time Machine is just too slow for it to be noticeable?

The dataset where the spikes occur is also encrypted with ZSTD compression, running as a plain ZFS dataset (with com.apple.mimic=hfs), on two unmirrored disks (no redundancy yet), so I don't see anything different that should account for such a big spike in CPU use.

For reference, here are the non-default ZFS properties for the dataset in question:

Code: Select all

zdata/media/Queue  recordsize             1M                            local
zdata/media/Queue  mountpoint             /Users/media/Downloads/Queue  local
zdata/media/Queue  compression            zstd                          inherited from zdata
zdata/media/Queue  readonly               off                           inherited from zdata/media
zdata/media/Queue  xattr                  sa                            inherited from zdata
zdata/media/Queue  copies                 1                             local
zdata/media/Queue  dnodesize              auto                          inherited from zdata
zdata/media/Queue  relatime               on                            inherited from zdata
zdata/media/Queue  com.apple.browse       on                            inherited from zdata
zdata/media/Queue  com.apple.ignoreowner  off                           inherited from zdata
zdata/media/Queue  com.apple.mimic        hfs                           inherited from zdata
zdata/media/Queue  com.apple.devdisk      on                            inherited from zdata

Changing compression to LZ4 or turning it off entirely didn't seem to make any meaningful difference to CPU usage.

In case it matters, I'm using rsync to perform the copy, copying all attributes (rsync -auishxAXNHP) however the target dataset was empty when I started so rsync shouldn't really be doing anything, i.e- there should be no checksumming of existing files or such. Copying in the Finder also exhibits the same CPU spike when the files are big enough.

I can't fault the transfer speed, as it's hovering around 170-200mb/sec and the two drives that make up the pool are rated for around 120mb/sec write each so that's near enough to their maximum theoretical speed.

Of course I don't anticipate so much write activity under normal use, but the spike in CPU usage seems extremely high; does OpenZFS on macOS support hardware encryption? What is the impact of having datasets with different encryption keys?

Update: Increasing the record size to 1M might have helped actually, now the CPU utilisation is around 500% for kernel_task, could the issue be checksum generation? That seems a bit excessive though, as surely checksums can be hardware accelerated as well?

[lundman]

What version? Since there was a new fix for encryption last week for the 2 line

[Haravikk]

I knew I'd forget something!

zfs version shows zfs-macOS-2.1.0-1, looks like the latest macOS release, or was this more of a hotfix (no new version number)?

[lundman]

The sysctl version will show git hash

[Haravikk]

I had a search but didn't find which sysctl property this is stored under? I did however find the following command:

Code: Select all

/usr/bin/strings /{System/,}Library/Extensions/zfs.kext/Contents/MacOS/zfs 2>/dev/null | grep -v 1234 | grep -E '[0-9]\.[0-9]\.[0-9]'

Which grabbed me a version of zfs-macOS-2.1.0-0-ga3ba8b30c1, is that what you're looking for? Is that the latest?

[FadingIntoBlue]

I had a search but didn't find which sysctl property this is stored under?

I think this is what you are looking for:

Code: Select all

sysctl -a | grep kext

Mine spits out this, though I am not sure it is the latest

Code: Select all

debug.kextlog: 4083
zfs.kext_version: 2.1.0-1

Seems like it might be a little more recent than 2.1.0-0-etc

[Haravikk]

That's the version number I get from zfs --version as well, so I'm not sure what that mismatch is about?

[lundman]

zfs-macOS-2.1.0-0-ga3ba8b30c1

is generally the sort of thing we care about, must admit I've lost track of what we are doing

[Haravikk]

zfs-macOS-2.1.0-0-ga3ba8b30c1

is generally the sort of thing we care about, must admit I've lost track of what we are doing

Is that the latest version with the encryption fixes? If so then it's the one I'm seeing the high CPU usage on; it's not so bad under normal use (as I'm not writing nearly as much data as I was during setup) but it still seems high so it might be good to find out what's causing it.

If I copy a large amount of data it still spikes, but I'm afraid I don't know how to debug it further to find out what exact feature might be to blame; like I said, I tried turning off compression to see if it was that and that didn't help (so I turned it back on again). I also increased record size on one of the datasets I was copying to (as it had a lot of larger files) and that did reduce CPU usage by around half (changing from 128K to 1M record size).

When I get a chance I'll try to run a copy of the same files to test datasets with different settings (encrypted/unencrypted, checksumming/no-checksumming etc.) to see if I can find a specific feature that has high CPU usage, but I do a lot of video transcoding that takes a long time to finish; this is why I was hoping to confirm I'm already on the latest version, as I have to wait for my queue to clear before I can restart to install an update.

[abc123]

There's a test package here viewtopic.php?f=26&t=3651&start=20#p11588 which adds extra assembler implementations for encryption and checksums. That may improve performance for you but it isn't an official build yet