Extreme Performance Issues with v2.1.6 (and v2.2.3)

[Haravikk]

The person who opened this issue, that you already commented on, seems to have eventually tracked it down to their enclosure. So I should probably also say that my ZFS drives are in a few of these Thunderbolt enclosures.

The enclosures I'm using are the four disk versions of yours; I should have said, I have two, with the disks split between them (I basically have three mirrored pairs, so I've split the disks between the two enclosures, with one bay free in each) so I should be protected against enclosure related issues (controller failure, power loss, loose cable etc.).

The enclosure being the problem was a bit of a strange end to that Github issue though, since they seemed to be getting different results with and without encryption; I can't think of any reason why that would make a difference if the enclosure were the problem, unless it was part of a new setup (copied their data across to a new encrypted dataset in the slower enclosure), i.e- their unencrypted test was a different enclosure, and they hadn't eliminated that difference already?

So while I did initially think the issues might be related I'm sceptical whether it's the same cause; they noticed no improvement when bypassing cache, whereas disabling ARC (primarycache=none, secondarycache=none) makes a huge and immediate difference whenever I test a version from 2.1.6 onwards, and others reporting the same problem have seen the same improvement.

And like I say, 2.1.0 works just fine, so the issue is definitely related to something that changed in ZFS.

[Haravikk]

Lundman, if you haven't got any time to look further into this, would it be possible to instead get a build of v2.1.0 for macOS versions after Big Sur (Montery, Ventura and Sonoma)?

Sonoma is likely be the last version of macOS that will be officially supported for Intel Macs anyway, and I'd really like to upgrade macOS but the fact that I can't do so without upgrading ZFS at the same time is a roadblock for me due to these performance issues. If I could at least upgrade macOS while keeping ZFS the same, I could then try newer ZFS versions on the newer OS without having to rollback the OS if it still doesn't work right.

Either that, or can anyone confirm whether I need a Sonoma specific build of ZFS, or will the Big Sur version of v2.1.0 work? Is there actually any difference between them?

[jawbroken]

I can't speak for that specific combination, but in general the releases are compatible with newer (but not older) operating systems, so you probably won't need a Sonoma specific build.. I haven't had any issues updating macOS to new major versions without upgrading OpenZFS at the same time, for example.

[Haravikk]

So I finally got around to upgrading to Sonoma, and can confirm v2.1.0 works just fine on it. In fact, the system as a whole feels more responsive, dunno if that's just a result of some UI enhancements or something, but operations that used to feel a bit slow, especially around ZFS, now seem to complete quicker.

I won't have a chance to test a newer version, but if there is some kind of Catalina specific speed issue then it's possible v2.1.6+ was merely making that worse? I'll try to update to the latest ZFS version next week to see if upgrading to Sonoma solved the problem.

[Haravikk]

Found some time earlier than expected, and I can confirm that these performance issues do not occur on macOS Sonoma.

Even with my ARC mostly filled at around 10gb I've noticed no performance degradation or unusually high CPU usage, where under Catalina I would have found the system completely unusable well before this point (and most likely crashed loginwindow by now).

Performance is also just generally better; I'm running a copy of the contents of a zvol into a regular dataset, which I've used before as a test, and kernel_task is barely hitting 20% CPU usage, where before it could jump up to 80-90% due to the lack of hardware encryption support. Spotlight is doing its usual trick of re-indexing everything, but it's not hammering the system into dust as it previously would have.

Since I've skipped several versions I can't confirm whether Big Sur, Monterey or Ventura might demonstrate the same correct behaviour – it seems possible as Big Sur was when the change to support M1/ARM began, so it may be a result of this work that the performance changed?

It is however very nice to finally be able to run with hardware accelerated encryption, and fully working xattrs. The only downside is now I have to try and remember all the hacks I implemented to workaround the broken xattrs, it seemed to involve giving a lot of things full disk access which shouldn't need it, so hopefully I can turn that off now.

[jawbroken]

it's good to know that it isn't a fundamental issue with some feature that you were using, and is likely some interaction with the old OS

[nodarkthings]

MacOS 10.14 here: while v2.2.3 is generally ok, I've started having system paralysis when putting the VMs on my dataset to sleep (meaning heavy writing).
After reading this thread, I've tried v2.1.0 instead and, even if there's still a lot of kernel activity, the system keeps on being usable.
As @Haravikk discovered, it looks like versions post 2.1.0 are more suited for recent MacOS.

[Sharko]

Hmm, I wonder where the break point is as far as: systems later than "X" we should use latest, and systems before "X" we should use 2.1.0... ?

[nodarkthings]

Hmm, I wonder where the break point is as far as: systems later than "X" we should use latest, and systems before "X" we should use 2.1.0... ?

Yes, I'd say it's very likely that the break point is between 10.15 and 11.0: a component of MacOS managing disks has probably been modified to accommodate a change in APFS and the biggest change in APFS happened in Big Sur (as we can read here: https://eclecticlight.co/2021/01/09/boot-disk-layout-on-intel-and-m1-macs-high-sierra-to-big-sur/).
And our issue is a side effect (just guessing ).