Extreme Performance Issues with v2.1.6 (and v2.2.3)

[Haravikk]

Nope. You are not the one with such trouble. I have the same problem on two different machines. Mac Pro 2008 with 32GB or RAM and Mac Pro 2010 with 128GB of RAM and it is the same thing when I started write operations on pools. The pools itself also different, at first I thought it was hard drive corruption somewhere else, but no, there isn't. I downgraded on MP2008 OpenZFS to version 2.1.0 from 2.2.2 and... everything starts to work smooth and nice. Of course I recreated pool.

On 2.1.6 through 2.2.2 the system caught I/O locks and almost unusable when I started writing anything... After several gigabytes write speed downs to 1-3 mb and huge system locks appears.

While that's not good news it's nice to know it's not just somehow my problem alone! You mention system locks as an issue, how are you monitoring these? I'm hoping my spindumps might show where the time is being wasted, but it can't hurt to get as much information as possible!

[armdn]

Further more. Today I started moving data from the pool to another disk in order to recreate the pool to version 2.1.0... And rsync showed me a gradual drop in performance when reading from a pool running ZFS version 2.2.2. A few hours later, the system began to slow down so much that it had to be forced to reboot. This is some kind of nightmare. I made a short copy via rsync with a drop in performance and did a spindump in parallel.

Update: i disabled primary cache and secondary cache, and then rsynced files.

Code: Select all

zfs set primarycache=none RAIDZ
zfs set secondarycache=none RAIDZ

Performance now seems normal. Caches are broken.

spindump.txt.zip

(1.21 MiB) Downloaded 30 times

[Haravikk]

Woah, so we have an actual possible culprit at last, awesome work!

I've just done the same on my affected system and I can confirm that it's much, much more responsive with primarycache and secondarycache both set to none, though of course this means it's not quite as responsive as v2.1.0 (as no cache means more requests hitting the drives).

I really wish I'd seen your post before spending the last two and bit hours trying everything else except disabling caching while trying to narrow this down. All I succeeded in doing was confirming that encryption and compression as definitely not the problem, by wasting time doing a huge send/receive of one of my affected datasets, removing encryption and compression so I could test it with neither of these enabled.

But yeah, the problem is definitely something to do with ARC. The fact that there aren't a load of Linux OpenZFS users complaining about the same issue makes me think this is macOS specific, either that or whatever the issue is is affected macOS worse than Linux.

[cgiard]

The note in the 2.2.0 release post mentioned "Fully adaptive ARC" as one of the changes, so there were big changes related to the ARC.

[cgiard]

And also "Persistent L2ARC fixes" in 2.1.6's release post.

[Haravikk]

Since we have a culprit and a reproducible workaround for the problem, I've posted a Github issue for this to try and narrow it down to the important bits; let me know if you think there's anything I've missed. There's an issue on the main OpenZFS Github that I'm currently tracking that sounds suspiciously similar, though their problem only affects an encrypted dataset so could be unrelated. Having an issue for the macOS problem will make it easier to reference it if it's confirmed to be the same problem (or at least similar enough to be noteworthy).

[Haravikk]

And also "Persistent L2ARC fixes" in 2.1.6's release post.

Hmm, while I do have an L2ARC configured for one of my pools, would fixes for persistence be likely to cause this kind of problem? Unfortunately it's getting too late in the day now for me to try testing with my L2ARC removed. @armdn, do you have an L2ARC device configured on any of your pools? Might confirm it as a possibility if we're both using one, could also give a hint as to why not everyone is seeing this problem as most probably don't bother with one.

It sounds like a possibility at least, I'll have to find some time to do additional testing.

Another weak possible cause from the v2.1.6 changelog that might also be worth looking into are the updates to lz4 and zstd; I know that the ARC used to only store decompressed records, but I don't remember which ZFS version changed it to retain the compressed records instead (to maximise space). But if records in the ARC were being decompressed often and it wasn't performing well, or there's some weird interaction, that could also be a possible cause? I just realised that when I was testing a decrypted dataset I removed zstd compression but probably would have still been using lz4, so if that's part of the problem somehow I wouldn't have seen an improvement. That's pure speculation though, it'll take quite a while to setup new test datasets, probably won't have time for a few days. I don't see any options for configuring lz4 or zstd implementations, are there any or is it just one default implementation each?

[armdn]

Since we have a culprit and a reproducible workaround for the problem, I've posted a Github issue for this to try and narrow it down to the important bits; let me know if you think there's anything I've missed. There's an issue on the main OpenZFS Github that I'm currently tracking that sounds suspiciously similar, though their problem only affects an encrypted dataset so could be unrelated. Having an issue for the macOS problem will make it easier to reference it if it's confirmed to be the same problem (or at least similar enough to be noteworthy).

Distributed version has a wide range from 10.13.6 to 10.15.6 at least.

[armdn]

And also "Persistent L2ARC fixes" in 2.1.6's release post.

Hmm, while I do have an L2ARC configured for one of my pools, would fixes for persistence be likely to cause this kind of problem? Unfortunately it's getting too late in the day now for me to try testing with my L2ARC removed. @armdn, do you have an L2ARC device configured on any of your pools? Might confirm it as a possibility if we're both using one, could also give a hint as to why not everyone is seeing this problem as most probably don't bother with one.

It sounds like a possibility at least, I'll have to find some time to do additional testing.

Another weak possible cause from the v2.1.6 changelog that might also be worth looking into are the updates to lz4 and zstd; I know that the ARC used to only store decompressed records, but I don't remember which ZFS version changed it to retain the compressed records instead (to maximise space). But if records in the ARC were being decompressed often and it wasn't performing well, or there's some weird interaction, that could also be a possible cause? I just realised that when I was testing a decrypted dataset I removed zstd compression but probably would have still been using lz4, so if that's part of the problem somehow I wouldn't have seen an improvement. That's pure speculation though, it'll take quite a while to setup new test datasets, probably won't have time for a few days. I don't see any options for configuring lz4 or zstd implementations, are there any or is it just one default implementation each?

Yes, I use L2ARC and ZIL devices (SAS SLC SSD). But I tried without them. That is, I threw L2ARC/ZIL devices out of the pool and nothing changed. Apparently this is a direct problem with the primary ARC and ZIL caches.

[armdn]

It seems this is the same problem.
https://openzfsonosx.org/forum/viewtopic.php?f=26&t=3833