Something wrong with the ARC memory code

[Bingo]

I recently created a large pool on a Toshiba helium-filled enterprise CMR drive. Single-disk pool, just for scratch stuff and such. It's connected via USB3 (as confirmed in the system report). No compression, no dedup.

I started rsyncing a ton of stuff from other drives, and after a few seconds, performance started tanking. Then macOS started tanking. Beach balls, stuttering, seconds-long lags, terminal lagging, apps lagging, mouse cursor locking up, the works.

It's exactly like this guy reported seven years ago: viewtopic.php?f=26&t=2873&p=7377&hilit=lag#p7377

Like him, I don't see any excessive memory usage, of my 16GB avail, only about 12GB was in use. Swap was stable.

And, the solution was the same as his: Actively limit the ARC size. In my case, I tried 11 GB. No difference. Tried half that. Slightly better. Still lock-ups, still lags, but not quite as bad. Tried halving again. Markedly better, but still lock-ups and beach balls. Then tried 1GB. Almost zero lag now, and when the lags appear, they are much shorter than before. Haven't seen a beach ball yet. The rsync is running much faster, so far almost twice as fast as with the ARC unlimited. The CPU fan being very audible now suggests that ZFS is actually able to work more instead of tripping on its own locks.

First of all, as long as the code is like this, clearly the default should be to limit the ARC. Out of the box, ZFS makes your system unusable during heavy writes, AND performance suffers.

Second of all, what is going on here? I'm not privy to the code, but I'm wondering how, and how often, ZFS issues synchronous writes to storage. It's the only thing I can see (again, not knowing the code) that would lock the kernel up like this, and by extension the system. And I wonder if this is also why SMR drives are unusable with ZFS (a sync will force the drive to write to a zone, which is very costly on an SMR drive) (the same goes for btrfs IME, though it isn't as bad as ZFS in this regard). And further, if the reason for this behaviour is also the reason that there seems to be a massive write amplification (IIRC, reddit reports were 3x or more) on the TBW of SSDs when using ZFS?

[BjarneDM]

Which version of OpenZFS on OS X do you use ?