ZFS memory usage is MUCH higher than zfs_arc_max

[Wowfunhappy]

Hello! Before I go any further, I'd like to note that I'm using OS X 10.9, and so I'm on version 1.9.2 of OpenZFS on OS X. If this means I can't get help here, I totally understand, but I figured I'd still ask!

My computer has 32 GB of memory, but for a variety of reasons, I'd like to limit ZFS to around 4 GB. So a few months ago, I created the file /etc/zfs/zsysctl.conf with the contents:

Code: Select all

kstat.zfs.darwin.tunable.zfs_arc_max=4294967296

as per the instructions in the wiki.

Unfortunately, I regularly see ZFS using ~ 16 GB of memory. For instance, as of when I'm writing this, if I run in the Terminal:

Code: Select all

sysctl kstat.spl.misc.spl_misc.os_mem_alloc

I get

Code: Select all

kstat.spl.misc.spl_misc.os_mem_alloc: 14750056448

That comes out to around 14 GB! I know actual usage can be somewhat greater than the limit, but that's way higher! Is the wiki outdated? Am I doing something stupid?

[leeb]

Yeah, with overhead actual will usually be higher than max, but that does seem excessive. Do you see the same behavior if you set it temporarily (sysctl -w)?

[mkush]

Perfect timing! I just came here looking to ask about the EXACT same thing, only a more extreme case.

My ARC Max is set to 1000000000 (1 billion). This has worked well for me in the past to get good performance on other systems. I don't think it was ever solved as to why setting a low number like this helped, but it did. See the old thread started by me on the subject.

Anyway, this is a new system, and a new disk array:
Mac Pro 2019
192GB memory
Version 1.9.4
MacOS 10.15.7
Samsung 860 EVO 4TB x16 attached by...
ATTO ExpressSAS H12F0
Configured as RAIDZ2

The ATTO is a PCIe host bus adapter (not RAID) with 16 external channels (each drive has its own channel).

Performance seems good enough... 488MB/s write, read not yet tested. I'd like more but I seem to recall that RAIDZ/2 performance ends up about the speed of a single drive, which is about what this is.

For fun, I pulled up the Activity Monitor and watched memory as I copied my first data to the volume... about 128GB of pictures from a CFexpress card. Memory just climbed and climbed... and now sits at over 121GB used. I'm not running anything else to speak of.

Running the command above shows:

Code: Select all

sysctl kstat.spl.misc.spl_misc.os_mem_alloc
kstat.spl.misc.spl_misc.os_mem_alloc: 117805416448


Which is nearly the entire amount of memory used. So what is going on?

[mkush]

Setting interactively with:

Code: Select all

sudo sysctl -w kstat.zfs.darwin.tunable.zfs_arc_max=1000000000

works perfectly! Perhaps something in how I did the config file is wrong. I will check later to see if I see anything obvious.

[mkush]

There is nothing wrong with my zsysctl.conf file. It seems that it is simply not being read. No idea why. The other thing is that my pools NEVER import automatically. So, is some startup process simply not working for me? In the end it's not a huge deal... I'll just make a script that sets the ARC and imports my pools (by serial, as I like it). Speaking of importing by serial... that's really hit or miss for some reason. I'll make another post about that...

[jawbroken]

If you're on Catalina then you might need to give “Full Disk Access” permission to /bin/bash to get automatic import working. Not sure if that will solve your other problem, though.

[mkush]

Sure enough, giving Full Disk Access to /bin/bash did fix the auto-import -- but not the reading of the conf file.

HOWEVER, two drives of my 8-HDD Backup pool (RAIDZ2) are now shown as having corrupt data and have been removed from the pool. See separate post on this problem.

[Wowfunhappy]

I fixed the problem by running

Code: Select all

sudo /usr/sbin/zsysctl -f /etc/zfs/zsysctl.conf

Just doing this one time appears to have fixed the problem permanently, across reboots and everything.

[Bingo]

OMG, I had forgotten how utterly broken OpenZFS was compared to ZFS until I "updated" to 2.2.2. Then this beachball-nightmare that I haven't seen since I neutered the garbage that is vanilla OpenZFS.

Thank you for the tip on

Code: Select all

sudo zsysctl -f /etc/zfs/zsysctl.conf

. Not only can I use my computer again, I don't accidentally delete files, and its fans stopped deafening me. Will this persist across reboots?

Everyone, make sure you don't enable auto-upgrade on zpools, or you'll be stuck when the OpenZFS clowns finally break everything. I can still go back, unfortunately not till pre-openzfs. Working on getting away from it altogether, though.

Does anyone know of a macOS version of ZFS-FUSE from when it didn't suck ass, preferably with fixes applied, like the recent cp bug?

[jawbroken]

That's a rude way to talk about volunteers developing software that you use for free.