lundman wrote:Heh you only got 14 cycles into the spindump.speed? peculiar, guess it is busy.
So of your busiest threads, you have
- Code: Select all
1 Thread 0x2a4a Thread name "z_wr_iss_1" 14 samples (1-14) priority 76 (base 76) cpu time 2.975s (11.6G cycles, 1961.3M instructions, 5.93c/i)
2 Spindump 797.6M
3 mdsync [5649]
4 com.apple.spindump.stackshot_parsing"(28) 1 sample (2) priority 31 (base 31) cpu time 0.061s (255.4M cycles, 319.4M instructions, 0.80c/i)
5 fseventsd [150]
6 Activity Monitor [6079]
Since #1, #5 (ZFS writing) is ending up in vm_map_find_space() your system is completely out of RAM, and is desperate to find more. I think you should tell ZFS
to use less (keep halving), and see if it improves. Could be there is a memory leak as well, that you trigger. The full output of sysctl for ZFS should show that, once it has started to slow down.
RAM isn't the issue; I have 64gb of RAM and I have ZFS configured to use up to 12gb for ARC. By the time I eventually gave up, kernel_task was on around 20gb RAM which is about normal (the SPL size usually reports around 16gb once the ARC has filled up so another 4gb for everything else isn't unreasonable), meanwhile I still had nearly 34gb of free RAM.
A memory leak also seems unlikely as performance was terrible from the moment v2.2.2 was installed and datasets were mounted and being used. Whatever is wrong the problem is immediate and fundamental to how ZFS is operating (or not), but what confuses me is why nobody else seems to be affected? For me the problem is immediate and so severe I have to downgrade to regain use of my system.
I'll try and get a full output from sysctl but is there more debug data I can grab at the same time? Upgrading and downgrading the ZFS version to get one piece of data at a time is extremely time consuming, that's why it's taken me so long to try again. And is it possible to tell ZFS to not use hardware acceleration? This could allow me to eliminate that as a possible cause, otherwise I'll have to try creating an dataset without compression or encryption, though IIRC hardware acceleration also includes some of the hashing methods as well?