Write performance in 2.0.0 RC1 (and prior?) has tanked?

All your general support questions for OpenZFS on OS X.

Write performance in 2.0.0 RC1 (and prior?) has tanked?

Postby Jimbo » Sun Apr 04, 2021 1:44 am

@lundman - thanks for the updated release, testing it now.

Big Sur 11.2.3 Intel on a couple of different machines.

One this painfully obvious right now, is that write performance to all my [upgraded] pools has tanked.

1Tb SSD pool, single drive, USB3, write speed of <10Mb/s where it was ~300Mb/s from local SSD -> SSD ZPool
4x12Tb RAIDZ write speed of ~10-30Mb/s when was in the 150Mb/s range
2 striped 4x4Tb RAIDZ write speed of ~20-40Mb/s when was in the 200-30Mb/s

Tested local copy and remote copy (SMB over 10GbE). This seems to have cropped up in the recent releases (March 14?).

Any of the new features or changes that could be causing this? Anything I could be experimenting with?

While I don't need stellar write speeds, this slow down is... a very inconvenient bottleneck.
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: Write performance in 2.0.0 RC1 (and prior?) has tanked?

Postby Jimbo » Sun Apr 04, 2021 2:53 am

Okay, a bit more observation. Local copy testing APFS SSD to 1TB USB3 SSD ZPool (pool created with 1.9.4).

Copy on Catalina running 1.9.4 just kicks off and runs ~300-400Mb/s beginning to end.

Copy on Big Sur running 2.0.0RC1 crawls for about 30 seconds - little bursts of activity in the single-digit Mb/s, then after about 30 seconds, the burst of activity get bigger and bigger until it starts running at a steady 400+Mb/s... for a while - 10, 20, 30 seconds and then tanks to 0 (or very low Mb/s) for about 10 seconds and then ramps back up to 400+Mb/s for another 10, 20 or 30 seconds and then tanks again, rinse, repeat until the very-large-file-here copies. At about the same time as the write I/O seems to tank, I see a freeing of memory ~100Mb and similarly, when starting a copy, I see the used memory slowly rampy up by 100-200Mb before sustained write I/O is "good".

Testing was done with a 50+Gb file, so just large streaming I/O.

Seems quite repeatable but my metrics are "Activity Monitor" quality. Is there a preferred way for me to monitor the ZFS memory usageand I/O?
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: Write performance in 2.0.0 RC1 (and prior?) has tanked?

Postby lundman » Sun Apr 04, 2021 3:51 pm

We had to gut all the memory pressure logic, as it no longer exists in BS and M1. Which means, we
made it think there is always pressure (so ARC is smallest, all the time) so it doesn't grow forever until it dies.

So, performance will have taken a hit. It was more about buying time for now, and get something working. We'll go
back and find different ways around memory pressure and export restrictions.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Write performance in 2.0.0 RC1 (and prior?) has tanked?

Postby Jimbo » Mon Apr 05, 2021 1:20 am

Yep, that'd explain it and I do recall seeing something about those changes. 8-)

Interestingly, performance over the network remains consitently low (5Mb/s < x < 40Mb/s) averaging about 30Mb/s for 50+Gb file over 10GbE without any of the bursty writes seen with a local copy. Network transfers to an SSD APFS share run at ~800Mb/s so its not a network thing (both using SMB on the same host).

I'll keep poking about.
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: Write performance in 2.0.0 RC1 (and prior?) has tanked?

Postby Jimbo » Mon Apr 05, 2021 4:21 am

Ok, my lat comment isn't correct - it is not the fact that it is going over the network to ZFS, it is the fact that the slow pools are spinning rust pools. A local copy from SSD to the pool also crawls along 5Mb/s < x < 40Mb/s (where it used to be 150 to 200Mb/s depending on the pool).

At least that makes some more sense.
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: Write performance in 2.0.0 RC1 (and prior?) has tanked?

Postby Jimbo » Sun May 16, 2021 8:30 pm

I thought I'd just close out this thread...

Running RC7 has *VASTLY* improved write performance for me. Things were still a bit slow while the ARC was growing and stabilizing, but then seemed to settle into a nice [fairly] consistent 300-400Mb/s write rate from a 10GbE client sending 50Gb files to a 2x RAIDZ1 stripe (4 drives each VDEV).

Thanks all for your efforts in getting this release to this point!
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am


Return to General Help

Who is online

Users browsing this forum: No registered users and 30 guests

cron