Trying to figure out performance issues on 8-drive Z-RAID 2

All your general support questions for OpenZFS on OS X.

Trying to figure out performance issues on 8-drive Z-RAID 2

Postby dguisinger01 » Thu Dec 03, 2020 9:36 pm

I'm trying to troubleshoot some performance issues and my software vendor keeps pointing me back at the fact I'm not using an SSD which is what they recommend.

I put this 8-drive Z-Raid 2 array together for video editing and I've not had an issue with it so far (knock on wood). The drives are MG07ACA14TE from Toshiba

But now that I've installed sample libraries for Logic Pro on it, I'm running into issues.
When switching instruments in logic, if its an instrument on the SSD, the load is instant.
On the ZFS volume, the good loads are either instant or 1-2 seconds of delay.

But some of these I am seeing take 30, 45 even almost 60 seconds to load..... which just doesn't make a whole lot of sense to me unless I'm missing something painfully obvious.

It looks like many of these .nkx files are either 1GB or 2GB in size, so it should be sequential reads... I'm pretty confused as to why it is choking.

This is:
8x MG07ACA14TE with 4KB sectors and 256MB Cache
Installed in a Thunderbay 8 enclosure over Thunderbolt 3
I'm running the latest ZFS 2.0 release build (as of an hour ago) on Mac OS Big Sur 11.0.1 on a 2019 MacBook Pro 16" with Intel i9 8-core 2.4GHz cpu and 32GB of ram.

Any ideas on what could be causing this thing to chug?
dguisinger01
 
Posts: 21
Joined: Fri Jun 12, 2020 9:51 am

Re: Trying to figure out performance issues on 8-drive Z-RAI

Postby lundman » Sat Dec 05, 2020 2:03 am

It is possible we have some lag issue with the new code, you can watch something like "zpool $pool iostat 1" and see how much IO it is doing each second. -v by disk.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Trying to figure out performance issues on 8-drive Z-RAI

Postby dguisinger01 » Fri Dec 11, 2020 6:00 pm

Alright, here are the numbers at the peak of the load I am seeing. I honestly don't know how to interpret it, 48MB/sec across 7200RPM drives seems slow to me. I don't know how random-access this activity is; it looks like the files are 1GB and 2GB in size, but it may be jumping around inside a file and not loading the entire thing into ram.

I was offloading video from a camera today and saw I had a sustained write rate of around 190MB/sec going from a SD Micro card to ZFS....
But then I fired up Final Cut Pro and imported that video, and the "Copying Files" task it spawned took around 30 minutes to copy the 24GB of video from one location on this ZFS volume to a second location on the same volume. In this case read & write bandwidth was about 34M on average, and i'm often seeing bandwidth of hundreds of K. I don't know if the read is the limitation here, obviously it can't write data faster than it reads...

Code: Select all
---------------------------------------------------------------------------------------------------------------  -----  -----  -----  -----  -----  -----
thunderbay8                                                                                                      4.29T  97.5T  2.10K      0  48.5M      0
  raidz2                                                                                                         4.29T  97.5T  2.10K      0  48.5M      0
    media-1F05BCAF-68AB-114B-BD1A-BA1F22032F11                                                                       -      -    266      0  6.04M      0
    PCI0@0-PEG1@1,1-UPSB@0-DSB1@1-UPS0@0-pci-bridge@4-pci-bridge@0-pci-bridge@2-pci197b,585@0-PRT0@0-PMP@0-@0:1      -      -    276      0  6.15M      0
    media-93B9E6A5-033F-5C47-B353-D6B48E3974A3                                                                       -      -    270      0  6.09M      0
    media-EA30491F-D5B0-2A42-A916-1190CA1AE5C0                                                                       -      -    267      0  6.05M      0
    media-E71C76CC-FFE8-0F4E-AADB-D6CD251B3707                                                                       -      -    264      0  5.87M      0
    media-A31C3A93-C451-7948-B896-BB6C0817EBF6                                                                       -      -    264      0  6.05M      0
    PCI0@0-PEG1@1,1-UPSB@0-DSB1@1-UPS0@0-pci-bridge@4-pci-bridge@0-pci-bridge@1-pci197b,585@0-PRT3@3-PMP@0-@0:1      -      -    271      0  6.23M      0
    PCI0@0-PEG1@1,1-UPSB@0-DSB1@1-UPS0@0-pci-bridge@4-pci-bridge@0-pci-bridge@2-pci197b,585@0-PRT3@3-PMP@0-@0:1      -      -    266      0  6.04M      0
---------------------------------------------------------------------------------------------------------------  -----  -----  -----  -----  -----  -----
dguisinger01
 
Posts: 21
Joined: Fri Jun 12, 2020 9:51 am

Re: Trying to figure out performance issues on 8-drive Z-RAI

Postby dguisinger01 » Fri Dec 11, 2020 6:13 pm

I also can't seem to scrub through video once its open in Final Cut.... trying to fast forward through video i get maybe one frame a second
dguisinger01
 
Posts: 21
Joined: Fri Jun 12, 2020 9:51 am

Re: Trying to figure out performance issues on 8-drive Z-RAI

Postby mike0810 » Sat Dec 12, 2020 1:36 am

Also with the 1.9.4 I experienced this also randomly on a 4 drive zraid-1. Copying from one location to another on the same zfs took ages. Copying from another drive to the zfs was instant. Didnt investigate further
mike0810
 
Posts: 55
Joined: Fri Jan 16, 2015 5:17 pm

Re: Trying to figure out performance issues on 8-drive Z-RAI

Postby dguisinger01 » Sat Dec 12, 2020 1:33 pm

I have found mentions that setting zfs_vdev_async_read_max_active=1 instead of the default of 3 has drastically increased read performance for some people. I'm not sure how to set it, I did find instructions for setting some settings, but it didn't look like the settings file existed on my machine so I don't know if its moved on MacOS or if I just need to create the file.
dguisinger01
 
Posts: 21
Joined: Fri Jun 12, 2020 9:51 am

Re: Trying to figure out performance issues on 8-drive Z-RAI

Postby dguisinger01 » Fri Dec 18, 2020 11:49 am

Any further instructions on what I could try?
I've now been unable to use this raid array for what its purpose is for the last month and a half.

I am getting very close to throwing in the towel, copying my data off, and going to SoftRaid + APFS
dguisinger01
 
Posts: 21
Joined: Fri Jun 12, 2020 9:51 am

Re: Trying to figure out performance issues on 8-drive Z-RAI

Postby dguisinger01 » Sat Dec 19, 2020 10:46 am

I reformatted overnight to SoftRAID 6 Beta + HFS+.

LogicPro sample load time dropped from 1 minute to instantaneous
Final Cut pro started working properly
and BlackMagic Speed Test is getting 2500MB/sec whereas it was struggling past 400MB/sec before and occasionally hitting 500.

Problem solved I suppose
dguisinger01
 
Posts: 21
Joined: Fri Jun 12, 2020 9:51 am


Return to General Help

Who is online

Users browsing this forum: Google [Bot] and 19 guests