Horrible Performance on newly created ZFS Raid

All your general support questions for OpenZFS on OS X.

Horrible Performance on newly created ZFS Raid

Postby thedarkhalf » Fri Nov 27, 2020 1:45 pm

So I just created a ZFSraid1 and I seem to be getting really bad write performance from it. I used the following to create

sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive -O normalization=formD UltraMega raidz disk2 disk3 disk4 disk5

So the drives seems to hang up or something. It takes about 90 seconds to duplicate a 930GB file. On a single external USB drive, it takes about 7 seconds to do the same copy.

Here are 3 different speed tests done within a 10 min time frame and I got some wildly different results.

https://ibb.co/D10mPDh
https://ibb.co/QDK524G
https://ibb.co/3YNhk9M

Dedup is off. Compression is off. No idea whats going on.. Any thoughts?

Also would adding a SSD cache drive work? A friend gave me the command of "zpool add <pool name> cache <drive name>" to add the cache drive

Thanks!!
thedarkhalf
 
Posts: 12
Joined: Thu Nov 19, 2020 5:43 pm

Re: Horrible Performance on newly created ZFS Raid

Postby Jimbo » Fri Nov 27, 2020 3:22 pm

More information please.

Version of O3X, type and size of drives, type of enclosure, enclosure connection type, version of macOS and hardware (make/model, CPU and RAM).

BTW, if you’re complaining about 930Gb in 90 seconds... I think you mean Mb? That said, ~10Mb/sec would seem very slow.

Generally, if you don’t have a workload that would benefit from a cache device (i.e. synchronous writes) you’re not going to get much (any) benefit without specific tuning.

Cheers!
James
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: Horrible Performance on newly created ZFS Raid

Postby thedarkhalf » Fri Nov 27, 2020 4:14 pm

Thanks for the reply James

O3X - 1.94 (latest)
2012 Mac Mini (Dual Core 2.5 Ghz i5, 8GB RAM) running Catalina (10.15.7)
Drives are Shucc'd WD 14TB White Labels (from 2x EasyStore and 2x Elements)
Enclosure is a 4 Bay Mediasonic ProBox HF2-SU3S2 with USB 3.0 B-Type connection

lol And yes.. 930MB.. not GB. I would not compain about a 930GB in a few minutes.

It's a plex server, btw. I'm currently watching a movie while I running clone of the ZFS Raid (just as a precaution) so it seems to work fine for what I need it for. But I went to copy around 20GB of new data to the drive and the copy time was all over the place (from 2 hours to 5 minutes.. It seems like it stalls, then copies, stalls, then copies. I know MacOS is pretty accurate with copy times, so I figured something was amiss.
thedarkhalf
 
Posts: 12
Joined: Thu Nov 19, 2020 5:43 pm

Re: Horrible Performance on newly created ZFS Raid

Postby Jimbo » Fri Nov 27, 2020 8:08 pm

Hi,

I know this isn't going to be particularly helpful, but...

I used to run a similar setup - MacMini quad core i7 with 16Gb driving 4x12Tb Seagate NAS drives in 5-bay thunderbolt enclosure where the 5th bay was a small SSD set up as an intent log. This was a fileserver for all-the-things (Time Machine, Plex, LMS, SMB, AFS, etc) all basically being served from one pool and various datasets therein running O3X 1.9.4. This has subsequently been upgraded but I'll talk about the old set up for reference.

Performance wise, this was never stellar, but I didn't expect it to be. Transferring to/from this setup would be able to saturate the 1Gbps network link for simple large-ish transfers. Throughput would wobble for many multi-Gb transfers in the 30-100Mb/sec range, but generally it was fine. So you should be able to get "acceptable" performance, but... things to consider or look at:

RAIDZ (with only 1 VDEV) - you're going to be limited (eventually) to about the maximum write performance of a single drive. I say "eventually" as ZFS will make extensive use of RAM to cache writes (and reads) prior to marshalling them and actually writting them out (making IO look a lot better than is physically is for a while), but when you hit the wall and have to rely on actually getting the bits to disk, performance can tank, hard. This might account for 20Gb data causing erratic write times (this can also be exacerbated by the size of the files).

Anyhoo... to the hardware.

The drives... those types of drives *may* be detuned for reliability and longevity (custom firmware). Dunno, maybe someone else can comment?
USB... is kinda arse for storage. Different people have vastly different experiences with ZFS on USB devices. I tend to only use ZFS on single USB disks but have been know to use it with striped USB enclosures and its been "ok". 4 or more drives I've always used a Thunderbolt enclosure (my 4x 12Tb drives are in an old Lacie 5-big TB1 box). Your Mac Mini might also lack UASP support.
The Mac Mini... RAM might be on the low side - the more data you throw at it in a short time will fill ARC fast and could cause stalls as ZFS has to scramble to actually push blocks to disk. The more concurrent activity you have going on (say loading new files, Plex transcoding, streaming, etc) the harder the disks will have to work and the more RAM will be required to smooth out that IO workload.

Genernally I find O3X performance to be ok for my needs (with the notable exception of when all the machines here are Time Machine backing up at the same time), but it certainly isn't stellar (the host now has 32Gb RAM and 10GbE and I can certainly push this host harder than the previous one). As @lundman might comment, for the Apple port, stability first and performance last in terms of priorities.

Oh and the benchmarks... always going to be a bit wonky due to how ZFS works. ;)

Cheers!
James
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: Horrible Performance on newly created ZFS Raid

Postby Jimbo » Fri Nov 27, 2020 8:12 pm

Oh, I also wanted to ask whether you've checked if the drives are being spun down agressively? If they are, are they spun up one at a time or in parallel? This is a performance killer too.
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: Horrible Performance on newly created ZFS Raid

Postby thedarkhalf » Sat Nov 28, 2020 7:34 am

Thanks again for the reply, James!

The drives are not being spun down (as far as I know). The case has drive light activity and it's like a Christmas Tree

I think I am going to boost the RAM on the MacMini to see if that helps. I can't even say they are matched sticks in there either. It was at 4Gb when I got it and I just tossed another 4 in when I put it together. I read that can cause issues.

I think i'm going to rebuild the zRAID on a Ubuntu machine. Do you know if they are cross compatible? I tried to mount the current one on ubuntu (zraid1, formatted on OSX) but it's not mounting for some reason. I figure I'll try to recreate it on Ubuntu and if I get better performance, then mount back on the Mac and see how it goes. Worst case is i'll just migrate my plex server over to ubuntu.

And if all else fails, I'll just live with the performance. It's more of an annoyance than anything else.
thedarkhalf
 
Posts: 12
Joined: Thu Nov 19, 2020 5:43 pm

Re: Horrible Performance on newly created ZFS Raid

Postby Jimbo » Sat Nov 28, 2020 7:00 pm

There are a few threads here and the wiki has some information on how to create “compatible pools” - sorry I don’t have links handy.

If you just run “sudo zpool import” without specifying the pool name it will give you information about whether it can import the pool and what issues there might be (i.e. unsupported features). You can often still import read-only. Just make sure that Ubuntu is running a similar or higher version (and be cautious about upgrading pools with new features).
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: Horrible Performance on newly created ZFS Raid

Postby thedarkhalf » Sun Nov 29, 2020 9:05 am

Welp.. I just backed up all of the data and destroyed the pool. Decided to just reformat on the Mac and see what happens. I ran tets and speeds/copy are looking good. 200Mb write and 2,000Mb read speeds. All looking consistant. I did several copies or 5GB files and they all finished within a minute.

So now it's time to copy all the data back and test again.

Thanks for all the help!
thedarkhalf
 
Posts: 12
Joined: Thu Nov 19, 2020 5:43 pm


Return to General Help

Who is online

Users browsing this forum: Google [Bot] and 25 guests