Long pauses during write

All your general support questions for OpenZFS on OS X.

Long pauses during write

Postby mkush » Tue Jan 02, 2018 7:50 pm

When I copy something *to* a ZFS volume, there are always long pauses in the copy progress. This is observed in the macOS Finder, watching the progress of how many bytes have been written. My observations about these pauses:

1. They are quite long... sometimes 10 seconds or more, but are of varying lengths.
2. They happen regularly, periodically, during the copy.
3. They only happen when *writing to* the ZFS volume; reading is linear.
4. The pool is created using settings recommended on the Wiki.
5. It does not matter what devices are used... even nice SSDs do the same thing.
6. My current test pool is a mirror of two Samsung 850 EVO 4TB SSDs.
7. It doesn't matter how the devices are connected to the computer (directly plugged into SATA on my 2010 Mac Pro or external, no difference).
8. If the same devices are configured as HFS or APFS, no pauses are observed.
9. The data is being copied from a PCIe SSD in the Mac Pro so as to not be a bottleneck.

As for actual performance data, the same two drives were first configured as a mirrored APFS pair, connected via USB3. My test folder is a 31GB folder of iPhone photos and videos. When copying to the APFS volume, it took 167s and there were no pauses observed in the copy progress. Then the APFS volume was destroyed and a ZFS pool/volume made in its place on the same disks. The copy then took 272s with frequent pauses observed. Although I will say that the drive lights on the enclosure never stopped flashing during writing. I moved the ZFS pool to the internal bays in the Mac Pro and the time dropped to 188s (not sure how APFS would have fared) but the long pauses were still there. In that configuration, moving the data the other direction (reading from ZFS, writing to boot PCIe SSD) the copy took 113s and showed no pauses.

Is there something I'm missing? Are the pauses for the checksums to be generated? I would have thought the machine should be pretty capable (12x 3.46GHz cores, 48GB).

Is this normal behavior?
mkush
 
Posts: 29
Joined: Tue Sep 30, 2014 1:17 pm

Re: Long pauses during write

Postby lundman » Fri Jan 05, 2018 12:25 am

No, we discovered this problem under the "bsdtar" issue page, and believe we have improved it in master. So hopefully it will be better for you with the next release.
User avatar
lundman
 
Posts: 402
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Long pauses during write

Postby mkush » Fri Jan 05, 2018 7:10 pm

lundman wrote:No, we discovered this problem under the "bsdtar" issue page, and believe we have improved it in master. So hopefully it will be better for you with the next release.


Great! Is that something I can build now to try?
mkush
 
Posts: 29
Joined: Tue Sep 30, 2014 1:17 pm

Re: Long pauses during write

Postby Brendon » Sat Jan 06, 2018 1:40 pm

Yes,

Follow the wiki instructions to build and install the current “master” code.

Regardless of your specific issue, master works quite a lot better than 1.7.0 in various areas.

Cheers
Brendon
Brendon
 
Posts: 279
Joined: Thu Mar 06, 2014 12:51 pm

Re: Long pauses during write

Postby mkush » Thu Jan 11, 2018 12:20 pm

OK, I built the master and installed. And the results are VERY encouraging, and VERY odd.

First of all, my hardware setup is totally different so in fairness I probably should try regular-old 1.7.0 also to see how it fares.

I am now testing on:
MacBook Pro 15, 2017 model, 2TB boot SSD, 16GB memory
Akitio Thunder3 Quad Mini
Samsung 850 EVO 4TB SSD x4

Instead of a 2-drive mirror, I'm testing a 4-drive raidz.

I created the pool with:
sudo zpool create -f \
-o ashift=13 \
-O compression=lz4 \
-O casesensitivity=insensitive \
-O atime=off \
-O normalization=formD \
Archives-2 raidz disk2 disk3 disk4 disk5

I then dragged a 31GB folder of JPEGs to the new volume. There were perhaps slight pauses during the copy but nothing objectionable. The time was estimated at 5 minutes. That would have resulted in a rather sad 103MB/s (I'd think four modern SSDs over Thunderbolt 3 could do a lot better).

HOWEVER, a strange thing happened. About halfway through the copy, still reporting about 5 minutes to go, the speed increased DRAMATICALLY.

I wish I could just upload a video of the Finder progress progress bar. It was something to behold. It just poked along for about half the copy and then ZOOM. In any case, I will illustrate it like this: The entire copy took 147s. At about 125s, it was nearing halfway finished and still reporting 5 minutes left. Then it finished the second half of the copy in about 20s.

Also: subsequent copies ON THE SAME BOOT seem to be lightning-quick too, right from the start. For example, I then copied a 7GB folder in 8s flat, fast from the beginning.

So, it almost seems like there is some sort of odd warm-up period. Once you've written a certain quantity of data, it's off to the races from that point forward.

Does that make any sense to any of you?
mkush
 
Posts: 29
Joined: Tue Sep 30, 2014 1:17 pm

Re: Long pauses during write

Postby Brendon » Thu Jan 11, 2018 1:44 pm

Thanks for the clearly documented steps. I'm not 100% sure about the ashift=13, there has been considerable debate in the past over 12 vs 13 for SSDs. I am happy to hear that "master" is performing much better for you, that mirrors my experience as well.

You could run "zpool iostat 1" in a shell while doing the big copy, it would illustrate how ZFS believes it is performing.

You are using hardware well beyond anything that we possess to test with, there is some possibility that tuning of the SW may be required in some way that we don't yet understand.

Cheers
Brendon
Brendon
 
Posts: 279
Joined: Thu Mar 06, 2014 12:51 pm

Re: Long pauses during write

Postby mkush » Thu Jan 11, 2018 4:55 pm

Here's a thought. Could it have something to do with filling up the ARC? I set the ARC limit at 12GB via the config file. It is about at the point that 12GB have been copied that things rapidly speed up, and then stay fast in subsequent copies. So maybe I should set the ARC to a really low number and try again? Maybe it will get fast quicker.
mkush
 
Posts: 29
Joined: Tue Sep 30, 2014 1:17 pm

Re: Long pauses during write

Postby mkush » Thu Jan 11, 2018 5:08 pm

I think that's it! I dropped the ARC to 1MB and the copy completed in 44s total (for 31GB, meaning 704MB/s write). If it was slow to begin with it was not easy to see; it seemed pretty fast right from the start. This is on a fresh reboot (after setting the ARC max in zsysctl.conf).

Since this is a pure SSD pool anyway, maybe ARC doesn't do me so much good? I'm not going to be reading the same things over and over anyway, so the point of a cache seems moot; it's all a bunch of pictures and videos.

What do you think?
mkush
 
Posts: 29
Joined: Tue Sep 30, 2014 1:17 pm

Re: Long pauses during write

Postby mkush » Thu Jan 11, 2018 5:09 pm

mkush wrote:I think that's it! I dropped the ARC to 1MB and the copy completed in 44s total (for 31GB, meaning 704MB/s write). If it was slow to begin with it was not easy to see; it seemed pretty fast right from the start. This is on a fresh reboot (after setting the ARC max in zsysctl.conf).

Since this is a pure SSD pool anyway, maybe ARC doesn't do me so much good? I'm not going to be reading the same things over and over anyway, so the point of a cache seems moot; it's all a bunch of pictures and videos.

What do you think?


Sorry! I meant 1GB. Can I just set it to zero or does that have some other meaning?
mkush
 
Posts: 29
Joined: Tue Sep 30, 2014 1:17 pm

Re: Long pauses during write

Postby mkush » Thu Jan 11, 2018 5:41 pm

Alright! Final post for the night but I feel really confident I've discovered a correlation between ARC setting and the point at which my writes become fast.

First, a correction. The test folder is 23.73GB. I had said 31GB. Not sure where I got that from. So the performance number in MB/s that I posted previously is wrong.

So, I ran a "scientific" test of the effect of ARC size as set in /etc/zfs/zsysctl.conf and rebooting, then trying the same copy each time. It seems that setting ARC too low causes it to be ignored. Here are my copy times:

ARC=0: 142s, vast speedup noted about halfway thru the copy
ARC=1000: 144s, vast speedup noted about halfway thru the copy (no difference from 0)
ARC=1000000000 (1 billion): 44s, vast speedup nearly immediately
ARC=5000000000 (5 billion): 98s, speedup clearly visible as copy reached about 5GB (the ARC size!)
ARC=12000000000 (12 billion): 147s, about the same as 0 and 1000 above.

From the above I draw these conclusions:

1. Writes are slow until ARC is filled up, as noted from the 1GB, 5GB and 12GB ARC tests that I ran.
2. Very small ARC settings seem to be ignored, and the system uses something around 12GB on my system (which has 16GB of memory if that matters).

Does this make sense? At this point, I'm REALLY happy. I'd like to know what's going on but if all I have to do is set the ARC to 1GB to get performance like this, then that's what I'm doing!

By the way, as for read performance, I can copy the same folder back to my boot SSD in 27s (879MB/s). But I don't think read was ever slow.
mkush
 
Posts: 29
Joined: Tue Sep 30, 2014 1:17 pm

Next

Return to General Help

Who is online

Users browsing this forum: No registered users and 2 guests