Write performance

Developer discussions.

Write performance

Postby lundman » Sun Nov 04, 2018 9:21 pm

So we sped up the unlinking a bit before, lets take a look at write performance. First I need to try to replicate what people experience so I can if I make it better,

So I have an ancient MacBookPro, and plugged in an Intel 300G SSD to it. Writing the same 1GB file, to 100 files gives me:

Code: Select all
ZFS:
1073741824 bytes transferred in 29.548205 secs (36338648 bytes/sec)
1073741824 bytes transferred in 31.414408 secs (34179916 bytes/sec)
1073741824 bytes transferred in 45.366121 secs (23668363 bytes/sec)
1073741824 bytes transferred in 28.718671 secs (37388284 bytes/sec)
1073741824 bytes transferred in 29.824061 secs (36002536 bytes/sec)
1073741824 bytes transferred in 33.351453 secs (32194754 bytes/sec)
1073741824 bytes transferred in 34.745797 secs (30902783 bytes/sec)
1073741824 bytes transferred in 33.533539 secs (32019937 bytes/sec)
1073741824 bytes transferred in 29.934258 secs (35870000 bytes/sec)
1073741824 bytes transferred in 32.869244 secs (32667068 bytes/sec)
1073741824 bytes transferred in 30.018757 secs (35769030 bytes/sec)
1073741824 bytes transferred in 30.330760 secs (35401085 bytes/sec)
1073741824 bytes transferred in 34.364843 secs (31245358 bytes/sec)
1073741824 bytes transferred in 29.871598 secs (35945242 bytes/sec)
1073741824 bytes transferred in 39.967623 secs (26865291 bytes/sec)
1073741824 bytes transferred in 29.906593 secs (35903181 bytes/sec)
1073741824 bytes transferred in 24.311085 secs (44166759 bytes/sec)
1073741824 bytes transferred in 35.506952 secs (30240327 bytes/sec)
1073741824 bytes transferred in 29.359566 secs (36572129 bytes/sec)
1073741824 bytes transferred in 30.261238 secs (35482416 bytes/sec)
1073741824 bytes transferred in 33.770840 secs (31794940 bytes/sec)
1073741824 bytes transferred in 30.132733 secs (35633735 bytes/sec)
1073741824 bytes transferred in 28.297547 secs (37944696 bytes/sec)
1073741824 bytes transferred in 34.076229 secs (31509995 bytes/sec)
1073741824 bytes transferred in 31.170477 secs (34447398 bytes/sec)
1073741824 bytes transferred in 26.228555 secs (40937895 bytes/sec)
1073741824 bytes transferred in 35.473493 secs (30268850 bytes/sec)
1073741824 bytes transferred in 33.797716 secs (31769656 bytes/sec)
1073741824 bytes transferred in 26.073193 secs (41181831 bytes/sec)
1073741824 bytes transferred in 38.908066 secs (27596895 bytes/sec)
1073741824 bytes transferred in 33.695992 secs (31865565 bytes/sec)
1073741824 bytes transferred in 34.348147 secs (31260546 bytes/sec)
1073741824 bytes transferred in 29.803245 secs (36027682 bytes/sec)
1073741824 bytes transferred in 32.881115 secs (32655274 bytes/sec)
1073741824 bytes transferred in 34.176195 secs (31417828 bytes/sec)
1073741824 bytes transferred in 30.003933 secs (35786702 bytes/sec)
1073741824 bytes transferred in 33.223096 secs (32319138 bytes/sec)
1073741824 bytes transferred in 29.236808 secs (36725686 bytes/sec)
1073741824 bytes transferred in 33.939400 secs (31637030 bytes/sec)
1073741824 bytes transferred in 30.726682 secs (34944932 bytes/sec)
1073741824 bytes transferred in 30.267948 secs (35474550 bytes/sec)
1073741824 bytes transferred in 39.794005 secs (26982502 bytes/sec)
1073741824 bytes transferred in 38.554431 secs (27850024 bytes/sec)
1073741824 bytes transferred in 29.862184 secs (35956574 bytes/sec)
1073741824 bytes transferred in 32.322123 secs (33220028 bytes/sec)
1073741824 bytes transferred in 33.129071 secs (32410864 bytes/sec)
1073741824 bytes transferred in 29.753261 secs (36088206 bytes/sec)
1073741824 bytes transferred in 29.698719 secs (36154483 bytes/sec)
1073741824 bytes transferred in 31.306076 secs (34298193 bytes/sec)
1073741824 bytes transferred in 37.512250 secs (28623765 bytes/sec)
1073741824 bytes transferred in 25.512924 secs (42086192 bytes/sec)
1073741824 bytes transferred in 40.135097 secs (26753189 bytes/sec)
1073741824 bytes transferred in 36.927949 secs (29076671 bytes/sec)
1073741824 bytes transferred in 28.803797 secs (37277787 bytes/sec)
1073741824 bytes transferred in 29.816472 secs (36011699 bytes/sec)
1073741824 bytes transferred in 33.957269 secs (31620382 bytes/sec)
1073741824 bytes transferred in 30.428104 secs (35287832 bytes/sec)
1073741824 bytes transferred in 35.931459 secs (29883057 bytes/sec)
1073741824 bytes transferred in 32.867031 secs (32669267 bytes/sec)
1073741824 bytes transferred in 31.537740 secs (34046251 bytes/sec)
1073741824 bytes transferred in 38.654583 secs (27777866 bytes/sec)
1073741824 bytes transferred in 35.984522 secs (29838991 bytes/sec)
1073741824 bytes transferred in 29.409050 secs (36510592 bytes/sec)
1073741824 bytes transferred in 34.300233 secs (31304214 bytes/sec)
1073741824 bytes transferred in 34.499185 secs (31123687 bytes/sec)
1073741824 bytes transferred in 33.824134 secs (31744843 bytes/sec)
1073741824 bytes transferred in 30.469279 secs (35240145 bytes/sec)
1073741824 bytes transferred in 39.691228 secs (27052371 bytes/sec)
1073741824 bytes transferred in 35.377033 secs (30351381 bytes/sec)
1073741824 bytes transferred in 32.751384 secs (32784624 bytes/sec)
1073741824 bytes transferred in 35.400002 secs (30331688 bytes/sec)
1073741824 bytes transferred in 29.610129 secs (36262653 bytes/sec)
1073741824 bytes transferred in 29.660297 secs (36201317 bytes/sec)
1073741824 bytes transferred in 30.314422 secs (35420165 bytes/sec)
1073741824 bytes transferred in 29.707132 secs (36144244 bytes/sec)
1073741824 bytes transferred in 28.840543 secs (37230291 bytes/sec)
1073741824 bytes transferred in 31.875907 secs (33685060 bytes/sec)
1073741824 bytes transferred in 34.413670 secs (31201026 bytes/sec)
1073741824 bytes transferred in 30.065147 secs (35713839 bytes/sec)
1073741824 bytes transferred in 30.026860 secs (35759378 bytes/sec)
1073741824 bytes transferred in 34.668755 secs (30971456 bytes/sec)
1073741824 bytes transferred in 29.900384 secs (35910637 bytes/sec)
1073741824 bytes transferred in 34.886119 secs (30778483 bytes/sec)
1073741824 bytes transferred in 35.998165 secs (29827682 bytes/sec)
1073741824 bytes transferred in 34.077433 secs (31508882 bytes/sec)
1073741824 bytes transferred in 35.185994 secs (30516171 bytes/sec)
1073741824 bytes transferred in 30.274068 secs (35467378 bytes/sec)
1073741824 bytes transferred in 34.764270 secs (30886362 bytes/sec)
1073741824 bytes transferred in 39.107077 secs (27456458 bytes/sec)
1073741824 bytes transferred in 35.199834 secs (30504173 bytes/sec)
1073741824 bytes transferred in 32.849073 secs (32687127 bytes/sec)
1073741824 bytes transferred in 29.667305 secs (36192766 bytes/sec)
1073741824 bytes transferred in 29.897006 secs (35914694 bytes/sec)


Code: Select all
HFS:
1073741824 bytes transferred in 27.987088 secs (38365614 bytes/sec)
1073741824 bytes transferred in 27.594821 secs (38910991 bytes/sec)
1073741824 bytes transferred in 27.506261 secs (39036270 bytes/sec)
1073741824 bytes transferred in 27.953763 secs (38411352 bytes/sec)
1073741824 bytes transferred in 28.037099 secs (38297180 bytes/sec)
1073741824 bytes transferred in 27.804322 secs (38617803 bytes/sec)
1073741824 bytes transferred in 27.509029 secs (39032342 bytes/sec)
1073741824 bytes transferred in 27.508027 secs (39033764 bytes/sec)
1073741824 bytes transferred in 27.932570 secs (38440495 bytes/sec)
1073741824 bytes transferred in 28.016971 secs (38324694 bytes/sec)
1073741824 bytes transferred in 27.780923 secs (38650330 bytes/sec)
1073741824 bytes transferred in 27.516863 secs (39021230 bytes/sec)
1073741824 bytes transferred in 27.493738 secs (39054050 bytes/sec)
1073741824 bytes transferred in 27.944166 secs (38424544 bytes/sec)
1073741824 bytes transferred in 28.048910 secs (38281053 bytes/sec)
1073741824 bytes transferred in 27.725003 secs (38728285 bytes/sec)
1073741824 bytes transferred in 27.521611 secs (39014498 bytes/sec)
1073741824 bytes transferred in 27.523356 secs (39012024 bytes/sec)
1073741824 bytes transferred in 27.984034 secs (38369801 bytes/sec)
1073741824 bytes transferred in 28.054239 secs (38273782 bytes/sec)
1073741824 bytes transferred in 27.666527 secs (38810141 bytes/sec)
1073741824 bytes transferred in 27.520963 secs (39015416 bytes/sec)
1073741824 bytes transferred in 27.508899 secs (39032526 bytes/sec)
1073741824 bytes transferred in 28.105552 secs (38203905 bytes/sec)
1073741824 bytes transferred in 28.075070 secs (38245383 bytes/sec)
1073741824 bytes transferred in 27.584938 secs (38924932 bytes/sec)
1073741824 bytes transferred in 27.515769 secs (39022781 bytes/sec)
1073741824 bytes transferred in 27.603091 secs (38899333 bytes/sec)
1073741824 bytes transferred in 28.043637 secs (38288251 bytes/sec)
1073741824 bytes transferred in 28.050496 secs (38278889 bytes/sec)
1073741824 bytes transferred in 27.571590 secs (38943776 bytes/sec)
1073741824 bytes transferred in 27.566323 secs (38951217 bytes/sec)
1073741824 bytes transferred in 27.668970 secs (38806715 bytes/sec)
1073741824 bytes transferred in 28.044560 secs (38286991 bytes/sec)
1073741824 bytes transferred in 28.061826 secs (38263434 bytes/sec)
1073741824 bytes transferred in 27.569282 secs (38947036 bytes/sec)
1073741824 bytes transferred in 27.522360 secs (39013436 bytes/sec)
1073741824 bytes transferred in 27.755314 secs (38685991 bytes/sec)
1073741824 bytes transferred in 28.050927 secs (38278301 bytes/sec)
1073741824 bytes transferred in 27.953804 secs (38411295 bytes/sec)
1073741824 bytes transferred in 27.539496 secs (38989160 bytes/sec)
1073741824 bytes transferred in 27.521976 secs (39013980 bytes/sec)
1073741824 bytes transferred in 27.836580 secs (38573051 bytes/sec)
1073741824 bytes transferred in 28.046696 secs (38284075 bytes/sec)
1073741824 bytes transferred in 27.893297 secs (38494618 bytes/sec)
1073741824 bytes transferred in 27.525119 secs (39009525 bytes/sec)
1073741824 bytes transferred in 27.494135 secs (39053486 bytes/sec)
1073741824 bytes transferred in 27.857376 secs (38544256 bytes/sec)
1073741824 bytes transferred in 28.043280 secs (38288739 bytes/sec)
1073741824 bytes transferred in 27.863579 secs (38535675 bytes/sec)


Looks like we can match HFS quite often, but ZFS has a lot of variation, highest at 45s.

Is this what people means when we talk about "poor performance" ? Is write speed the worst? Or read speed? Listing?
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Write performance

Postby jawbroken » Tue Nov 06, 2018 3:27 am

Thanks for looking into it. This is going to be frustratingly vague, but I think I have more trouble with what I think of as read performance than write. But it's not a straightforward issue, because my scrub performance is very good (e.g. ~80 MB/s per drive). It's more a vague feeling that things are slow, like listing files in a directory (often Finder will just spin for a long time on first access before showing the file list), seeking in a video file (sometimes immediate, sometimes a long period of buffering), things like that. I have another vague feeling that ARC isn't quite working right, also, because I don't seem to get a great hit rate (I seem to recall a better hit rate with Zevo with a similar setup and workload) and I think some people on the forums report improved performance from lowering their ARC size. Sometimes situations just seem counterintuitive, e.g. I'm scrubbing my pool at the moment and streaming video to the Air Video HD app on my iPad seems to be working better than when I am not scrubbing. Not sure if this is at all helpful, I know if the problem was simple you probably would have found it by now.
jawbroken
 
Posts: 61
Joined: Wed Apr 01, 2015 4:46 am

Re: Write performance

Postby lundman » Tue Nov 06, 2018 4:06 pm

Is that the consensus? Should focus on read performance over write? Or any other for that matter.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Write performance

Postby tangles » Thu Nov 08, 2018 3:09 am

Hi Lundy,

Thank you so much for focusing your time on this.
Personally, I'm happy for anyone to tackle either.
Is it at all possible you can start your investigation using multiple vdevs as well?
I'm willing to donate the cash so you can have identical external Firewire/eSata drives for testing. (maybe 3 to include raidz testing)
I have an old MacPro 1,1 I can donate to you if you want it, but as you know, it can't run anything past macOS 10.11 (using bootloader hack)
Alternatively, if you have a static IP, I'm happy to punch a whole through my router if you want ssh access to some test boxes.

I also use a Kanex Thunderbolt to eSATA adapter with laptops to connect disks as it it PM capable.
https://www.apple.com/shop/product/HGC8 ... 30-adapter

Just yell out Lundy if you do want some more hardware… that I can help you with no probs.
tangles
 
Posts: 195
Joined: Tue Jun 17, 2014 6:54 am

Re: Write performance

Postby lundman » Thu Nov 08, 2018 6:14 pm

Cheers, always appreciated. The patreon donations are high enough that I will buy a thunderbolt something so I can do some high performance testing. Disks and SSDs are not a problem, I just never had a great way to attach them (while avoiding USB). Thunderbolt things don't appear available in Japan, so I'll get something from the states.

I'd be curious in testing the #666 hack test, since it affects writes, and reads, and would in fact be worse for multiple vdev. Although, admittedly it doesn't fix the actual problem, but perhaps alleviates some of it in the IO path. I will need to clean up the hack, so it's nicer, and maybe you can give it a test, to see if it is worth looking it at, or not.

I used to have a macpro1,1, loved that machine, currently have a 4,1 at home. At work I have a trashcan, but obviously it sucks to dev on the machine you test on - maybe i should pick up a 2nd hand mac to do real testing. VMs are great for dev, terrible for performance troubleshooting.

Oh hey, thunderbolt to esata? That changes everything!
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Write performance

Postby lundman » Wed Nov 14, 2018 1:23 am

For those that can, try branch "avoid_alloc" to see if IO is a bit more level
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Write performance

Postby tangles » Tue Nov 27, 2018 11:08 pm

Hey Lundy,

what's the commands to get the avoid_alloc build?

do I gitclone or use zfsadmin?

Happy to test a before and after on the same hardware for you if you like...
tangles
 
Posts: 195
Joined: Tue Jun 17, 2014 6:54 am

Re: Write performance

Postby lundman » Wed Nov 28, 2018 6:41 pm

With git, you would check it out:

git clone http://github.com/openzfsonosx/spl.git
cd spl
./autogen.sh
./configure
make

cd ../
git clone http://github.com/openzfsonosx/zfs.git
cd zfs
git checkout avoid_alloc
./autogen.sh
./configure
make
./load.sh

I can probably make you a pkg if that is easier, although, if you know how to build master, you'll always be able to run the latest :)
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Write performance

Postby tangles » Thu Nov 29, 2018 12:38 am

okay,

So before I pull down the latest, do you have a test in mind that we should use for consistency?

Those tests you carried out with lots of small files to = nGB versus few large files that also = nGB sounds like a winner to me.

Perhaps it's time to set a new benchmark wiki page? I don't think I'll be selling my 2012 MacPro any time soon.
I also have a 2013 MacPro that I could use but sadly Apple may end its life in a couple of years also. :cry:
tangles
 
Posts: 195
Joined: Tue Jun 17, 2014 6:54 am


Return to OpenZFS on OS X Development

Who is online

Users browsing this forum: No registered users and 14 guests

cron