Slow Write Performance

[Brendon]

mac pro 2013, OSX 10.11.something - copy single large file ~2G from internal ssd to exernal 5 disk raidz2.

tank 4.23T 14.0T 8 1.44K 35.9K 109M
tank 4.23T 14.0T 7 9.10K 104K 388M
tank 4.23T 14.0T 0 10.2K 0 435M
tank 4.23T 14.0T 4 10.1K 20.0K 432M
tank 4.23T 14.0T 0 7.43K 3.99K 314M
tank 4.23T 14.0T 22 8.50K 91.8K 345M
tank 4.23T 14.0T 11 3.30K 47.9K 142M
(truncated some entries, you get the idea)

mac pro 2008, 10.11.5, same file copied to internal raidz of 3 disks

big 237G 923G 0 414 0 51.9M
big 237G 923G 3 1.13K 15.9K 138M
big 237G 923G 0 607 0 75.5M
big 238G 922G 0 1.31K 0 159M
big 238G 922G 0 1.36K 0 175M
big 238G 922G 0 1.47K 0 188M
big 238G 922G 0 1.39K 0 178M
big 238G 922G 0 1.31K 0 167M
big 238G 922G 0 1.32K 0 168M
big 238G 922G 0 670 0 83.8M
big 238G 922G 0 1.17K 0 149M
big 238G 922G 0 292 0 24.2M
big 238G 922G 0 1.27K 0 163M
big 238G 922G 0 1.27K 0 162M
big 238G 922G 0 1.27K 0 163M
big 238G 922G 0 1.13K 0 145M
big 238G 922G 0 1.05K 0 135M
big 238G 922G 0 1.03K 0 132M
big 238G 922G 0 1.19K 0 152M
big 238G 922G 0 1.04K 0 133M
big 238G 922G 0 1.06K 0 136M

So of course none of this really helps you. You haven't been tuning things have you - thinking /etc/zfs/zsysctl.conf. Also make sure your O3X install is clean, no mismatched versions etc.

- Brendon

[Brendon]

Other thoughts - are you using dedup?

Perhaps post zpool status, zfs get all, and whatever commands you used to create the pools.

Another thing, try current code from repo.

Cheers
Brendon

[mstan]

mac pro 2013, OSX 10.11.something - copy single large file ~2G from internal ssd to exernal 5 disk raidz2.
You haven't been tuning things have you - thinking /etc/zfs/zsysctl.conf. Also make sure your O3X install is clean, no mismatched versions etc.

- Brendon

That helps in that at least I know that ZFS on a MAcPro is capable of decent writes. No, I have done no tuning at all. O3X install was clean.

[mstan]

Here is latest pool status:

zpool status
pool: MacProZFSPool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0 in 12h41m with 0 errors on Mon Aug 1 20:17:14 2016
config:

NAME STATE READ WRITE CKSUM
MacProZFSPool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
media-BC984D67-CE7E-9D4C-ABDE-48592A18ACE3 ONLINE 0 0 0
media-09A0B1BC-52A7-5249-9BA0-AB0945DF91D2 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
media-9BCA8A4C-C23A-F949-A246-CA3C6412B431 ONLINE 0 0 0
media-82F2ABE3-6C14-504E-8BE5-65DD65452F01 ONLINE 0 0 0

errors: No known data errors

[mstan]

Here is the get all result - I see that I didn't turn on compression - could that be a factor?

Last login: Tue Aug 2 19:46:43 on ttys000
Stan-Hendrixs-Mac-Pro:~ $ zfs get all
NAME PROPERTY VALUE SOURCE
MacProZFSPool type filesystem -
MacProZFSPool creation Tue Dec 15 6:41 2015 -
MacProZFSPool used 4.33T -
MacProZFSPool available 2.69T -
MacProZFSPool referenced 969G -
MacProZFSPool compressratio 1.00x -
MacProZFSPool mounted yes -
MacProZFSPool quota none default
MacProZFSPool reservation none default
MacProZFSPool recordsize 128K default
MacProZFSPool mountpoint /Volumes/MacProZFSPool default
MacProZFSPool sharenfs off default
MacProZFSPool checksum on default
MacProZFSPool compression off default
MacProZFSPool atime on default
MacProZFSPool devices on default
MacProZFSPool exec on default
MacProZFSPool setuid on default
MacProZFSPool readonly off default
MacProZFSPool zoned off default
MacProZFSPool snapdir hidden default
MacProZFSPool aclmode passthrough default
MacProZFSPool aclinherit restricted default
MacProZFSPool canmount on default
MacProZFSPool xattr on default
MacProZFSPool copies 1 default
MacProZFSPool version 5 -
MacProZFSPool utf8only on -
MacProZFSPool normalization formD -
MacProZFSPool casesensitivity sensitive -
MacProZFSPool vscan off default
MacProZFSPool nbmand off default
MacProZFSPool sharesmb off default
MacProZFSPool refquota none default
MacProZFSPool refreservation none default
MacProZFSPool primarycache all default
MacProZFSPool secondarycache all default
MacProZFSPool usedbysnapshots 0 -
MacProZFSPool usedbydataset 969G -
MacProZFSPool usedbychildren 3.38T -
MacProZFSPool usedbyrefreservation 0 -
MacProZFSPool logbias latency default
MacProZFSPool dedup off default
MacProZFSPool mlslabel none default
MacProZFSPool sync standard default
MacProZFSPool refcompressratio 1.00x -
MacProZFSPool written 969G -
MacProZFSPool logicalused 4.32T -
MacProZFSPool logicalreferenced 966G -
MacProZFSPool filesystem_limit none default
MacProZFSPool snapshot_limit none default
MacProZFSPool filesystem_count none default
MacProZFSPool snapshot_count none default
MacProZFSPool snapdev hidden default
MacProZFSPool com.apple.browse on default
MacProZFSPool com.apple.ignoreowner off default
MacProZFSPool com.apple.mimic_hfs off default
MacProZFSPool redundant_metadata all default
MacProZFSPool overlay off default
MacProZFSPool/Media type filesystem -
MacProZFSPool/Media creation Tue Dec 15 6:43 2015 -
MacProZFSPool/Media used 3.38T -
MacProZFSPool/Media available 2.69T -
MacProZFSPool/Media referenced 3.38T -
MacProZFSPool/Media compressratio 1.00x -
MacProZFSPool/Media mounted yes -
MacProZFSPool/Media quota none default
MacProZFSPool/Media reservation none default
MacProZFSPool/Media recordsize 128K default
MacProZFSPool/Media mountpoint /Volumes/MacProZFSPool/Media default
MacProZFSPool/Media sharenfs off default
MacProZFSPool/Media checksum on default
MacProZFSPool/Media compression off default
MacProZFSPool/Media atime on default
MacProZFSPool/Media devices on default
MacProZFSPool/Media exec on default
MacProZFSPool/Media setuid on default
MacProZFSPool/Media readonly off default
MacProZFSPool/Media zoned off default
MacProZFSPool/Media snapdir hidden default
MacProZFSPool/Media aclmode passthrough default
MacProZFSPool/Media aclinherit restricted default
MacProZFSPool/Media canmount on default
MacProZFSPool/Media xattr on default
MacProZFSPool/Media copies 1 default
MacProZFSPool/Media version 5 -
MacProZFSPool/Media utf8only on -
MacProZFSPool/Media normalization formD -
MacProZFSPool/Media casesensitivity sensitive -
MacProZFSPool/Media vscan off default
MacProZFSPool/Media nbmand off default
MacProZFSPool/Media sharesmb off default
MacProZFSPool/Media refquota none default
MacProZFSPool/Media refreservation none default
MacProZFSPool/Media primarycache all default
MacProZFSPool/Media secondarycache all default
MacProZFSPool/Media usedbysnapshots 0 -
MacProZFSPool/Media usedbydataset 3.38T -
MacProZFSPool/Media usedbychildren 0 -
MacProZFSPool/Media usedbyrefreservation 0 -
MacProZFSPool/Media logbias latency default
MacProZFSPool/Media dedup off default
MacProZFSPool/Media mlslabel none default
MacProZFSPool/Media sync standard default
MacProZFSPool/Media refcompressratio 1.00x -
MacProZFSPool/Media written 3.38T -
MacProZFSPool/Media logicalused 3.38T -
MacProZFSPool/Media logicalreferenced 3.38T -
MacProZFSPool/Media filesystem_limit none default
MacProZFSPool/Media snapshot_limit none default
MacProZFSPool/Media filesystem_count none default
MacProZFSPool/Media snapshot_count none default
MacProZFSPool/Media snapdev hidden default
MacProZFSPool/Media com.apple.browse on default
MacProZFSPool/Media com.apple.ignoreowner off default
MacProZFSPool/Media com.apple.mimic_hfs off default
MacProZFSPool/Media redundant_metadata all default
MacProZFSPool/Media overlay off default

[mstan]

Other thoughts - are you using dedup?

Perhaps post zpool status, zfs get all, and whatever commands you used to create the pools.

Another thing, try current code from repo.

Cheers
Brendon

Not using dedup. What do you mean current code from repo? The download image repository?

[mstan]

Here are the pool properties:
zpool get all MacProZFSPool
NAME PROPERTY VALUE SOURCE
MacProZFSPool size 7.25T -
MacProZFSPool capacity 59% -
MacProZFSPool altroot - default
MacProZFSPool health ONLINE -
MacProZFSPool guid 8657621597889387473 default
MacProZFSPool version - default
MacProZFSPool bootfs - default
MacProZFSPool delegation on default
MacProZFSPool autoreplace off default
MacProZFSPool cachefile - default
MacProZFSPool failmode wait default
MacProZFSPool listsnapshots off default
MacProZFSPool autoexpand on local
MacProZFSPool dedupditto 0 default
MacProZFSPool dedupratio 1.00x -
MacProZFSPool free 2.92T -
MacProZFSPool allocated 4.33T -
MacProZFSPool readonly off -
MacProZFSPool ashift 12 local
MacProZFSPool comment - default
MacProZFSPool expandsize - -
MacProZFSPool freeing 0 default
MacProZFSPool fragmentation 24% -
MacProZFSPool leaked 0 default
MacProZFSPool feature@async_destroy enabled local
MacProZFSPool feature@empty_bpobj active local
MacProZFSPool feature@lz4_compress active local
MacProZFSPool feature@multi_vdev_crash_dump disabled local
MacProZFSPool feature@spacemap_histogram active local
MacProZFSPool feature@enabled_txg active local
MacProZFSPool feature@hole_birth active local
MacProZFSPool feature@extensible_dataset enabled local
MacProZFSPool feature@embedded_data active local
MacProZFSPool feature@bookmarks enabled local
MacProZFSPool feature@filesystem_limits enabled local
MacProZFSPool feature@large_blocks enabled local
MacProZFSPool feature@sha512 disabled local
MacProZFSPool feature@skein disabled local
MacProZFSPool feature@edonr disabled local

[Brendon]

You could turn atime off.
You should turn compression on, its always an advantage.

Beyond that things look ok.

I'd be carefully checking your drives to see if any of them are underperforming. We have had cases recently where a dying hdd has resulted in very poor performance of an array.

- Brendon

[mstan]

Thanks for the support.

Will iostat show performance of individual drives?
Will SMART report anything while these drives are in an active ZFS pool.
Would a different raid configuration make any difference? On both my pools I use mirrored pairs which I thought would be fast but maybe not?

[Brendon]

zpool iostat can show individual drives. If a drive is slow, then the next step may be to inspect with SMART as you say.

I don't think a mirror of stripes should be slow. Up to you as to whether you try a different setup, certainly can't hurt to try.

- Brendon