Slow reads and writes

Moderators: jhartley, MSR734, nola

Slow reads and writes

Post by marcomwest » Thu Sep 20, 2012 8:52 pm

Hi,

I've been experiencing really slow reads and writes. There is a long pause before stuff shows up in the "open file" window, for example, or even when making a screen capture. It can take a minute or so for new screen captures to show up on the desktop.

I've attached a screenshot of what I mean. Lots of really slow or tiny reads.

I don't know how to fix this and it makes the machine painfully slow to use.

Help.

PS- I see that someone else is having a similar problem on the thread "Set ARC cache size".

slow_zfs_mirror.png
slow_zfs_mirror.png (35.27 KiB) Viewed 325 times
marcomwest Offline


 
Posts: 4
Joined: Sat Sep 15, 2012 12:03 pm

more detail please

Post by grahamperrin » Sat Sep 22, 2012 2:38 am

Seagate ST2000DL004 also known as Samsung HD204UI, yes?

A little more detail please:

* model of Mac
* memory
* output from the following commands.

diskutil list

zfs list
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Slow reads and writes

Post by Jimbo » Sat Sep 22, 2012 6:07 am

I'm seeing this right now with an rsync from an external USB3 HFS+ HDD to an external Firewire 800 Zevo drive. The Zevo drive is Seagate Momentus ST9750420AS. This is an Advanced Format drive however the ashift value returned from zdb -l was 9, so when I created the pool I specified ashift=12.

This is on a Retina MBP with 16Gb RAM and an i7.

Issuing commands against zfs or zpool stall for ages before completing. If I kill the rsync, they're ok.

There is minimal CPU usage.

Output:
Code: Select all
zfs list
NAME          USED   AVAIL   REFER  MOUNTPOINT
WorkBackup   122Gi   563Gi   122Gi  /Volumes/WorkBackup

Code: Select all
diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *251.0 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS Macintosh HD            250.1 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.5 TB     disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                  Apple_HFS Work                    1.5 TB     disk1s2
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *750.2 GB   disk2
   1:                        EFI                         209.7 MB   disk2s1
   2:                        ZFS                         749.8 GB   disk2s2
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:             zfs_pool_proxy WorkBackup             *747.3 GB   disk3


Some I/O stats for copying one big file with rsync (sample):
Code: Select all
zpool iostat 1
                capacity       operations       bandwidth
pool         alloc    free    read   write    read   write
----------  ------  ------  ------  ------  ------  ------
WorkBackup   122Gi   574Gi       0      16       0  2.12Mi
WorkBackup   122Gi   574Gi       0     396       0  45.1Mi
WorkBackup   122Gi   574Gi       0     443       0  46.1Mi
WorkBackup   122Gi   574Gi       0      84       0  10.2Mi
WorkBackup   122Gi   574Gi       0      21       0  2.75Mi
WorkBackup   122Gi   574Gi       0      18       0  2.37Mi
WorkBackup   122Gi   574Gi       0      15       0  2.00Mi
WorkBackup   122Gi   574Gi       0      67       0  8.49Mi
WorkBackup   122Gi   574Gi       0       5       0   767Ki
WorkBackup   122Gi   574Gi       0      29       0  3.75Mi
WorkBackup   122Gi   574Gi       0       0       0   128Ki
WorkBackup   122Gi   574Gi       0       9       0  1.25Mi
WorkBackup   122Gi   574Gi       0      16       0  2.12Mi
WorkBackup   122Gi   574Gi       0      11       0  1.50Mi
WorkBackup   122Gi   574Gi       0       0       0   128Ki
WorkBackup   122Gi   574Gi       0       0       0       0
WorkBackup   122Gi   574Gi       0      24       0  3.12Mi
WorkBackup   122Gi   574Gi       0      19       0  2.50Mi
WorkBackup   122Gi   574Gi       0       4       0   639Ki
WorkBackup   122Gi   574Gi       0       0       0       0
WorkBackup   122Gi   574Gi       0       0       0       0
WorkBackup   122Gi   574Gi       0       0       0       0
WorkBackup   122Gi   574Gi       0      91       0  2.79Mi


Update: forgot, this is 10.8.2 - never tried this prior to upgrading...
Jimbo Offline


 
Posts: 5
Joined: Tue Sep 18, 2012 4:59 am

rsync, Carbon Copy Cloner, bandwidth and performance

Post by grahamperrin » Sat Sep 22, 2012 8:07 am

Please note: parts of this post have moved to
rsync, Carbon Copy Cloner, bandwidth and performance under
Need Help with a Backup Solution

Jimbo wrote:… minimal CPU usage. … 10.8.2 …


With this modern OS, for some tasks you may find that where CPU usage is relatively low: GPU usage is relatively high. For at-a-glance views of GPU usage, I like iStat Menus but I don't leave it installed at all times.

Jimbo, please: for test/comparison purposes, does use of bandwidth seem better if you use (say) ditto to copy a comparable set of data to ZFS?

(I don't suggest ditto as an alternative to rsync. Just aiming to see whether issues are more noticeable with rsync than with other processes that may write to a ZFS file system.)
Last edited by grahamperrin on Sun Nov 11, 2012 4:21 am, edited 4 times in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Slow reads and writes

Post by Jimbo » Sat Sep 22, 2012 8:27 am

Well, I see very erratic I/O performance even with a straight 'cp'. Copying a 3+Gb file from HFS+ to ZGS results in these sort of numbers (nothing else going on with the system that I am aware of):
Code: Select all
 zpool iostat 1
                capacity       operations       bandwidth
pool         alloc    free    read   write    read   write
----------  ------  ------  ------  ------  ------  ------
WorkBackup  2.81Gi   693Gi       0      61       0  7.74Mi
WorkBackup  2.86Gi   693Gi       0     367       0  40.9Mi
WorkBackup  2.90Gi   693Gi       0     437       0  44.6Mi
WorkBackup  2.90Gi   693Gi       0     238  3.99Ki  21.8Mi
WorkBackup  2.94Gi   693Gi       0       3       0  16.0Ki
WorkBackup  2.94Gi   693Gi       0       0       0       0
WorkBackup  2.94Gi   693Gi       0       0       0       0
WorkBackup  2.94Gi   693Gi       0      38       0  4.79Mi
WorkBackup  2.94Gi   693Gi       0      46       0  5.87Mi
WorkBackup  2.94Gi   693Gi       0      36       0  4.62Mi
WorkBackup  2.94Gi   693Gi       0      38       0  4.75Mi
WorkBackup  2.94Gi   693Gi       0      70       0  5.11Mi
WorkBackup  2.97Gi   693Gi       0       3       0  16.0Ki
Jimbo Offline


 
Posts: 5
Joined: Tue Sep 18, 2012 4:59 am

Re: Slow reads and writes

Post by Jimbo » Sat Sep 22, 2012 8:32 am

FWIW, copying the 3+Gb file from the ZFS volume back to the internal (SSD) drive, I still see erratic I/O read times:
Code: Select all
zpool iostat 1
                capacity       operations       bandwidth
pool         alloc    free    read   write    read   write
----------  ------  ------  ------  ------  ------  ------
WorkBackup  2.97Gi   693Gi      42       0  5.37Mi       0
WorkBackup  2.97Gi   693Gi     415       0  51.2Mi       0
WorkBackup  2.97Gi   693Gi     115       0  14.5Mi       0
WorkBackup  2.97Gi   693Gi     105       0  13.0Mi       0
WorkBackup  2.97Gi   693Gi     137       0  17.2Mi       0
WorkBackup  2.97Gi   693Gi     261       0  32.5Mi       0
WorkBackup  2.97Gi   693Gi     527       0  65.4Mi       0
WorkBackup  2.97Gi   693Gi     513       0  63.8Mi       0
WorkBackup  2.97Gi   693Gi     476       0  59.2Mi       0
WorkBackup  2.97Gi   693Gi      51       0  6.26Mi       0
WorkBackup  2.97Gi   693Gi      71       0  8.99Mi       0
WorkBackup  2.97Gi   693Gi      43       0  5.49Mi       0
WorkBackup  2.97Gi   693Gi      48       0  6.12Mi       0
WorkBackup  2.97Gi   693Gi      62       0  7.86Mi       0
WorkBackup  2.97Gi   693Gi      45       0  5.74Mi       0
WorkBackup  2.97Gi   693Gi      70       0  8.86Mi       0
WorkBackup  2.97Gi   693Gi     452       0  55.7Mi       0
Jimbo Offline


 
Posts: 5
Joined: Tue Sep 18, 2012 4:59 am

Re: Slow reads and writes

Post by Jimbo » Sun Sep 23, 2012 8:37 am

I think this might be drive-with-zfs related rather than enclosure related. I've tried the ST9750420AS drive in different enclosures (which work fine with other drives) on different interfaces (USB and Firewire) and performance is still spotty. Performance of the drive was fine using HFS+. I've got Zevo running on other (different model) drives and they don't exhibit this problem.
Jimbo Offline


 
Posts: 5
Joined: Tue Sep 18, 2012 4:59 am

Re: more detail please

Post by marcomwest » Sun Sep 23, 2012 1:12 pm

grahamperrin wrote:Seagate ST2000DL004 also known as Samsung HD204UI, yes?

A little more detail please:

* model of Mac
* memory
* output from the following commands.

diskutil list

zfs list


Hi there,

I have a Mac Pro 4,1 (Early 2009) 2 x 2.26 GHz Quad-Core Intel Xeon. 24GB of RAM. Running 10.7.4

diskutil list
Code: Select all
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *240.1 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS SSD_Start               239.2 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                        ZFS                         2.0 TB     disk1s2
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk2
   1:                        EFI                         209.7 MB   disk2s1
   2:                        ZFS                         2.0 TB     disk2s2


zfs list
Code: Select all
NAME         USED   AVAIL   REFER  MOUNTPOINT
userdataz   454Gi  1.35Ti   454Gi  /Volumes/userdataz
marcomwest Offline


 
Posts: 4
Joined: Sat Sep 15, 2012 12:03 pm

Re: Slow reads and writes

Post by grahamperrin » Sat Sep 29, 2012 11:38 pm

Jimbo wrote:Well, I see very erratic I/O performance even with a straight 'cp'. …


Taking a hint from viewtopic.php?p=482#p482

You could use spindump(8) to profile the system at the time when performance with cp is not as you expect it.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Slow reads and writes

Post by Jimbo » Sun Sep 30, 2012 9:31 pm

Log for spindump attached.
Attachments
spindump.zip
(139.17 KiB) Downloaded 6 times
Jimbo Offline


 
Posts: 5
Joined: Tue Sep 18, 2012 4:59 am


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 0 guests

cron