Slow IO on ElCapitan

[sebu06]

Hi,
I'm running OSX ElCapitan and created a ZFS pool on an external USB3.0 drive. But somehow it became very slow. I don't need that drive to be really fast, I know that ZFS is not designed to be fast, but to be safe - but it wasn't that slow before my OSX update, I think.

Code: Select all

$ zpool iostot
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
extpool   819G   109G    122     52  11.3M  4.39M


Code: Select all

$ zpool status
  pool: extpool
 state: ONLINE
  scan: scrub repaired 0 in 68h10m with 0 errors on Mon Oct 12 19:31:24 2015
config:

   NAME                                          STATE     READ WRITE CKSUM
   extpool                                       ONLINE       0     0     0
     media-D152463E-B753-4C01-9B27-5F738F28274B  ONLINE       0     0     0

errors: No known data errors


Code: Select all

$ zpool get all extpool
NAME        PROPERTY                    VALUE                       SOURCE
extpool     size                        928G                        -
extpool     capacity                    88%                         -
extpool     altroot                     -                           default
extpool     health                      ONLINE                      -
extpool     guid                        16673695283323271852        default
extpool     version                     -                           default
extpool     bootfs                      -                           default
extpool     delegation                  on                          default
extpool     autoreplace                 off                         default
extpool     cachefile                   -                           default
extpool     failmode                    wait                        default
extpool     listsnapshots               off                         default
extpool     autoexpand                  off                         default
extpool     dedupditto                  0                           default
extpool     dedupratio                  1.00x                       -
extpool     free                        106G                        -
extpool     allocated                   822G                        -
extpool     readonly                    off                         -
extpool     ashift                      0                           default
extpool     comment                     -                           default
extpool     expandsize                  -                           -
extpool     freeing                     0                           default
extpool     fragmentation               33%                         -
extpool     leaked                      0                           default
extpool     feature@async_destroy       enabled                     local
extpool     feature@empty_bpobj         enabled                     local
extpool     feature@lz4_compress        active                      local
extpool     feature@spacemap_histogram  active                      local
extpool     feature@enabled_txg         active                      local
extpool     feature@hole_birth          active                      local
extpool     feature@extensible_dataset  enabled                     local
extpool     feature@embedded_data       active                      local
extpool     feature@bookmarks           enabled                     local
extpool     feature@filesystem_limits   enabled                     local
extpool     feature@large_blocks        enabled                     local


Is there anything to speed this up, or are these speeds normal?

Best, Sebastian

P.S.: I just realised that I have a quite high fragmentation and my pool is relatively full - could this explain my significant performance drop?

[Brendon]

I have not noticed any change in performance, El Capitan or otherwise. Your numbers do seem very low.

Cheers
Brendon

[lundman]

ZFS gets slow when you are in the high 90s of being full, like 95% etc. Maybe that is what you are experiencing ?

[sebu06]

Hi,
I assumed that this might be the reason, therefore I deleted some files and it got a little bit better. But still it is not that fast. What is the recommended size I should keep free?

Code: Select all

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
extpool   681G   247G    264      6  26.6M   189K


Best,
Sebastian

[Brendon]

thats heaps of free space, probably not your problem.

Brendon

[haer22]

I suggest you run "zpool iostat 30" so you can get a running average. With only "zpool iostat" I think you get the average since reboot.

Also, how are you reading the data from zfs? Database? file access? small files? big files?

Hi,
I assumed that this might be the reason, therefore I deleted some files and it got a little bit better. But still it is not that fast. What is the recommended size I should keep free?

Code: Select all

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
extpool   681G   247G    264      6  26.6M   189K


Best,
Sebastian

[nekton34]

I had a similar situation + other Finder hanging issues with an external USB3 disk array that started perhaps after upgrading from 10.10 to El Cap.
I don't know if Apple changed something in USB3 drivers with El Cap but the problem was solved by switching to a Thunderbolt-to-eSata connection (which the case had in addition to USB3). The difference in speed was astonishing. It got even better when we moved to a pure Thunderbolt case.
Perhaps try a different I/F if you can?

[Brendon]

There is obviously something going on, but who knows what. Nekton certainly had abysmal performance with his USB gear, as demonstrated by migrating to thunderbolt. I have both thunderbolt and USB drives, but my USB 3 hosted drive is only a backup drive. It does however seem to still deliver ~180MB/sec sequential IO, this is running the latest everything, on a MacPro.

We have an unverified claim, that moving from OSX version X to version Y slowed a USB disk drive unacceptably. I would be very handy if OP was to test that assertion by reverting back to the original configuration. Proof would be in that the drive immediately returns to full speed.

Cheers
Brendon

[nekton34]

Sadly, I cannot regress to the previous OS X and cabinet because this array is in constant office use, but here is an indication of the speed difference. I don't know why the USB3 was so slow after updating to 10.11.4, but the TB2 array makes a BIG difference.

In a USB3 cabinet, the 4 x 4 TB raidz-1 connected to core i7 MacMini with 16 GB of RAM running 10.11.4 took 20+ hours to scrub 4.7 TB of data.

With the EXACT same setup but in a TB2 cabinet using the EXACT same disks, the same 4.7 TB of data took 3.5 hours to scrub today.

One thing I did notice was the USB3 setup always had several accumulating checksum errors at every status check, which people here suggested might have been due to poor USB3 cables. We changed the standard USB3 cables to best-quality shielded cables and the checksum errors decreased in frequency a lot but never reached 0. With the TB2 I/F, there have been no checksum errors so far.

The scrubs found no problems when the disks were in the USB3 case and in the TB2 case.

Just my 2c—hope it helps.

[Brendon]

Not you nekton! The other guy. Buy I seriously expect nobody to revert. Was just thinking aloud.

Brenodn