Page 1 of 1

Read speeds dropping to zero

PostPosted: Wed Oct 05, 2016 2:45 am
by RobRehnmark
I have recently started having this VERY unwanted problem where read speed drops to zero and I simply have to wait for the requested data.
This leads to a complete stop in playback/work when watching a video or editing in Final Cut Pro for example... I can get a spinning beach ball and then it just picks up when data comes from the pool again.
I'm not sure if it started with the update to El Capitan 10.11.6 but that's what I'm running now.
It's a 6-core 4 GHz Xeon with 24 GB of RAM (not ECC) and to my knowledge I never had any performance problems that where caused by hardware.
The pool is a 6 drive Raidz2 made from 3 TB Seagate Barracudas.
At one point I had some jerky mouse/UI so I built from source to get the larger memory chunks allocated by ZFS, 512 instead of 128.
Arc max is at 8 GB and Arc meta limit is at 6 GB.
I will post some screen shots to illustrate my problem and the severity of it.

This is how it behave when there is no read stop, scrubbing the pool at speeds of almost 300 MB to almost 700 MB per second:
Image

This shows the very irregular read speeds:
Image

Coming to a complete stop:
Image

And then picking it up again after about 40 seconds:
(the stoppage isn't always this long)
Image

Any help or ideas would be greatly appreciated.

Re: Read speeds dropping to zero

PostPosted: Wed Oct 05, 2016 11:22 am
by haer22
I have the same problem. Even if I put arc_max to 8 GB I get freezes from time to time. Most probably a memory allocator problem.

I am eagerly awaiting the next version where this hopefully is fixed.

Re: Read speeds dropping to zero

PostPosted: Wed Oct 05, 2016 3:44 pm
by Brendon
This will probably alleviate some of the pausing.

sudo syctl -w kstat.zfs.darwin.tuneable.zfs_dirty_data_max=$((128*1024*1024)).

Cheers
Brendon

Re: Read speeds dropping to zero

PostPosted: Thu Oct 06, 2016 5:34 am
by RobRehnmark
Brendon wrote:This will probably alleviate some of the pausing.

sudo sysctl -w kstat.zfs.darwin.tuneable.zfs_dirty_data_max=$((128*1024*1024)).

[BGH - edited syctl -> sysctl]

Cheers
Brendon


I will try this but what does it actually do?

[BGH - further limits an aspect of ZFS that drives the allocator hard]

And will it stick between reboots, export and import?

[BGH -> only if you edit into zsysctl.conf which should be under /etc/zfs/...]

EDIT:
syctl: command not found

EDIT2:
changed syctl to zsysctl but got this error..
"top level name kstat in kstat.zfs.darwin.tuneable.zfs_dirty_data_max is invalid"

Re: Read speeds dropping to zero

PostPosted: Thu Oct 06, 2016 5:35 am
by RobRehnmark
Also, I just noticed that I got 1 checksum error on one of the drives.
It was repaired and it never happened before so I'm not worried.

Image

Re: Read speeds dropping to zero

PostPosted: Wed Oct 12, 2016 7:46 pm
by lundman
"sysctl"

Re: Read speeds dropping to zero - Oh CRAP!

PostPosted: Thu Oct 13, 2016 11:56 am
by RobRehnmark
EDIT:
Deleted and started a new thread.

Re: Read speeds dropping to zero

PostPosted: Thu Nov 03, 2016 12:39 am
by RobRehnmark
The complete stop in read speed was due to faulty drives.
It seemed like the drive(s) halted all SATA traffic when their SMART status was queried.
After replacing the drives it's all working ok again.