Performance degrades until reboot

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

Performance degrades until reboot

Post by erico » Sun Mar 03, 2013 3:02 pm

Dear all,

I'm finding that with a raids-0 (3 disk internal) array, performance degrades in relation to the time expired (and number of I/Os performed) since the last reboot. And when I say degrade....I mean really badly degrade: a 500mb duplicate than normally takes 8 seconds can take 35 minutes, but after a reboot is back to 8 seconds again.

I assume this is some sort of bug. But how do I help track it down and report it? Are there things in zstat or elsewhere that I should look for and report? Is this a known bug, and I just am missing something?

The rig is a Mac Pro 2009 with 32gb of memory and 3 3TB seagate drives in the default bays, running macosx mountain lion server with the liberate hack.

Any thoughts are much appreciated.

best,
Erico
Last edited by erico on Sun Mar 03, 2013 3:17 pm, edited 1 time in total.
erico Offline


 
Posts: 10
Joined: Sun Nov 25, 2012 6:31 pm

Leopard?

Post by grahamperrin » Sun Mar 03, 2013 3:10 pm

Leopard? Please double-check. ZEVO Community Edition 1.1.1 requires Snow Leopard (10.6.8) or greater
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Performance degrades until reboot

Post by erico » Sun Mar 03, 2013 3:17 pm

oops, i'm sorry, I mean mountain lion, with all latest patches. I must be brain dead. I'll edit the original post.
erico Offline


 
Posts: 10
Joined: Sun Nov 25, 2012 6:31 pm

Re: Performance degrades until reboot

Post by raattgift » Mon Mar 04, 2013 3:54 am

"zpool status -v Poolname" would be useful so people know what you mean by "raids-0", which is not standard zfs terminology. It would also indicate whether you've configured any cache or log vdevs, whether there's a resilver or scrub in progress, or any vdev faults or degradation of the pool.

It would also help for you to be explicit about exactly you're doing when your "500mb duplicate".
Do you mean command-D at the Finder? Or a drag-and-drop copy at the Finder? Or "cp" or "rsync" or "ditto" at the comand line?

The output of "mdutil -vas" would be useful too. Mds and mdworker hit storage hard particularly when scanning through many small files. It is the biggest generator of unexpected post-boot I/O that most people will encounter.

Some of the output of "zpool iostat -v Poolname 1" during your '500mb duplicate [which] normally takes 8 seconds [slows down to] 35 minutes, but after a reboot is back to 8 seconds again' tests likely would also be useful.

A couple of lines of "zstat | egrep 'arc_'" and a cut and paste of the WIRED and PEAK lines from just "zstat", all taken during the slow performance period, also would be handy.

Finally, the output of "top -l 2" when you're suffering slow performance is likely to be useful.

Things you can try other than just rebooting to see if they make a qualitative difference to performance:

1. /usr/bin/purge (this cleans out Mac OS X's Unified Buffer Cache)
2. zfs umount <dataset>; zfs mount <dataset>
3. zpool export <dataset>; sleep 10; zpool import <dataset> (this cleans out the UBC and the zfs ARC)
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Link

Post by grahamperrin » Tue Mar 05, 2013 1:59 pm

The suggestion to purge is pleasantly thought-provoking.

UBC (Unified Buffer Cache), purge, performance – thoughts please …
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Performance degrades until reboot

Post by raattgift » Tue Mar 05, 2013 4:39 pm

In this case the idea is simply to return some of the OS to a state approximating that at (and near) boot time.

This is entirely because of "performance degrades [with time elapsed since] reboot. And when I say degrade....I mean really badly degrade: a 500mb duplicate than normally takes 8 seconds can take 35 minutes, but after a reboot is back to 8 seconds again."

Rebooting clears out a lot of state. Purge clears out a small subset of that, and does so without destructively interfering with applications by putting resources entirely out of their reach.

Unmounting/remounting and exporting/reimporting respectively clear out larger amounts of state, and necessarily prevents applications from using data in the affected datasets for a period of time. (In practice one may even have to quit/kill some processes).

The idea is to instrument the 500mb duplicate (easy, use /usr/bin/time or a wall clock) to see if performance returns from 35 MINUTES to only a few seconds after purge, or after a zfs unmount/mount or after a zpool export/import, as it does with a system restart.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: Performance degrades until reboot

Post by raattgift » Thu Mar 21, 2013 10:25 pm

It would be useful for you to look at the numbers of reads and writes happening to your pool(s) during the slowdown.

You can do this with "zpool iostat 1" and the Disk Activity panel of Activity Monitor.

If there are "lots", where that is in the literal thousands of writes/sec, please write back.

(Additionally, open a tall terminal window and run zstat and look for running searches at the very bottom of the refreshing text).
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

active searchfs jobs

Post by grahamperrin » Fri Mar 22, 2013 8:51 pm

raattgift wrote:… zstat and look for running searches at the very bottom of the refreshing text).


For what it's worth: in my OS X 10.8.3 environment, active searchfs jobs are noticeable only after a forced stop or forced restart of the Mac, or after a kernel panic.

I allow indexing of all ZFS file systems, including children, and I don't find indexing disruptive. YMMV
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to Troubleshooting

Who is online

Users browsing this forum: bileyqrkq, ilovezfs and 0 guests

cron