Large number of fseventsd errors during send

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

Large number of fseventsd errors during send

Post by shuman » Thu Feb 14, 2013 4:06 pm

While running sends to another pool, my system continually degrades to a point of a forced shut down; can't even ssh into the box. I don't get any panic files but I do get a whole list fseventsd errors. Suggestions?

Screen Shot 2013-02-11 at 7.42.17 PM.jpg
Screen Shot 2013-02-11 at 7.42.17 PM.jpg (193.73 KiB) Viewed 131 times
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

More details please

Post by grahamperrin » Thu Feb 14, 2013 6:46 pm

From the 16 GB pool to another pool at the same Mac? Or remote?

How are the drives connected?

In the Energy Saver pane of System Preferences, is sleep of hard disks allowed?

Any other details that might help to troubleshoot?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Large number of fseventsd errors during send

Post by shuman » Thu Feb 14, 2013 10:42 pm

Both pools connected via USB. Sender USB 3, receiver USB 2 (although I had the same issue when sender and receiver where 2.0)

The ONLY thing I allow to sleep is the display, although the drive enclosure must have a sleep mode as I do hear the receiver drives spinning up, HOWEVER. . . the issue occurs well after spin up has happened. In fact, it successfully sends a good amount of data to receiver filesystems before having any issues.

I'm running another scrub now since I read elsewhere that the fseventsd errors could mean corruption. My last scrub was in late January with zero errors.
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

Re: Large number of fseventsd errors during send

Post by shuman » Sat Feb 16, 2013 11:52 am

Here we go again. . .
Screen Shot 2013-02-16 at 10.45.50 AM.png
Screen Shot 2013-02-16 at 10.45.50 AM.png (209.08 KiB) Viewed 121 times

506199 copy.txt
(5.61 KiB) Downloaded 6 times


Code: Select all
localadmins-Mac-mini:~ chris$ zpool status
  pool: zTM2
 state: ONLINE
 scan: resilvered 42Ki in 0h0m with 0 errors on Mon Dec 17 00:29:12 2012
config:

   NAME                                         STATE     READ WRITE CKSUM
   zTM2                                         ONLINE       0     0     0
     GPTE_A0D25D3B-A9C9-44E0-B251-87AE54AB3647  ONLINE       0     0     0  at disk2s2

errors: No known data errors

  pool: zdata
 state: ONLINE
 scan: scrub repaired 0 in 11h54m with 0 errors on Fri Feb 15 03:42:55 2013
config:

   NAME                                           STATE     READ WRITE CKSUM
   zdata                                          ONLINE       0     0     0
     mirror-0                                     ONLINE       0     0     0
       GPTE_59D9A15C-8290-4B54-9BAC-C9D8306FD760  ONLINE       0     0     0  at disk5s2
       GPTE_1A52F1BF-D18F-4310-B304-1AF7F74CC944  ONLINE       0     0     0  at disk6s2
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

Spin

Post by grahamperrin » Sat Feb 16, 2013 2:28 pm

Without a send and receive, see how the disk drive behaves with another approach …

Keep Drive Spinning
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Large number of fseventsd errors during send

Post by shuman » Mon Feb 25, 2013 11:59 am

Applied the "keep drive spinning" job to all involved devices. Running send/receive now. An important note is that I have not done a send/receive since my last posting on the subject. I have had zero issues during that time. My first few send/receives should work ok. If the same behavior still occurs, it won't be until after I have done several send/receives that the system will quit responding.
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

Re: Large number of fseventsd errors during send

Post by raattgift » Mon Mar 04, 2013 6:11 am

The nature of ZFS three-phase TXG system essentially guarantees large bursts of writes in this sort of use.

In modern Mac OS X, when the target device atttaches with IOUserClientClass != IOUSBInterfaceUserClientV3 these bursts will almost always lead to head-of-line blocking that will affect all traffic on the HCI.

This may lead to timeouts (for nonblocking clients) and I/O errors for all users of all devices on that particular HCI or USB bus. The fseventsd log entries are a symptom of this.

ZFS tries to avoid this with its write throttle, but zfs receive in turn has to create metadata writes during after the TXG transitions from open. This will put pressure on the ARC and on the USB driver stack, particularly in the face of lots of small objects in the stream. FInally, this also puts pressure on the draining of dirty pages from Mac OS X's standard Unified Buffer Cache out through the ARC (and through the write schedulers that operate on other filesystem stacks competing for the same USB HCI), and that can lead to memory starvation and ultimately a hang.

Really, you don't want to be dumping loads of data into a ZFS pool where one or more of the vdevs comprises a physical device connected via USB 2. If you *have* to then you should throttle the writes at the application level as much as possible. pv can do this with the --rate-limit option. pv can be found at http://www.ivarch.com/programs/pv.shtml or via macports. You will want to keep the rate low (single-digit Mbps, most likely) and lower it further as you increase the number of USB 2 attached physical devices on *either* side of the send/receive pipeline.

Additionally, you should use mdutil -vas to check to see whether indexing is adding to the USB 2 traffic.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: Large number of fseventsd errors during send

Post by raattgift » Mon Mar 04, 2013 6:20 am

(Incidentally, zevo's port deals with power management remarkably well; disks being put to sleep by the system (man pmset) should never cause it troubles, and it copes at least as well as the autofs/HFS+/CoreStorage stacks do with external disks entering low power states of their own accord. Keeping drives spinning is only going to stop you from having to wait for them to spin up on demand; it can take several seconds for an I/O call to complete if eight FW800-attached drives in a raidz3 have to be awakened, for example... It is unlikely that a drive will be put into (or independently enter into) a low power state if it's part of a pool that is the target of a zfs receive, except by manual intervention (e.g. pmset sleepnow)).
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: Large number of fseventsd errors during send

Post by grahamperrin » Mon Mar 04, 2013 9:38 pm

raattgift wrote:… unlikely that a drive will be put into (or independently enter into) a low power state if it's part of a pool that is the target of a zfs receive, except by manual intervention (e.g. pmset sleepnow)).


That's good to know.

raattgift, big thanks for your recent posts – in this topic and in others. Exceptionally helpful!
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to Troubleshooting

Who is online

Users browsing this forum: bileyqrkq, ilovezfs and 0 guests

cron