Write failures using Apple Logic to RAIDZ array

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

Write failures using Apple Logic to RAIDZ array

Post by JoelRichards » Tue Jan 08, 2013 5:02 pm

Hello all, I'm at my wit's end with this one--any help/insight will be greatly appreciated. I have a:

5,1 Mac Pro 1X Quad core 2.66ghz CPU
24GB Ram
64GB SSD (system drive)
640GB HDD (user data)
5x 2TB RAIDZ1 array (mostly audio, some video)

I do mostly voice-overs with a bit of audio production for video (and on the rare occasion video editing). I had a Solaris/OI box that housed the RAIDZ but with ZEVO I finally migrated everything over to the Pro (boy that was a pain!). Now everything seems to be working and working well. I get acceptable performance and much lower latency using the RAID locally instead of over a 1Gb connection.

However, Apple Logic is starting to do screwy things. Occasionally it will stop recording (bear in mind this is laying down one track of audio) and tell me my system can't keep up. I've also have some intermittent distortions that sound like write failures. Everything is intermittent and I'm pulling my hair out--ok, mostly bald already so instead I'm just vigorously rubbing my scalp but you get the idea.

Would adding an SLC SSD ZIL fix the issue? Honestly in all other instances the setup runs like a champ. I scrub and get no problems. Read/writes are fast enough for my needs, etc.
JoelRichards Offline


 
Posts: 4
Joined: Mon Sep 24, 2012 4:33 pm

Re: Write failures using Apple Logic to RAIDZ array

Post by mk01 » Tue Jan 08, 2013 5:56 pm

try running this script (I found somewhere when I was solving similar issues) and let's see, what is doing some long io. maybe you find something. it takes one number as parameter and will output each io, which lasts longer than that (in ms).

Code: Select all
#!/usr/sbin/dtrace -s

#pragma D option quiet
#pragma D option defaultargs
#pragma D option switchrate=10hz

dtrace:::BEGIN
{
    printf("%-20s %-16s %1s %4s %6s %s\n", "TIME", "PROCESS",
        "D", "KB", "ms", "FILE");
    min_ns = $1 * 1000000;
}

/* see bsd/hfs/hfs_vnops.c */

fbt::hfs_vnop_read:entry
{
    this->read = (struct vnop_read_args *)arg0;
    self->path = this->read->a_vp->v_name;
    self->kb = this->read->a_uio->uio_resid_64 / 1024;
    self->start = timestamp;
}

fbt::hfs_vnop_write:entry
{
    this->write = (struct vnop_write_args *)arg0;
    self->path = this->write->a_vp->v_name;
    self->kb = this->write->a_uio->uio_resid_64 / 1024;
    self->start = timestamp;
}

fbt::hfs_vnop_read:return,
fbt::hfs_vnop_write:return
/self->start && (timestamp - self->start) >= min_ns/
{
    this->iotime = (timestamp - self->start) / 1000000;
    this->dir = probefunc == "hfs_vnop_read" ? "R" : "W";
    printf("%-20Y %-16s %1s %4d %6d %s\n", walltimestamp,
        execname, this->dir, self->kb, this->iotime,
    self->path != NULL ? stringof(self->path) : "<null>");
}

fbt::hfs_vnop_read:return,
fbt::hfs_vnop_write:return
{
    self->path = 0; self->kb = 0; self->start = 0;
}



mk
mk01 Offline


 
Posts: 65
Joined: Mon Sep 17, 2012 1:16 am

Re: Write failures using Apple Logic to RAIDZ array

Post by JoelRichards » Tue Jan 08, 2013 9:23 pm

Wow, cool little script. I'm running it but I'm not really seeing any surprises--but then I also can't replicate the problem now that I'm really trying to either. Is there a recommended time to run the script at? I've found that 0 gives me lots of data but I'm not sure any of these small io would be problematic. Nothing *seems* to be hogging the io...although to be honest I don't really know what I am looking for?

EDIT: Sorry, that first response was a little rambling. Basically, I don't see any long io. I am using a mix of Seagate and WD "green" drives. Honestly I've never had a problem with them (even performance wise with 4K set up properly) but I know some people have. Could this be the issue? I can't even seem to replicate the problem now. It is nerve wracking having an undiagnosed intermittent problem in your work machine...
JoelRichards Offline


 
Posts: 4
Joined: Mon Sep 24, 2012 4:33 pm

Re: Write failures using Apple Logic to RAIDZ array

Post by JoelRichards » Thu Jan 10, 2013 7:34 am

Attached is a screen shot from AJA's disk benchmarking utility. Would the dips in the write graph indicate the problem?
Attachments
Screen Shot 2013-01-10 at 7.28.20 AM.png
Screen Shot 2013-01-10 at 7.28.20 AM.png (78.39 KiB) Viewed 57 times
JoelRichards Offline


 
Posts: 4
Joined: Mon Sep 24, 2012 4:33 pm

Re: Write failures using Apple Logic to RAIDZ array

Post by JoelRichards » Thu Jan 10, 2013 10:21 am

Ok, I *think* the issue may be one of two things:

1. My "Audio" zpool was set up for compression and that is somehow introducing an occasional slight latency
2. Apple Logic uses synchronous writes and doesn't play nice with ZFS because it wants to hog the IO

The problem is I still can't get this problem to consistently surface so while I've taken care of #1 by turning off compression, I can't say for certain if that will make a difference. #2 could be solved with an SSD ZIL I imagine but I don't know how to confirm if a program is making sync or async calls to write. Using the suggested script I do see that Logic makes lot of very short read/write calls (most less than 1ms) whereas Soundtrack Pro (an EOL Apple audio program) tends to make larger, less frequent calls to IO. Am I correct interpreting this to mean that Logic is probably making synchronous calls and Soundtrack asynchronous? In which case a ZIL should make a small but critical difference in my case.
JoelRichards Offline


 
Posts: 4
Joined: Mon Sep 24, 2012 4:33 pm


Return to Troubleshooting

Who is online

Users browsing this forum: bileyqrkq, ilovezfs and 0 guests

cron