Pool to Pool Copies - Finder Lag - Copy Won't Finish

All your general support questions for OpenZFS on OS X.

Pool to Pool Copies - Finder Lag - Copy Won't Finish

Postby mkush » Fri Dec 04, 2015 9:22 am

I've been dabbling with OpenZFS for a little while but there are a few nagging things that have kept me from wholesale adopting it. I will admit up front that I haven't tried as hard as I probably could to figure these things out. Hopefully there are some obvious things that someone can just quickly point out.

My main issue is that it seems very slow in several situations:

1. Large Pool To Pool Copies: Since I've never fully settled on a solution for my data, I've ended up copying it around quite a bit, from pool to pool. What I have noticed is that this process (pool to pool copy) gets very slow very quickly. It starts off OK but quickly becomes horribly slow. These are typically large pools. For example, a mirrored pair of 6TB drives copying to another similar pool. Zevo never seemed nearly that slow. HFS is not anywhere near that slow. Also, it seems that the entire memory of the computer, no matter how large, is quickly consumed. There have been discussions in the past about using a Terminal command to limit the use of memory as cache but the last I looked there wasn't a good way to make it persistent, nor do I know if this is truly the answer to my issue.

2. Folders Take Long To Display Contents: This seems true even with smaller pools. For example, while on a long vacation recently, I brought along two 2TB USB3 drives and ran them as a mirror. For each camera with me, I made a folder in the root of the pool to hold photos and videos from that device. After mounting the pool, when I click on a folder that has any decent amount of content, it takes a long time before the list of files is displayed in the Finder. For example, I just clicked a folder that contains 270GB in 6,300 items. It took over TWO MINUTES before Finder showed me the contents of the folder. This is a fairly simple config of a top-config 2015 15" MacBook Pro (16GB mem) with the pool consisting of just the two 2TB USB3 drives. No other pools mounted. Is this normal behavior? Makes it a real pain to use. It is definitely tied to the size of the folder clicked. Smaller folders show their contents much more quickly, although still much more slowly than HFS. One final thing: once the contents of the folder have been displayed, clicking that folder again results in nearly instantaneous display of contents. It's just the first click after import that's horrible.

3. Here's another issue that just happened (but has happened many times in the past). I'm copying a card full of pics and videos to a zpool. In this case it's the same pool described in number 2 above. It gets to the end of the copy and never finishes. I have to end up killing Finder. In the current case, it was copying about 128GB of data from a card to the pool. It has been stuck on finishing in "About 5 seconds" for about 15 minutes or more. It says "Copying 0 items" and lists the amount copied so far as "127.80 GB of 127.8 GB" so it gives the appearance of being done. At the same time, the fans on my computer are running very fast (audible, which they almost never are on my MBP). As soon as I relaunch Finder, the fans begin to slow down. I check my pictures, and the final one, which it was supposedly working on at the end, is copied just fine. So I'm not sure what it thinks it hasn't done yet.

I know I'm short on details here so please just ask. And sorry if the solutions to these issues are super obvious. Thanks for any help.
mkush
 
Posts: 53
Joined: Tue Sep 30, 2014 1:17 pm

Re: Pool to Pool Copies - Finder Lag - Copy Won't Finish

Postby JasonBelec » Tue Dec 08, 2015 12:08 pm

Well, not fully going with something tends to always leave a bad taste.

1) Are you trying to copy through the Finder large amounts of data from PoolA to PoolB all on the same system bus? Slow, hell yeah. You're asking something that is all about data integrity to move to something else that is all about data integrity all through he same pipe. It's like a super protective and hungry dog shoving a bratwurst through a McDonald's straw to another such dog. That said, you can set things up to be more efficient but more info about your hardware and OS might be needed and your really going to hate this USB3 is not really fast used they way you are using it. Personally I don't find it very easy at all.

2) Not sure the Folder issue is specifically ZFS, as I don't seem to have much of a wait for massive amount of content on several pools with about 53 mount points. I can look in any mount or folder within and see everything pretty fast. Maybe a second depending on network activity or other system tasks. Non of my clients are experiencing this issue even over networks, several dumping large amounts of images daily.

3) This seems like something very old, not current. But I'd be interested in more details, as I have 4 clients currently doing just that daily with out any issues.

Hopefully we can get you sorted. :)
JasonBelec
 
Posts: 32
Joined: Mon Oct 26, 2015 1:07 pm

Re: Pool to Pool Copies - Finder Lag - Copy Won't Finish

Postby mkush » Tue Dec 08, 2015 1:17 pm

Thanks for answering! I was getting concerned that I'd asked such silly questions that nobody even wanted to bother... But you did, so thanks.

As far as not fully going with it, it's just that, to be honest, it has never felt as polished as Zevo. I didn't get this kind of strange stuff with that. But it's now unsupported on current OSs, and supposedly has that 16TB limit. Although... I've made pools far larger than that and it never complained, so I'm not sure what to make of that. I really want this to work. My situation is that I believe in ZFS and can't bring myself to trust my data to anything else but I don't have vast amounts of time to tinker with things to get them to work. I need something that just works (but yet uses ZFS). I know I could go with a separate server (FreeNAS, FreeBSD, OmniOS, etc.) and I may end up doing that, but I was just hopeful that this product would be reasonably trouble-free.

1. No, not all on same bus. For example, I have copied from 2-drive mirror of 6TB drives to another similar pool with the drives housed in separate OWC 2-drive Thunderbolt enclosures, each hooked to a separate Thunderbolt bus on a 2013 Mac Pro (which I no longer have). Same thing... all system memory gets consumed and things slow to a crawl. Was there ever any progress at making the setting for the ARC size "sticky"? When I'd set that with the command line after fresh booting, that would prevent the memory from getting exhausted. It still was slow as I recall.

2. This only happens on my ZFS drives, things are nearly instantaneous on HFS drives.

3. I'm currently running 1.4.2 on the computer which just experienced the issue. I can try 1.4.5 but I don't recall reading anything that leads me to believe it was fixed. I will say that this does not happen often, but enough that it's irritating. Gives me a bad feeling to have to kill the Finder.

Any debugging steps greatly appreciated. I'm wondering if I might be better off with just a separate file server running FreeBSD or OmniOS.
mkush
 
Posts: 53
Joined: Tue Sep 30, 2014 1:17 pm

Re: Pool to Pool Copies - Finder Lag - Copy Won't Finish

Postby lundman » Tue Dec 08, 2015 7:37 pm

1. You should use "zfs send" "zfs recv" to do pool to pool copies, that will be the fastest.
But yes, we have been focusing on porting the complete ZFS, then stability, and eventually, on speed.

2. Yep, listing is slower, we only have readdir, which means VFS layer calls getattr on everything afterwards. We should implement getattrlist, and getattrlistbulk, this will
speed listings up considerably. But, not quite had time for it yet.

3. Probably a bug, not seen it reported before either. See if latest version improves it, otherwise you should file a bug issue for it in github.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Pool to Pool Copies - Finder Lag - Copy Won't Finish

Postby JasonBelec » Wed Dec 09, 2015 7:49 am

1. As Lundman pointed out... However I do a few complete Pool to Pool transfers a year and client snapshots are sent daily (remotely) and things have been pretty good. One does not want to sacrfice quality for speed in the case of data. Well I guess some do. ;)

2. I can see how 'slower' is going to rather interesting to quantify, I don't see much delay unless I compare to an SSD which really isn't fair. ;) But as mentioned improvements are coming.

3) I'm interested in your OS version and any funny third party stuff you have running. I'm running almost all client stuff and all these machines staring at me on Mac Mini's (2012-2013) each does have at least 8 GB RAM and at least 1 SSD. A few have SSD striped OS X for fun. :0 I have systems backing up various Windows VMs to disk images, and OS X systems to images, on the ZFS servers daily, along with thousands of images and I'm not seeing the issue so I would really like to track down what might be interfering. Files are being copied back and forth depending on the office and workflow from the ZFS arrays that are mounted locally to each system. Running all the latest OS X 10.10 and 10.11 with the latest OpenZFSOSX. So I am very interested in catching a potential issue.
JasonBelec
 
Posts: 32
Joined: Mon Oct 26, 2015 1:07 pm

Re: Pool to Pool Copies - Finder Lag - Copy Won't Finish

Postby mkush » Thu Dec 17, 2015 6:10 am

To be honest, I'm heading quite strongly now in the direction of a ZFS-based NAS (FreeNAS). But, I did have one thing to add, hopefully it might shed some light.

Going back to the "slow directory listing" issue, the original pool I was seeing it on was comprised of a pair of portable HDDs (2.5") connected via USB3 (each directly to a laptop port). The time to list a particular directory was measurable in minutes. Last night I copied that pool to another one, same computer but this new pool is desktop HDDs (3.5") housed in a Thunderbolt enclosure. The time to list the same directory is measurable now in seconds, probably under 10. I'm guessing this is at least 20x faster.

I find that odd because I would not think the difference in the speed of the HDDs or the connecting bus would account for such a vast difference.
mkush
 
Posts: 53
Joined: Tue Sep 30, 2014 1:17 pm

Re: Pool to Pool Copies - Finder Lag - Copy Won't Finish

Postby mstan » Thu Dec 17, 2015 6:44 am

Well, that is faster but it still takes a minute or more for a top level size calculation of a complicated large directory to appear on my 8TB pool (half full) not to mention about 30 minutes for a dataset rollback to appear at all. My pool consists of mirrored pairs of 4TB HDDs inside a '09 MacPro with 32 GB ECC Ram and a larger backup pool connected via eSATA. So I don't think relying on brute force speed of hardware to make up for the lag is good enough. I think Lundman et al said they are working on this...in their spare time.

I have thought about the FreeNAS server too but since I have a paid for MacPro and the last couple of Mac OS' finally support SMB sharing with Windows without issues it is hard to justify a new server. I do like the polish and features of FreeNAS, though but OpenZ gets the job done for now.
Mac Pro 4,1->5,1; single 2.66 cpu; 32GB ecc ram; Sonnet usb 3.1 PCIe card; Firmtek eSATA PCIe card; Radeon 5770
mstan
 
Posts: 38
Joined: Sun May 18, 2014 2:36 pm

Re: Pool to Pool Copies - Finder Lag - Copy Won't Finish

Postby lundman » Thu Dec 17, 2015 5:53 pm

not to mention about 30 minutes for a dataset rollback to appear at all.


I haven't done much testing with rollback, but it is entirely possible that we need to invalidate the cache after a rollback, so it doesn't think the "old" entries are still around. Certainly a unmount / remount after rollback would show if that is the case.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan


Return to General Help

Who is online

Users browsing this forum: No registered users and 21 guests