tim.rohrer wrote:Thanks leeb.
Thanks for the details as well!
- Code: Select all
spl.kext_version: 1.7.2-1
zfs.kext_version: 1.7.2-1
I probably sound like a broken record given I just said it in another thread, but seriously start by upgrading to 1.7.4. There have
significant changes in the last 6 months since 1.7.2 was released, and you're on 10.13 anyway.
Here is my ctl file:
Nothing radical looking, though I actually reverted to pure vanilla despite having 64 GiB or memory during the last few updates to see how the newer memory balancers worked. I don't have enough uptime though yet to really say how 1.7.4 works there.
tank0 (from where I did the large delete that took 32 hours) is a RAIDZ1 pool using a 4-bay enclosure (OWC Mercury Rack Pro) connected via USB3. The disks, 4x3TB, are Hitachi Deskstar 7K3000s. Default settings were used in setting up the pool; some datasets have quotas, but that is about all.
I assume you're using it in dock mode, so none of its RAID5 hardware involved? I assume it's reliable enough there but have no experience with that one in particular.
So looking at this overall obviously a vanilla create. I assume the Deskstars do nothing funky that would trick autodetect so ashift value should be fine. Having said that a few things to consider about the defaults FWIW, some performance related though I don't think they'd account for your observed behavior. I've chopped out irrelevant stuff:
- Code: Select all
[...]
tank0 compression off default
tank0 atime on default
[...]
tank0 utf8only off -
tank0 normalization none -
tank0 casesensitivity sensitive -
So in order-
Compression: with OpenZFS and the integration of LZ4 you should basically always enable compression outside of specific tuning situations where you know you have a good reason not to. LZ4 is extremely fast and also avoids incompressible failure modes. Old advice about enabling/disabling per filesystem is niche or obsolete now, just enable LZ4 all the time. Compression can increase performance by trading a small amount of CPU (cheap and abundant) for disk/subsystem latency/bandwidth (way more restricted). Worst case it should do no harm. And of course even a 10 or 20% space savings isn't nothing over enough terabytes, particularly since it's generally a good idea to keep a pool from getting more then 75%-85% full.
atime: This can impose a significant write performance penalty and serves only a few unusual and niche old unix apps these days (are you running a mail server maybe?). If relatime was implemented under O3X might be worth still keeping that, but between on and off normal recommendation is off.
utf8/normalization: Mac OS X under HFS always traditionally used normalization formD and I found subtle but significant errors in the past running ZFS without it (way back in the ZEVO days actually). AFPS mixed that up, but still is pure UTF8. I don't know how strictly it's required now but I've stuck to just running formD myself for native usage (utf8 will auto go on if normalization is used). Not relevant to zvol.
casesensitivity: though it's always been possible to format case-sensitive under OS X/macOS, the default is insensitive and some apps have screwy behavior under case sensitive, if only because it's non-default (Adobe is particularly scummy/infamous in this regard). Since ZFS supports this per-FS anyway under the Mac I usually just match the default and then enable case sensitivity for code repos and the like specifically. You may have your own reasons there though! Obviously also not relevant to a zvol, whatever FS is formatted on top will be dealing with that.
The data I deleted that took so long were the User folders I'd been trying to rsync (You may see my other posts), although I skip ~/Library/caches in my copying. So, some user documents, a lot of email messages, A LOT of photos in the Photos library, and quite a bit of music; some movies. Unfortunately, I didn't analyze exactly what was there before I started the process, but that should give you an idea.
It was a while ago and under an older version of O3X, but when I used rsync to bring over all my data from my old system I do remember that my Library folder specifically took an inordinately long time vs everything else, though it was a one-time issue. Even ignoring caches there are tons and tons of tiny little files in there (Mail stores everything that way for example), and perhaps older versions at least (maybe improved? haven't stress tested that aspect yet under the most recent ones) have trouble in that area. Unfortunately I don't have hard numbers for you, but I will say yeah, what you describe does in fact ring a bell
.
I'm actually running my main account with the home folder native under a somewhat complex multipool setup. I haven't experimented in a year with comparing to a formatted zvol or the like I'm sorry to say. However I'll see if I can't find some time to experiment with this a bit, I've been meaning to set back up some ongoing rsync jobs anyway since I'm running encryption and won't be able to do any send/receives until either O3X 1.8 or I get my other system upgraded to something later then 10.12 as well.