Page 1 of 1

A couple of questions (SMR and features compared to ZoL)

PostPosted: Fri May 10, 2019 6:27 am
by DanielSmedegaardBuus
Hey :)

I hope it's okay that I ask two unrelated questions here. The forum is pretty quiet, so... :)

The first one is the quicker one to answer, I guess. I've been trying to use a mac Mini as a NAS basically, with a bunch of 4TB drives in a RAIDZ-3 configuration. I created the pool on macOS with encryption enabled, but I've had a lot of problems getting stability on macOS, with the pool eventually not even being importable without crashing the Mac unless I import it read-only, so I've now switched back to Linux. That's not my question directly, but I've put Ubuntu Server on an older MacBook Air, and when I tried to import the pool, I couldn't use the -l flag, since to my surprise, encryption wasn't in the stable ZoL 0.7.x branch. I compiled the 0.8.0 RC4 binaries (which was a bit of a hassle, as the instructions provided are for 0.7 and below, and in addition the shared libs needed for the compiled binaries are not where the binaries expect them to be, but so far I'm hacking my way to getting the pool up :D also, not the question). Sorry. The question is: How has O3X been offering encryption for almost half a year now when the ZoL repo, which I'm assuming is the source of O3X, hasn't pulled it out of beta yet? Have I been unknowingly running beta here, or is there some kind of git cherry-picking going on here WRT features, or something entirely different?

Second question: This is an open one. The drives I'm using are 11 4TB 2.5" USB drives, all SMR (clearly no other way to get 4TB in that small of a space). Firstly, I've read a lot of people online suggesting that the COW nature of ZFS ought to be perfect for SMR drives, but in my experience, this is definitely not the case (at least with O3X). I actually recreated this pool from a previous one that wasn't compatible with macOS (OMG, also not the question, but WTF is the default not to NOT enable any OS-exclusive features when creating a pool? OMG, the pain, the months of pain), and it was completely empty. I've been adding back backups, 2TB at a time, just writes, no reads, and every time the writes quickly dipped in transfer speeds to around 4 MB/s. This all while individual drives tested with dd are capable of about 50 MB/s I/O each — which is pretty lame, too, but seems to be the performance of these sub-par WD SMR drives. But I just don't get how ZFS triggers this behavior. Some kind of sync operations forced often that forces some SMR-related firmware operation? Something else? I'm not trying to bash ZFS here, I'm just looking for some way to get around this issue. Oh, and somewhere in my adventure, I added an SLOG device on an SSD in attempt to alleviate the problem, but to no avail.

I know I have a tendency to be a bit ranty when I write like this, so sorry about that. Hope I'm making sense. Also, thank you so much, anyone involved with bringing this beautiful filesystem to my almost every OS. Lundman, I've noticed while watching YouTube and seeing the ZoL commit logs that you're a natural force here, and I'm really grateful.

Oh, and even if these drives I'm using suck, I just replaced two dying ones at the store where I bought them (Elgiganten). The procedure: "So, you're saying they do what?" "Well, I wrote it here, but basically, I cannot read or write to them, even format them, they're just causing USB hubs to reset and throw I/O errors." "Okay, so you take this piece of paper, go back in the store and pick up two new drives, then show it to the lady at the checkout." Yeah, I like that sort of RMA. 5-minute RMA. :D

Re: A couple of questions (SMR and features compared to ZoL)

PostPosted: Mon May 13, 2019 7:02 am
by Sharko
Hey Daniel,

I'm in no way qualified to answer any of your questions (and non-questions!), but out of curiosity: what are 'SMR' drives?



PS Actually I can comment on one thing, the fact that Linux hasn't released 0.8 with encryption yet... I think it's just the dynamic that Linux is used in mission critical machines in millions of applications, whereas we are kind of a niche community of (mostly) hobby enthusiasts... Linux has to get it 99.9999% right, whereas if we waited for the volume of testing necessary to assure that kind of reliability then the time scale for advancing O3X would be measured in decades, not months. Our user base just isn't that big. So yes, it is kind of implied and hopefully understood that we are somewhat beta. At least that is the way I've always approached it - I always have a rotating set of HFS+ backups in play.

Re: A couple of questions (SMR and features compared to ZoL)

PostPosted: Mon May 13, 2019 11:59 am
by DanielSmedegaardBuus
Hi Kurt :)

Thanks for replying! I wasn't aware of this implied being in beta thing, but thankfully I haven't experienced any data-damaging problems either :) TBH, 0.8.0 by some reports being a massive update isn't something I really like to hear WRT a filesystem :D So perhaps the O3X approach is saner, I dunno :)

Anyway, about the "SMR" thing; this is an acronym for Shingled Magnetic Recording ( and is a somewhat recent trick by the drive manufacturers to push the limits of how much data can be put on a platter. Anyone who's using ZFS on consumer-grade drives should be aware of this technology and the fact that its usage is very rarely mentioned in datasheets, let alone put on boxes or on the drives themselves.

These 2.5" 4TB drives of mine — I searched extensively to find out if any 2.5" drives in this capacity were on offer in non-SMR configurations, but there's just zero data revealed out there. I even contacted WD and Seagate (WD actually phoned me from the states to reply to me) and both of them were "unable" (I'd say unwilling) to disclose this information, unless I offered them specific serial numbers that they could then check.

I eventually just settled on there most likely not existing any 2.5" 4TB non-SMR drives simply because the data I collected along the way led me to conclude that it just isn't feasible given the very limited platter area compared to 3.5" drives (which are also often SMR in 4TB versions, though they seem to be easier to recognise from carrying "archive" or similar designations in their names).

What I've found in practice is that these drives do not play nice with ZFS as a lot of people only have hypothesised due to ZFS being COW. Even during initial data restoral where I'm 100% writing sequentially, the write speeds achieved were abysmal. Though, to be fair, I've just now gotten the pool up-and-running, ready for new data on an Ubuntu Server installation, and I'm looking at 8x the write speed I was getting on macOS, and this even with a scrub running at the same time, so it may be a macOS-only thing. But, the "8x" the speed is still not impressive, at 42 MB/s (11 USB 3.0 drives in RAIDZ-3).

Re: A couple of questions (SMR and features compared to ZoL)

PostPosted: Mon May 13, 2019 2:35 pm
by Sharko
Interesting, thank you for the link on SMR drives. I have a couple 2.5" 2TB Toshiba drives that are formatted ZFS, and they don't seem especially slow so I'm guessing that they aren't SMR drives. So maybe the 2TB to 4TB boundary is where you fall into SMR drives right now.

Re: A couple of questions (SMR and features compared to ZoL)

PostPosted: Mon May 13, 2019 5:14 pm
by lundman
The encryption went into ZOL 0.7 master about a year ago, but ZOL have very few releases, and only now are pushing for 0.8. People in ZOL have run with encryption just as long as we have, by compiling it themselves. However, we did have a release 1.8.0 last year, which then included encryption support. It was not entirely cooked then, but looks to be good now. Encryption is from ZOL and has already been that, we just have more releases.

Re: A couple of questions (SMR and features compared to ZoL)

PostPosted: Tue May 14, 2019 12:39 am
by DanielSmedegaardBuus
Kurt, yeah, I too have a 2TB 2.5" drive that definitely isn't SMR. Interestingly, it's a "WD Elements SE" one, the same name as 10 of my new 4TB drives have, which all are SMR. Annoying that they don't help us figure it out, especially because it seems like a no-brainer to start using SMR across the board, even if not necessary due to space constraints, simply to cut down costs.

Lundman, thanks for the clarification! When puzzled that I couldn't import my pool in the 19.04 Ubuntu Server install which carried the latest ZoL version, I started googling and did notice several posts from 2017 where people were running encryption just fine on self-built binaries :)

I definitely wouldn't recommend this USB-based setup I'm doing now to anyone, BTW. I've spent much more time resilvering, scrubbing, handling dropped USB connections and resetting hubs, while moving data back and forth than I have actually enjoying a stable working setup. My old 19-drive SATA setup was so much more stable. If anyone does go down this rabbit hole, here's just one tip you'll come to appreciate for SMR disks (at least at the current ZFS version): If you ever need to replace a drive, use an interim non-SMR drive (or RAID up a few of them to get the target capacity), and resilver onto that, then mirror the drive onto the target with dd and re-import the pool. Resilvering with SMR drives is excruciatingly slow, because AFAIU, the resilvering process is simply traversing the block tree, so it's not going to be sequential, and it's random writes in particular that really suffer with SMR drives. Also, and worse, the nature of SMR will make the drives sometimes take a very long time to respond to a request, due to house-keeping on the platters, which may cause the kernel to deem the drive or hub hung, and issue a hub reset, which will affect any other drives on that hub and consequentially mess with the resilvering process, sometimes even offlining drives that aren't actually problematic, just really, really slow. In my latest resilver, I used mdadm to JBOD two 2-TB non-SMR drives for the resilver and then dd'ed the result onto my replacement 4TB drive, and instead of spending weeks on this, it took "just" about three days all in all. Oh, and if you do this, make sure after mirroring the resilvered data onto your target drive that you open it up in gdisk and re-write the GPT, as the backup GPT is likely to be misaligned on the new drive.