Thunderbolt 3 NVMe SLOG?

All your general support questions for OpenZFS on OS X.

Thunderbolt 3 NVMe SLOG?

Postby tangent » Wed Aug 25, 2021 7:11 am

I've got a Thunderbolt 3 enclosure with SATA connectors, which allows both 3.5" and 2.5" drives. I just put a 2.5" enterprise-grade SSD in as a separate ZIL (SLOG), and it does help, but not as much as I'd like.

Since I believe this is because of the 6 Gbit/sec SATA bottleneck, I got to wondering if I could attach an NVMe drive via Thunderbolt 3 instead. I'm asking here because most of the offerings I'm finding seem rather cheesy, not capable of exposing the full speed of the drive within. For instance, this one is spec'd to go to about 11.2 Gbit/sec, nowhere near the 40 Gbit/sec of the interface. I can get drives with write speeds much higher than that limit, so it bugs me that I won't get full benefit of either the drive or the interface.

It also bothers me that I'm not finding any with daisy-chaining ability. While I do want to put this SLOG onto its own Thunderbolt port to maximize throughput in write-heavy operations, I'd like the option to chain other low-speed devices off it, to get some use out of the port while it's idle.

Has anyone tried this? Did you find it a worthwhile improvement?
tangent
 
Posts: 47
Joined: Tue Nov 11, 2014 6:58 pm

Re: Thunderbolt 3 NVMe SLOG?

Postby tangent » Mon Sep 06, 2021 6:12 pm

The answer is, "yes, this is worthwhile."

Without a SLOG, I was getting this sort of result from the AJA system test's disk benchmark with my 24 TB ZFS pool, based on three pairs of 8 TB mirrors:

no-SLOG.jpg
no-SLOG benchmark result
no-SLOG.jpg (82.68 KiB) Viewed 2208 times


The test program is set to try and disable the OS cache, but I believe the high read speed shows it doesn't know how to disable ZFS's L2ARC. That in turn means the following results aren't surprising in that AJA apparently doesn't know how to disable the ZIL, either.

Adding a Separate ZFS Intent Log (SLOG) device gives these results:

  • Intel Enterprise-grade SATA SSD: ~260 MB/sec writes
  • OWC Envoy Pro SX (m.2 NVMe SSD over Thunderbolt 3): ~700 MB/sec writes
This is with a 40 GB slice of the drive dedicated to the SLOG, since I believe the flushing behavior of the ZIL means there's a practical limit of how much value you can get from a SLOG device: at some point, the rate of writing to the SLOG balances with the speed of reading from it, so making the queue longer only lets writes stack up longer before stalling. For my tests, dedicating the entire 240 GB of the SSD to the SLOG gave no improvement.

I therefore formatted the rest of the SSD as an APFS scratch drive. It sustains about 1100 MB/sec writes in this test. That's lower than I was led to believe by OWC's marketing department, but it is a marked improvement.

It also shows there's performance left on the table, if ZFS can work out how to approach the speed of APFS here.
tangent
 
Posts: 47
Joined: Tue Nov 11, 2014 6:58 pm


Return to General Help

Who is online

Users browsing this forum: No registered users and 27 guests