Using zvol as virtual machine disks

[abc123]

I have a number of virtual machines I run, through both VMWare Fusion and Virtual Box and I've migrated from storing them on HFS+ to a second partition formatted with ZFS. Currently they're using 2GB split/expanding VMDKs. I'm currently storing each VM in it's own zfs filesystem so I can manage snapshots and clones for each individually.

With ZFS compression, I don't need to create expanding disks anymore, a single-file fixed-size disk won't actually occupy all it's space until used.

But I'm wondering if there would be an advantage to using a ZVOL rather than a fixed-size disk image? Both Fusion and VirtualBox support raw VMDK disks and I've set up test VMs in both platforms using zvols for the disks.

Has anyone been through this already? My hope is that there would be a slight performance advantage to using zvols but this is just hope rather than based on any knowledge.

Are thee any properties I should change on the zvol to give better performance? I had read a post on using them for VirtualBox that setting the blocks size to 64k would help but I can't find the reference anymore. What about cache setting or any other properties?

The only downsides I've noticed in my testing so far are:

1) The /dev/diskX devices created for each zvol revert to root ownership when rebooting - is there a way to change the owner so that it persists across reboots?
2) Packing up a virtual machine to give to others is harder because you need to convert the raw disk to a disk image but that is doable.

Thanks

Russell

[rottegift]

We lack a way to prevent a r/w zvol (with an OSX-mountable volume inside) from mounting at pool import.

zfs recv suffers from this in two ways: firstly, a full send that creates a new zvol will cause a mount; and secondly, a -I sendstream is almost certain to fail as each snapshot when received causes a mount, and an rw mount will dirty the target zvol.

These will be fixed eventually.

If you have an osx-mountable filesystem inside the zvol you are almost certainly doing extra caching (i.e., in the ARC and also in the fs cache that osx maintains). So the same data may be in memory in the guest OS's caching, any caching the virtual machine system software is doing, the operating system cache, and ARC. When using data in files (including vmdks and the like) in datasets, although the OS may be doing some caching (some of that depends on whether the OS is given mmap calls or read calls, and that significantly depends on choice of virtual machine system software), it's less likely. We don't have a great view of this in the zfs subsystem; we get a list of addresses from the OS and copy ARC data into them, but don't (and pretty much can't) track the fate of those pages once the data is copied over.

The opposite side of this coin is that synchronizations from memory to disk must do more work. If your vm software system gives you robustness against crashes by propagating sync writes (or turning all writes into synchronous ones) then you're taking an additional I/O performance hit; but if it doesn't, you run the risk of fsck_hfs+ and its equivalents being unable to put your stack of filesystems into useful shape after a crash when something in your guest is writing.

[abc123]

Hi rottegift, thanks for your reply.

With the extra caching on a zvol (if the filesystems are os x-mountable, which for me they currently aren't as they're ext4, rtfs and ufs, and sometimes FreeBSD on zfs), wouldn't this be negated by unmounting the volume before booting the VM?

And just to clarify the synchronisation issue, are you saying that using a vmdk image has to do more work when the guest does a sync write (depending on how the VM software treats them) to avoid corrupt guest fs and so this could lead to a slight performance gain for a guest using a zvol?

I guess I'd need to do some benchmarking on this but if the extra caching can't be avoided then I'll probably stick with disk images for now as memory is always the limiting factor for me.

Thanks

Russell

[4ever6]

These will be fixed eventually.

One of the fundamental design cores of ZFS is a private memory pool. If the OS can't support unifying that with its own VM subsystem, it will always be a little island. People really should look at ZFS (on OSX at least) as a kernel mode application. Pinning memory, etc. It's an uphill battle.