@rottegift
Thanks for the long reply.. and it sounds like you need more detailed info on what I am doing.. so here you go!! Thanks again for helping out.
Firstly.. i use to run ZFS on and old OSX 10.6 server hackintosh using apples unreleased code.. it was on a e8400 and that server was over 6 years old and did a great job but upgrading to 10.11 and using openzfs OS X was OK but it was clear that I needed to do something new since the hardware was just too slow and it was just waiting to die with 6+ years of 24/7 uptime i was shocked at just how reliable it all was.
So I found a nice half depth server that ran dual l5640s and was going to be perfect for my needs as a plex server/file server etc.. but the issue I had hackintoshing it was a lack of (non existent drivers) for LSI cards and was using a 4e4i card to provide some addl drives to the 12 internal and also connect to the external 24 drive shelf over 8088.
So, I decided to install esxi on it... and set up an all-in-one where omnios/napp-it (solaris based) VM that is installed on the only local vmfs store (sata connector) ... it is loaded by esxi and it provides ZFS backed NFS shares to esxi to host the rest of my VMs. I have a 2x intel s3500 super cap ssd stripe (for speed) as the machine pool running on the LSI card passed to omnios and a datapool of regular drives... and of course a pool on the backup shelf that gets booted for backup runs.
So.. now the meat... since all NFS in esxi is sync and the VMs... just a couple like about 7 mix of linux, OS X, vcenter... are running on that NFS served ZFS backed vmfs ... i want to reduce the overhead.... and keep the SSDs from having to write all VM data twice to SSD.. once for the zil sync write and the second time when zfs lays it back down to 'perminant' storage. An external zil (SLOG) would keep the 2 intel s3500 SSDs from doing 2 writes but it seems like a waste. Everything here is internal to a server on ups backup... so unless something panics there should be no interruption.. and the SSDs are all supercap anyway.
I still dont feel like setting sync=never on the datastore... but as I read it.. oracle designed logbias specifically for situations where there databases write 2 different datasets to the storage pool.. data and log.. and allows the data to be written logbias=throughout which commits the writes directly to disk rather than any zil.. thus freeing the zil device to serve the more critical and smaller latency critical log writes.
So i was thinking with 2 stripped SSDs .. I should be able to directly write the data without zil as it shoul be fast enough since there is no seek penalty on SSD... I feel that ZFS is still optimized for spindles and not really tuned or aware of SDD...
that way the SSD sees half the write traffic since zil is not written first.. it should even speed it up.. since again I dont see how a zil write on SSD can be any faster than commiting the data .. zil writes to spindle disks are faster as zfs doesn't try to place the zil write it just writes it wherever the heads are at the time to reduce latency so the ack gets back to the sync flush... but this behavior makes no sence in a all flash pool
So in sum... an all flash pool on stripped SSD being served via internal virtual network which runs at 10G+ speeds, via NFS requireing sync is writhing the data twice so looking for a good way to optimize this to prevent writing twice the data to the storage and to keep sync from increasing the fragmentation that sync imparts on a pool without SLOG zil.
The reason I am not really looking to use a SLOG is that I cant afford Zeus drive or some other storage that is faster than the stripped SSDs.. and just adding a small intel 80gb s3500 as SLOG would fix the frag on the 2x s3500 main pool but might actually be slower than just commiting the writes.
So logbias=throughput tells zfs to not use ZIL at all and commit all writes directly to disk (which on all SSD should be fast enough and just as fast if not faster than using zil on pool) and since the disk is supercap protected the writes will get written
Another way of doing this is tuning the zil.. since zil only get writes smaller than a x value .. tuning that number would force all writes to bypass ZIL... but tuning is behind the scenes and logbias is a property that can be set on a dataset level granularity
Thoughts?
As you can see this is not really a osx zfs question but a 'how zfs works' kinda question.
I wish I could have stuck to a OSX on bare metal and use OS X zfs but it just didnt work out due to lack of HBA drivers. And now that I am running certain workloads in esxi with VMs I really cant go back... its pretty slick.. really stable..
Another issue I have is that in my all mac home.. now that all storage is being provided by solaris/napp-it SMB.. it uses windows alike SMB NFS4 permissions and ACL models so trying to set up my SMB shares for mac users is complicated for me as understanding the POSIX vs ACL vs SMB permissions levels is giving me a headache!!!
Thanks
- Code: Select all
N-Bytes N-Bytes/s N-Max-Rate B-Bytes B-Bytes/s B-Max-Rate ops <=4kB 4-32kB >=32kB
62384 31192 62384 1310720 655360 1310720 10 0 0 10
148304 74152 148304 786432 393216 786432 6 0 0 6
10632 5316 10632 532480 266240 532480 6 2 0 4
Here is a zilstat 2 3 of the server running idle