ZFS worth using with non-ECC RAM? Can ARC be disabled?

Moderators: jhartley, MSR734, nola

ZFS worth using with non-ECC RAM? Can ARC be disabled?

Post by royfactorial » Sat Jan 05, 2013 1:51 pm

Hello,

I've been reading these forums for the past few months, considering whether or not to make the move to ZFS. However, my system currently has standard, non-ECC RAM, and pursuant to this hardforum thread (http://hardforum.com/showthread.php?t=1689724), I'm wondering whether ZFS' aggressive RAM-based ARC cache will expose my data to RAM-based corruption more so than HFS+. Is it true that files caches in RAM are not checked for integrity? On the flip side, I'm not sure how much HFS+ likes to cache files in memory (I imagine Don might have some insight on this?), so I'm not sure if a 'tradeoff' is the proper way to conceptualize it.

For those who do not have ECC memory, is it possible to minimize corruption risk by disabling the primary ARC cache, perhaps relying on an SSD for L2ARC instead? My understanding is that L2ARC is subject to the same integrity checks as the main ZFS pool.

I realize that end-to-end data integrity is not possible on non-ECC systems (even if all caching were disabled, something could happen to an open file while in memory), but I'm wondering if disabling the primary ARC cache is a reasonable step for non-ECC systems.
royfactorial Offline


 
Posts: 11
Joined: Sun Sep 16, 2012 1:10 am

Re: ZFS worth using with non-ECC RAM? Can ARC be disabled?

Post by si-ghan-bi » Sat Jan 05, 2013 5:02 pm

I have been using a Mac Pro since 2007 and I never experienced any ECC error. If you have good RAM and you check it once per moth with something like memtest86, you are fine.
si-ghan-bi Offline


 
Posts: 145
Joined: Sat Sep 15, 2012 5:55 am

UBC

Post by grahamperrin » Sun Jan 06, 2013 10:05 am

royfactorial wrote:… how much HFS+ likes to cache files in memory …


"… The Unified Buffer Cache (UBC) is a kernel optimization that combines the file-system cache and the virtual-memory (VM) cache. …"

That's not specific to HFS Plus.

Things such as ditto can work without the UBC – "… Files read and written will not be cached …".

Whether ZEVO makes use of the UBC, I don't know.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: ZFS worth using with non-ECC RAM? Can ARC be disabled?

Post by zslg01 » Sun Jan 06, 2013 10:45 am

If you REALLY want to get paranoid realize that most modern chips sacrifice internal parity checks in the name of speed - good old mainframe cpus had parity carried right thru - registers, adders, etc ,,, toy cpus did away with it.
zslg01 Offline


 
Posts: 18
Joined: Wed Nov 14, 2012 7:34 pm

Re: ZFS worth using with non-ECC RAM? Can ARC be disabled?

Post by royfactorial » Sun Jan 06, 2013 4:30 pm

si-ghan-bi wrote:I have been using a Mac Pro since 2007 and I never experienced any ECC error. If you have good RAM and you check it once per moth with something like memtest86, you are fine.


Good to know. My RAM passed several runs of memtest when I first installed them, but I guess I should make a point of testing them on an ongoing basis.

grahamperrin wrote:
royfactorial wrote:… how much HFS+ likes to cache files in memory …


"… The Unified Buffer Cache (UBC) is a kernel optimization that combines the file-system cache and the virtual-memory (VM) cache. …"

That's not specific to HFS Plus.


Thanks for that info... the details are kind of lost on me, but at a high level it seems that the kernel (not the filesystem) is in charge of maintaining a single file+VM cache in order to avoid duplication of information between them. I wonder how/if the UBC interacts with ZFS-specific ARC?

zslg01 wrote:If you REALLY want to get paranoid realize that most modern chips sacrifice internal parity checks in the name of speed - good old mainframe cpus had parity carried right thru - registers, adders, etc ,,, toy cpus did away with it.


Can't say I'm surprised... but at the same time, the cosmic-ray-resistant 200MHz PowerPC G3 board in the Mars Rover costs around $200,000, so I guess not having in-chip parity on my $230 quad-core Xeon is a reasonable compromise ;)
royfactorial Offline


 
Posts: 11
Joined: Sun Sep 16, 2012 1:10 am

Re: ZFS worth using with non-ECC RAM? Can ARC be disabled?

Post by zslg01 » Mon Jan 07, 2013 12:59 pm

The lack of parity protection in most workstation chips originated IIRC as a combination of speed and cost - an extra bit for every 8 plus the parity generation in adders and so on was viewed as not needed and would add clock cycles - especially in the Intel 8088 timeframe. I was always amused by financial types worrying about disk and tape errors when unnoticed and uncorrected errors in the cpu were possible. IIRC some of the high end Macs at least had ECC memory ...
In mainframe times the problem was always "ok - we had a parity hit ... now what do we do about it?" Some systems would retry the operation in microcode which necessitated checkpoint data for each instruction be saved. Other systems turned on a red light and hard stopped. At least you knew it happened. I read an article about Cray eliminating all parity and ECC to get speed (he maintained up to his death that it was not necessary) - so some code had to be run multiple times and the results compared - especially when modeling things like airplane wings and bridges.
zslg01 Offline


 
Posts: 18
Joined: Wed Nov 14, 2012 7:34 pm

Re: ZFS worth using with non-ECC RAM? Can ARC be disabled?

Post by royfactorial » Tue Mar 12, 2013 12:07 am

So I finally got around to doing a bit of web searching on this issue, and found a research paper (http://research.cs.wisc.edu/wind/Publications/zfs-corruption-fast10.pdf) showing that ZFS does not perform any checksum/integrity analysis once a file gets placed in ARC. In their experiment, they were able to have corrupted data written back to the disk while files were in ARC.

Given that ZFS tends to use available memory for ARC, it would appear that non-ECC systems would have a greater risk of memory-based data corruption vs HFS+ which, so far as I can tell, does not cache files in-memory as aggressively as ZFS. Higher-spec'd systems with a larger amount of non-ECC RAM (current 27" iMac models can support 32GB) would be exposing a greater amount of data to potential memory-based corruption vs. more RAM-constrained systems. Am I right in interpreting the above paper in this manner?
royfactorial Offline


 
Posts: 11
Joined: Sun Sep 16, 2012 1:10 am

Re: ZFS worth using with non-ECC RAM? Can ARC be disabled?

Post by raattgift » Tue Mar 12, 2013 5:32 am

UBC sits above the vfs layer and caches pages for all virtual file systems (including ZFS).

None of the filesystems presently available to Mac OS X -- from Apple or from other parties -- is resilient to memory corruption that affects application memory, kernel structures, machine code, or clean pages within the Unified Buffer Cache.

ZFS "robs" some memory that would ordinarily be used by the UBC for the ARC; it does not protect you from memory corruption in clean pages in the ARC.

Dirty pages can also suffer corruption, but the dirty->clean transition for some workloads will be much quicker for ZFS-destined writes than for JHFS+-destined ones, in spite of how ZFS keeps transaction groups open for several seconds. For other workloads, JHFS+-destined pages will transition from dirty->clean more quickly.

Unfortunately pretty much all corruption in a dirty cache page will go unnoticed by all filesystems (how can they know the data is bad?) and is likely to wind up being stored in corrupted form. Future reads that reach down into the secondary storage will return the corrupted data.

Likewise, pretty much all corruption in a clean cache page will go unnoticed by all filesystems, and may be delivered up to an application in corrupt form. On the other hand, the page may wind up evicted (or lost to an unmount, reboot, etc.), with some future read coming from good data on secondary storage.

zstat will tell you how much of system memory is being used for ARC at any given moment. Memory not being used for ARC is liable to be used for UBC instead. Reads will use UBC first and ARC second, as the former sits above all vfses, including zfs.

ZFS assumes a vanishing error rate in system memory, the processor, and the plumbing between them.

JHFS+ makes the same assumption, and also assumes error-free secondary I/O busses, controllers, cables, devices, and the media in which the data resides.

Filling ECC memory with unchecked reads along a path with errors isn't helpful.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: ZFS worth using with non-ECC RAM? Can ARC be disabled?

Post by royfactorial » Wed Mar 13, 2013 10:00 pm

Thanks for providing some more detail :). I guess the question I have is, how much RAM does the UBC normally take up on a system that isn't under memory pressure? For example, I have 32GB, with only about 10GB being used at the moment. Is the 'free' RAM listed in Activity Monitor being used for UBC behind the scenes? If so, how much? My concern with ZFS was that, since it likes to use free RAM for ARC, that it might be caching a significantly greater quantity of data vs HFS+ and the UBC. However, if the UBC normally caches a similar amount of data compared with ARC, then the concern in my last post about ZFS exposing more data to potential memory corruption would be unfounded.
royfactorial Offline


 
Posts: 11
Joined: Sun Sep 16, 2012 1:10 am

Link

Post by grahamperrin » Thu Mar 14, 2013 12:58 am

royfactorial wrote:… I guess the question I have is …


Replies to parts of that question are under memory usage, kernel_task and zstat …
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Next

Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 1 guest

cron