memory usage, kernel_task and zstat

Moderators: jhartley, MSR734, nola

memory usage, kernel_task and zstat

Post by grahamperrin » Tue Nov 20, 2012 6:19 pm

Spun off from viewtopic.php?p=3129#p3129

Use cases vary but generally, ZFS needs lots of memory.

ZEVO includes:


Other memory-related topics include:


Reference material:

grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Output from zstat

Post by grahamperrin » Thu Feb 07, 2013 1:30 pm

For anyone who would like to compare

On a MacBookPro5,2 with Mountain Lion and 8 GB memory, after taking a small L2ARC device (USB flash drive) offline then online, during a send and receive, the first few lines of output:

Code: Select all
 v2012.09.23    343 threads        5 mounts       50051 vnodes     18:27:56
____________________________________________________________________________
             KALLOC      KERNEL/MAPS        TOTAL         EQUITY
  WIRED     106 MiB    2825 MiB/2848         2932 MiB      35.79%
  PEAK      166 MiB    3026 MiB              3192 MiB
  VMPAGE     131420 (IN)      19961 (OUT)      19951 (SYNC)        176 (MDS)
____________________________________________________________________________
                     HITS                  MISSES
  ARC overall:        93% (156800060)          7% (10414010)
  ARC demand data:    98% (77505168)           2% (1159925)
  ARC demand meta:    92% (70594286)           8% (5322811)
  ARC prefetch data:  84% (3387775)           16% (615254)
  ARC prefetch meta:  61% (5312831)           39% (3316020)
  DMU zfetch:         62% (196799363)         38% (117691236)


Postscript, 2013-03-14

Code: Select all
 v2012.09.23    293 threads        6 mounts        5156 vnodes     05:25:47
____________________________________________________________________________
             KALLOC      KERNEL/MAPS        TOTAL         EQUITY
  WIRED      60 MiB    1294 MiB/1329         1355 MiB      16.54%
  PEAK      109 MiB    2717 MiB              2826 MiB
  VMPAGE       4826 (IN)       2512 (OUT)       2512 (SYNC)        197 (MDS)
____________________________________________________________________________
                     HITS                  MISSES
  ARC overall:        92% (47494685)           8% (3612996)
  ARC demand data:    98% (26098415)           2% (502895)
  ARC demand meta:    94% (18867852)           6% (1192876)
  ARC prefetch data:  29% (165193)            71% (388496)
  ARC prefetch meta:  60% (2363225)           40% (1528729)
  DMU zfetch:         60% (59685301)          40% (39604973)


  • equity is pleasantly low
  • I might wish for lower percentages of misses, but see below.

Maybe worth mentioning:

  • two of four pools are being scrubbed concurrently (the ZEVO norm is no more than one)
  • for the file system at the root of one of the other pools, there's a non-default value for primarycache.

Code: Select all
macbookpro08-centrim:~ gjp22$ zpool list
NAME      SIZE   ALLOC    FREE     CAP  HEALTH  ALTROOT
gjp22    572Gi   448Gi   124Gi     78%  ONLINE  -
tall    4.55Ti  1.84Ti  2.70Ti     40%  ONLINE  -
twoz     780Gi   348Gi   432Gi     44%  ONLINE  -
zhandy   592Gi   508Gi  84.1Gi     85%  ONLINE  -


Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get primarycache zhandy
NAME    PROPERTY      VALUE         SOURCE
zhandy  primarycache  metadata      local
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Memory, UBC (unified buffer cache), ARC

Post by grahamperrin » Thu Mar 14, 2013 12:57 am

Spun off from viewtopic.php?p=4236#p4236

royfactorial wrote:… how much RAM does the UBC normally take up on a system that isn't under memory pressure? For example, I have 32GB, with only about 10GB being used at the moment. Is the 'free' RAM listed in Activity Monitor being used for UBC behind the scenes?


To the Activity Monitor question: no.

My understanding:

  • free memory is truly free – not used for the unified buffer cache or any other purpose
  • the memory used for UBC is classified as inactive.

royfactorial wrote:… how much?


Sorry, I can't recall whether there's an easy way to measure the amount or percentage of memory that's used for UBC.

royfactorial wrote:My concern with ZFS was that, since it likes to use … RAM for ARC, that it might be caching a significantly greater quantity of data vs HFS+ and the UBC. …


In the opening post here –

generally, ZFS needs lots of memory.


Why might an operating system with ZFS use more memory than if the same data was stored with HFS Plus?

There are probably good technical explanations beyond this forum.

Less technical, echoed from viewtopic.php?p=3812#p3812

grahamperrin wrote:With HFS Plus, there's the catalog file and so on. Ready made, if you like.

As I understand it … with ZFS there's nothing like that catalogue. And so, more need to traverse things. …


– and I guess that less equity (in output from zstat) may be seen when, for example, there's less traversal.

Links

I/O Kit Fundamentals: Managing Data: More on Memory Cursors: DMA and System Memory, https://developer.apple.com/library/mac ... aMgmt.html#//apple_ref/doc/uid/TP0000019-BCIFHFJJ

> … The Unified Buffer Cache (UBC) is a kernel optimization
> that combines the file-system cache and the virtual-memory
> (VM) cache. The UBC eliminates the situation where the same
> page is duplicated in both caches. Instead, there is just
> one image in memory …

Activity Monitor: View system memory usage (2012-05-08)

Using Activity Monitor to read System Memory and determine how much RAM is being used (2012-05-03)

  • includes a link to another part of the Mac Developer Library, although that part doesn't mention UBC.

Last but not least, there's discussion of ARC and UBC under Large number of fseventsd errors during send.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

A question about dynamic_pager

Post by grahamperrin » Sun Apr 14, 2013 11:32 pm

In Ask Different, a question (driven partly by the amount of memory that I usually find wired for ZEVO):

Mac virtual memory: how to make dynamic_pager create swap (page) files with a normal boot as freely as it does with a safe boot (swapfile1 etc)
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: memory usage, kernel_task and zstat

Post by raattgift » Mon Apr 15, 2013 8:35 am

Why do you want to do that?

"As performance with safe boot is remarkably good" is unlikely to have anything to do with how dynamic_pager is invoked. However, its man page gives you everything you need to know other than a pointer to the plist in /System/Library/LaunchDaemons.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: memory usage, kernel_task and zstat

Post by zslg01 » Mon Apr 15, 2013 7:43 pm

Code: Select all
WIRED GOES NUTS AFTER 2 GB COPY TO ZFS POOL


 v2012.09.23    148 threads        1 mount        17828 vnodes     20:39:41
____________________________________________________________________________
             KALLOC      KERNEL/MAPS        TOTAL         EQUITY
  WIRED    7494 KiB    1691 MiB/1699         1698 MiB      10.37%
  PEAK     9578 KiB    1691 MiB              1700 MiB
  VMPAGE        253 (IN)         83 (OUT)         83 (SYNC)         50 (MDS)
____________________________________________________________________________
                     HITS                  MISSES
  ARC overall:        97% (85513)              3% (2258)
  ARC demand data:    99% (48805)              1% (218)
  ARC demand meta:    96% (36614)              4% (1468)
  ARC prefetch data:  12% (68)                88% (483)
  ARC prefetch meta:  22% (26)                78% (89)
  DMU zfetch:         62% (935536)            38% (572621)
____________________________________________________________________________
     SIZE     SLAB    AVAIL    INUSE    TOTAL     PEAK  KMEM CACHE NAME
       72     4096      149    21356    21505    21505  kmem_slab_cache
       24     4096     1595    90255    91850    92017  kmem_bufctl_cache
       88     4096       74     1006     1080     1080  taskq_ent_cache
      360     4096        4       18       22       22  taskq_cache
      824     8192        8        1        9     5004  zio_cache
       80     4096       22    17828    17850    17850  sa_cache
      840     8192        4    17906    17910    17910  dnode_t
      216     4096        4    44132    44136    44136  dmu_buf_impl_t
Last edited by zslg01 on Tue Apr 16, 2013 6:59 am, edited 2 times in total.
zslg01 Offline


 
Posts: 18
Joined: Wed Nov 14, 2012 7:34 pm

Re: memory usage, kernel_task and zstat

Post by grahamperrin » Mon Apr 15, 2013 10:44 pm

(Post deleted by author.)
Last edited by grahamperrin on Tue Apr 16, 2013 4:23 pm, edited 1 time in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: memory usage, kernel_task and zstat

Post by raattgift » Tue Apr 16, 2013 5:37 am

Sure, whenever you do a write(2) or similar system call to something with backing store behind the ZFS subsystem, one or more dirty pages are placed into UBC and into the ARC, and those pages are locked as WIRED.

These blocks are liable to remain locked until the data is written to the pool.

Under enormous write pressure and with any vfs that writes slowly, seeing large WIRED is normal. ZFS (generically) is clever about safely speeding up the remarking of dirty blocks as clean when there is system memory pressure, but it all depends on the speed of the devices the pool is using.

Write throttling exists in the ZFS subsystem, and will trigger on subsequent writes/page-dirtyings, so the large memory occupancy should not be chronic; however if the problem is created by a truly huge single write(2) call, throttling can only block the writing process from creating more dirty pages for a while. (Well, it could return a write error, but nobody would be happy about that...).

Moreover, if there is no memory pressure, the now-clean pages referenced by the ARC remain WIRED so as to avoid being disturbed by the system's memory management. As memory pressure increases, the ARC does its normal eviction and un-wires pages in freed memory slabs.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: memory usage, kernel_task and zstat

Post by zslg01 » Tue Apr 16, 2013 7:04 am

Sort of defeats the purpose of a combined disk/paging cache as in Linux, FreeBSD, OSX - especially the keeping WIRED pages after they are written. It certainly makes zfs irrelevant for a workstation or laptop.
I've seen WIRED blow up from 1.5 GB to over 8 GB.... and then only drop back to around 4 GB. Leaks?
zslg01 Offline


 
Posts: 18
Joined: Wed Nov 14, 2012 7:34 pm

Re: memory usage, kernel_task and zstat

Post by raattgift » Tue Apr 16, 2013 3:11 pm

The ARC is a system data structure, and system data structures are WIRED in practically all cases to avoid being paged, purged, or otherwise interfered with.

The only substantial exception is the UBC, which is not WIRED but INACTIVE, and it relies upon the system to page, purge and otherwise interfere with its pages. That's OK, because the system's activities in those areas are tuned for UBC, with which it co-evolved.

Any other kernel subsystem (including several Apple-written ones) that requires a different memory management approach must wire down its pages. The ZFS subsystem manages its own (sizable) caches and structures, but also takes some care to also provide some efficiencies in terms of avoiding page mapping churn and memory to memory copying. The slab allocation system that originated with Sun (and informs the on-disk layout of ZFS) is less wasteful of memory on its own than practically all its competitors, including the UBC. Additionally, ARC > UBC for scheduling of the use of memory that depends on the contents and activity level of the imported zfs pools, simply because it is specialized for that (i.e., it's not just the generic IBM algorithm).

ARC evictions and other releases of memory happen whenever there is memory pressure, and ARC does not try to take all of UBC's memory away, which might starve other vfs-es of their backing store.

Inflating to a large amount and deflating again is evidence of good design. Unless you spot a bug wherein a lot of pageouts and subsequent pageins are triggered by the pattern that triggers the "blow up" (or alternatively that there is a reduction in cache performace for other vfses), you should take it as evidence that your memory is being put to good use, rather than being left idle or holding stale data.

Certainly a single memory stat wandering around is a lot better than TXG state changes (to quiescing and syncing) and zio rewriting activity blocking HFS, AFP, NFS, and other writes, or noticeably throttling the performance of threads that do a write to a busy zfs pool.

"It certainly makes zfs irrelevant for a workstation or laptop"

Irrelevant? Really? Don't you think that's overstating your complaint a bit?
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Next

Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 0 guests

cron