macos-2.0.0-rc6 Memory Leak?

Developer discussions.

Re: macos-2.0.0-rc6 Memory Leak?

Postby Jimbo » Tue May 04, 2021 9:42 pm

lundman wrote:
Code: Select all
2021-05-03 18:26:24.764 Df kernel[0:d897] (zfs) <zfs`spl_kmem_fini> SPL: Leaks, numnodes 0
2021-05-03 18:26:24.764 Df kernel[0:d897] (zfs) <zfs`spl_kmem_fini> SPL: End of leaks (if any)
2021-05-03 18:26:24.767 Df kernel[0:d897] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_24' (0xffffffc180011040) not empty


Yeah run the memleak24 posted above, and that should print some leak nodes (instead of numnodes 0), and the exact place they were found.

We are also tracking a problem with raidz, which is turning out to be really mischievous.
https://github.com/openzfsonosx/openzfs/issues/48


That output was from the memleak24 version...
Code: Select all
 % zfs version
zfs-2.0.0-rc6
zfs-kmod-macos-2.0.0-rc6-3-gbd2d850a38-dirty


I'll grab another dump to make sure.

Edit: BTW, both pools are RAIDZ, one is just a single RAIDZ VDEV, the other is two striped RAIDZ VDEVs.
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: macos-2.0.0-rc6 Memory Leak?

Postby Jimbo » Tue May 04, 2021 10:37 pm

After a few days uptime and a bit of activity:
Code: Select all
% zfs version
zfs-2.0.0-rc6
zfs-kmod-macos-2.0.0-rc6-3-gbd2d850a38-dirty


Code: Select all
2021-05-05 14:35:03.317 Df kernel[0:6b768] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> zfs_boot_fini no pool_list to clear
2021-05-05 14:35:03.317 Df kernel[0:6b768] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> ZFS: Attempting to unload ...
2021-05-05 14:35:07.515 Df kernel[0:6b768] (zfs) <zfs`spl_tsd_fini> SPL: tsd unloading 0
2021-05-05 14:35:07.515 Df kernel[0:6b768] (zfs) <zfs`spl_kmem_fini> SPL: Leaks, numnodes 0
2021-05-05 14:35:07.515 Df kernel[0:6b768] (zfs) <zfs`spl_kmem_fini> SPL: End of leaks (if any)
2021-05-05 14:35:07.530 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_24' (0xffffffc180011040) not empty
2021-05-05 14:35:07.664 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_128' (0xffffffc18001a040) not empty
2021-05-05 14:35:07.666 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_160' (0xffffffc18001b040) not empty
2021-05-05 14:35:07.668 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_192' (0xffffffc18001c040) not empty
2021-05-05 14:35:07.670 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_224' (0xffffffc18001d040) not empty
2021-05-05 14:35:07.672 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_256' (0xffffffc18001e040) not empty
2021-05-05 14:35:08.159 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_320' (0xffffffc18001f040) not empty
2021-05-05 14:35:08.160 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_384' (0xffffffc180020040) not empty
2021-05-05 14:35:08.588 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_40960' (0xffffffc180036040) not empty
2021-05-05 14:35:10.263 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_va_4096' (0xffffffc18000c040) not empty
2021-05-05 14:35:11.052 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_bufctl_cache' (0xffffffc18000a040) not empty
2021-05-05 14:35:11.477 Df kernel[0:6b768] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_slab_cache' (0xffffffc180009040) not empty
2021-05-05 14:35:11.485 Df kernel[0:6b768] (zfs) <zfs`spl_kmem_fini> SPL: Released 5448 slabs
2021-05-05 14:35:11.485 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_default'): leaked 4664561664 bytes
2021-05-05 14:35:11.530 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_va'): leaked 4739194880 bytes
2021-05-05 14:35:11.568 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_msb'): leaked 291618816 bytes
2021-05-05 14:35:11.591 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_metadata'): leaked 291618816 bytes
2021-05-05 14:35:11.661 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('heap'): leaked 5363818496 bytes
2021-05-05 14:35:11.725 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking spl_heap_arena, aka bucket_heap (pass 2)
2021-05-05 14:35:11.748 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking bucket arenas...
2021-05-05 14:35:11.784 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_32768'): leaked 726564864 bytes
2021-05-05 14:35:11.789 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_65536'): leaked 4199587840 bytes
2021-05-05 14:35:11.810 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking vmem metadata-related arenas...
2021-05-05 14:35:11.819 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_heap'): leaked 5363818496 bytes
2021-05-05 14:35:11.863 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_seg'): leaked 456294400 bytes
2021-05-05 14:35:11.871 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_hash'): leaked 393216 bytes
2021-05-05 14:35:11.871 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_metadata'): leaked 456687616 bytes
2021-05-05 14:35:11.878 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('spl_default_arena'): leaked 456753152 bytes
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy_internal> SPL: vmem_destroy('vmem_vmem'): leaked 7680 bytes
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_destroy_internal> SPL: vmem_destroy('vmem_vmem'): STILL 7680 bytes at kstat_delete() time
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> SPL: arenas removed, now try destroying mutexes...
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> vmem_xnu_alloc_lock
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> vmem_panic_lock
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> vmem_pushpage_lock
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> vmem_nosleep_lock
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> vmem_sleep_lock
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> vmem_segfree_lock
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> vmem_list_lock
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini>
SPL: vmem_fini: walking list of live slabs at time of call to vmem_fini
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> SPL: WOULD HAVE released 0 bytes (0 spans) from arenas
2021-05-05 14:35:11.880 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> SPL: vmem_fini: Brief delay for readability...
2021-05-05 14:35:12.875 Df kernel[0:6b768] (zfs) <zfs`vmem_fini> SPL: vmem_fini: done!
2021-05-05 14:35:12.875 Df kernel[0:6b768] (zfs) <zfs`spl_mutex_subsystem_fini> Dumping leaked mutex allocations...
2021-05-05 14:35:12.875 Df kernel[0:6b768] (zfs) <zfs`spl_mutex_subsystem_fini> Dumped 0 leaked allocations. Wait for watchdog to exit..
2021-05-05 14:35:15.134 Df kernel[0:ea9] (zfs) <zfs`spl_wdlist_check> SPL: watchdog thread exit
2021-05-05 14:35:15.134 Df kernel[0:6b768] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> ZFS: Unloaded module v2.0.0-rc6


Not seeing anything.
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: macos-2.0.0-rc6 Memory Leak?

Postby Jaw3000 » Thu May 06, 2021 11:24 pm

I'm not sure if this is related to the memory leak, but after installing rc6, I get an "internal error: out of memory" when importing any pool. There is no memory issue with the system and plenty of memory is available.

The last build I have been able to get to work properly enough on Big Sur was zfs-2.0.0-1 zfs-kmod-macOS-2.0.0-2-g0197ddc9c5-dirty from December 2020. The subsequent builds have all had immediate issues. RC6 with the 'out of memory' error. Previous builds had issues with not auto-remounting pools on reboot. And each installer since has had errors with installing, where it fails and I have to restart and install again for the installer to complete and get the security prompt to approve the new extension. The December build has no issues with installing the first time. I have 'boot-args keepsyms=1' set.
Jaw3000
 
Posts: 4
Joined: Thu May 06, 2021 11:15 pm

Re: macos-2.0.0-rc6 Memory Leak?

Postby lundman » Thu May 06, 2021 11:52 pm

Ok probably something to do with the name change. But we did find a leak, so let's try again once the new version is out.
User avatar
lundman
 
Posts: 1334
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: macos-2.0.0-rc6 Memory Leak?

Postby FunMiles » Wed May 12, 2021 6:27 am

Jaw3000 wrote:I'm not sure if this is related to the memory leak, but after installing rc6, I get an "internal error: out of memory" when importing any pool. There is no memory issue with the system and plenty of memory is available.

The last build I have been able to get to work properly enough on Big Sur was zfs-2.0.0-1 zfs-kmod-macOS-2.0.0-2-g0197ddc9c5-dirty from December 2020. The subsequent builds have all had immediate issues. RC6 with the 'out of memory' error. Previous builds had issues with not auto-remounting pools on reboot. And each installer since has had errors with installing, where it fails and I have to restart and install again for the installer to complete and get the security prompt to approve the new extension. The December build has no issues with installing the first time. I have 'boot-args keepsyms=1' set.

I've experienced the same issue of 'out of memory'. I am running the April release which suits most of my needs except it can't reliably do a send/receive. I will try the next version when it is ready and report the results.
FunMiles
 
Posts: 27
Joined: Sat Sep 30, 2017 12:05 pm

Re: macos-2.0.0-rc6 Memory Leak?

Postby jwiegley » Fri May 14, 2021 3:29 pm

I just installed rc6, but have memory issues also:

Code: Select all
Vulcan ~ $ zpool --version
zfs-2.0.0-rc6
zfs-kmod-zfs-2.0.0-rc1-452-g964b2a6de7
Vulcan ~ $ sudo zpool import tank
internal error: out of memory
jwiegley
 
Posts: 7
Joined: Fri May 14, 2021 9:06 am

Re: macos-2.0.0-rc6 Memory Leak?

Postby dereed999 » Mon May 17, 2021 2:18 pm

With RC7, I've moved 10TB to/from the ZFS pool and everything seems to have stabilized. My Pages wired down no longer grows uncontrolled and no use of swapping so memory pressure seems stable. The system would have panicked by now under rc6 so while it's possible there are still edge cases, thank you so much for finding the memory leak!
dereed999
 
Posts: 21
Joined: Thu Sep 10, 2020 6:33 am

Previous

Return to OpenZFS on OS X Development

Who is online

Users browsing this forum: No registered users and 3 guests