macos-2.0.0-rc6 Memory Leak?

Developer discussions.

macos-2.0.0-rc6 Memory Leak?

Postby dereed999 » Fri Apr 30, 2021 11:16 am

I have a raidz3 pool (90TB ish) and seem to have a memory leak. When accessing files on either the native ZFS file system or on zvols formatted w/ HFS+ the system will eventually reboot after exhausting all memory. A vm_stat in a while loop shows the wired down pages increasing more or less monotonically and then eventually swapins/outs starting to happen and then an eventual panic. If I run the system but with the pool exported I never run into any issues. So it seems to be related to the zfs subsystem/extension.

I can reproduce it by doing a read loop of copying files from the filesystem.
For instance, on a ZVOL formatted with HFS+ filled with media files, the following
Code: Select all
find . -type f -exec cat {} > /dev/null \;

will trigger the eventual crash. I did all I could to quiesce all other activity and looking at lsof saw no other activity on the volume.

Thoughts on how I can help track down where the memory leak could be coming from?

If it helps, my pool doesn't have all the latest features enabled. In particular, a zpool upgrade informs me the following aren't enabled:
userobj_accounting
project_quota
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress
draid


Thanks!
-david
dereed999
 
Posts: 21
Joined: Thu Sep 10, 2020 6:33 am

Re: macos-2.0.0-rc6 Memory Leak?

Postby lundman » Fri Apr 30, 2021 2:09 pm

# sysctl kstat | grep inuse

Would show all the memory areas and how much memory they are using. Usually they will go up initially, but eventually find some balance and hover around some amount. If you find one "ever-increasing" then we should take a look at it.

Additionally, kstat.vmem.vmem.spl_default_arena.mem_inuse is the grand total memory we have taken from XNU.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: macos-2.0.0-rc6 Memory Leak?

Postby dereed999 » Sat May 01, 2021 11:30 am

The last check before panic gave the following. The device only has 64GB of RAM...

Code: Select all
kstat.vmem.vmem.spl_default_arena_parent.mem_inuse: 0
kstat.vmem.vmem.spl_default_arena.mem_inuse: 1821442048
kstat.vmem.vmem.bucket_4096.mem_inuse: 0
kstat.vmem.vmem.bucket_8192.mem_inuse: 0
kstat.vmem.vmem.bucket_16384.mem_inuse: 109793280
kstat.vmem.vmem.bucket_32768.mem_inuse: 1991430144
kstat.vmem.vmem.bucket_65536.mem_inuse: 35545858048
kstat.vmem.vmem.bucket_131072.mem_inuse: 81625088
kstat.vmem.vmem.bucket_262144.mem_inuse: 1409024
kstat.vmem.vmem.bucket_524288.mem_inuse: 0
kstat.vmem.vmem.bucket_1048576.mem_inuse: 0
kstat.vmem.vmem.bucket_2097152.mem_inuse: 4194304
kstat.vmem.vmem.bucket_4194304.mem_inuse: 0
kstat.vmem.vmem.bucket_8388608.mem_inuse: 8388608
kstat.vmem.vmem.bucket_16777216.mem_inuse: 0
kstat.vmem.vmem.bucket_heap.mem_inuse: 38808993792
kstat.vmem.vmem.heap.mem_inuse: 38767706112
kstat.vmem.vmem.vmem_metadata.mem_inuse: 747675648
kstat.vmem.vmem.vmem_seg.mem_inuse: 716750848
kstat.vmem.vmem.vmem_hash.mem_inuse: 30838272
kstat.vmem.vmem.vmem_vmem.mem_inuse: 238080
kstat.vmem.vmem.kmem_metadata.mem_inuse: 163889152
kstat.vmem.vmem.kmem_msb.mem_inuse: 147140608
kstat.vmem.vmem.kmem_cache.mem_inuse: 459200
kstat.vmem.vmem.kmem_hash.mem_inuse: 15504384
kstat.vmem.vmem.kmem_log.mem_inuse: 396864
kstat.vmem.vmem.kmem_oversize.mem_inuse: 67272704
kstat.vmem.vmem.kmem_va.mem_inuse: 38535839744
kstat.vmem.vmem.kmem_default.mem_inuse: 38277607424
kstat.vmem.vmem.taskq_id_arena.mem_inuse: 54
kstat.vmem.vmem.abd_cache.mem_inuse: 18530304
kstat.unix.kmem_cache.kmem_magazine_1.buf_inuse: 537
kstat.unix.kmem_cache.kmem_magazine_3.buf_inuse: 598
kstat.unix.kmem_cache.kmem_magazine_7.buf_inuse: 335
kstat.unix.kmem_cache.kmem_magazine_15.buf_inuse: 306
kstat.unix.kmem_cache.kmem_magazine_31.buf_inuse: 288
kstat.unix.kmem_cache.kmem_magazine_47.buf_inuse: 96
kstat.unix.kmem_cache.kmem_magazine_63.buf_inuse: 24
kstat.unix.kmem_cache.kmem_magazine_95.buf_inuse: 24
kstat.unix.kmem_cache.kmem_magazine_143.buf_inuse: 287
kstat.unix.kmem_cache.kmem_slab_cache.buf_inuse: 1094620
kstat.unix.kmem_cache.kmem_bufctl_cache.buf_inuse: 1897413
kstat.unix.kmem_cache.kmem_bufctl_audit_cache.buf_inuse: 0
kstat.unix.kmem_cache.kmem_va_4096.buf_inuse: 532187
kstat.unix.kmem_cache.kmem_va_8192.buf_inuse: 23344
kstat.unix.kmem_cache.kmem_alloc_8.buf_inuse: 151
kstat.unix.kmem_cache.kmem_alloc_16.buf_inuse: 577
kstat.unix.kmem_cache.kmem_alloc_24.buf_inuse: 19779
kstat.unix.kmem_cache.kmem_alloc_32.buf_inuse: 65
kstat.unix.kmem_cache.kmem_alloc_40.buf_inuse: 14
kstat.unix.kmem_cache.kmem_alloc_48.buf_inuse: 597
kstat.unix.kmem_cache.kmem_alloc_56.buf_inuse: 199
kstat.unix.kmem_cache.kmem_alloc_64.buf_inuse: 20275
kstat.unix.kmem_cache.kmem_alloc_80.buf_inuse: 135
kstat.unix.kmem_cache.kmem_alloc_96.buf_inuse: 12121
kstat.unix.kmem_cache.kmem_alloc_112.buf_inuse: 7481
kstat.unix.kmem_cache.kmem_alloc_128.buf_inuse: 250067
kstat.unix.kmem_cache.kmem_alloc_160.buf_inuse: 781004
kstat.unix.kmem_cache.kmem_alloc_192.buf_inuse: 545688
kstat.unix.kmem_cache.kmem_alloc_224.buf_inuse: 252596
kstat.unix.kmem_cache.kmem_alloc_256.buf_inuse: 365186
kstat.unix.kmem_cache.kmem_alloc_320.buf_inuse: 461198
kstat.unix.kmem_cache.kmem_alloc_384.buf_inuse: 225543
kstat.unix.kmem_cache.kmem_alloc_448.buf_inuse: 3
kstat.unix.kmem_cache.kmem_alloc_512.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_576.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_640.buf_inuse: 6157
kstat.unix.kmem_cache.kmem_alloc_768.buf_inuse: 2
kstat.unix.kmem_cache.kmem_alloc_832.buf_inuse: 5
kstat.unix.kmem_cache.kmem_alloc_896.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_1024.buf_inuse: 3
kstat.unix.kmem_cache.kmem_alloc_1152.buf_inuse: 11
kstat.unix.kmem_cache.kmem_alloc_1344.buf_inuse: 12
kstat.unix.kmem_cache.kmem_alloc_1600.buf_inuse: 72
kstat.unix.kmem_cache.kmem_alloc_2048.buf_inuse: 482
kstat.unix.kmem_cache.kmem_alloc_2688.buf_inuse: 41
kstat.unix.kmem_cache.kmem_alloc_4096.buf_inuse: 7380
kstat.unix.kmem_cache.kmem_alloc_8192.buf_inuse: 20
kstat.unix.kmem_cache.kmem_alloc_12288.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_16384.buf_inuse: 12
kstat.unix.kmem_cache.kmem_alloc_24576.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_32768.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_40960.buf_inuse: 871719
kstat.unix.kmem_cache.kmem_alloc_49152.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_57344.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_65536.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_73728.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_81920.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_90112.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_98304.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_106496.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_114688.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_122880.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_131072.buf_inuse: 0
kstat.unix.kmem_cache.taskq_ent_cache.buf_inuse: 1607
kstat.unix.kmem_cache.taskq_cache.buf_inuse: 55
kstat.unix.kmem_cache.kmem_defrag_cache.buf_inuse: 1
kstat.unix.kmem_cache.kmem_move_cache.buf_inuse: 0
kstat.unix.kmem_cache.zfs_btree_leaf_cache.buf_inuse: 918
kstat.unix.kmem_cache.metaslab_alloc_trace_cache.buf_inuse: 0
kstat.unix.kmem_cache.ddt_cache.buf_inuse: 14
kstat.unix.kmem_cache.ddt_entry_cache.buf_inuse: 0
kstat.unix.kmem_cache.zio_cache.buf_inuse: 16
kstat.unix.kmem_cache.zio_link_cache.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_512.buf_inuse: 7
kstat.unix.kmem_cache.zio_data_buf_512.buf_inuse: 1
kstat.unix.kmem_cache.zio_buf_1024.buf_inuse: 2
kstat.unix.kmem_cache.zio_data_buf_1024.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1536.buf_inuse: 1
kstat.unix.kmem_cache.zio_data_buf_1536.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_2048.buf_inuse: 10
kstat.unix.kmem_cache.zio_data_buf_2048.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_2560.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_2560.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_3072.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_3072.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_3584.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_3584.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_4096.buf_inuse: 7377
kstat.unix.kmem_cache.zio_data_buf_4096.buf_inuse: 3
kstat.unix.kmem_cache.zio_buf_5120.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_5120.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_6144.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_6144.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_7168.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_7168.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_8192.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_8192.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_10240.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_10240.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_12288.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_12288.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_14336.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_14336.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_16384.buf_inuse: 7398
kstat.unix.kmem_cache.zio_data_buf_16384.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_20480.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_20480.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_24576.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_24576.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_28672.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_28672.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_32768.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_32768.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_40960.buf_inuse: 1
kstat.unix.kmem_cache.zio_data_buf_40960.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_49152.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_49152.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_57344.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_57344.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_65536.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_65536.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_81920.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_81920.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_98304.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_98304.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_114688.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_114688.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_131072.buf_inuse: 137
kstat.unix.kmem_cache.zio_data_buf_131072.buf_inuse: 216
kstat.unix.kmem_cache.zio_buf_163840.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_163840.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_196608.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_196608.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_229376.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_229376.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_262144.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_262144.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_327680.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_327680.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_393216.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_393216.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_458752.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_458752.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_524288.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_524288.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_655360.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_655360.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_786432.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_786432.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_917504.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_917504.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1048576.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_1048576.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1310720.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_1310720.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1572864.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_1572864.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1835008.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_1835008.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_2097152.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_2097152.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_2621440.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_2621440.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_3145728.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_3145728.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_3670016.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_3670016.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_4194304.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_4194304.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_5242880.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_5242880.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_6291456.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_6291456.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_7340032.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_7340032.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_8388608.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_8388608.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_10485760.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_10485760.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_12582912.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_12582912.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_14680064.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_14680064.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_16777216.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_16777216.buf_inuse: 0
kstat.unix.kmem_cache.lz4_cache.buf_inuse: 0
kstat.unix.kmem_cache.abd_chunk.buf_inuse: 4227
kstat.unix.kmem_cache.sa_cache.buf_inuse: 165756
kstat.unix.kmem_cache.dnode_t.buf_inuse: 204592
kstat.unix.kmem_cache.arc_buf_hdr_t_full.buf_inuse: 17921
kstat.unix.kmem_cache.arc_buf_hdr_t_full_crypt.buf_inuse: 0
kstat.unix.kmem_cache.arc_buf_hdr_t_l2only.buf_inuse: 7909613
kstat.unix.kmem_cache.arc_buf_t.buf_inuse: 7765
kstat.unix.kmem_cache.dmu_buf_impl_t.buf_inuse: 212282
kstat.unix.kmem_cache.zil_lwb_cache.buf_inuse: 3
kstat.unix.kmem_cache.zil_zcw_cache.buf_inuse: 0
kstat.unix.kmem_cache.sio_cache_0.buf_inuse: 0
kstat.unix.kmem_cache.sio_cache_1.buf_inuse: 0
kstat.unix.kmem_cache.sio_cache_2.buf_inuse: 0
kstat.unix.kmem_cache.zfs_znode_cache.buf_inuse: 165758
kstat.unix.kmem_cache.zfs_znode_hold_cache.buf_inuse: 0
kstat.unix.kmem_cache.mod_hash_entries.buf_inuse: 35
kstat.unix.kmem_cache.kcf_sreq_cache.buf_inuse: 0
kstat.unix.kmem_cache.kcf_areq_cache.buf_inuse: 0
kstat.unix.kmem_cache.kcf_context_cache.buf_inuse: 0


About an hour before, during the "while loop copy" this is the result:
Code: Select all
kstat.vmem.vmem.spl_default_arena_parent.mem_inuse: 0
kstat.vmem.vmem.spl_default_arena.mem_inuse: 1630437376
kstat.vmem.vmem.bucket_4096.mem_inuse: 0
kstat.vmem.vmem.bucket_8192.mem_inuse: 0
kstat.vmem.vmem.bucket_16384.mem_inuse: 211300352
kstat.vmem.vmem.bucket_32768.mem_inuse: 1715433472
kstat.vmem.vmem.bucket_65536.mem_inuse: 24014233600
kstat.vmem.vmem.bucket_131072.mem_inuse: 166510592
kstat.vmem.vmem.bucket_262144.mem_inuse: 2064384
kstat.vmem.vmem.bucket_524288.mem_inuse: 0
kstat.vmem.vmem.bucket_1048576.mem_inuse: 0
kstat.vmem.vmem.bucket_2097152.mem_inuse: 6291456
kstat.vmem.vmem.bucket_4194304.mem_inuse: 0
kstat.vmem.vmem.bucket_8388608.mem_inuse: 0
kstat.vmem.vmem.bucket_16777216.mem_inuse: 0
kstat.vmem.vmem.bucket_heap.mem_inuse: 27183534080
kstat.vmem.vmem.heap.mem_inuse: 27071336448
kstat.vmem.vmem.vmem_metadata.mem_inuse: 556654592
kstat.vmem.vmem.vmem_seg.mem_inuse: 532021248
kstat.vmem.vmem.vmem_hash.mem_inuse: 24546816
kstat.vmem.vmem.vmem_vmem.mem_inuse: 238080
kstat.vmem.vmem.kmem_metadata.mem_inuse: 136900608
kstat.vmem.vmem.kmem_msb.mem_inuse: 126406656
kstat.vmem.vmem.kmem_cache.mem_inuse: 459200
kstat.vmem.vmem.kmem_hash.mem_inuse: 9242112
kstat.vmem.vmem.kmem_log.mem_inuse: 396864
kstat.vmem.vmem.kmem_oversize.mem_inuse: 67272704
kstat.vmem.vmem.kmem_va.mem_inuse: 26864865280
kstat.vmem.vmem.kmem_default.mem_inuse: 26786512896
kstat.vmem.vmem.taskq_id_arena.mem_inuse: 54
kstat.vmem.vmem.abd_cache.mem_inuse: 31277056
kstat.unix.kmem_cache.kmem_magazine_1.buf_inuse: 563
kstat.unix.kmem_cache.kmem_magazine_3.buf_inuse: 636
kstat.unix.kmem_cache.kmem_magazine_7.buf_inuse: 396
kstat.unix.kmem_cache.kmem_magazine_15.buf_inuse: 344
kstat.unix.kmem_cache.kmem_magazine_31.buf_inuse: 326
kstat.unix.kmem_cache.kmem_magazine_47.buf_inuse: 150
kstat.unix.kmem_cache.kmem_magazine_63.buf_inuse: 36
kstat.unix.kmem_cache.kmem_magazine_95.buf_inuse: 0
kstat.unix.kmem_cache.kmem_magazine_143.buf_inuse: 425
kstat.unix.kmem_cache.kmem_slab_cache.buf_inuse: 867687
kstat.unix.kmem_cache.kmem_bufctl_cache.buf_inuse: 1710034
kstat.unix.kmem_cache.kmem_bufctl_audit_cache.buf_inuse: 0
kstat.unix.kmem_cache.kmem_va_4096.buf_inuse: 498905
kstat.unix.kmem_cache.kmem_va_8192.buf_inuse: 30583
kstat.unix.kmem_cache.kmem_alloc_8.buf_inuse: 70
kstat.unix.kmem_cache.kmem_alloc_16.buf_inuse: 577
kstat.unix.kmem_cache.kmem_alloc_24.buf_inuse: 38929
kstat.unix.kmem_cache.kmem_alloc_32.buf_inuse: 66
kstat.unix.kmem_cache.kmem_alloc_40.buf_inuse: 26
kstat.unix.kmem_cache.kmem_alloc_48.buf_inuse: 615
kstat.unix.kmem_cache.kmem_alloc_56.buf_inuse: 199
kstat.unix.kmem_cache.kmem_alloc_64.buf_inuse: 39415
kstat.unix.kmem_cache.kmem_alloc_80.buf_inuse: 162
kstat.unix.kmem_cache.kmem_alloc_96.buf_inuse: 31180
kstat.unix.kmem_cache.kmem_alloc_112.buf_inuse: 13974
kstat.unix.kmem_cache.kmem_alloc_128.buf_inuse: 67062
kstat.unix.kmem_cache.kmem_alloc_160.buf_inuse: 306802
kstat.unix.kmem_cache.kmem_alloc_192.buf_inuse: 235119
kstat.unix.kmem_cache.kmem_alloc_224.buf_inuse: 72007
kstat.unix.kmem_cache.kmem_alloc_256.buf_inuse: 176438
kstat.unix.kmem_cache.kmem_alloc_320.buf_inuse: 425488
kstat.unix.kmem_cache.kmem_alloc_384.buf_inuse: 144371
kstat.unix.kmem_cache.kmem_alloc_448.buf_inuse: 55
kstat.unix.kmem_cache.kmem_alloc_512.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_576.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_640.buf_inuse: 6157
kstat.unix.kmem_cache.kmem_alloc_768.buf_inuse: 3
kstat.unix.kmem_cache.kmem_alloc_832.buf_inuse: 5
kstat.unix.kmem_cache.kmem_alloc_896.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_1024.buf_inuse: 3
kstat.unix.kmem_cache.kmem_alloc_1152.buf_inuse: 11
kstat.unix.kmem_cache.kmem_alloc_1344.buf_inuse: 12
kstat.unix.kmem_cache.kmem_alloc_1600.buf_inuse: 101
kstat.unix.kmem_cache.kmem_alloc_2048.buf_inuse: 482
kstat.unix.kmem_cache.kmem_alloc_2688.buf_inuse: 41
kstat.unix.kmem_cache.kmem_alloc_4096.buf_inuse: 13738
kstat.unix.kmem_cache.kmem_alloc_8192.buf_inuse: 20
kstat.unix.kmem_cache.kmem_alloc_12288.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_16384.buf_inuse: 12
kstat.unix.kmem_cache.kmem_alloc_24576.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_32768.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_40960.buf_inuse: 590121
kstat.unix.kmem_cache.kmem_alloc_49152.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_57344.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_65536.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_73728.buf_inuse: 1
kstat.unix.kmem_cache.kmem_alloc_81920.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_90112.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_98304.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_106496.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_114688.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_122880.buf_inuse: 0
kstat.unix.kmem_cache.kmem_alloc_131072.buf_inuse: 0
kstat.unix.kmem_cache.taskq_ent_cache.buf_inuse: 1617
kstat.unix.kmem_cache.taskq_cache.buf_inuse: 55
kstat.unix.kmem_cache.kmem_defrag_cache.buf_inuse: 1
kstat.unix.kmem_cache.kmem_move_cache.buf_inuse: 0
kstat.unix.kmem_cache.zfs_btree_leaf_cache.buf_inuse: 911
kstat.unix.kmem_cache.metaslab_alloc_trace_cache.buf_inuse: 0
kstat.unix.kmem_cache.ddt_cache.buf_inuse: 14
kstat.unix.kmem_cache.ddt_entry_cache.buf_inuse: 0
kstat.unix.kmem_cache.zio_cache.buf_inuse: 122
kstat.unix.kmem_cache.zio_link_cache.buf_inuse: 110
kstat.unix.kmem_cache.zio_buf_512.buf_inuse: 7
kstat.unix.kmem_cache.zio_data_buf_512.buf_inuse: 84
kstat.unix.kmem_cache.zio_buf_1024.buf_inuse: 2
kstat.unix.kmem_cache.zio_data_buf_1024.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1536.buf_inuse: 1
kstat.unix.kmem_cache.zio_data_buf_1536.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_2048.buf_inuse: 9
kstat.unix.kmem_cache.zio_data_buf_2048.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_2560.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_2560.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_3072.buf_inuse: 1
kstat.unix.kmem_cache.zio_data_buf_3072.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_3584.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_3584.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_4096.buf_inuse: 12740
kstat.unix.kmem_cache.zio_data_buf_4096.buf_inuse: 1
kstat.unix.kmem_cache.zio_buf_5120.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_5120.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_6144.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_6144.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_7168.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_7168.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_8192.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_8192.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_10240.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_10240.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_12288.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_12288.buf_inuse: 75
kstat.unix.kmem_cache.zio_buf_14336.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_14336.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_16384.buf_inuse: 13756
kstat.unix.kmem_cache.zio_data_buf_16384.buf_inuse: 3
kstat.unix.kmem_cache.zio_buf_20480.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_20480.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_24576.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_24576.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_28672.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_28672.buf_inuse: 6
kstat.unix.kmem_cache.zio_buf_32768.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_32768.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_40960.buf_inuse: 1
kstat.unix.kmem_cache.zio_data_buf_40960.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_49152.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_49152.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_57344.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_57344.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_65536.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_65536.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_81920.buf_inuse: 1
kstat.unix.kmem_cache.zio_data_buf_81920.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_98304.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_98304.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_114688.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_114688.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_131072.buf_inuse: 163
kstat.unix.kmem_cache.zio_data_buf_131072.buf_inuse: 223
kstat.unix.kmem_cache.zio_buf_163840.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_163840.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_196608.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_196608.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_229376.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_229376.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_262144.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_262144.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_327680.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_327680.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_393216.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_393216.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_458752.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_458752.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_524288.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_524288.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_655360.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_655360.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_786432.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_786432.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_917504.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_917504.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1048576.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_1048576.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1310720.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_1310720.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1572864.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_1572864.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_1835008.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_1835008.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_2097152.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_2097152.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_2621440.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_2621440.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_3145728.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_3145728.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_3670016.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_3670016.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_4194304.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_4194304.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_5242880.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_5242880.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_6291456.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_6291456.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_7340032.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_7340032.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_8388608.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_8388608.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_10485760.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_10485760.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_12582912.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_12582912.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_14680064.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_14680064.buf_inuse: 0
kstat.unix.kmem_cache.zio_buf_16777216.buf_inuse: 0
kstat.unix.kmem_cache.zio_data_buf_16777216.buf_inuse: 0
kstat.unix.kmem_cache.lz4_cache.buf_inuse: 0
kstat.unix.kmem_cache.abd_chunk.buf_inuse: 6874
kstat.unix.kmem_cache.sa_cache.buf_inuse: 241228
kstat.unix.kmem_cache.dnode_t.buf_inuse: 271534
kstat.unix.kmem_cache.arc_buf_hdr_t_full.buf_inuse: 25257
kstat.unix.kmem_cache.arc_buf_hdr_t_full_crypt.buf_inuse: 0
kstat.unix.kmem_cache.arc_buf_hdr_t_l2only.buf_inuse: 7902818
kstat.unix.kmem_cache.arc_buf_t.buf_inuse: 14148
kstat.unix.kmem_cache.dmu_buf_impl_t.buf_inuse: 285608
kstat.unix.kmem_cache.zil_lwb_cache.buf_inuse: 3
kstat.unix.kmem_cache.zil_zcw_cache.buf_inuse: 0
kstat.unix.kmem_cache.sio_cache_0.buf_inuse: 0
kstat.unix.kmem_cache.sio_cache_1.buf_inuse: 0
kstat.unix.kmem_cache.sio_cache_2.buf_inuse: 0
kstat.unix.kmem_cache.zfs_znode_cache.buf_inuse: 241229
kstat.unix.kmem_cache.zfs_znode_hold_cache.buf_inuse: 0
kstat.unix.kmem_cache.mod_hash_entries.buf_inuse: 35
kstat.unix.kmem_cache.kcf_sreq_cache.buf_inuse: 0
kstat.unix.kmem_cache.kcf_areq_cache.buf_inuse: 0
kstat.unix.kmem_cache.kcf_context_cache.buf_inuse: 0


Honestly not 100% sure what to look for...
dereed999
 
Posts: 21
Joined: Thu Sep 10, 2020 6:33 am

Re: macos-2.0.0-rc6 Memory Leak?

Postby lundman » Sat May 01, 2021 4:25 pm

This is certainly interesting:
Code: Select all
kstat.vmem.vmem.bucket_16384.mem_inuse: 109793280.   = 00.1G
kstat.vmem.vmem.bucket_32768.mem_inuse: 1991430144. = 01.8G
kstat.vmem.vmem.bucket_65536.mem_inuse: 35545858048 = 33.0G
kstat.vmem.vmem.bucket_131072.mem_inuse: 81625088
kstat.vmem.vmem.bucket_262144.mem_inuse: 1409024


So it certainly looks like a leak in 64KB blocks. You can probably confirm that by running;

sudo log stream --source --predicate 'sender == "zfs"' --style compact

in a window, then run;

zpool export -a
kextunload -b org.openzfsonosx.zfs

to unload the kext, it will complain about leaked memory it detects.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: macos-2.0.0-rc6 Memory Leak?

Postby lundman » Sat May 01, 2021 4:52 pm

Can we also confirm you dont have anything in /etc/zfs/zsysctl.conf ?
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: macos-2.0.0-rc6 Memory Leak?

Postby dereed999 » Sat May 01, 2021 5:33 pm

I don't have that file.

My /etc/zfs directory only has:
Code: Select all
$ ls -ltRF
total 80
-rw-r--r--   1 root  wheel  5076 May  1 20:50 zpool.cache
drwxr-xr-x  19 root  wheel   608 Apr 30 10:15 zed.d/
drwxr-xr-x  44 root  wheel  1408 Apr 30 10:15 zpool.d/
-rw-r--r--   1 root  wheel   165 Apr 29 21:33 vdev_id.conf.alias.example
-rw-r--r--   1 root  wheel   166 Apr 29 21:33 vdev_id.conf.multipath.example
-rw-r--r--   1 root  wheel   616 Apr 29 21:33 vdev_id.conf.sas_direct.example
-rw-r--r--   1 root  wheel   152 Apr 29 21:33 vdev_id.conf.sas_switch.example
-rw-r--r--   1 root  wheel   254 Apr 29 21:33 vdev_id.conf.scsi.example
-rwxr-xr-x   1 root  wheel  9813 Apr 29 21:33 zfs-functions*

./zed.d:
total 40
lrwxr-xr-x  1 root  wheel     50 Apr 30 10:15 snapshot_mount.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/snapshot_mount.sh
lrwxr-xr-x  1 root  wheel     51 Apr 30 10:15 statechange-led.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/statechange-led.sh
lrwxr-xr-x  1 root  wheel     63 Apr 30 10:15 resilver_finish-start-scrub.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/resilver_finish-start-scrub.sh
lrwxr-xr-x  1 root  wheel     54 Apr 30 10:15 statechange-notify.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/statechange-notify.sh
lrwxr-xr-x  1 root  wheel     52 Apr 30 10:15 snapshot_unmount.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/snapshot_unmount.sh
lrwxr-xr-x  1 root  wheel     51 Apr 30 10:15 vdev_attach-led.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/vdev_attach-led.sh
lrwxr-xr-x  1 root  wheel     58 Apr 30 10:15 resilver_finish-notify.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/resilver_finish-notify.sh
lrwxr-xr-x  1 root  wheel     55 Apr 30 10:15 scrub_finish-notify.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/scrub_finish-notify.sh
lrwxr-xr-x  1 root  wheel     51 Apr 30 10:15 pool_import-led.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/pool_import-led.sh
lrwxr-xr-x  1 root  wheel     47 Apr 30 10:15 zvol.create.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/zvol.create.sh
lrwxr-xr-x  1 root  wheel     46 Apr 30 10:15 all-syslog.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/all-syslog.sh
lrwxr-xr-x  1 root  wheel     47 Apr 30 10:15 data-notify.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/data-notify.sh
lrwxr-xr-x  1 root  wheel     65 Apr 30 10:15 history_event-zfs-list-cacher.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/history_event-zfs-list-cacher.sh
lrwxr-xr-x  1 root  wheel     50 Apr 30 10:15 vdev_clear-led.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/vdev_clear-led.sh
lrwxr-xr-x  1 root  wheel     47 Apr 30 10:15 zvol.remove.sh@ -> /usr/local/zfs/libexec/zfs/zed.d/zvol.remove.sh
-rw-r--r--  1 root  wheel  14320 Apr 29 21:34 zed-functions.sh
-rw-------  1 root  wheel   3590 Apr 29 21:34 zed.rc

./zpool.d:
total 0
lrwxr-xr-x  1 root  wheel  40 Apr 30 10:15 media@ -> /usr/local/zfs/libexec/zfs/zpool.d/media
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 vendor@ -> /usr/local/zfs/libexec/zfs/zpool.d/vendor
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 serial@ -> /usr/local/zfs/libexec/zfs/zpool.d/serial
lrwxr-xr-x  1 root  wheel  42 Apr 30 10:15 dm-deps@ -> /usr/local/zfs/libexec/zfs/zpool.d/dm-deps
lrwxr-xr-x  1 root  wheel  43 Apr 30 10:15 nvme_err@ -> /usr/local/zfs/libexec/zfs/zpool.d/nvme_err
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 w_ucor@ -> /usr/local/zfs/libexec/zfs/zpool.d/w_ucor
lrwxr-xr-x  1 root  wheel  38 Apr 30 10:15 ses@ -> /usr/local/zfs/libexec/zfs/zpool.d/ses
lrwxr-xr-x  1 root  wheel  39 Apr 30 10:15 slot@ -> /usr/local/zfs/libexec/zfs/zpool.d/slot
lrwxr-xr-x  1 root  wheel  39 Apr 30 10:15 size@ -> /usr/local/zfs/libexec/zfs/zpool.d/size
lrwxr-xr-x  1 root  wheel  40 Apr 30 10:15 upath@ -> /usr/local/zfs/libexec/zfs/zpool.d/upath
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 nonmed@ -> /usr/local/zfs/libexec/zfs/zpool.d/nonmed
lrwxr-xr-x  1 root  wheel  48 Apr 30 10:15 test_progress@ -> /usr/local/zfs/libexec/zfs/zpool.d/test_progress
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 r_proc@ -> /usr/local/zfs/libexec/zfs/zpool.d/r_proc
lrwxr-xr-x  1 root  wheel  45 Apr 30 10:15 iostat-10s@ -> /usr/local/zfs/libexec/zfs/zpool.d/iostat-10s
lrwxr-xr-x  1 root  wheel  40 Apr 30 10:15 model@ -> /usr/local/zfs/libexec/zfs/zpool.d/model
lrwxr-xr-x  1 root  wheel  44 Apr 30 10:15 iostat-1s@ -> /usr/local/zfs/libexec/zfs/zpool.d/iostat-1s
lrwxr-xr-x  1 root  wheel  45 Apr 30 10:15 smart_test@ -> /usr/local/zfs/libexec/zfs/zpool.d/smart_test
lrwxr-xr-x  1 root  wheel  42 Apr 30 10:15 pwr_cyc@ -> /usr/local/zfs/libexec/zfs/zpool.d/pwr_cyc
lrwxr-xr-x  1 root  wheel  43 Apr 30 10:15 pend_sec@ -> /usr/local/zfs/libexec/zfs/zpool.d/pend_sec
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 iostat@ -> /usr/local/zfs/libexec/zfs/zpool.d/iostat
lrwxr-xr-x  1 root  wheel  38 Apr 30 10:15 enc@ -> /usr/local/zfs/libexec/zfs/zpool.d/enc
lrwxr-xr-x  1 root  wheel  45 Apr 30 10:15 locate_led@ -> /usr/local/zfs/libexec/zfs/zpool.d/locate_led
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 r_ucor@ -> /usr/local/zfs/libexec/zfs/zpool.d/r_ucor
lrwxr-xr-x  1 root  wheel  43 Apr 30 10:15 off_ucor@ -> /usr/local/zfs/libexec/zfs/zpool.d/off_ucor
lrwxr-xr-x  1 root  wheel  45 Apr 30 10:15 test_ended@ -> /usr/local/zfs/libexec/zfs/zpool.d/test_ended
lrwxr-xr-x  1 root  wheel  43 Apr 30 10:15 hours_on@ -> /usr/local/zfs/libexec/zfs/zpool.d/hours_on
lrwxr-xr-x  1 root  wheel  40 Apr 30 10:15 label@ -> /usr/local/zfs/libexec/zfs/zpool.d/label
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 cmd_to@ -> /usr/local/zfs/libexec/zfs/zpool.d/cmd_to
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 smartx@ -> /usr/local/zfs/libexec/zfs/zpool.d/smartx
lrwxr-xr-x  1 root  wheel  40 Apr 30 10:15 smart@ -> /usr/local/zfs/libexec/zfs/zpool.d/smart
lrwxr-xr-x  1 root  wheel  42 Apr 30 10:15 ata_err@ -> /usr/local/zfs/libexec/zfs/zpool.d/ata_err
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 health@ -> /usr/local/zfs/libexec/zfs/zpool.d/health
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 defect@ -> /usr/local/zfs/libexec/zfs/zpool.d/defect
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 encdev@ -> /usr/local/zfs/libexec/zfs/zpool.d/encdev
lrwxr-xr-x  1 root  wheel  43 Apr 30 10:15 rep_ucor@ -> /usr/local/zfs/libexec/zfs/zpool.d/rep_ucor
lrwxr-xr-x  1 root  wheel  42 Apr 30 10:15 realloc@ -> /usr/local/zfs/libexec/zfs/zpool.d/realloc
lrwxr-xr-x  1 root  wheel  40 Apr 30 10:15 lsblk@ -> /usr/local/zfs/libexec/zfs/zpool.d/lsblk
lrwxr-xr-x  1 root  wheel  46 Apr 30 10:15 test_status@ -> /usr/local/zfs/libexec/zfs/zpool.d/test_status
lrwxr-xr-x  1 root  wheel  44 Apr 30 10:15 test_type@ -> /usr/local/zfs/libexec/zfs/zpool.d/test_type
lrwxr-xr-x  1 root  wheel  44 Apr 30 10:15 fault_led@ -> /usr/local/zfs/libexec/zfs/zpool.d/fault_led
lrwxr-xr-x  1 root  wheel  39 Apr 30 10:15 temp@ -> /usr/local/zfs/libexec/zfs/zpool.d/temp
lrwxr-xr-x  1 root  wheel  41 Apr 30 10:15 w_proc@ -> /usr/local/zfs/libexec/zfs/zpool.d/w_proc
dereed999
 
Posts: 21
Joined: Thu Sep 10, 2020 6:33 am

Re: macos-2.0.0-rc6 Memory Leak?

Postby dereed999 » Sat May 01, 2021 6:02 pm

I followed the directions, but had to modify the kextunload command to be: kextunload /Library/Extensions/zfs.kext as the "-n" option didn't work for some reason.

Code: Select all
2021-05-01 21:57:00.168 Df kernel[0:a7d] (zfs) <zfs`zvolRemoveDeviceTerminate> zvolRemoveDeviceTerminate
2021-05-01 21:57:00.168 Df kernel[0:a7e] (zfs) <zfs`zvolRemoveDeviceTerminate> zvolRemoveDeviceTerminate
2021-05-01 21:57:00.168 Df kernel[0:a7f] (zfs) <zfs`zvolRemoveDeviceTerminate> zvolRemoveDeviceTerminate
2021-05-01 21:57:00.168 Df kernel[0:173d] (zfs) <zfs`__dprintf> spa_history.c:295:spa_history_log_sync(): command: zpool export -a
2021-05-01 21:57:00.470 Df kernel[0:d769] (zfs) <zfs`__dprintf> spa_misc.c:413:spa_load_note(): spa_load(zfsPool, config trusted): UNLOADING
2021-05-01 21:57:00.904 Df kernel[0:d769] (zfs) <zfs`__dprintf> mmp.c:261:mmp_thread_stop(): MMP thread stopped pool 'zfsPool' gethrtime 3994613076333
2021-05-01 22:00:19.550 Df kernel[0:dbae] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> zfs_boot_fini no pool_list to clear
2021-05-01 22:00:19.550 Df kernel[0:dbae] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> ZFS: Attempting to unload ...
2021-05-01 22:00:27.186 Df kernel[0:dbae] (zfs) <zfs`spl_tsd_fini> SPL: tsd unloading 0
2021-05-01 22:00:27.196 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_24' (0xffffffc200011040) not empty
2021-05-01 22:00:27.225 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_128' (0xffffffc20001a040) not empty
2021-05-01 22:00:27.227 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_160' (0xffffffc20001b040) not empty
2021-05-01 22:00:27.227 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_192' (0xffffffc20001c040) not empty
2021-05-01 22:00:27.228 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_224' (0xffffffc20001d040) not empty
2021-05-01 22:00:27.229 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_256' (0xffffffc20001e040) not empty
2021-05-01 22:00:27.292 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_320' (0xffffffc20001f040) not empty
2021-05-01 22:00:27.304 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_384' (0xffffffc200020040) not empty
2021-05-01 22:00:27.335 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_40960' (0xffffffc200036040) not empty
2021-05-01 22:00:27.635 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_va_4096' (0xffffffc20000c040) not empty
2021-05-01 22:00:28.038 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_bufctl_cache' (0xffffffc20000a040) not empty
2021-05-01 22:00:28.439 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_slab_cache' (0xffffffc200009040) not empty
2021-05-01 22:00:28.468 Df kernel[0:dbae] (zfs) <zfs`spl_kmem_fini> SPL: Released 20975 slabs
2021-05-01 22:00:28.468 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_default'): leaked 15797891072 bytes
2021-05-01 22:00:28.594 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_va'): leaked 16319913984 bytes
2021-05-01 22:00:28.710 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_msb'): leaked 131522560 bytes
2021-05-01 22:00:28.719 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_metadata'): leaked 131522560 bytes
2021-05-01 22:00:28.852 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('heap'): leaked 16462258176 bytes
2021-05-01 22:00:29.030 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking spl_heap_arena, aka bucket_heap (pass 2)
2021-05-01 22:00:29.092 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking bucket arenas...
2021-05-01 22:00:29.195 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_32768'): leaked 984317952 bytes
2021-05-01 22:00:29.203 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_65536'): leaked 15359057920 bytes
2021-05-01 22:00:29.276 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking vmem metadata-related arenas...
2021-05-01 22:00:29.284 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_heap'): leaked 16462258176 bytes
2021-05-01 22:00:29.397 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_seg'): leaked 381272064 bytes
2021-05-01 22:00:29.403 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_hash'): leaked 393216 bytes
2021-05-01 22:00:29.403 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_metadata'): leaked 381665280 bytes
2021-05-01 22:00:29.408 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('spl_default_arena'): leaked 381714432 bytes
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy_internal> SPL: vmem_destroy('vmem_vmem'): leaked 7680 bytes
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_destroy_internal> SPL: vmem_destroy('vmem_vmem'): STILL 7680 bytes at kstat_delete() time
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> SPL: arenas removed, now try destroying mutexes...
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> vmem_xnu_alloc_lock
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> vmem_panic_lock
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> vmem_pushpage_lock
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> vmem_nosleep_lock
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> vmem_sleep_lock
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> vmem_segfree_lock
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> vmem_list_lock
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini>
SPL: vmem_fini: walking list of live slabs at time of call to vmem_fini
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> SPL: WOULD HAVE released 0 bytes (0 spans) from arenas
2021-05-01 22:00:29.410 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> SPL: vmem_fini: Brief delay for readability...
2021-05-01 22:00:30.402 Df kernel[0:dbae] (zfs) <zfs`vmem_fini> SPL: vmem_fini: done!
2021-05-01 22:00:30.402 Df kernel[0:dbae] (zfs) <zfs`spl_mutex_subsystem_fini> Dumping leaked mutex allocations...
2021-05-01 22:00:30.402 Df kernel[0:dbae] (zfs) <zfs`spl_mutex_subsystem_fini> Dumped 0 leaked allocations. Wait for watchdog to exit..
2021-05-01 22:00:30.542 Df kernel[0:a42] (zfs) <zfs`spl_wdlist_check> SPL: watchdog thread exit
2021-05-01 22:00:30.542 Df kernel[0:dbae] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> ZFS: Unloaded module v2.0.0-rc6
dereed999
 
Posts: 21
Joined: Thu Sep 10, 2020 6:33 am

Re: macos-2.0.0-rc6 Memory Leak?

Postby lundman » Sat May 01, 2021 6:33 pm

Ok that should have been "-b" and not "-n" (I updated my post to be correct now)

2021-05-01 22:00:27.196 Df kernel[0:dbae] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_24' (0xffffffc200011040) not empty


Yes, that is a leak. Let me try to see if get any here, but if I can not, I could build you a leak-tracer version and it will show exactly where it leaks. But it will only
do one range at a time.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: macos-2.0.0-rc6 Memory Leak?

Postby Jimbo » Sat May 01, 2021 8:25 pm

I think I'm seeing this too. I've had a number of watchdog panics after extended periods of activity on a 6-core i7 32Gb Mac Mini with two large-ish pools.

Just unloading the kext today shows (after just a bit more than 24hr up time):

Code: Select all
2021-05-02 12:15:00.290 Df kernel[0:89116] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> zfs_boot_fini no pool_list to clear
2021-05-02 12:15:00.290 Df kernel[0:89116] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> ZFS: Attempting to unload ...
2021-05-02 12:15:01.874 Df kernel[0:89116] (zfs) <zfs`spl_tsd_fini> SPL: tsd unloading 0
2021-05-02 12:15:01.875 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_24' (0xffffffc280011040) not empty
2021-05-02 12:15:01.897 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_128' (0xffffffc28001a040) not empty
2021-05-02 12:15:01.899 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_160' (0xffffffc28001b040) not empty
2021-05-02 12:15:01.900 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_192' (0xffffffc28001c040) not empty
2021-05-02 12:15:01.901 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_224' (0xffffffc28001d040) not empty
2021-05-02 12:15:01.902 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_256' (0xffffffc28001e040) not empty
2021-05-02 12:15:01.931 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_320' (0xffffffc28001f040) not empty
2021-05-02 12:15:01.945 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_384' (0xffffffc280020040) not empty
2021-05-02 12:15:02.005 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_alloc_40960' (0xffffffc280036040) not empty
2021-05-02 12:15:02.094 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_va_4096' (0xffffffc28000c040) not empty
2021-05-02 12:15:02.564 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_bufctl_cache' (0xffffffc28000a040) not empty
2021-05-02 12:15:03.053 Df kernel[0:89116] (zfs) <zfs`vcmn_err> SPL: Warning: kmem_cache_destroy: 'kmem_slab_cache' (0xffffffc280009040) not empty
2021-05-02 12:15:03.059 Df kernel[0:89116] (zfs) <zfs`spl_kmem_fini> SPL: Released 2982 slabs
2021-05-02 12:15:03.059 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_default'): leaked 10336202752 bytes
2021-05-02 12:15:03.166 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_va'): leaked 10541121536 bytes
2021-05-02 12:15:03.244 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_msb'): leaked 177160192 bytes
2021-05-02 12:15:03.261 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('kmem_metadata'): leaked 177160192 bytes
2021-05-02 12:15:03.372 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('heap'): leaked 10738581504 bytes
2021-05-02 12:15:03.494 Df kernel[0:89116] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking spl_heap_arena, aka bucket_heap (pass 2)
2021-05-02 12:15:03.537 Df kernel[0:89116] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking bucket arenas...
2021-05-02 12:15:03.602 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_32768'): leaked 754974720 bytes
2021-05-02 12:15:03.610 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_65536'): leaked 9143705600 bytes
2021-05-02 12:15:03.654 Df kernel[0:89116] (zfs) <zfs`vmem_fini> SPL: vmem_fini: walking vmem metadata-related arenas...
2021-05-02 12:15:03.663 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('bucket_heap'): leaked 10738581504 bytes
2021-05-02 12:15:03.745 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_seg'): leaked 410255360 bytes
2021-05-02 12:15:03.751 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_hash'): leaked 196608 bytes
2021-05-02 12:15:03.752 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('vmem_metadata'): leaked 410451968 bytes
2021-05-02 12:15:03.757 Df kernel[0:89116] (zfs) <zfs`vmem_destroy> SPL: vmem_destroy('spl_default_arena'): leaked 410517504 bytes
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_destroy_internal> SPL: vmem_destroy('vmem_vmem'): leaked 7680 bytes
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_destroy_internal> SPL: vmem_destroy('vmem_vmem'): STILL 7680 bytes at kstat_delete() time
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> SPL: arenas removed, now try destroying mutexes...
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> vmem_xnu_alloc_lock
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> vmem_panic_lock
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> vmem_pushpage_lock
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> vmem_nosleep_lock
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> vmem_sleep_lock
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> vmem_segfree_lock
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> vmem_list_lock
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini>
SPL: vmem_fini: walking list of live slabs at time of call to vmem_fini
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> SPL: WOULD HAVE released 0 bytes (0 spans) from arenas
2021-05-02 12:15:03.759 Df kernel[0:89116] (zfs) <zfs`vmem_fini> SPL: vmem_fini: Brief delay for readability...
2021-05-02 12:15:04.758 Df kernel[0:89116] (zfs) <zfs`vmem_fini> SPL: vmem_fini: done!
2021-05-02 12:15:04.758 Df kernel[0:89116] (zfs) <zfs`spl_mutex_subsystem_fini> Dumping leaked mutex allocations...
2021-05-02 12:15:04.758 Df kernel[0:89116] (zfs) <zfs`spl_mutex_subsystem_fini> Dumped 0 leaked allocations. Wait for watchdog to exit..
2021-05-02 12:15:14.057 Df kernel[0:9ad] (zfs) <zfs`spl_wdlist_check> SPL: watchdog thread exit
2021-05-02 12:15:14.057 Df kernel[0:89116] (zfs) <zfs`org_openzfsonosx_zfs_zvol::stop(IOService*)> ZFS: Unloaded module v2.0.0-rc6


Happy to help debug or test further.
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: macos-2.0.0-rc6 Memory Leak?

Postby lundman » Sun May 02, 2021 12:54 am

Did you say your macOS version? BigSur? I'll build a memleak test size == 24 first, run for a bit, then do the unload and it should say where it is leaked from.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Next

Return to OpenZFS on OS X Development

Who is online

Users browsing this forum: No registered users and 15 guests