Development - Revision history

Lundman: /* Merging PRs */

2021-04-13T08:21:51Z

‎Merging PRs

Lundman at 11:28, 15 May 2017

2017-05-15T11:28:25Z

Lundman at 05:24, 2 April 2017

2017-04-02T05:24:06Z

Lundman: /* Debugging with LLDB */

2016-12-15T11:46:51Z

‎Debugging with LLDB

192.48.8.20 at 21:00, 27 July 2016

2016-07-27T21:00:25Z

192.48.8.21 at 20:59, 27 July 2016

2016-07-27T20:59:26Z

192.48.8.20: added comments on reasons behind large KMEM_QUANTUM

2016-07-27T20:56:33Z

added comments on reasons behind large KMEM_QUANTUM

Ilovezfs: /* Debugging with GDB */

2015-10-26T23:13:42Z

‎Debugging with GDB

Lundman: /* VFS nolocks */

2015-10-26T08:57:46Z

‎VFS nolocks

Lundman: /* Reclaim */

2015-10-26T08:51:45Z

‎Reclaim

← Older revision		Revision as of 08:21, 13 April 2021
Line 612:		Line 612:
	And view the commits that you want		And view the commits that you want

−	[[w]]	+	[[zfs.exports]]

← Older revision		Revision as of 11:28, 15 May 2017
Line 611:		Line 611:

	And view the commits that you want		And view the commits that you want
		+
		+	[[w]]

← Older revision		Revision as of 05:24, 2 April 2017
Line 560:		Line 560:

	Care is also needed in vnop_remove, to clean out the hardlink AVL node in both trees, as well as in vnop_rename, to update the new source mapping (parent id, file id, name). The AVL trees are unloaded at unmount.		Care is also needed in vnop_remove, to clean out the hardlink AVL node in both trees, as well as in vnop_rename, to update the new source mapping (parent id, file id, name). The AVL trees are unloaded at unmount.
		+
		+
		+	== Merging with OpenZFS ==
		+
		+	Add upstream OpenZFS repo as a remote source;
		+
		+	[remote "upstream"]
		+	url = git@github.com:openzfs/openzfs.git
		+	fetch = +refs/heads/:refs/remotes/upstream/
		+
		+	Set the rename limit really high to make it find our locations
		+
		+	# git config merge.renameLimit 999999
		+
		+	[merge]
		+	renameLimit = 999999
		+
		+
		+	Make sure it is up to date
		+
		+	# git fetch upstream
		+
		+	Check whats new:
		+
		+	# git log --stat upstream/master
		+
		+	For each new commit, bring it in, for example f4a6fedc42535abef5f0584fa0c6cb2af46b9ddf
		+
		+	# git cherry-pick f4a6fedc42535abef5f0584fa0c6cb2af46b9ddf
		+
		+	fix any clashes, if any, make sure it compiles with no new errors or warnings.
		+
		+	Always update the commit message
		+
		+	# git commit --amend
		+
		+	and delete any lines that are like
		+
		+	Closes #324
		+
		+	since they do not match our issues.
		+
		+
		+	=== Merging PRs ===
		+
		+	Check out the PR branch, for example PR 124
		+
		+	# git fetch upstream pull/124/head:pr124
		+	# git checkout pr124
		+
		+	And view the commits that you want

@@ Line 59: / Line 59: @@
 (lldb) addkext -F /tmp/zfs.kext/Contents/MacOS/zfs 0xffffff7f8ebbf000
 </syntaxhighlight>
 Then follow the guide for GDB above.

@@ Line 214: / Line 214: @@
 As of 1.5.2 we switched the KMEM_QUANTUM to 128k based on feedback from a user. It was believed at the time that some tuning in the allocator had enabled this improvement. Surprisingly this has lead to reduced performance and some stuttering/beachballing on various machines. There is no apparent predictability to which class of machine will suffer from this, i.e. newer fast machines are apparently susceptible to this over the reference machine (a mac mini) around which the 128k opinion was formed. It also seems that allowing wired memory to become very large can (does?) result in performance problems.
-There has been further investigation into exactly why we need to gain large blocks of memory from the page allocator, when the kernels own level 2 allocator does not. It turns out that vmem does not return memory to the page allocator in general on Illumos as it is the system wide allocator. In our case we do have to release memory back to the OS under pressure situations. To achieve this we need to configure vmem to act more like libumem does in user space, that is to know that it has an upstream allocator that must be cooperated with. Furthermore it turns out that the "quantum caches" in the heap vmem arena were not active, due to the vmem arena chaining not working at all. The size of the KMEM_QUANTUM is a proxy for frequency of memory allocations/frees via the kernel page allocator. High frequency is not good - the page allocator is slow and heavily impacts operation of the machine (TLB shootdowns etc).
+There has been further investigation into exactly why we need to gain large blocks of memory from the page allocator, when the kernels own level 2 allocator does not. It turns out that vmem does not return memory to the page allocator in general on Illumos as it is the system wide allocator. In our case we do have to release memory back to the OS under pressure situations. To achieve this we need to configure vmem to act more like libumem does in user space, that is to know that it has an upstream allocator that must be cooperated with. Furthermore it turns out that the "quantum caches" in the heap vmem arena were not active, due to the vmem arena chaining not working at all (this is a bug). While this bug remains, the size of KMEM_QUANTUM is a proxy for frequency of memory allocations/frees via the kernel page allocator. High frequency is not good - the page allocator is slow and heavily impacts operation of the machine (TLB shootdowns etc).
 References:

@@ Line 210: / Line 210: @@
 Our strategy was to determine how much of the Illumos allocator could be implemented on OS X. After a series of experiments where we implemented significant portions of the kmem code from illumos on top of bmalloc, we had learned enough to take the final step of essentially copying the entire kmem/vmem allocator stack from Illumos. Some portions of the kmem code have been disabled in kmem such as logging, and hot swap CPU support have been disabled due to architectural differences between OS X and Illumos.
-By default kmem/vmem require a certain level of performance from the OS page allocator. It is easy to overwhelm the OS X page allocator. We tuned vmem to use 512Kb chunks of memory from the page allocator rather than the smaller allocations that vmem prefers. This is less than ideal as it reduces the ability for vmem to smoothly release memory to the page allocator when the machine is under pressure. While we have an adequately performing solution now, there will always be a tension between our allocator and OS X itself. OS X only provides minimal mechanisms to observe and respond to memory pressure in the machine, so we are somewhat limited in what can be achieved in this regard.
+By default kmem/vmem require a certain level of performance from the OS page allocator. It is easy to overwhelm the OS X page allocator. We tuned vmem to use a KMEM_QUANTUM of 512Kb chunks of memory from the page allocator rather than the smaller allocations that vmem prefers. This is less than ideal as it reduces the ability for vmem to smoothly release memory to the page allocator when the machine is under pressure. While we have an adequately performing solution now, there will always be a tension between our allocator and OS X itself. OS X only provides minimal mechanisms to observe and respond to memory pressure in the machine, so we are somewhat limited in what can be achieved in this regard.
 References:

@@ Line 13: / Line 13: @@
 <syntaxhighlight lang="bash">
-$ sudo nvram boot-args="-v keepsyms=y debug=0x144"
+$ sudo nvram boot-args="-v keepsyms=1 debug=0x144"
 </syntaxhighlight>

@@ Line 496: / Line 496: @@
 attached. We should look into a better delay, perhaps a condvar with wakeup.
-There is further work in zil_lwb_commit() to ensure we call zfs_get_data without locks, and vnode attached. This this area also differs somewhat from IllumOS.
+There is further work in zil_lwb_commit() to ensure we call zfs_get_data without locks, and vnode attached. This this area also differs somewhat from IllumOS. The zget() call has been extended to allow flags to specify if
 === Fastpath vs Recycle ===

@@ Line 489: / Line 489: @@
 This section is an attempt to outline the differences from ZFS versions of other platforms, as compared to OS X. To assist developers new to the Apple platform, who wishes to assist, or understand, development of the O3X version.
-=== Reclaim ===
+=== VFS nolocks ===
-One of the biggest hassles with OS X is the VFS layer's handling of reclaim. First it is worth noting that "struct vnode" is an opaque type, so we are not allowed to see, nor modify, the contents of a vnode.
+To avoid deadlocking complications, we do not call VFS with any locks, at any time. This means we differ a little bit from original IllumOS code. In particular, the calls that create a znode (zfs_mknode and zfs_znode_alloc)
-(Of course, we could craft a mirror struct of vnode and tailor it to each OS X version where vnode changes. But that is rather hacky.)
+we do not attach the vnode here, as we are inside that of a dmu_tx. For example, the VNOPs zfs_create, zfs_mkdir, zfs_symlink and zfs_make_xattrdir, have instead been patched to call zfs_znode_getvnode() to attach the vnode '''after'''
-Following that, the '''only''' place where you can set the '''vtype''' (VREG, VDIR), '''vdata''' (user pointer to hold the ZFS znode), '''vfsops''' (list of filesystem calls "vnops") etc, is '''only in calling vnode_create()'''.
+There is further work in zil_lwb_commit() to ensure we call zfs_get_data without locks, and vnode attached. This this area also differs somewhat from IllumOS.
-So there is no way to "allocate an empty vnode, and set its values later". The FreeBSD method of pre-allocating vnodes, to avoid reclaim, can not be done.
-ZFS will start a new dmu_tx, then call zfs_mknode which will eventually call vnode_create, so we can not do anything with dmu_tx in those vnops.
-−
-The problem is, if vnode_create decides to reclaim, it will do so directly, as the same thread. It will end up in vclean() which can call vnop_fsync, vnop_pageout, vnop_inactive and vnop_reclaim. The first three of these calls, we can
-use the API call vnode_isrecycled() to detect if these vnops are called "the normal way", or from vclean. If we come from vclean, and the vnode is doomed, we will do as little as possible. We can not open a new TX, and
-we can not use mutex locks (panic: locking against ourselves).
-−
-Nor is there any way to defer, or delay, a doomed vnode. If vnop_reclaim returns anything but 0, you find the lovely XNU code of
-         if (VNOP_RECLAIM(vp, ctx))
-                 panic("vclean: cannot reclaim");
-in vfs_subr.c
-−
-−
-So, at the moment there is some extra logic in '''zfs_vnop_reclaim''' to handle that we might be re-entrant as the '''vnode_create''' thread.
-−
-    exception = ((zp->z_sa_hdl != NULL) &&
-        zp->z_unlinked) ? B_TRUE : B_FALSE;
-    fastpath = zp->z_fastpath;
-−
-if both exception and fastpath are FALSE, we can call direct reclaim right there. As in those cases, no final dmu_tx is caused. Following
-the zfs_rmnode->zfs_purgedir->zget and similar paths, exception is set to TRUE.
-−
-If exception is TRUE, we add the zp to the reclaim_list, and the separate reclaim_thread will call zfs_rmnode(zp). As a separate thread it can handle calling
-dmu_tx.
-−
-If fastpath is TRUE, we do no more/nothing in zfs_vnop_reclaim. See below.
 === Fastpath vs Recycle ===