OpenZFS on OS X - User contributions [en]

FAQ

2016-08-17T09:53:37Z

Brendon: /* Q) I compiled from source, how do I know that I am running what I compiled? */

[[Category:About O3X]]

Besides the questions covered below, you may find [[Documentation]] and [[OpenZFS on OS X]] helpful. Both articles contain a good deal of information about OpenZFS on OS X.

== General ==

===Q) What is OpenZFS on OS X?===
'''A)''' See the article entitled [[OpenZFS on OS X]].

===Q) What does O3X mean?===
'''A)''' O3X = O O O X = '''O'''penZFS '''o'''n '''O'''S '''X'''.

===Q) What version of ZFS do you use?===
'''A)''' OpenZFS. Pool version 5000. File system version 5. Pool version 5000 is pool version 28 plus support for feature flags. We support pool version 5000 and pool versions less than or equal to 28. We do not support the closed-source Oracle Solaris ZFS pool versions 29 and up.

===Q) Can I set up a test pool using files instead of disks?===
'''A)''' Yes. The example uses simple files, its also possible to use disk images.

$ cd /tmp
$ mkfile 10G aaa
$ mkfile 10G bbb
$ mkfile 10G ccc
$ sudo zpool create tank raidz /tmp/aaa /tmp/bbb /tmp/ccc
$

===Q) I compiled from source, how do I know that I am running what I compiled?===
'''A)''' There are a couple of sysctls that can be read to determine what commit was compiled into your custom kexts. Assuming you used zfsadm to build, or that you executed the autoconf, and configure steps before compiling the following works.

$ sysctl -a | grep kext
spl.kext_version: 1.5.2-2_g'''115aa2f'''
zfs.kext_version: 1.5.2-33_g'''9ac66a7'''

$ cd <path to source code>/zfs
$ git log -n 1
commit '''9ac66a7'''1e53636eec04f4718b0b3870a18f07840
Merge: 3326995 890ef86
Author: zadmin <zadmin@jerry.local>
Date: Thu Jun 16 17:19:24 2016 +1000

Merge branch 'master' of https://github.com/openzfsonosx/zfs

$ cd <path to source code>/spl
$ git log -n 2
commit f1ff660a2f1fa340d451c2afa5f726f9bd3e609d
Author: Brendon Humphrey <brendon.humphrey@mac.com>
Date: Sat Jun 18 20:25:09 2016 -0700
...
commit '''115aa2f'''05b6f843e0d39d4f6bf999602db120113
Author: Jorgen Lundman <lundman@lundman.net>
Date: Thu May 12 09:48:31 2016 +0900
...

I have highlighted the relevant text to match. You can see that this machine is running the latest ZFS, and is one commit behind the latest SPL.

If you make a small local change to the code then this technique will not work. One workaround for that is to edit your file(s) and commit them in your repository clone, that way it will have a commit id.

== Best practices ==

===Q) Do I have to use mirrors or raidz? ===
'''A)''' Have to? No. Should you? Virtually always. ZFS will not be able to repair errors it finds unless you have redundancy at the vdev level.

===Q) Can I set copies=2 in lieu of using mirrors or raidz? ===
Setting copies=2 is a poor substitute for vdev-level redundancy. Two copies on a broken drive are worthless. That being said, yes, you can set copies=2. Do so at your own risk.

== Administration ==

===Q) How can I access the .zfs snapshot directories?===
'''A)''' You need to set snapdir visible and manually mount a snapshot.
<syntaxhighlight lang="bash">
$ sudo zfs set snapdir=visible tank/bob
$ sudo zfs mount tank/bob@yesterday
$ ls -l /tank/bob/.zfs/snapshot/yesterday/
</syntaxhighlight>
You can see existing snapshots via:
<syntaxhighlight lang="bash">
$ zfs list -t snapshot
</syntaxhighlight>

===Q) Is .zfs snapdir auto-mounting supported?===
'''A)''' No, not at this time. You must manually "zfs mount" snapshots manually to see them in the snapdir.

===Q) OK, I manually mounted my snapshot but still cannot see it in Finder. What gives?===
'''A)''' Currently mounted snapshots are only visible from Terminal, not from Finder.
<syntaxhighlight lang="bash">
$ ls -l /tank/bob/.zfs/snapshot/yesterday/
</syntaxhighlight>

===Q) Why does OSX server not allow the server storage to be on ZFS?===
'''A)''' OSX Server has been coded in such a way as to only allow the server storage area to be an HFS formatted drive. O3X offers a feature that causes ZFS datasets to identify themselves as HFS. This is sufficient for OSX server to allow storage on a ZFS filesystem. HFS mimic is enabled by setting the com.apple.mimic_hfs property on a per dataset basis.

In addition as of OSX Server 5.x it seems that the Application Store Caching server can only store its cache on HFS. This is a new behaviour. I just used an HFS formatted ZVOL to get around this.

sudo zfs create -o volblocksize=1m -s -V 250g tank/cachingzvol

== Interoperability ==

===Q) How do I create an O3X compatible pool on another OpenZFS platform? ===
'''A)''' Only enable feature flags supported by O3X, as discussed [https://openzfsonosx.org/wiki/Zpool#Feature_flags here].

===Q) Can I import my ZEVO pools?===
'''A)''' Yes. O3X can import pool version 28, which means it can import ZEVO pools.

===Q) Can I import my MacZFS pools?===
'''A)''' Yes. O3X can import pool version 8, which means it can import MacZFS pools.

===Q) Do HFS only applications such as Photos, OSX server and others work on ZFS?===
'''A)''' Sometimes. Apple codes some software to only work when stored on HFS. We can't change that. We have provided a property that causes ZFS filesystems to identify themselves as a HFS when enabled. We are unable to guarantee that the application will work 100% correctly as HFS may have specific behaviours that the application depends on, and ZFS may not behave identically.

sudo zfs set com.apple.mimic_hfs=on <dataset>

===Q) Does Spotlight work? ===
'''A)''' Yes. Spotlight works on O3X 1.3.1+.

===Q) Can Time Machine backups be stored on ZFS? ===
'''A)''' Yes. It is possible to host a TimeMachine backup within a SparseImage on ZFS, or an HFS formatted ZVOL On ZFS.

== Limitations ==

===Q) Can I use finder permissions aka ACLs? ===
'''A)''' Not yet. There is work to go in this area. See https://github.com/openzfsonosx/zfs/issues/275

===Q) Can I boot my computer off of O3X?===
'''A)''' No. O3X cannot be used as your main system partition.

===Q) So if I use O3X, that means I don't need to back up, right? ===
'''A)''' Wrong. Wrong. Wrong.

===Q) Can TimeMachine backup the contents of a ZFS volume? ===
'''A)''' No. We believe that when "Issue 116" is resolved it may be supportable. At the present time TimeMachine excludes ZFS filesystems from the list of available backup targets.

FAQ

2016-08-17T09:46:57Z

Brendon:

[[Category:About O3X]]

Besides the questions covered below, you may find [[Documentation]] and [[OpenZFS on OS X]] helpful. Both articles contain a good deal of information about OpenZFS on OS X.

== General ==

===Q) What is OpenZFS on OS X?===
'''A)''' See the article entitled [[OpenZFS on OS X]].

===Q) What does O3X mean?===
'''A)''' O3X = O O O X = '''O'''penZFS '''o'''n '''O'''S '''X'''.

===Q) What version of ZFS do you use?===
'''A)''' OpenZFS. Pool version 5000. File system version 5. Pool version 5000 is pool version 28 plus support for feature flags. We support pool version 5000 and pool versions less than or equal to 28. We do not support the closed-source Oracle Solaris ZFS pool versions 29 and up.

===Q) Can I set up a test pool using files instead of disks?===
'''A)''' Yes. The example uses simple files, its also possible to use disk images.

$ cd /tmp
$ mkfile 10G aaa
$ mkfile 10G bbb
$ mkfile 10G ccc
$ sudo zpool create tank raidz /tmp/aaa /tmp/bbb /tmp/ccc
$

===Q) I compiled from source, how do I know that I am running what I compiled?===
'''A)''' There are a couple of sysctls that can be read to determine what commit was compiled into your custom kexts. Assuming you used zfsadm to build, or that you executed the autoconf, and configure steps before compiling the following works.

$ sysctl -a | grep kext
spl.kext_version: 1.5.2-2_g'''115aa2f'''
zfs.kext_version: 1.5.2-33_g'''9ac66a7'''

$ cd <path to source code>/zfs
$ git log -n 1
commit '''9ac66a7'''1e53636eec04f4718b0b3870a18f07840
Merge: 3326995 890ef86
Author: zadmin <zadmin@jerry.local>
Date: Thu Jun 16 17:19:24 2016 +1000

Merge branch 'master' of https://github.com/openzfsonosx/zfs

$ cd <path to source code>/spl
$ git log -n 2
commit f1ff660a2f1fa340d451c2afa5f726f9bd3e609d
Author: Brendon Humphrey <brendon.humphrey@mac.com>
Date: Sat Jun 18 20:25:09 2016 -0700
...
commit '''115aa2f'''05b6f843e0d39d4f6bf999602db120113
Author: Jorgen Lundman <lundman@lundman.net>
Date: Thu May 12 09:48:31 2016 +0900
...

I have highlighted the relevant text to match. You can see that this machine is running the latest ZFS, and is one commit behind the latest SPL.

== Best practices ==

===Q) Do I have to use mirrors or raidz? ===
'''A)''' Have to? No. Should you? Virtually always. ZFS will not be able to repair errors it finds unless you have redundancy at the vdev level.

===Q) Can I set copies=2 in lieu of using mirrors or raidz? ===
Setting copies=2 is a poor substitute for vdev-level redundancy. Two copies on a broken drive are worthless. That being said, yes, you can set copies=2. Do so at your own risk.

== Administration ==

===Q) How can I access the .zfs snapshot directories?===
'''A)''' You need to set snapdir visible and manually mount a snapshot.
<syntaxhighlight lang="bash">
$ sudo zfs set snapdir=visible tank/bob
$ sudo zfs mount tank/bob@yesterday
$ ls -l /tank/bob/.zfs/snapshot/yesterday/
</syntaxhighlight>
You can see existing snapshots via:
<syntaxhighlight lang="bash">
$ zfs list -t snapshot
</syntaxhighlight>

===Q) Is .zfs snapdir auto-mounting supported?===
'''A)''' No, not at this time. You must manually "zfs mount" snapshots manually to see them in the snapdir.

===Q) OK, I manually mounted my snapshot but still cannot see it in Finder. What gives?===
'''A)''' Currently mounted snapshots are only visible from Terminal, not from Finder.
<syntaxhighlight lang="bash">
$ ls -l /tank/bob/.zfs/snapshot/yesterday/
</syntaxhighlight>

===Q) Why does OSX server not allow the server storage to be on ZFS?===
'''A)''' OSX Server has been coded in such a way as to only allow the server storage area to be an HFS formatted drive. O3X offers a feature that causes ZFS datasets to identify themselves as HFS. This is sufficient for OSX server to allow storage on a ZFS filesystem. HFS mimic is enabled by setting the com.apple.mimic_hfs property on a per dataset basis.

In addition as of OSX Server 5.x it seems that the Application Store Caching server can only store its cache on HFS. This is a new behaviour. I just used an HFS formatted ZVOL to get around this.

sudo zfs create -o volblocksize=1m -s -V 250g tank/cachingzvol

== Interoperability ==

===Q) How do I create an O3X compatible pool on another OpenZFS platform? ===
'''A)''' Only enable feature flags supported by O3X, as discussed [https://openzfsonosx.org/wiki/Zpool#Feature_flags here].

===Q) Can I import my ZEVO pools?===
'''A)''' Yes. O3X can import pool version 28, which means it can import ZEVO pools.

===Q) Can I import my MacZFS pools?===
'''A)''' Yes. O3X can import pool version 8, which means it can import MacZFS pools.

===Q) Do HFS only applications such as Photos, OSX server and others work on ZFS?===
'''A)''' Sometimes. Apple codes some software to only work when stored on HFS. We can't change that. We have provided a property that causes ZFS filesystems to identify themselves as a HFS when enabled. We are unable to guarantee that the application will work 100% correctly as HFS may have specific behaviours that the application depends on, and ZFS may not behave identically.

sudo zfs set com.apple.mimic_hfs=on <dataset>

===Q) Does Spotlight work? ===
'''A)''' Yes. Spotlight works on O3X 1.3.1+.

===Q) Can Time Machine backups be stored on ZFS? ===
'''A)''' Yes. It is possible to host a TimeMachine backup within a SparseImage on ZFS, or an HFS formatted ZVOL On ZFS.

== Limitations ==

===Q) Can I use finder permissions aka ACLs? ===
'''A)''' Not yet. There is work to go in this area. See https://github.com/openzfsonosx/zfs/issues/275

===Q) Can I boot my computer off of O3X?===
'''A)''' No. O3X cannot be used as your main system partition.

===Q) So if I use O3X, that means I don't need to back up, right? ===
'''A)''' Wrong. Wrong. Wrong.

===Q) Can TimeMachine backup the contents of a ZFS volume? ===
'''A)''' No. We believe that when "Issue 116" is resolved it may be supportable. At the present time TimeMachine excludes ZFS filesystems from the list of available backup targets.

Getting involved

2015-09-18T21:49:08Z

Brendon:

[[Category:About O3X]]
Getting involved and contributing benefits not only the community member and their fellow O3X users, but all users of [[Wikipedia:Free and open source software|free and open source software]].

This article describes how both new and experienced O3X users can contribute to the community. Note that this is not an exhaustive list.

== Official OpenZFS on OS X projects ==

=== Post on the forums ===

One of the easiest ways to get involved is participating in the [https://openzfsonosx.org/forum/ O3X Forums], which allow O3X users to get to know the community and help each other.

=== Improve this wiki ===

[[AboutWiki|O3XWiki]] is collaboratively maintained O3X documentation. All users are encouraged to [[O3XWiki:Contributing|contribute]].

=== Join the chatroom ===

You can help other users to solve problems in the [[IRC channel|IRC Channel]].

=== Fix and report bugs ===

Reporting and fixing bugs on the [https://github.com/zfs-osx/zfs/issues bug tracker] is one of the possible ways to help the community.

When reporting errors, please ensure that you include sufficient contextual information to assist with reproduction, diagnosis and resolution of the issue. At a minimum consider describing what you were doing, how repeatable the issue is, all text from the stack dump, what OSX version, and what O3X version you are running.

If you are experiencing panics, please turn keepsyms on (https://openzfsonosx.org/wiki/Install). This will make the stack dumps human readable, thereby improving our ability to diagnose.

If you are deadlocking O3X, please run the spindump command after O3X deadlocks and before the machine becomes unusable and post the resulting output. There is often a narrow window of time to do this. Spindump needs to run with elevated permissions using sudo. Normally only administrators can do this.

#sudo spindump
Password:
Sampling all processes for 10 seconds with 10 milliseconds of run time between samples
Sampling completed, processing symbols...
Spindump analysis written to file /tmp/spindump.txt

Copy the output file to an hfs filesystem and reboot.

For real time assistance please see us on IRC, there is usually someone online that can help (https://openzfsonosx.org/wiki/IRC_channel)

=== Code ===

Developers of all levels of experience and skill are encouraged to come talk about how they can contribute in the [[IRC channel|IRC Channel]] and on the [https://openzfsonosx.org/forum/viewforum.php?f=24 OpenZFS on OS X Development] forum.

FAQ

2015-09-18T21:30:24Z

Brendon: add all comments

Development

2014-12-05T23:13:09Z

Brendon: /* Kernel */

[[Category:O3X development]]
You should also familiarize yourself with the [[Project_roadmap|project roadmap]] so that you can put the technical details here in context.

== Kernel ==

=== Debugging with GDB ===

Dealing with [[Panic|panics]].

Apple's documentation: https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KEXTConcept/KEXTConceptDebugger/debug_tutorial.html

Boot target VM with

<syntaxhighlight lang="bash">
$ sudo nvram boot-args="-v keepsyms=y debug=0x144"
</syntaxhighlight>

Make it panic.

On your development machine, you will need the Kernel Debug Kit. Download it from Apple [https://developer.apple.com/downloads/index.action?q=Kernel%20Debug%20Kit here].

<syntaxhighlight lang="text">
$ gdb /Volumes/Kernelit/mach_kernel
(gdb) source /Volumes/KernelDebugKit/kgmacros
(gdb) target remote-kdp
(gdb) kdp-reattach 192.168.30.133 # obviously use the IP of your target / crashed VM
(gdb) showallkmods
</syntaxhighlight>

Find the addresses for ZFS and SPL modules.

<code>^Z</code> to suspend gdb, or, use another terminal

<syntaxhighlight lang="bash">
^Z
$ sudo kextutil -s /tmp -n \
-k /Volumes/KernelDebugKit/mach_kernel \
-e -r /Volumes/KernelDebugKit module/zfs/zfs.kext/ \
../spl/module/spl/spl.kext/
</syntaxhighlight>

Then resume gdb, or go back to gdb terminal.
<syntaxhighlight lang="text">
$ fg
(gdb) set kext-symbol-file-path /tmp
(gdb) add-kext /tmp/spl.kext
(gdb) add-kext /tmp/zfs.kext
(gdb) bt
</syntaxhighlight>

=== Debugging with LLDB ===

<syntaxhighlight lang="bash">
$ echo "settings set target.load-script-from-symbol-file true" >> ~/.lldbinit
$ lldb /Volumes/KernelDebugKit/mach_kernel
(lldb) kdp-remote 192.168.30.146
(lldb) showallkmods
(lldb) addkext -F /tmp/spl.kext/Contents/MacOS/spl 0xffffff7f8ebb0000 (Address from showallkmods)
(lldb) addkext -F /tmp/zfs.kext/Contents/MacOS/zfs 0xffffff7f8ebbf000
</syntaxhighlight>

Then follow the guide for GDB above.

=== Non-panic ===

If you prefer to work in GDB, you can always panic a kernel with

<syntaxhighlight lang="bash">
$ sudo dtrace -w -n "BEGIN{ panic();}"
</syntaxhighlight>

But this was revealing:

<syntaxhighlight lang="bash">
$ sudo /usr/libexec/stackshot -i -f /tmp/stackshot.log
$ sudo symstacks.rb -f /tmp/stackshot.log -s -w /tmp/trace.txt
$ less /tmp/trace.txt
</syntaxhighlight>

Note that my hang is here:

<syntaxhighlight lang="text">
PID: 156
Process: zpool
Thread ID: 0x4e2
Thread state: 0x9 == TH_WAIT |TH_UNINT
Thread wait_event: 0xffffff8006608a6c
Kernel stack:
machine_switch_context (in mach_kernel) + 366 (0xffffff80002b3d3e)
0xffffff800022e711 (in mach_kernel) + 1281 (0xffffff800022e711)
thread_block_reason (in mach_kernel) + 300 (0xffffff800022d9dc)
lck_mtx_sleep (in mach_kernel) + 78 (0xffffff80002265ce)
0xffffff8000569ef6 (in mach_kernel) + 246 (0xffffff8000569ef6)
msleep (in mach_kernel) + 116 (0xffffff800056a2e4)
0xffffff7f80e52a76 (0xffffff7f80e52a76)
0xffffff7f80e53fae (0xffffff7f80e53fae)
0xffffff7f80e54173 (0xffffff7f80e54173)
0xffffff7f80f1a870 (0xffffff7f80f1a870)
0xffffff7f80f2bb4e (0xffffff7f80f2bb4e)
0xffffff7f80f1a9b7 (0xffffff7f80f1a9b7)
0xffffff7f80f1b65f (0xffffff7f80f1b65f)
0xffffff7f80f042ee (0xffffff7f80f042ee)
0xffffff7f80f45c5b (0xffffff7f80f45c5b)
0xffffff7f80f4ce92 (0xffffff7f80f4ce92)
spec_ioctl (in mach_kernel) + 157 (0xffffff8000320bfd)
VNOP_IOCTL (in mach_kernel) + 244 (0xffffff8000311e84)
</syntaxhighlight>

It is a shame that it only shows the kernel symbols, and not inside SPL and ZFS, but we can ask it to load another sym file. (Alas, it cannot handle multiple symbols files. Fix this Apple.)

<syntaxhighlight lang="bash">
$ sudo kextstat #grab the addresses of SPL and ZFS again
$ sudo kextutil -s /tmp -n -k /Volumes/KernelDebugKit/mach_kernel \
-e -r /Volumes/KernelDebugKit module/zfs/zfs.kext/ ../spl/module/spl/spl.kext/

$ sudo symstacks.rb -f /tmp/stackshot.log -s -k /tmp/net.lundman.spl.sym
0xffffff800056a2e4 (0xffffff800056a2e4)
spl_cv_wait (in net.lundman.spl.sym) + 54 (0xffffff7f80e52a76)
taskq_wait (in net.lundman.spl.sym) + 78 (0xffffff7f80e53fae)
taskq_destroy (in net.lundman.spl.sym) + 35 (0xffffff7f80e54173)
0xffffff7f80f1a870 (0xffffff7f80f1a870)

$ sudo symstacks.rb -f /tmp/stackshot.log -s -k /tmp/net.lundman.zfs.sym
0xffffff7f80e54173 (0xffffff7f80e54173)
vdev_open_children (in net.lundman.zfs.sym) + 336 (0xffffff7f80f1a870)
vdev_root_open (in net.lundman.zfs.sym) + 94 (0xffffff7f80f2bb4e)
vdev_open (in net.lundman.zfs.sym) + 311 (0xffffff7f80f1a9b7)
vdev_create (in net.lundman.zfs.sym) + 31 (0xffffff7f80f1b65f)
spa_create (in net.lundman.zfs.sym) + 878 (0xffffff7f80f042ee)
</syntaxhighlight>

Voilà!

=== Memory leaks ===

In some cases, you may suspect memory issues, for instance if you saw the following panic:

<syntaxhighlight lang="text">
panic(cpu 1 caller 0xffffff80002438d8): "zalloc: \"kalloc.1024\" (100535 elements) retry fail 3, kfree_nop_count: 0"@/SourceCache/xnu/xnu-2050.7.9/osfmk/kern/zalloc.c:1826
</syntaxhighlight>

To debug this, you can attach GDB and use the zprint command:

<syntaxhighlight lang="text">
(gdb) zprint
ZONE COUNT TOT_SZ MAX_SZ ELT_SZ ALLOC_SZ TOT_ALLOC TOT_FREE NAME
0xffffff8002a89250 1620133 18c1000 22a3599 16 1000 125203838 123583705 kalloc.16 CX
0xffffff8006306c50 110335 35f000 4ce300 32 1000 13634985 13524650 kalloc.32 CX
0xffffff8006306a00 133584 82a000 e6a900 64 1000 26510120 26376536 kalloc.64 CX
0xffffff80063067b0 610090 4a84000 614f4c0 128 1000 50524515 49914425 kalloc.128 CX
0xffffff8006306560 1070398 121a2000 1b5e4d60 256 1000 72534632 71464234 kalloc.256 CX
0xffffff8006306310 399302 d423000 daf26b0 512 1000 39231204 38831902 kalloc.512 CX
0xffffff80063060c0 100404 6231000 c29e980 1024 1000 22949693 22849289 kalloc.1024 CX
0xffffff8006305e70 292 9a000 200000 2048 1000 77633725 77633433 kalloc.2048 CX
</syntaxhighlight>

In this case, kalloc.256 is suspect.

Reboot kernel with zlog=kalloc.256 on the command line, then we can use

<syntaxhighlight lang="text">
(gdb) findoldest
oldest record is at log index 393:

--------------- ALLOC 0xffffff803276ec00 : index 393 : ztime 21643824 -------------
0xffffff800024352e <zalloc_canblock+78>: mov %eax,-0xcc(%rbp)
0xffffff80002245bd <get_zone_search+23>: jmpq 0xffffff80002246d8 <KALLOC_ZINFO_SALLOC+35>
0xffffff8000224c39 <OSMalloc+89>: mov %rax,-0x18(%rbp)
0xffffff7f80e847df <zfs_kmem_alloc+15>: mov %rax,%r15
0xffffff7f80e90649 <arc_buf_alloc+41>: mov %rax,-0x28(%rbp)
and indeed, list any index

(gdb) zstack 394

--------------- ALLOC 0xffffff8032d60700 : index 394 : ztime 21648810 -------------
0xffffff800024352e <zalloc_canblock+78>: mov %eax,-0xcc(%rbp)
0xffffff80002245bd <get_zone_search+23>: jmpq 0xffffff80002246d8 <KALLOC_ZINFO_SALLOC+35>
0xffffff8000224c39 <OSMalloc+89>: mov %rax,-0x18(%rbp)
0xffffff7f80e847df <zfs_kmem_alloc+15>: mov %rax,%r15
0xffffff7f80e90649 <arc_buf_alloc+41>: mov %rax,-0x28(%rbp)
How many times was zfs_kmem_alloc involved in the leaked allocs?

(gdb) countpcs 0xffffff7f80e847df
occurred 3999 times in log (100% of records)
</syntaxhighlight>

At least we know it is our fault.

How many times is it arc_buf_alloc?

<syntaxhighlight lang="text">
(gdb) countpcs 0xffffff7f80e90649
occurred 2390 times in log (59% of records)
</syntaxhighlight>

=== Memory Architecture ===

ZFS is designed to aggressively cache filesystem data in main memory. The result of this caching can be significant filesystem performance improvement.

Selection of an allocator has been very challenging on OSX. In the last year we have evolved from:
* Direct call to OSMalloc - a very low level allocator in the kernel - rejected because of slow performance and because the minimum allocation size is one page (4k)
* Direct call to zalloc - the OSX zones allocator - rejected because only 25% of the machines memory can be accessed (50% under some circumstances), and because the result of exceeding this limit is a kernel panic with no other feedback mechanisms available.
* Direct call to bmalloc - bmalloc was a home grown slice allocator that allocated slices of memory from the kernel page allocator, and subdivided into smaller units of allocation to use by ZFS. This was quite successful but very space inefficient. Was used in O3X 1.27 and 1.3.0. At this stage we had no real response to memory pressure in the machine, so the total memory allocation to O3X was kept to 50% of the machine.
* Implementation of kmem and vmem allocators using code from Illumos. Provision of a memory pressure monitor mechanism - we are now able to allocate most of the machines memory to ZFS, and scale that back when the machine experiences memory pressure.

O3X has the Solaris Porting Layer (SPL). The SPL has long since provided the Illumos kmem.h API for use by ZFS. In O3X releases up to 1.3.0 the kmem implementation was been a stub that passes allocation requests to an underlying allocator. In O3X 1.3.0 we were still missing some key behaviours in the allocator - efficient lifecycle control of objects, and an effective response to memory pressure in the machine, and the allocator was not very space efficient because of metadata overheads in bmalloc. We were also not convinced that bmalloc represented the state of the art.

Our strategy was to determine how much of the Illumos allocator could be implemented on OSX. After a series of experiments where we implemented significant portions of the kmem code from illumos on top of bmalloc, we had learned enough to take the final step of essentially copying the entire kmem/vmem allocator stack from Illumos. Some portions of the kmem code have been disabled in kmem such as logging, and hot swap CPU support have been disabled due to architectural differences between OSX and Illumos.

By default kmem/vmem require a certain level of performance from the OS page allocator. It is easy to overwhelm the OSX page allocator. We tuned vmem to use 512Kb chunks of memory from the page allocator rather than the smaller allocations that vmem prefers. This is less than ideal as it reduces the ability for vmem to smoothly release memory to the page allocator when the machine is under pressure. While we have an adequately performing solution now, there will always be a tension between our allocator and OSX itself. OSX only provides minimal mechanisms to observe and respond to memory pressure in the machine, so we are somewhat limited in what can be achieved in this regard.

References:

Jeff Bonwicks paper - kmem and vmem implement this design. https://www.usenix.org/legacy/event/usenix01/full_papers/bonwick/bonwick_html/

== Flamegraphs ==

Huge thanks to [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html BrendanGregg] for so much of the dtrace magic.

dtrace the kernel while running a command:

<syntaxhighlight lang="bash">
$ sudo dtrace -x stackframes=100 -n 'profile-997 /arg0/ {
@[stack()] = count(); } tick-60s { exit(0); }' -o out.stacks
</syntaxhighlight>

It will run for 60 seconds.

Convert it to a flamegraph:

<syntaxhighlight lang="bash">
$ ./stackcollapse.pl out.stacks > out.folded
$ ./flamegraph.pl out.folded > out.svg
</syntaxhighlight>

This is <code>rsync -a /usr/ /BOOM/deletea/</code> running:

[[File:rsyncflamegraph.svg|thumb|rsync flamegraph]]

Or running '''Bonnie++''' in various stages:

<gallery mode="packed-hover">
File:create.svg|Create files in sequential order|alt=[[File:create.svg]]
File:stat.svg|Stat files in sequential order|alt=Stat files in sequential order
File:delete.svg|Delete files in sequential order|alt=Delete files in sequential order
</gallery>

[[File:VX_create.svg|thumb|Create files in sequential order]]

<br>

[[File:iozone.svg|thumb|IOzone flamegraph]]

[[File:iozoneX.svg|thumb|IOzone flamegraph (untrimmed)]]

<br>

------

== Iozone ==

Quick peek at how they compare, just to see how much we should improve it by.

HFS+ and ZFS were created on the same virtual disk in VMware. Of course, this is not ideal testing specs, but should serve as an indicator.

The pool was created with

<syntaxhighlight lang="bash">
$ sudo zpool create -f -o ashift=12 \
-O atime=off \
-O casesensitivity=insensitive \
-O normalization=formD \
BOOM /dev/disk1
</syntaxhighlight>

and the HFS+ file system was created with the standard OS X Disk Utility.app, with everything default (journaled, case-insensitive).

'''Iozone''' was run with standard automode:

<syntaxhighlight lang="bash">
sudo iozone -a -b outfile.xls
</syntaxhighlight>

[[File:hfs2_read.png|thumb|HFS+ read]]
[[File:hfs2_write.png|thumb|HFS+ write]]
[[File:zfs2_read.png|thumb|ZFS read]]
[[File:zfs2_write.png|thumb|ZFS write]]

As a guess, writes need to double, and reads need to triple.

=== VFS ===

[[VFS]]

== File-based zpools for testing==

* create 2 files (each 100 MB) to be used as block devices:
<syntaxhighlight lang="bash">
$ dd if=/dev/zero bs=1m count=100 of=vdisk1
$ dd if=/dev/zero bs=1m count=100 of=vdisk2
</syntaxhighlight>

* attach files as raw disk images:
<syntaxhighlight lang="bash">
$ hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount vdisk1
/dev/disk2
$ hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount vdisk2
/dev/disk3
</syntaxhighlight>

* create mirrored zpool:
<syntaxhighlight lang="bash">
$ sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive -O normalization=formD tank mirror disk2 disk3
</syntaxhighlight>

* show zpool:
<syntaxhighlight lang="bash">
$ sudo zpool status
pool: tank
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
disk2 ONLINE 0 0 0
disk3 ONLINE 0 0 0

errors: No known data errors
</syntaxhighlight>

* test ZFS features, find bugs, ...

* export zpool:
<syntaxhighlight lang="bash">
$ sudo zpool export tank
</syntaxhighlight>

* detach raw images:
<syntaxhighlight lang="bash">
$ hdiutil detach disk2
"disk2" unmounted.
"disk2" ejected.
$ hdiutil detach disk3
"disk3" unmounted.
"disk3" ejected.
</syntaxhighlight>

== Platform differences ==

This section is an attempt to outline the differences from ZFS versions of other platforms, as compared to OSX. To assist developers new to the Apple platform, who wishes to assist, or understand, development of the O3X version.

=== Reclaim ===

One of the biggest hassles with OSX is the VFS layer's handling of reclaim. First it is worth noting that "struct vnode" is an opaque type, so we are not allowed to see, nor modify, the contents of a vnode.
(Of course, we could craft a mirror struct of vnode and tailor it to each OSX version where vnode changes. But that is rather hacky.)

Following that, the '''only''' place where you can set the '''vtype''' (VREG, VDIR), '''vdata''' (user pointer to hold the ZFS znode), '''vfsops''' (list of filesystem calls "vnops") etc, is '''only in calling vnode_create()'''.
So there is no way to "allocate an empty vnode, and set its values later". The FreeBSD method of pre-allocating vnodes, to avoid reclaim, can not be done.
ZFS will start a new dmu_tx, then call zfs_mknode which will eventually call vnode_create, so we can not do anything with dmu_tx in those vnops.

The problem is, if vnode_create decides to reclaim, it will do so directly, as the same thread. It will end up in vclean() which can call vnop_fsync, vnop_pageout, vnop_inactive and vnop_reclaim. The first three of these calls, we can
use the API call vnode_isrecycled() to detect if these vnops are called "the normal way", or from vclean. If we come from vclean, and the vnode is doomed, we will do as little as possible. We can not open a new TX, and
we can not use mutex locks (panic: locking against ourselves).

Nor is there any way to defer, or delay, a doomed vnode. If vnop_reclaim returns anything but 0, you find the lovely XNU code of
2205 if (VNOP_RECLAIM(vp, ctx))
2206 panic("vclean: cannot reclaim");
in vfs_subr.c

As for vnop_reclaim, it requires a little more work. We need to detect if curthread is coming via the inside of vnode_create. Currently, the call to vnode_create looks like:

reentry.id = curthread;
mutex_enter(&zfsvfs->z_reentry_lock);
'''list_insert_head'''(&zfsvfs->z_reentry_threads, &reentry);
mutex_exit(&zfsvfs->z_reentry_lock);
'''vnode_create'''(VNCREATE_FLAVOR, VCREATESIZE, &vfsp, vpp);
mutex_enter(&zfsvfs->z_reentry_lock);
'''list_remove'''(&zfsvfs->z_reentry_threads, &reentry);
mutex_exit(&zfsvfs->z_reentry_lock);

That is to say, add a node (carrying only curthread id) to the list z_reentry_threads in zfsvfs. Then at the start of vnop_reclaim, we can test curthread against this list to see if the current thread is inside a call to
vnode_create.
If we are not inside vnode_create (zp->z_reclaim_reentry = FALSE) we can handle the reclaim as upstream, either calling zfs_znode_free or zfs_zinactive depending on the value of z_sa_hdl.
However, if we are inside vnode_create (zp->z_reclaim_reentry = TRUE) we call zfs_zinactive, but IF zp->unlinked is set, and we end up in zfs_rmnode(), this function will return early, before it does the "final TX".
At the end of vnop_reclaim, we then add the znode (zp) to a zfsvfs->z_reclaim_list. Note that the znode has not finished reclaim, but its z_vnode is now set to NULL (since vnop_reclaim was called, we have no choice but to release it)
nor has it called zfs_znode_free().

The reclaim_thread (started from zfs_vfsops creation of zfsvfs) will wake up, see the znode on the reclaim_list, and do the final TX, lifted from zfs_rmnode. It then finally calls zfs_znode_free(). As it is run as a separate thread, dmu_tx and dmu_tx_commit will succeed.

=== Fastpath vs Recycle ===

Another interesting aspect is that IllumOS has a delete fastpath. In zfs_remove, if it is detected that the znode can be "deleted_now", it marks the vnode as free and directly calls zfs_znode_delete(), if it can not, adds it to zfs_unlinked_add().

In OSX, there is no way to directly release a vnode. Ie, XNU always has full control of the vnodes. Even if you call vnode_recycle(), the vnode is not released '''until''' vnop_reclaim is called. The vnode can just be marked for later reclaim, but remain active (especially if you are racing against other threads using the same vnode). So in zfs_remove, we attempt to call vnode_recycle(), and only if this returns "1" do we know that vnop_reclaim was called, and we can directly call zfs_znode_delete(). Note that the O3X vnop_reclaim handler then has special code to not do anything with the vnode (zp->z_fastpath) but to only clear out the z_vnode and return.

There is also a little special lock-handling in zfs_zinactive, since we can call it from inside of a vnode_create() which is called by ZFS with locks held. If this is the case, we do not attempt to acquire locks in zfs_zinactive.

Performance

2014-04-13T01:07:50Z

Brendon: /* Memory Utilization */

== Status ==

Currently OpenZFS on OS X is active development, with priority being given to stability and integration enhancements, before performance.

== Performance Tuning ==

=== Memory Utilization ===

ZFS uses memory caching to improve performance. Filesystem meta-data, data, and the De-duplication Tables (DDT) are stored in the ARC, allowing frequently accessed data to be accessed many times faster than possible from hard disk media. The performance that you experience with ZFS can be directly linked to the amount of memory that is allocated to the ARC.

O3X 1.2.0 sets the ARC size to just 6% of your computers main memory. This design decision was made for stability reasons as its not hard to exhaust the built in kernel memory allocator, which allocates memory from a pool that defaults to 25% of your computers main memory.

The O3X project will soon release a version of the code that contains a replacement memory allocator that can access much more of your computers memory. The allocator was recently integrated into the 'master' branch in the source repository. With the introduction of the new allocator, the default ARC size has been raised to 25% of main memory. This has resulted in significant performance improvements. Due to space efficiency limitations in the allocator, this will result in up to 40% of your computers memory being occupied by ZFS.

O3X uses Wired (non-pageable) memory for the ARC. Activity Monitor and Top can display the total amount of Wired memory allocated. The spl.kmem_bytes_total sysctl contains the amount of memory allocated by ZFS. The difference between these figures reflects memory use by the kernel and the efficiency factor of the allocator. A real word example, a 32GB iMac, has ARC set to 8GB. Approximately 7GB RAM allocated by ZFS, results in approximately 10.5GB of Wired RAM in use.

The size of the ARC is controllable by altering the value stored in the sysctl zfs.arc_max which can be set by:

::sudo sysctl -w zfs.arc_max=<arc size in bytes>

When configuring large amount of ARC storage, the total allocated memory should be carefully monitored as the machines performance may suffer if too much memory is allocated.

Future improvements to the allocator will improve its space efficiency for small allocations, which will address some of the limitations present in current code.

== Current Benchmarks ==

In order to establish a baseline of current performance of OpenZFS on OS X, measurements have been made using the iozone [http://www.iozone.org] benchmarking tool. The benchmark consists of running various ZFS implementations inside a VMware 6.0.2 VM on a 2011 iMac. Each VM is provisioned with 8GB of RAM, an OS boot drive, and a 5 GB second HDD containing a ZFS dataset. The HDDs are standard VMware .vmx files.

The test zpool was created using the following command:

::zpool create -o ashift=12 -f tank <disk device name>

The benchmark consists of the following steps:

# Start the VM.
# Import the tank dataset
# Execute -> mkdir /tank/tmp && cd /tank/tmp
# Execute -> time iozone -a
# Record Time 1
# Execute -> time iozone -a
# Record time 2
# Terminate the VM and VMware before moving to the next ZFS implementation/OS combination.

The results are as follows:

[[File:ZFS_iozone_time.png|left|frame|Comparison of ZFS implementations]]

Performance

2014-04-13T01:02:41Z

Brendon: /* Memory Utilization */

== Status ==

Currently OpenZFS on OS X is active development, with priority being given to stability and integration enhancements, before performance.

== Performance Tuning ==

=== Memory Utilization ===

ZFS uses memory caching to improve performance. Filesystem meta-data, data, and the De-duplication Tables (DDT) are stored in the ARC, allowing frequently accessed data to be accessed many times faster than possible from hard disk media. The performance that you experience with ZFS can be directly linked to the amount of memory that is allocated to the ARC.

O3X 1.2.0 sets the ARC size to just 6% of your computers main memory. This design decision was made for stability reasons as its not hard to exhaust the built in kernel memory allocator, which allocates memory from a pool that defaults to 25% of your computers main memory.

The O3X project will soon release a version of the code that contains a replacement memory allocator that can access much more of your computers memory. The allocator was recently integrated into the 'master' branch in the source repository. With the introduction of the new allocator, the default ARC size has been raised to 25% of main memory. This has resulted in significant performance improvements. Due to space efficiency limitations in the allocator, this will result in up to 40% of your computers memory being occupied by ZFS.

O3X uses Wired (non-pageable) memory for the ARC. Activity Monitor and Top can display the total amount of Wired memory allocated. The spl.kmem_bytes_total sysctl contains the amount of memory allocated by ZFS. The difference between these figures reflects memory use by the kernel and the efficiency factor of the allocator. A real word example, a 32GB iMac, has ARC set to 8GB. Approximately 7GB RAM allocated by ZFS, results in Approximately 10.5GB of Wired RAM in use.

The size of the ARC is controllable by altering the value stored in the sysctl zfs.arc_max which can be set by:

::sudo sysctl -w zfs.arc_max=<arc size in bytes>

When configuring large amount of ARC storage, the total allocated memory should be carefully monitored as the machines performance may suffer if too much memory is allocated.

Future improvements to the allocator will improve its space efficiency for small allocations, which will address some of the limitations present in current code.

== Current Benchmarks ==

In order to establish a baseline of current performance of OpenZFS on OS X, measurements have been made using the iozone [http://www.iozone.org] benchmarking tool. The benchmark consists of running various ZFS implementations inside a VMware 6.0.2 VM on a 2011 iMac. Each VM is provisioned with 8GB of RAM, an OS boot drive, and a 5 GB second HDD containing a ZFS dataset. The HDDs are standard VMware .vmx files.

The test zpool was created using the following command:

::zpool create -o ashift=12 -f tank <disk device name>

The benchmark consists of the following steps:

# Start the VM.
# Import the tank dataset
# Execute -> mkdir /tank/tmp && cd /tank/tmp
# Execute -> time iozone -a
# Record Time 1
# Execute -> time iozone -a
# Record time 2
# Terminate the VM and VMware before moving to the next ZFS implementation/OS combination.

The results are as follows:

[[File:ZFS_iozone_time.png|left|frame|Comparison of ZFS implementations]]

Performance

2014-04-12T23:46:16Z

Brendon:

== Status ==

Currently OpenZFS on OS X is active development, with priority being given to stability and integration enhancements, before performance.

== Performance Tuning ==

=== Memory Utilization ===

ZFS uses memory caching to improve performance. Filesystem meta-data, data, and the De-duplication Tables (DDT) are stored in the ARC, allowing frequently accessed data to be accessed many times faster than possible from hard disk media. The performance that you experience with ZFS can be directly linked to the amount of memory that is allocated to the ARC.

O3X 1.2.0 sets the ARC size to just 6% of your computers main memory. This design decision was made for stability reasons as its not hard to exhaust the built in kernel memory allocator, which allocates memory from a pool that defaults to 25% of your computers main memory.

The O3X project will soon release a version of the code that contains a replacement memory allocator that can access much more of your computers memory. The allocator was recently integrated into the 'master' branch in the source repository. With the introduction of the new allocator, the default ARC size has been raised to 25% of main memory. This has resulted in significant performance improvements. Due to space efficiency limitations in the allocator, this will result in up to 40% of your computers memory being occupied by ZFS.

O3X uses Wired (non-pageable) memory for the ARC. Activity Monitor and Top can display the total amount of Wired memory allocated. the spl.kmem_bytes_total sysctl contains the amount of memory allocated by ZFS. The difference between these figures reflects memory use by the kernel and the efficiency factor of the allocator. A real word example, a 32GB iMac, has ARC set to 8GB. Approximately 7GB RAM allocated by ZFS, results in Approximately 10.5GB of Wired RAM in use.

The size of the ARC is controllable by altering the value stored in the sysctl zfs.arc_max which can be set by:

::sudo sysctl -w zfs.arc_max=<arc size in bytes>

When configuring large amount of ARC storage, the total allocated memory should be carefully monitored as the machines performance may suffer if too much memory is allocated.

Future improvements to the allocator will improve its space efficiency for small allocations, which will address some of the limitations present in current code.

== Current Benchmarks ==

In order to establish a baseline of current performance of OpenZFS on OS X, measurements have been made using the iozone [http://www.iozone.org] benchmarking tool. The benchmark consists of running various ZFS implementations inside a VMware 6.0.2 VM on a 2011 iMac. Each VM is provisioned with 8GB of RAM, an OS boot drive, and a 5 GB second HDD containing a ZFS dataset. The HDDs are standard VMware .vmx files.

The test zpool was created using the following command:

::zpool create -o ashift=12 -f tank <disk device name>

The benchmark consists of the following steps:

# Start the VM.
# Import the tank dataset
# Execute -> mkdir /tank/tmp && cd /tank/tmp
# Execute -> time iozone -a
# Record Time 1
# Execute -> time iozone -a
# Record time 2
# Terminate the VM and VMware before moving to the next ZFS implementation/OS combination.

The results are as follows:

[[File:ZFS_iozone_time.png|left|frame|Comparison of ZFS implementations]]

File:ZFS iozone time.png

2014-04-11T06:21:54Z

Brendon: Brendon uploaded a new version of "File:ZFS iozone time.png"

Simple benchmarking of ZFS implementations for comparison purposes.

File:ZFS iozone time.png

2014-04-11T06:21:53Z

Brendon: Brendon uploaded a new version of "File:ZFS iozone time.png"

Simple benchmarking of ZFS implementations for comparison purposes.

Performance

2014-04-10T10:04:25Z

Brendon:

== Status ==

Currently OpenZFS on OSX is under heavy development, with priority being given to completion of features over performance.

== Current Benchmarks ==

In order to establish a baseline of current performance of OpenZFS on OSX, measurements have been made using the iozone [http://www.iozone.org] benchmarking tool. The benchmark consists of running various ZFS implementations inside a VMware 6.0.2 VM on a 2011 iMac. Each VM is provisioned with 8GB of RAM, an OS boot drive, and a 5 GB second HDD containing a ZFS dataset. The HDDs are standard VMware .vmx files.

The test zpool was created using the following command:

::zpool create -o ashift=12 -f tank <disk device name>

The benchmark consists of the following steps:

# Start the VM.
# Import the tank dataset
# Execute -> mkdir /tank/tmp && cd /tank/tmp
# Execute -> time iozone -a
# Record Time 1
# Execute -> time iozone -a
# Record time 2
# Terminate the VM and VMware before moving to the next ZFS implementation/OS combination.

The results are as follows:

[[File:ZFS_iozone_time.png|left|frame|Comparision of ZFS implementations]]

File:ZFS iozone time.png

2014-04-10T09:57:44Z

Brendon: Brendon uploaded a new version of "File:ZFS iozone time.png"

Simple benchmarking of ZFS implementations for comparison purposes.

File:ZFS iozone time.png

2014-04-10T09:55:56Z

Brendon: Brendon uploaded a new version of "File:ZFS iozone time.png"

Simple benchmarking of ZFS implementations for comparison purposes.

Performance

2014-04-10T09:53:56Z

Brendon: Created page with " == Status == Currently OpenZFS on OSX is under heavy development, with priority being given to completion of features over performance. == Current Benchmarks == In order t..."

== Status ==

Currently OpenZFS on OSX is under heavy development, with priority being given to completion of features over performance.

== Current Benchmarks ==

In order to establish a baseline of current performance of OpenZFS on OSX, measurements have been made using the iozone [http://www.iozone.org] benchmarking tool. The benchmark consists of running various ZFS implementations inside a VMware 6.0.2 VM on a 2011 iMac. Each VM is provisioned with 8GB of RAM, an OS boot drive, and a 5 GB second HDD containing a ZFS dataset. The HDDs are standard VMware .vmx files.

::zpool create -o ashift=12 -f tank <disk device name>

The benchmark consists of the following steps:

# Start the VM.
# Import the tank dataset
# mkdir /tank/tmp && cd /tank/tmp
# time iozone -a
# time iozone -a
# Terminate the VM and VMware before moving to the next ZFS implementation/OS combination.

The results are as follows:

[[File:ZFS_iozone_time.png|frame|Comparision of ZFS implementations]]

Main Page

2014-04-10T09:33:30Z

Brendon: Add Performance/benchmarking page.

__NOTOC__ __NOEDITSECTION__
'''Welcome to the [[O3XWiki:About|O3XWiki]]: your source for OpenZFS on OS X documentation on the web.'''

'''O'''penZFS '''o'''n '''O'''S '''X''' → '''OOOX''' → '''O3X'''

Please note that if you are running MacZFS, you should use their site instead: [http://maczfs.org maczfs.org].

==The software==

<div style="float: left; margin-right: 1%; width: 49%;">

; [[OpenZFS on OS X]]: Brief overview of OpenZFS on OS X describing what to expect when using O3X.

; [[Downloads]]: Latest release of the Mac installer.

; [[Documentation|O3X documentation]]: Index of popular articles and often-referenced information about O3X and OpenZFS.

; [[Source code]]: Show me the code.

; [[Performance]]: Performance and Benchmarking

</div>
<div style="float: left; width: 50%;">
; [[Install|Installation guide]]: Detailed guide through the whole process of installing and configuring OpenZFS on OS X.

; [[Uninstall|Uninstallation guide]]: How to remove O3X from your computer.

; [[Development]]: Information of particular interest to developers.

; [[FAQ]]: List of common and frequently asked questions.

</div>

<div style="clear: both;"></div>

==Our community==

<div style="float: left; margin-right: 1%; width: 49%;">
; [[Getting involved]]: Describes various ways O3X users can contribute to the O3X community.

; [[O3XWiki:Contributing]]: If willing and able to contribute to the wiki, please see this article for ideas.

; [[Help:Editing]]: Technical Information about editing and contributing to the OpenZFS on OS X Wiki.
</div>

<div style="float: left; width: 50%;">

; [[O3X forums]]: General technical assistance for users and discussion of all things O3X including development.

; [[IRC channel]]: Live problem-solving and conversation with your fellow O3X users and developers '''#openzfs-osx'''.

; [[Beyond our walls]]: How to get connect more broadly with users and developers of OpenZFS on other platforms.
</div>
<div style="clear: both;"></div>

== Contact us ==

<div style="float: left; margin-right: 1%; width: 49%;">

; [[O3XWiki:Administrators]]: Contact one of the admins by email.

</div>

<div style="float: left; width: 50%;">

; [[O3XWiki:Donations]]: Donations and other ways to contribute
</div>
<div style="clear: both;"></div>

==Twitter==
[https://twitter.com/openzfsonosx @openzfsonosx] on Twitter.

<html><a class="twitter-timeline" href="https://twitter.com/openzfsonosx" data-widget-id="444275713776951296">Tweets by @OpenZFSonOSX</a>
<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+"://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script>

<span>
<br>
<a href="https://twitter.com/openzfsonosx" class="twitter-follow-button" data-show-count="true">Follow @openzfsonosx</a>
<br>
<a href="https://twitter.com/share" class="twitter-share-button" data-via="OpenZFSonOSX">Tweet</a>
</span>
<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script></html>

File:ZFS iozone time.png

2014-04-10T09:26:31Z

Brendon: Simple benchmarking of ZFS implementations for comparison purposes.

Simple benchmarking of ZFS implementations for comparison purposes.