The panics occur without any clearly visible pattern, although running Thunderbird and performing IMAP synchronization with large mailboxes ( > 3 GB per mailbox file) seems to increase the probability of an immediate kernel panic. However, panics do also occur when Thunderbird is not running.
Another program significantly increasing the crash risk is Spotlight (more precisely: mds and mdimporter) trying to rebuild the Spotlight index after a crash. I think Spotlight is responsible for most if not all crashes shortly after boot after a previous crash.
Relatively often (approx. 1/4 of all cases) I see two panics right after each other, usually first a page fault, followed by an assertion failed. See below for full list. The panics started around 6 days after installing ZEVO.
All panics of type "page fault" show similar backtraces and register values.
System configuration:
- Macbook Pro from late 2010
- MacOSX Snow Leopard 10.6.8, fully updated
- 8 GB of RAM
- internal SSD disk with four partitions (five, counting the EFI partition):
s2 is system / boot partition,
s3 is ZFS pool
s4 is L2ARC for that pool,
s5 is another HFS+ volume - ZEVO community edition 1.1.1 (build 2012-09-23)
- MacPorts version 2.1.3
The ZFS pool has three child file systems, with mount points set to /Developer /Users/bj (my main account) and /opt (for holding MacPorts).
On all dataset, I have enable compression. DeDup is Off. Copies is set to 1 (default), normalization is formD.
Here all non-default properties:
- Code: Select all
bj $ zfs get all | grep -v -e '@2013' | grep -v -e 'default$' -e '-$'
NAME PROPERTY VALUE SOURCE
ZFSStore/Developer mountpoint /Developer local
ZFSStore/Developer compression on local
ZFSStore/bj mountpoint /Users/bj local
ZFSStore/bj compression on local
ZFSStore/bj snapdir visible local
ZFSStore/opt mountpoint /opt local
ZFSStore/opt compression on local
A cron job takes automatic snapshots of the Users/bj dataset once every hour between 7:00 and 23:00. Currently the system has 454 snapshots.
The pool size is 288GB, with 188GB allocated, i.e 65% used. 115GB are used by file systems, the rest is used by snapshots.
List of panics, including link to panic file
1) zevo-crash-2013-02-25.txt uptime 157h58 Kernel trap at 0xffffff7f8135a768, type 14=page fault
2) zevo-crash-2013-02-25_02.txt uptime 00h54 zio.c:474 ZFS assertion failed: *countp > 0 (0x0 > 0x0)
3) zevo-crash-2013-02-26_01.txt uptime 11h37 Kernel trap at 0xffffff7f81a7c768, type 14=page fault
4) zevo-crash-2013-02-26_02.txt uptime 00h11 zio.c:474 ZFS assertion failed: *countp > 0 (0x0 > 0x0)
5) zevo-crash-2013-02-26_03.txt uptime 03h07 Kernel trap at 0xffffff7f80c740a6, type 14=page fault
6) zevo-crash-2013-02-26_04.txt uptime 02h44 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
7) zevo-crash-2013-02-26_05.txt uptime 00h00'26 Kernel trap at 0xffffff7f81a3e365, type 14=page fault
8) zevo-crash-2013-02-27_01.rtf uptime 15h39 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
9) zevo-crash-2013-02-27_02.rtf uptime 00h16 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
10) zevo-crash-2013-03-01_01.rtf uptime 23h44 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf576718 0 0xffffff80bf5769a0 … zio.c:478
11) zevo-crash-2013-03-04_01.txt uptime 43h27 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
12) zevo-crash-2013-03-04_02.txt uptime 00h00'7 metaslab.c:1428 ZFS assertion failed: DVA_IS_VALID(dva)
13) zevo-crash-2013-03-04_03.txt uptime 01h57 Kernel trap at 0xffffff7f819ff55d, type 14=page fault
14) zevo-crash-2013-03-04_04.txt uptime 08h53 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
15) zevo-crash-2013-03-04_05.txt uptime 00h15 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
16) zevo-crash-2013-03-05_01.txt uptime 22h07 Kernel trap at 0xffffff7f819ff55d, type 14=page fault
17) zevo-crash-2013-03-06_01.txt uptime 14h42 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf444400 0 0xffffff80bf444688 ... zio.c:478
18) zevo-crash-2013-03-07_01.txt uptime 14h26 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf5db720 0xffffff80bf5db9d8 0xffffff80bf5db9a8 ... zio.c:478
19) zevo-crash-2013-03-07_02.txt uptime 01h23 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf63ca58 0xffffff80bf63cd10 0xffffff80bf63cce0 ... zio.c:478
20) zevo-crash-2013-03-07_03.txt uptime 09h05 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff810824a568 0 0xffffff810824a7f0 ... zio.c:478
21) zevo-crash-2013-03-08_01.txt uptime 06h13 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf57e730 0 0xffffff80bf57e9b8 ... zio.c:478
22) zevo-crash-2013-03-08_02.txt uptime 00h25 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
23) zevo-crash-2013-03-08_03.txt uptime 00h18 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf4fb0b8 0xffffff80bf4fb378 0xffffff80bf4fb340 ... zio.c:478
24) zevo-crash-2013-03-09_01.txt uptime 05h02 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf8cbd60 0 0xffffff80bf8cbfe8 ... zio.c:478
25) zevo-crash-2013-03-09_02.txt uptime 08h43 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80cd4bd7f8 0 0xffffff80cd4bda80 ... zio.c:478
26) zevo-crash-2013-03-13_01.txt uptime 38h46 Kernel trap at 0xffffff7f819e2a96, type 14=page fault
27) zevo-crash-2013-03-13_02.txt uptime 10h04 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf7c20a0 0 0xffffff80bf7c2328 ... zio.c:478
28) zevo-crash-2013-03-13_03.txt uptime 04h23 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf613720 0xffffff80bf6139e0 0xffffff80bf6139a8 ... zio.c:478
29) zevo-crash-2013-03-13_04.txt uptime 04h04 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
30) zevo-crash-2013-03-14_01.txt uptime 15h07 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
31) zevo-crash-2013-03-14_02.txt uptime 04h20 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
32) zevo-crash-2013-03-15_01.txt uptime 12h17 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf4ab708 0 0xffffff80bf4ab990 ... zio.c:478
33) zevo-crash-2013-03-15_02.txt uptime 02h24 metaslab.c:1428 ZFS assertion failed: DVA_IS_VALID(dva)
34) zevo-crash-2013-03-15_03.txt uptime 00h00'6 metaslab.c:1428 ZFS assertion failed: DVA_IS_VALID(dva)
35) zevo-crash-2013-03-15_04.txt uptime 00h00'5 metaslab.c:1428 ZFS assertion failed: DVA_IS_VALID(dva)
36) zevo-crash-2013-03-15_05.txt uptime 03h55 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff811fbf2ae0 0xffffff811fbf2d98 0xffffff811fbf2d68 ... zio.c:478
37) zevo-crash-2013-03-15_06.txt uptime 00h01'3 Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault
38) zevo-crash-2013-03-15_07.txt uptime 01h50 metaslab.c:1428 ZFS assertion failed: DVA_IS_VALID(dva)
39) zevo-crash-2013-03-15_08.txt uptime 00h13 zio_notify_parent: zero io_children [0, 0], err 0, 0xffffff80bf1bc400 0xffffff80bf1bc6c0 0xffffff80bf1bc688 ... zioc:478
Almost 40 crashes in 18 days renders Zevo pretty unusable. Even the old Z-410 beta have been more stable.
@ Don Brady et.al. : If you need more information, please let me know. I'll also happy to run any extra debug version you may have or to provide any help in solving this that I can give. I have several years of programming experience and some experience in Mac OSX kernel debugging. I needed, I can setup a remote machine for interactive kernel tracing and provide remote access. Feel free to contact me.
The 18 crashes with "Kernel trap at ???, type 14=page fault" are distributed over 7 different backtraces:
- Code: Select all
panic(cpu 1 caller 0xffffff80002d1208): Kernel trap at 0xffffff7f8135a768, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0xffffff90c88bc148, CR3: 0x0000000000100000, CR4: 0x0000000000000660
RAX: 0x0000000000000000, RBX: 0xffffff90c88bbec0, RCX: 0x0000000002000000, RDX: 0xffffff80c88bc9c0
RSP: 0xffffff80a9493e80, RBP: 0xffffff80a9493eb0, RSI: 0x0000000000000000, RDI: 0xffffff80c88bc9c0
R8: 0x36377cf923d094de, R9: 0x002a008e2682d60d, R10: 0x000015b7ffd69702, R11: 0x000000000000001f
R12: 0xffffff90c88bc148, R13: 0xffffff80c88bc6c0, R14: 0xffffff80c88bc9c0, R15: 0xffffff80c88bc9f8
RFL: 0x0000000000010646, RIP: 0xffffff7f8135a768, CS: 0x0000000000000008, SS: 0x0000000000000010
Error code: 0x0000000000000000
Backtrace (CPU 1), Frame : Return Address
0xffffff80a9493b20 : 0xffffff8000204d15
0xffffff80a9493c20 : 0xffffff80002d1208
0xffffff80a9493d70 : 0xffffff80002e3f4a
0xffffff80a9493d80 : 0xffffff7f8135a768
0xffffff80a9493eb0 : 0xffffff7f813593f9
0xffffff80a9493ee0 : 0xffffff7f81356a31
0xffffff80a9493f20 : 0xffffff7f812c7d78
0xffffff80a9493fa0 : 0xffffff80002c8527
Kernel Extensions in backtrace (with dependencies):
com.getgreenbytes.filesystem.zfs(2012.09.14)@0xffffff7f812b9000->0xffffff7f813f9fff
dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f80884000
- Code: Select all
panic(cpu 0 caller 0xffffff80002d1208): Kernel trap at 0xffffff7f80c740a6, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0xffffff90aa8bc818, CR3: 0x0000000000100000, CR4: 0x0000000000000660
RAX: 0x0000000000000000, RBX: 0xffffff90aa8bc590, RCX: 0x0000000002000000, RDX: 0xffffff80aa8bd090
RSP: 0xffffff80a96abe70, RBP: 0xffffff80a96abeb0, RSI: 0x0000000000000000, RDI: 0xffffff80aa8bd090
R8: 0x26df3cd00d9a2ee1, R9: 0x6d435214141e401a, R10: 0x035abb5351b1b0d4, R11: 0xffffff7f80861bf7
R12: 0xffffff80aa8bcd90, R13: 0xffffff90aa8bc818, R14: 0xffffff80aa8bc720, R15: 0x0000000000000000
RFL: 0x0000000000010646, RIP: 0xffffff7f80c740a6, CS: 0x0000000000000008, SS: 0x0000000000000010
Error code: 0x0000000000000000
Backtrace (CPU 0), Frame : Return Address
0xffffff80a96abb10 : 0xffffff8000204d15
0xffffff80a96abc10 : 0xffffff80002d1208
0xffffff80a96abd60 : 0xffffff80002e3f4a
0xffffff80a96abd70 : 0xffffff7f80c740a6
0xffffff80a96abeb0 : 0xffffff7f80c72d31
0xffffff80a96abee0 : 0xffffff7f80c70369
0xffffff80a96abf20 : 0xffffff7f80be168c
0xffffff80a96abfa0 : 0xffffff80002c8527
Kernel Extensions in backtrace (with dependencies):
com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f80bd3000->0xffffff7f80d12fff
dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f80884000
- Code: Select all
panic(cpu 1 caller 0xffffff80002d1208): Kernel trap at 0xffffff7f81a7c0a6, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0xffffff90bf6caae8, CR3: 0x0000000000100000, CR4: 0x0000000000000660
RAX: 0x0000000000000000, RBX: 0xffffff90bf6ca860, RCX: 0x0000000002000000, RDX: 0xffffff80bf6cb360
RSP: 0xffffff80aa433e70, RBP: 0xffffff80aa433eb0, RSI: 0x0000000000000000, RDI: 0xffffff80bf6cb360
R8: 0xd23609d776221ee5, R9: 0x9df596023bf9f8cc, R10: 0x0877d9306729e32c, R11: 0x000000000000001f
R12: 0xffffff80bf6cb060, R13: 0xffffff90bf6caae8, R14: 0xffffff80f2ee9d60, R15: 0x0000000000000000
RFL: 0x0000000000010646, RIP: 0xffffff7f81a7c0a6, CS: 0x0000000000000008, SS: 0x0000000000000010
Error code: 0x0000000000000000
Backtrace (CPU 1), Frame : Return Address
0xffffff80aa433b10 : 0xffffff8000204d15
0xffffff80aa433c10 : 0xffffff80002d1208
0xffffff80aa433d60 : 0xffffff80002e3f4a
0xffffff80aa433d70 : 0xffffff7f81a7c0a6
0xffffff80aa433eb0 : 0xffffff7f81a7ad31
0xffffff80aa433ee0 : 0xffffff7f81a78369
0xffffff80aa433f20 : 0xffffff7f819e968c
0xffffff80aa433fa0 : 0xffffff80002c8527
Kernel Extensions in backtrace (with dependencies):
com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f819db000->0xffffff7f81b1afff
dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f810f0000
- Code: Select all
panic(cpu 1 caller 0xffffff80002d1208): Kernel trap at 0xffffff7f81a3e365, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000428, CR3: 0x0000000000100000, CR4: 0x0000000000000660
RAX: 0x0000000000000000, RBX: 0x0000000000ff5800, RCX: 0x0000000001000006, RDX: 0x0000000000000001
RSP: 0xffffff80bf80bcc0, RBP: 0xffffff80bf80bce0, RSI: 0x000000000000007f, RDI: 0xffffff8014956000
R8: 0x0000000000000000, R9: 0x000000000000007f, R10: 0x0000000000000000, R11: 0x000000000000001f
R12: 0x0000000000000600, R13: 0xffffff8014956388, R14: 0xffffff8015c4eb20, R15: 0xffffff8014956000
RFL: 0x0000000000010206, RIP: 0xffffff7f81a3e365, CS: 0x0000000000000008, SS: 0x0000000000000010
Error code: 0x0000000000000000
Backtrace (CPU 1), Frame : Return Address
0xffffff80bf80b960 : 0xffffff8000204d15
0xffffff80bf80ba60 : 0xffffff80002d1208
0xffffff80bf80bbb0 : 0xffffff80002e3f4a
0xffffff80bf80bbc0 : 0xffffff7f81a3e365
0xffffff80bf80bce0 : 0xffffff7f81a3e39d
0xffffff80bf80bd10 : 0xffffff7f819f6581
0xffffff80bf80bde0 : 0xffffff7f81a39841
0xffffff80bf80bdf0 : 0xffffff7f819f53c1
0xffffff80bf80be30 : 0xffffff7f81a391e8
0xffffff80bf80bf10 : 0xffffff7f81a40070
0xffffff80bf80bfa0 : 0xffffff80002c8527
Kernel Extensions in backtrace (with dependencies):
com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f819db000->0xffffff7f81b1afff
dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f810f0000
- Code: Select all
panic(cpu 1 caller 0xffffff80002d1208): Kernel trap at 0xffffff7f819ff55d, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x00000000001265b2, CR3: 0x0000000000100000, CR4: 0x0000000000000660
RAX: 0x00000000001265b2, RBX: 0xffffff801f8d7d38, RCX: 0xffffff801f8d7d68, RDX: 0xffffff801426ba00
RSP: 0xffffff80bff6bad0, RBP: 0xffffff80bff6bb20, RSI: 0xffffff80ab394e00, RDI: 0xffffff801f8d7d58
R8: 0x0000000000000001, R9: 0x0000000000000000, R10: 0x000000000000000a, R11: 0x000000000000005f
R12: 0xffffff801f98b200, R13: 0xffffff81174e02f0, R14: 0xffffff8107302838, R15: 0xffffff801f8bc700
RFL: 0x0000000000010202, RIP: 0xffffff7f819ff55d, CS: 0x0000000000000008, SS: 0x0000000000000010
Error code: 0x0000000000000000
Backtrace (CPU 1), Frame : Return Address
0xffffff80bff6b770 : 0xffffff8000204d15
0xffffff80bff6b870 : 0xffffff80002d1208
0xffffff80bff6b9c0 : 0xffffff80002e3f4a
0xffffff80bff6b9d0 : 0xffffff7f819ff55d
0xffffff80bff6bb20 : 0xffffff7f819ff504
0xffffff80bff6bb80 : 0xffffff7f819ff504
0xffffff80bff6bbe0 : 0xffffff7f819f779e
0xffffff80bff6bcc0 : 0xffffff7f81a07fda
0xffffff80bff6bd00 : 0xffffff7f81a07c6d
0xffffff80bff6bdb0 : 0xffffff7f81a1ca8d
0xffffff80bff6be30 : 0xffffff7f81a39178
0xffffff80bff6bf10 : 0xffffff7f81a40070
0xffffff80bff6bfa0 : 0xffffff80002c8527
Kernel Extensions in backtrace (with dependencies):
com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f819db000->0xffffff7f81b1afff
dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f810f0000
- Code: Select all
panic(cpu 1 caller 0xffffff80002d1208): Kernel trap at 0xffffff7f819ff55d, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x000000000019ab00, CR3: 0x0000000000100000, CR4: 0x0000000000000660
RAX: 0x000000000019ab00, RBX: 0xffffff80194d8238, RCX: 0xffffff80194d8268, RDX: 0xffffff8014eba000
RSP: 0xffffff80c240bb30, RBP: 0xffffff80c240bb80, RSI: 0xffffff8116fae038, RDI: 0xffffff80194d8258
R8: 0x0000000000000001, R9: 0x000000000000000b, R10: 0x0000000000000002, R11: 0x000000000000005f
R12: 0xffffff8019918900, R13: 0xffffff80c1e81610, R14: 0xffffff80daf43b30, R15: 0xffffff8019633a00
RFL: 0x0000000000010202, RIP: 0xffffff7f819ff55d, CS: 0x0000000000000008, SS: 0x0000000000000010
Error code: 0x0000000000000000
Backtrace (CPU 1), Frame : Return Address
0xffffff80c240b7d0 : 0xffffff8000204d15
0xffffff80c240b8d0 : 0xffffff80002d1208
0xffffff80c240ba20 : 0xffffff80002e3f4a
0xffffff80c240ba30 : 0xffffff7f819ff55d
0xffffff80c240bb80 : 0xffffff7f819ff504
0xffffff80c240bbe0 : 0xffffff7f819f779e
0xffffff80c240bcc0 : 0xffffff7f81a07fda
0xffffff80c240bd00 : 0xffffff7f81a07c6d
0xffffff80c240bdb0 : 0xffffff7f81a1ca8d
0xffffff80c240be30 : 0xffffff7f81a39178
0xffffff80c240bf10 : 0xffffff7f81a40070
0xffffff80c240bfa0 : 0xffffff80002c8527
Kernel Extensions in backtrace (with dependencies):
com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f819db000->0xffffff7f81b1afff
dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f810f0000
- Code: Select all
panic(cpu 1 caller 0xffffff80002d1208): Kernel trap at 0xffffff7f819e2a96, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x000000000022c3b8, CR3: 0x000000001c87d000, CR4: 0x0000000000000660
RAX: 0x0000000000000000, RBX: 0xffffff801d295e00, RCX: 0x000000000005eb45, RDX: 0xffffff81081f4e68
RSP: 0xffffff80c1f0b750, RBP: 0xffffff80c1f0b760, RSI: 0xffffff801ca33400, RDI: 0x000000000022c3a0
R8: 0x0000000000000000, R9: 0xffffff80c1f0b758, R10: 0x0000000000000000, R11: 0x0000000000000001
R12: 0xffffff801d295e00, R13: 0xffffff80cfb296b8, R14: 0x000000000022c348, R15: 0x000000000005eb45
RFL: 0x0000000000010217, RIP: 0xffffff7f819e2a96, CS: 0x0000000000000008, SS: 0x0000000000000010
Error code: 0x0000000000000000
Backtrace (CPU 1), Frame : Return Address
0xffffff80c1f0b3f0 : 0xffffff8000204d15
0xffffff80c1f0b4f0 : 0xffffff80002d1208
0xffffff80c1f0b640 : 0xffffff80002e3f4a
0xffffff80c1f0b650 : 0xffffff7f819e2a96
0xffffff80c1f0b760 : 0xffffff7f819fc3fa
0xffffff80c1f0b7a0 : 0xffffff7f819fd3af
0xffffff80c1f0b830 : 0xffffff7f81a04b8b
0xffffff80c1f0b8e0 : 0xffffff7f81a71813
0xffffff80c1f0b9e0 : 0xffffff800030c78f
0xffffff80c1f0ba40 : 0xffffff80004ca970
0xffffff80c1f0baf0 : 0xffffff80002a0f26
0xffffff80c1f0bb70 : 0xffffff80002a15f4
0xffffff80c1f0bb80 : 0xffffff8000210e1d
0xffffff80c1f0bdc0 : 0xffffff80002a21b9
0xffffff80c1f0be20 : 0xffffff80002a50c3
0xffffff80c1f0bf10 : 0xffffff800047ffa9
0xffffff80c1f0bf40 : 0xffffff80004e8168
0xffffff80c1f0bfa0 : 0xffffff80002e4874
Kernel Extensions in backtrace (with dependencies):
com.getgreenbytes.filesystem.zfs(2012.09.23)@0xffffff7f819db000->0xffffff7f81b1afff
dependency: com.apple.iokit.IOStorageFamily(1.6.3)@0xffffff7f810f0000
Best regards
Björn