Kernel Panics on import since 1.8.2 on Mojave

All your general support questions for OpenZFS on OS X.

Kernel Panics on import since 1.8.2 on Mojave

Postby joeljacobs » Tue Jun 18, 2019 1:29 pm

Since or around the time I installed 1.8.2 I've had occasional kernel panics that I suspected were ZFS related. But they happened at most a couple times a day. I didn't worry about it. But it started getting worse lately, so I re-installed 1.8.2 and it didn't fix it. Finally, every time I import my pool, the machine panics and immediately reboots.

Steps taken:
I first did a dd from one disk to another to ensure no hardware issues - I didn't get any issues in the copy.
I then tried importing again, and failing on both disks.
I installed 1.9.0 - same problem.
I setup a VM in Parallels with Mojave 10.14.5 (my host OS). When I installed 1.9.0 on it, I saw the same kernel panicking.
I reverted to a Parallels snapshot before installing ZFS and re-installed using an older 1.8.2. Saw the same kernel panic.
I reverted to the pre-zfs snapshot again, and installed 1.7.2. THIS TIME IT WORKED! My pool imported!
I immediately uninstalled 1.9.0 from my main machine and installed 1.7.2 and my pool imported. HOWEVER after it running for just a few minutes, sure enough, the system panicked and rebooted,. From that point on, no matter what version of zfs I installed (down to 1.7.2) I couldn't get it to import without panic.
I then tried the DD'd copy I tried earlier. As before, on 1.7.2(but not on anything later) I COULD import it. So I did. I then let it sit a while, and it finally froze. After a long while, I saw that forcing a reboot would be the only way to gain control again, so I force-rebooted. After that, I can't get the dd'd copy to import either without a kernel panic.

Finally, I connected my drive to Linux using Ubuntu and the latests ZoL I could install. I would see the pool, but it wouldn't import it due to incompatible features.

I am now stuck with both drives unimportable. I still have my Mojave snapshot to pre-zfs and can install and get KP's. So here is the latest KP from installing 1.9.0:


Tue Jun 18 15:05:26 2019

*** Panic Report ***
panic(cpu 0 caller 0xffffff7f98ad018e): zfs: freeing free segment (offset=400276344832 size=12288)
Backtrace (CPU 0), Frame : Return Address
0xffffff887e35b5a0 : 0xffffff80163aea2d mach_kernel : _handle_debugger_trap + 0x47d
0xffffff887e35b5f0 : 0xffffff80164e9e95 mach_kernel : _kdp_i386_trap + 0x155
0xffffff887e35b630 : 0xffffff80164db70a mach_kernel : _kernel_trap + 0x50a
0xffffff887e35b6a0 : 0xffffff801635bb40 mach_kernel : _return_from_trap + 0xe0
0xffffff887e35b6c0 : 0xffffff80163ae447 mach_kernel : _panic_trap_to_debugger + 0x197
0xffffff887e35b7e0 : 0xffffff80163ae293 mach_kernel : _panic + 0x63
0xffffff887e35b850 : 0xffffff7f98ad018e net.lundman.spl : _vcmn_err + 0x8e
0xffffff887e35b970 : 0xffffff7f99d6808a net.lundman.zfs : _zfs_panic_recover + 0x6a
0xffffff887e35b9d0 : 0xffffff7f99d21c7e net.lundman.zfs : _range_tree_remove_impl + 0xe3
0xffffff887e35ba70 : 0xffffff7f99d39bce net.lundman.zfs : _space_map_load_callback + 0x74
0xffffff887e35ba90 : 0xffffff7f99d39629 net.lundman.zfs : _space_map_iterate + 0x270
0xffffff887e35bb30 : 0xffffff7f99d39b37 net.lundman.zfs : _space_map_load_length + 0x85
0xffffff887e35bb80 : 0xffffff7f99d1ba80 net.lundman.zfs : _metaslab_load + 0x90
0xffffff887e35bbb0 : 0xffffff7f99d200d5 net.lundman.zfs : _metaslab_activate + 0xe6
0xffffff887e35bbf0 : 0xffffff7f99d1dd43 net.lundman.zfs : _metaslab_alloc_dva + 0x7f8
0xffffff887e35bd00 : 0xffffff7f99d1f220 net.lundman.zfs : _metaslab_alloc + 0xa1
0xffffff887e35bd90 : 0xffffff7f99d9bcae net.lundman.zfs : _zio_dva_allocate + 0xf5
0xffffff887e35beb0 : 0xffffff7f99d98f1f net.lundman.zfs : ___zio_execute + 0xf1
0xffffff887e35bf10 : 0xffffff7f98adbed5 net.lundman.spl : _taskq_thread + 0x1ba
0xffffff887e35bfa0 : 0xffffff801635b0ce mach_kernel : _call_continuation + 0x2e
Kernel Extensions in backtrace:
net.lundman.spl(1.9)[C6A1ACB1-D9AD-3D42-B5F6-10EBD56F0850]@0xffffff7f98ace000-

# zpool import
pool: joelpool
id: 3034255057909438324
state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit 'zpool upgrade'.
config:

joelpool ONLINE
media-1E6F3E37-E145-E347-8472-459EA3EBE9FC ONLINE



Help!



Update:

I just tried it again installing an even EARLIER version (1.7.1) and again, it mounted! Now I'm afraid to leave it alone, so I'm desperately trying to copy data off.
Any ideas what I should try next? Ideally, I'd like to keep using ZFS (and this SSD.) I'd like to somehow get it back to a usable state.
joeljacobs
 
Posts: 2
Joined: Tue Jun 18, 2019 1:15 pm

Re: Kernel Panics on import since 1.8.2 on Mojave

Postby lundman » Tue Jun 18, 2019 3:44 pm

Looks like you have spacemap corruption of some kind, where it tries to free something already freed.

One option is to enable "sysctl kstat.zfs.darwin.tunable.zfs_recover=1" which will let you carry on past that error.
Then either run with it on - or copy your pool to a fresh install.

I believe you can also boot one of the illumos flavours, and run zdb command to repair it. But
I don't think OsX has a way to set "zfs_recovery" for the zdb process - unless you compile it on
yourself.

https://sigtar.com/2009/10/19/opensolar ... nel-panic/
User avatar
lundman
 
Posts: 622
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Kernel Panics on import since 1.8.2 on Mojave

Postby joeljacobs » Tue Jun 18, 2019 5:18 pm

Thank you SO much!

When I got the dd'd version to import on 1.7.1, I did a zfs list and discovered an older encrypted filesystem I was experimenting with a few months ago. It never mounts or asks to mount, as expected, so I forgot about it.
I removed that file system and a couple others (since I didn't need them), then proceeded to copy most of my data off.

Once I finished with that, I decided to try the original SSD again. This time it did NOT work with 1.7.1 or even 1.7.0. I guess it was a fluke that the DD one worked with the lower version.

Anyway, I then saw your post and tried adding the "sysctl kstat.zfs.darwin.tunable.zfs_recover=1" and it didn't work. But then I remembered I was still on 1.7.0, and switched to a new Parallels snapshot with 1.9.0 again. This time it worked!

I immediately deleted the same filesystems that I had done with the dd copy. I exported the pool, then imported it into my main OS (without the "sysctl kstat.zfs.darwin.tunable.zfs_recover=1" ) and it came up just fine!

So it appears that the encrypted filesystem might have been the problem. Though it could have also had something to do with space. Either way, I'm back up and running.

I can't thank you enough lundman!
joeljacobs
 
Posts: 2
Joined: Tue Jun 18, 2019 1:15 pm

Re: Kernel Panics on import since 1.8.2 on Mojave

Postby lundman » Wed Jun 19, 2019 3:56 pm

Phew that is a relief, good you have it running again.
User avatar
lundman
 
Posts: 622
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan


Return to General Help

Who is online

Users browsing this forum: No registered users and 2 guests