ARM64 Mac Mini Panic on zfs receive or large total write

Developer discussions.

ARM64 Mac Mini Panic on zfs receive or large total write

Postby monkeyvoodoo » Fri May 07, 2021 1:20 am

ARM64 (Mac Mini) kernel panic instantly on zfs receive, or after anywhere between 80GB-400GB write. This is on a 4-disk pool on a USB-C enclosure that exposes a SATA bridge for each device.

I'm not sure what I can provide to help with this, but it's instantly reproducible with zfs receive, and though the total write amount varies, writing too much to the pool definitely causes kernel panic/reboot. I've also noticed after about 40GB or so of write, the speed gets ridiculously low (<100KiB/s).
monkeyvoodoo
 
Posts: 12
Joined: Tue Oct 22, 2019 3:27 pm

Re: ARM64 Mac Mini Panic on zfs receive or large total write

Postby lundman » Mon May 10, 2021 11:34 pm

I've just spent 3 days debugging and fixing both zfs send and zfs recv, so there is a good chance that it might be fixed. Although, not sure they would cause a panic.

You can look in /Library/Logs/Diagno* and see if you have any *.panic files.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: ARM64 Mac Mini Panic on zfs receive or large total write

Postby monkeyvoodoo » Tue May 18, 2021 3:16 pm

Still getting an insta-crash on my M1 Mac Mini with rc7 when I do a ZFS receive from an Ubuntu system. I do have a .panic file at /Library/Logs/DiagnosticReports/panic-full-2021-05-18-153600.0002.panic. I'm attaching that. Dunno what else I can do to help. So far I haven't had it panic due to read/write throughput, but will update if that happens.
Attachments
panic-full-2021-05-18-153600.0002.panic.txt
Panic! At the Disco?
(924.55 KiB) Downloaded 287 times
monkeyvoodoo
 
Posts: 12
Joined: Tue Oct 22, 2019 3:27 pm

Re: ARM64 Mac Mini Panic on zfs receive or large total write

Postby monkeyvoodoo » Tue May 18, 2021 3:30 pm

Just did it again from a clean reboot, and interestingly, this time, the pool wasn't even imported. It definitely was imported the last time. The pool doesn't auto-import at boot, and I'd forgotten this time around. So I tried a couple more times, both with pool imported and not, and it always crashes and reboots the system. I rm'd the .panic files from theDiagnosticReports folder, and ran it again after a clean boot without the pool imported. Attached is the .panic file from that.

Command I ran from the Ubuntu system was

Code: Select all
sudo zfs send rpool/swap@test | pv -ratb | ssh monkey@nao.local 'sudo /usr/local/zfs/bin/zfs receive storage/swap'


This seems to have instantly frozen the system, and it rebooted. Attached is the .panic file from that.

I can attach the disks in this pool to an ARM Ubuntu 21.04 system, import the pool there, and zfs receive works fine.

Anyways, cheers! o/
Attachments
panic-full-2021-05-18-182012.000.panic.txt
Wew lad.
(927.92 KiB) Downloaded 281 times
monkeyvoodoo
 
Posts: 12
Joined: Tue Oct 22, 2019 3:27 pm

Re: ARM64 Mac Mini Panic on zfs receive or large total write

Postby lundman » Tue May 18, 2021 6:26 pm

Is it possible to turn keepsym=1 ? When I try to lookup the symbols I just get weirdness:

Code: Select all
# atos -o /Library/Extensions/zfs.kext/Contents/MacOS/zfs -arch arm64e -l 0xfffffe001a460000 0xfffffe001b43e920 0xfffffe001b43e704 0xfffffe001b567338 0xfffffe001b558b98 0xfffffe001b3f77e8 0xfffffe001b43e394 0xfffffe001b43e394 0xfffffe001bbee930 0xfffffe001bbf2dd0 0xfffffe001b55af74 0xfffffe001b558edc 0xfffffe001b3f77e8 0xfffffe001a6889d4 0xfffffe001a54eb60 0xfffffe001a55a4c8 0xfffffe001a5592b0 0xfffffe001a551c50 0xfffffe001a55b798 0xfffffe001b6bf834


vmem_init.initial_default_block (in zfs) + 10316064
vmem_init.initial_default_block (in zfs) + 10315524
vmem_init.initial_default_block (in zfs) + 11531064
vmem_init.initial_default_block (in zfs) + 11471768
vmem_init.initial_default_block (in zfs) + 10024936
vmem_init.initial_default_block (in zfs) + 10314644
vmem_init.initial_default_block (in zfs) + 10314644
0xfffffe001bbee930
0xfffffe001bbf2dd0
vmem_init.initial_default_block (in zfs) + 11480948
vmem_init.initial_default_block (in zfs) + 11472604
vmem_init.initial_default_block (in zfs) + 10024936
ZSTD_compressBlock_btopt (in zfs) + 140
dsl_dataset_get_holds (in zfs) (dsl_userhold.c:677)
handle_media_terminate_cb(void*, void*, IOService*, IONotifier*) (in zfs) (ldi_iokit.cpp:258)
ldi_open_media_by_path (in zfs) (ldi_iokit.cpp:1532)
hkdf_sha512 (in zfs) (hkdf.c:160)
handle_unmap_iokit (in zfs) (ldi_iokit.cpp:2008)
vmem_init.initial_default_block (in zfs) + 12941364


Using the load address and stack addresses from your panic log.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: ARM64 Mac Mini Panic on zfs receive or large total write

Postby lundman » Wed May 19, 2021 11:07 pm

OK tried it on my own MBP, and yes, zfs send is very easy to trigger. But I can not get any symbols from it, nor am I able to connect to the
debugger. Quite disappointing. Not entirely sure how to find out what goes wrong.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: ARM64 Mac Mini Panic on zfs receive or large total write

Postby lundman » Fri May 21, 2021 9:55 pm

Fixed that panic, and posted new build.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: ARM64 Mac Mini Panic on zfs receive or large total write

Postby monkeyvoodoo » Tue May 25, 2021 9:57 am

I can confirm this fixed things! Thanks!
monkeyvoodoo
 
Posts: 12
Joined: Tue Oct 22, 2019 3:27 pm


Return to OpenZFS on OS X Development

Who is online

Users browsing this forum: No registered users and 18 guests

cron