2.1.6.rc3 immediate panic on scrub

Developer discussions.

2.1.6.rc3 immediate panic on scrub

Postby pgdh » Thu Oct 27, 2022 4:05 am

I'm running this Ventura 13.0 (Edit: on arm64) with ...

Code: Select all
zfs-macOS-2.1.6-3_g84dbe5990
zfs-kmod-2.1.6-3_g84dbe5990


I have a pool with these features ...

Code: Select all
tank  feature@async_destroy          enabled                        local
tank  feature@empty_bpobj            active                         local
tank  feature@lz4_compress           active                         local
tank  feature@multi_vdev_crash_dump  enabled                        local
tank  feature@spacemap_histogram     active                         local
tank  feature@enabled_txg            active                         local
tank  feature@hole_birth             active                         local
tank  feature@extensible_dataset     active                         local
tank  feature@embedded_data          active                         local
tank  feature@bookmarks              enabled                        local
tank  feature@filesystem_limits      enabled                        local
tank  feature@large_blocks           enabled                        local
tank  feature@large_dnode            enabled                        local
tank  feature@sha512                 enabled                        local
tank  feature@skein                  enabled                        local
tank  feature@edonr                  enabled                        local
tank  feature@userobj_accounting     active                         local
tank  feature@encryption             enabled                        local
tank  feature@project_quota          active                         local
tank  feature@device_removal         enabled                        local
tank  feature@obsolete_counts        enabled                        local
tank  feature@zpool_checkpoint       enabled                        local
tank  feature@spacemap_v2            active                         local
tank  feature@allocation_classes     enabled                        local
tank  feature@resilver_defer         enabled                        local
tank  feature@bookmark_v2            enabled                        local
tank  feature@redaction_bookmarks    enabled                        local
tank  feature@redacted_datasets      enabled                        local
tank  feature@bookmark_written       enabled                        local
tank  feature@log_spacemap           active                         local
tank  feature@livelist               enabled                        local
tank  feature@device_rebuild         enabled                        local
tank  feature@zstd_compress          enabled                        local
tank  feature@draid                  enabled                        local
tank  feature@zilsaxattr             disabled                       local
tank  feature@head_errlog            disabled                       local
tank  feature@blake3                 disabled                       local


i.e. I have not upgraded it yet

It import, and seems to work for reads (haven't tested writes yet) but panics as soon as I start a 'zpool scrub'. Panic report output attached.

If I reinstall 2.1.0-1 from the BigSur package (and do the trigger-panic medic dance), the scrub completes without error.

Edit: I get upgrade advice from 'zpool status', but nothing ominous. The pool is a single Samsung T5 1TB drive connected via USB3 over USBC and 95% full. The machine is a first generation M1 MacBook Pro 13.
Attachments
panic.txt
(17.27 KiB) Downloaded 160 times
Last edited by pgdh on Thu Oct 27, 2022 9:19 am, edited 2 times in total.
pgdh
 
Posts: 12
Joined: Fri Jun 26, 2015 3:58 am

Re: 2.1.6.rc3 panic on scrub

Postby Jimbo » Thu Oct 27, 2022 4:49 am

No panic here (scrub in progress). Ventura 13.0 x86:

Code: Select all
% zpool version
zfs-macOS-2.1.6-3_g84dbe5990
zfs-kmod-2.1.6-3_g84dbe5990

% sudo zpool status -v
  pool: dataroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
   The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
   the pool may no longer be accessible by software that does not support
   the features. See zpool-features(7) for details.
  scan: scrub in progress since Thu Oct 27 20:39:32 2022
   573G scanned at 5.03G/s, 64.0G issued at 575M/s, 573G total
   0B repaired, 11.17% done, 00:15:06 to go
config:

   NAME                                          STATE     READ WRITE CKSUM
   dataroot                                      ONLINE       0     0     0
     media-7D0140E7-C5A1-4543-A202-0499BE18210E  ONLINE       0     0     0

errors: No known data errors

  pool: workroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
   The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
   the pool may no longer be accessible by software that does not support
   the features. See zpool-features(7) for details.
  scan: scrub in progress since Thu Oct 27 20:39:24 2022
   471G scanned at 3.86G/s, 43.5G issued at 365M/s, 471G total
   0B repaired, 9.23% done, 00:19:59 to go
config:

   NAME                                          STATE     READ WRITE CKSUM
   workroot                                      ONLINE       0     0     0
     media-E789C13E-4845-C340-A373-50EB43B3BFD3  ONLINE       0     0     0

errors: No known data errors


Features:
Code: Select all
workroot  feature@async_destroy          enabled                        local
workroot  feature@empty_bpobj            active                         local
workroot  feature@lz4_compress           active                         local
workroot  feature@multi_vdev_crash_dump  enabled                        local
workroot  feature@spacemap_histogram     active                         local
workroot  feature@enabled_txg            active                         local
workroot  feature@hole_birth             active                         local
workroot  feature@extensible_dataset     active                         local
workroot  feature@embedded_data          active                         local
workroot  feature@bookmarks              enabled                        local
workroot  feature@filesystem_limits      enabled                        local
workroot  feature@large_blocks           enabled                        local
workroot  feature@large_dnode            enabled                        local
workroot  feature@sha512                 enabled                        local
workroot  feature@skein                  enabled                        local
workroot  feature@edonr                  enabled                        local
workroot  feature@userobj_accounting     active                         local
workroot  feature@encryption             active                         local
workroot  feature@project_quota          active                         local
workroot  feature@device_removal         enabled                        local
workroot  feature@obsolete_counts        enabled                        local
workroot  feature@zpool_checkpoint       enabled                        local
workroot  feature@spacemap_v2            active                         local
workroot  feature@allocation_classes     enabled                        local
workroot  feature@resilver_defer         enabled                        local
workroot  feature@bookmark_v2            enabled                        local
workroot  feature@redaction_bookmarks    enabled                        local
workroot  feature@redacted_datasets      enabled                        local
workroot  feature@bookmark_written       enabled                        local
workroot  feature@log_spacemap           active                         local
workroot  feature@livelist               enabled                        local
workroot  feature@device_rebuild         enabled                        local
workroot  feature@zstd_compress          enabled                        local
workroot  feature@draid                  enabled                        local
workroot  feature@zilsaxattr             disabled                       local
workroot  feature@head_errlog            disabled                       local
workroot  feature@blake3                 disabled                       local


No issues show with zpool status -v?

Edit: I should add that these are single disk pools, so no mirror or raidz vdevs.
Jimbo
 
Posts: 149
Joined: Sun Sep 17, 2017 5:12 am

Re: 2.1.6.rc3 immediate panic on scrub

Postby lundman » Thu Oct 27, 2022 2:43 pm

Invalid kernel stack pointer (probable overflow).

Interesting, thanks for reporting it, probably related to arm64 in this case.
You can avoid this by doubling the kernel stack space, but it is better for us
to fix it in the code. I'll make sure the stack-limiting work is active.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: 2.1.6.rc3 immediate panic on scrub

Postby lundman » Thu Oct 27, 2022 4:43 pm

Ah yep, I see it. On Ventura/arm64 it was compiled -O0 (no optimization) for debugging reasons, but the Makefile has
a special line to change it to -O2 for dsl_scan.c. The reason for this is we know dsl_scan.c is heavy on the stack, so it
must always use -O2. Unfortunately, the Makefile fix was incorrect - upstream changed how all Makefiles work, and
that work was not reflected.

I have fixed the -O0 -> -O2 mapping, and, I have changed Ventura/arm64 default compile to -O2. Double fixed if you will.
All other builds use -O2 anyway.

If you have the patience, could you try:

OpenZFSonOsX-2.1.6rc3-Ventura-13-arm64.pkg
(19.76 MiB) Downloaded 156 times


bcca9c1a22334d6e0f9fc84dccd12f57 OpenZFSonOsX-2.1.6rc3-Ventura-13-arm64.pkg
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: 2.1.6.rc3 immediate panic on scrub

Postby pgdh » Thu Oct 27, 2022 9:01 pm

lundman wrote:If you have the patience, could you try:

OpenZFSonOsX-2.1.6rc3-Ventura-13-arm64.pkg

bcca9c1a22334d6e0f9fc84dccd12f57 OpenZFSonOsX-2.1.6rc3-Ventura-13-arm64.pkg


Yes, that does it. Thanks!

Code: Select all
# zfs version
zfs-macOS-2.1.6-5_g5917e9b58
zfs-kmod-2.1.6-5_g5917e9b58
#
pgdh
 
Posts: 12
Joined: Fri Jun 26, 2015 3:58 am


Return to OpenZFS on OS X Development

Who is online

Users browsing this forum: No registered users and 8 guests