panic immediately on mount on ventura

[jollyjinx]

I've created and used a pool on ubuntu lts 20.4 on a raspberry and used it the last years.
Now one of the disks failed and I can't mount the pool any more.
The pool is a ( 4 x 5TB usb disks with raid z2, one disk failed) and I thought I would use it on my real machine a mac book pro.

But on the mac the problem is even harder - as soon as I try to import the pool macOS panics. Machine is running the latest ventura, m1-max 32gb.

Any suggestions ?

patrick aka jolly

[jollyjinx]

Mounting readonly:

zpool import -N -o readonly=on -f -F poolname

I was able to mount the pool on macos.

[jollyjinx]

well I thought it mounted - but it did only show me a degraded zpool on issuing that command. zpool list was clear afterwards.

I've now installed freebsd on a raspberry and was able to issue the same command, and there it stayed so zpool list did list the pool. But as soon as I mounted the thing not readonly it kernel panics ? - at least does not longer react on the network and then boots. Weird.

[jollyjinx]

So the current state is as follows:

I can import the pool in readonly mode and there with zpool status it lists me the degraded pool and 3693 files which are broken. But I can't run a scrub in readonly mode nor can I do more with that information.

Importing it on freedbsd 13.2 and macOS ventura zfs-macOS-2.1.7-rc3 it will panic the machine , on linux ubuntu 20-4 and ubuntu 22-4 it will hang zpool and zfs related commands but the machine itself stays usable.

Using zdb works, but currently zdb on all three os will not support -B switch, which would allow me to export part of the pool.

I really thought that using 4 disks with raid-z2 would help me until three disks fail, but now it's not even usable with just one disk fail ?

Is there any way for me to get the data from the zpool ? which os does support zdb -B as described on https://openzfs.github.io/openzfs-docs/man/8/zdb.8.html which runs on a raspberry ? (my only computer besides the mac)

[FadingIntoBlue]

Is there any way for me to get the data from the zpool ?

Possibly a silly question, have you removed the failed disk from the pool and tried to mount it in its absence?

[jollyjinx]

Yes, the broken disk is no longer connected and the remaining disks work find reading their contents with dd .

Btw. the pool looks like this (now connected on a linux machine for the dump)

Code: Select all

root@ubuntu:/home/ubuntu# zpool status
  pool: ocean
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
   continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Jul  1 10:01:24 2023
   0B / 11.6T scanned, 0B / 11.6T issued
   0B resilvered, 0.00% done, no estimated completion time
config:

   NAME                      STATE     READ WRITE CKSUM
   ocean                     DEGRADED     0     0     0
     raidz2-0                DEGRADED     0     0     0
       18276338438250889807  FAULTED      0     0     0  was /dev/sdd2
       sdc1                  ONLINE       0     0     0
       sdb2                  ONLINE       0     0     0
       sdd2                  ONLINE       0     0     0

errors: 3962 data errors, use '-v' for a list

And no the pool is not being resilvered as it's in readonly mode:

Code: Select all

root@ubuntu:/home/ubuntu# zpool iostat -v 1
                            capacity     operations     bandwidth
pool                      alloc   free   read  write   read  write
------------------------  -----  -----  -----  -----  -----  -----
ocean                         0  18.1T     78      0  1.28M      0
  raidz2-0                    0  18.1T     78      0  1.28M      0
    18276338438250889807      -      -      1      0  74.7K      0
    sdc1                      -      -     23      0   406K      0
    sdb2                      -      -     28      0   418K      0
    sdd2                      -      -     26      0   408K      0
------------------------  -----  -----  -----  -----  -----  -----
                            capacity     operations     bandwidth
pool                      alloc   free   read  write   read  write
------------------------  -----  -----  -----  -----  -----  -----
ocean                         0  18.1T      0      0      0      0
  raidz2-0                    0  18.1T      0      0      0      0
    18276338438250889807      -      -      0      0      0      0
    sdc1                      -      -      0      0      0      0
    sdb2                      -      -      0      0      0      0
    sdd2                      -      -      0      0      0      0
------------------------  -----  -----  -----  -----  -----  -----


[FadingIntoBlue]

The faulted drive is no longer physically present, but still referred to? Can you use zpool offline to take that offline and then try to mount the pool.

What happens if you import it without trying to mount it, can you run a scrub or a resilver on it like that? Or maybe import and if it auto mounts, explicitly unmount it.

[jollyjinx]

The problem is that importing without readonly will panic the machine. (and yes I tried the no mount option)

I'm currently trying it with the current zfs version to get back things with the new zdb --backup switch.
I installed ubuntu on a new stick , compiled the current version of zfs and try it right now. Let's see how it works.

[jollyjinx]

Just a quick update as it's starting to get better.

I

Code: Select all

root@ubuntu:/home/ubuntu# zdb -u -e ocean

Uberblock:
magic = 0000000000bab10c
version = 5000
txg = 18360509
guid_sum = 15765822683839940218
timestamp = 1688241822 UTC = Sat Jul  1 20:03:42 2023
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0


root@ubuntu:/home/ubuntu# zdb -dP -e ocean
Dataset mos [META], ID 0, cr_txg 4, 1092430080, 6633 objects
Dataset ocean/tmtinkerbell@2023-06-24T00:00:01 [ZPL], ID 5114, cr_txg 18302756, 1111802459136, 24930 objects
.
.
.
Dataset ocean/media [ZPL], ID 527, cr_txg 105, 64094481024, 223493 objects
Dataset ocean/movies [ZPL], ID 772, cr_txg 1430, 4065675030144, 9979 objects
.
.
.
dmu_bonus_hold(os, object, dl, &dl->dl_dbuf) == 0 (0x34 == 0)
ASSERT at module/zfs/dsl_deadlist.c:308:dsl_deadlist_open()Aborted (core dumped)



Now I'm getting media off the zpool with the following command:

Code: Select all

root@ubuntu:/home/ubuntu# zdb -B -e ocean/527 Lecw | zfs receive -F newocean/media


Right now it seems to work as it's starting to appear on newocean. But It's not finished yet.

[jollyjinx]

zdb -B worked fine on ocean/527  (media) but failed on ocean/722 (movies).

I was able to issue

Code: Select all

zdb -ddddd -K $pass  -e ocean/movies 5268
Unlocked encryption root: ocean/movies
Dataset ocean/movies [ZPL], ID 772, cr_txg 1430, 3.70T, 9979 objects, rootbp DVA[0]=<0:2400014a000:3000> DVA[1]=<0:f20081e4000:3000> [L0 DMU objset] fletcher4 uncompressed authenticated LE contiguous unique double size=800L/800P birth=18360433L/18360433P fill=9979 cksum=00000021c040a809:000025b93e7be9da:00190442ec3d9de4:0c1fdfcfea494199

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
      5268    1   128K   105K  40.5K     512   105K  100.00  ZFS plain file
                                               168   bonus  System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path   /Pictures/Testbilder/NecPattern.png
uid     1000
gid     1000
atime   Tue Dec 31 08:24:53 2019
mtime   Thu Feb 11 09:05:10 2016
ctime   Wed Jan 27 17:59:02 2021
crtime   Tue Jan 26 20:40:27 2021
gen   4641518
mode   100644
size   107030
parent   5267
links   1
pflags   40800000004
Indirect blocks:
               0 L0 0:10a1263d2000:15000 1a400L/9000P F=1 B=4641518/4641518 cksum=246684c5894cb36e:ecb92ef5e414d48a:13123f3d0f3e78df:9619bdcef80cd664

segment [0000000000000000, 000000000001a400) size  105K


 but using 

Code: Select all

zdb -R -K $pass -e ocean 0:10a1263d2000:15000:d |head
or
zdb -R -K $pass -e ocean/movies 0:10a1263d2000:15000:d |head


did not get me the data but something completly other from the disk.

I tried that on mac as well ( as I thought maybe the raspberry , even though running in 64bit maybe had a problem parsing 64 bit numbers ) , but the same incorrect result.

( The file starts with a PNG header) as I still have that file that's why I use the |head command.