can't import new pool that disappeared after reboot

[kgreene]

After running zfs for years on my old 5,1 Mac Pro I finally got an m1 Mac Studio and an external nvme case (owc express 4m2 which holds 4 m2 ssds). I got 2 wd black 4tb sn850X nvme ssds for the case. Running Sonoma (14.3.1). zfs 2.2.2

% zpool version
zfs-2.2.2-1
zfs-kmod-2.2.2-1


After doing the various steps to get kernel extensions, etc loaded I created a mirror pool. Did various testing and different ashift values (12 was faster than 13 oddly) and mostly been testing it. This morning I rebooted and the pool didn't mount, I figured it was some auto mount issue and then went to manually mount it but it isn't showing up.

I tried all the
sudo zpool import -d /var/run/disk/by* foo
variants, with and without the pool name, -a -f, -d /dev, etc.

It just gives me 'no such pool available' or 'no pools available to import'

I'm wondering if this is an invariant id issue, I did create them using diskN diskM (not /dev/diskN). The ids on the devices appear to be gone. Even so, I'd still think I could import it as mentioned.

I've only copied a couple things to the drives to test so I can still wipe everything and restart if I didn't do something correctly. Mostly I'm surprised I can't seem to manually import it.

I don't think it's a kext loading issue, I manually loaded it just to make sure and it didn't make a difference.

[cgiard]

You may have already done it, but make sure you've given Terminal full disk access in System Settings | Privacy & Security | Full Disk Access. I've found it can't read the drives to find the partitions if you haven't done that.

[kgreene]

Thanks, yeah I had done that.

[jawbroken]

What do all the disks look like in "diskutil list"? I don't think using the diskn syntax is the issue because that's always how I have created my pools, e.g.

Code: Select all

zpool create -f -o ashift=12 -O casesensitivity=insensitive -O atime=off -O normalization=formD estorage raidz2 disk6 disk8 disk9 disk10 disk13 disk14 disk15 disk16 disk18 disk21 disk24 disk25

As to Full Disk Access, I've given that to bash, sh, zfs, zpool and zsh, but I don't know which of those are actually required (I believe at least one of the shell ones was required for automount to work, at one point).

[kgreene]

This seems wrong, at least I wouldn't expect an apfs partition to be sitting around on disk4s2, etc. I did format the disks as apfs first to do some performance testing. Maybe zfs didn't grab the whole device for some reason? It was working fine before reboot though. Before when I created them they were disk6 and disk7 I think.


/dev/disk4 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *4.0 TB disk4
1: EFI EFI 209.7 MB disk4s1
2: Apple_APFS 4.0 TB disk4s2

/dev/disk5 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *4.0 TB disk5
1: EFI EFI 209.7 MB disk5s1
2: Apple_APFS 4.0 TB disk5s2


diskutil info output:
**********

Device Identifier: disk4
Device Node: /dev/disk4
Whole: Yes
Part of Whole: disk4
Device / Media Name: WD_BLACK SN850X 4000GB

Volume Name: Not applicable (no file system)
Mounted: Not applicable (no file system)
File System: None

Content (IOContent): GUID_partition_scheme
OS Can Be Installed: No
Media Type: Generic
Protocol: PCI-Express
SMART Status: Verified

Disk Size: 4.0 TB (4000787030016 Bytes) (exactly 7814037168 512-Byte-Units)
Device Block Size: 512 Bytes

Media OS Use Only: No
Media Read-Only: No
Volume Read-Only: Not applicable (no file system)

Device Location: External
Removable Media: Fixed

Solid State: Yes
Virtual: No
Hardware AES Support: No

**********

Device Identifier: disk4s1
Device Node: /dev/disk4s1
Whole: No
Part of Whole: disk4

Volume Name: EFI
Mounted: No

Partition Type: EFI
File System Personality: MS-DOS FAT32
Type (Bundle): msdos
Name (User Visible): MS-DOS (FAT32)

OS Can Be Installed: No
Media Type: Generic
Protocol: PCI-Express
SMART Status: Verified
Volume UUID: 0E239BC6-F960-3107-89CF-1C97F78BB46B
Disk / Partition UUID: BD6643C8-023B-4506-9F0D-C35D2BFFD31B
Partition Offset: 20480 Bytes (40 512-Byte-Device-Blocks)

Disk Size: 209.7 MB (209715200 Bytes) (exactly 409600 512-Byte-Units)
Device Block Size: 512 Bytes

Volume Total Space: 0 B (0 Bytes) (exactly 0 512-Byte-Units)
Volume Free Space: 0 B (0 Bytes) (exactly 0 512-Byte-Units)

Media OS Use Only: No
Media Read-Only: No
Volume Read-Only: Not applicable (not mounted)

Device Location: External
Removable Media: Fixed

Solid State: Yes
Hardware AES Support: No

**********

Device Identifier: disk4s2
Device Node: /dev/disk4s2
Whole: No
Part of Whole: disk4

Volume Name: Not applicable (no file system)
Mounted: Not applicable (no file system)
File System: None

Partition Type: Apple_APFS
OS Can Be Installed: No
Media Type: Generic
Protocol: PCI-Express
SMART Status: Verified
Disk / Partition UUID: 762C5BC1-7843-4509-9E43-6DFA180BB208
Partition Offset: 209735680 Bytes (409640 512-Byte-Device-Blocks)

Disk Size: 4.0 TB (4000577273856 Bytes) (exactly 7813627488 512-Byte-Units)
Device Block Size: 512 Bytes

Media OS Use Only: No
Media Read-Only: No
Volume Read-Only: Not applicable (no file system)

Device Location: External
Removable Media: Fixed

Solid State: Yes
Hardware AES Support: No

This disk is an APFS Physical Store. APFS Information:
Unavailable

**********

Device Identifier: disk5
Device Node: /dev/disk5
Whole: Yes
Part of Whole: disk5
Device / Media Name: WD_BLACK SN850X 4000GB

Volume Name: Not applicable (no file system)
Mounted: Not applicable (no file system)
File System: None

Content (IOContent): GUID_partition_scheme
OS Can Be Installed: No
Media Type: Generic
Protocol: PCI-Express
SMART Status: Verified

Disk Size: 4.0 TB (4000787030016 Bytes) (exactly 7814037168 512-Byte-Units)
Device Block Size: 512 Bytes

Media OS Use Only: No
Media Read-Only: No
Volume Read-Only: Not applicable (no file system)

Device Location: External
Removable Media: Fixed

Solid State: Yes
Virtual: No
Hardware AES Support: No

**********

Device Identifier: disk5s1
Device Node: /dev/disk5s1
Whole: No
Part of Whole: disk5

Volume Name: EFI
Mounted: No

Partition Type: EFI
File System Personality: MS-DOS FAT32
Type (Bundle): msdos
Name (User Visible): MS-DOS (FAT32)

OS Can Be Installed: No
Media Type: Generic
Protocol: PCI-Express
SMART Status: Verified
Volume UUID: 0E239BC6-F960-3107-89CF-1C97F78BB46B
Disk / Partition UUID: 958E33DA-E941-4C89-8F5C-F6A1EA52D52D
Partition Offset: 20480 Bytes (40 512-Byte-Device-Blocks)

Disk Size: 209.7 MB (209715200 Bytes) (exactly 409600 512-Byte-Units)
Device Block Size: 512 Bytes

Volume Total Space: 0 B (0 Bytes) (exactly 0 512-Byte-Units)
Volume Free Space: 0 B (0 Bytes) (exactly 0 512-Byte-Units)

Media OS Use Only: No
Media Read-Only: No
Volume Read-Only: Not applicable (not mounted)

Device Location: External
Removable Media: Fixed

Solid State: Yes
Hardware AES Support: No

**********

Device Identifier: disk5s2
Device Node: /dev/disk5s2
Whole: No
Part of Whole: disk5

Volume Name: Not applicable (no file system)
Mounted: Not applicable (no file system)
File System: None

Partition Type: Apple_APFS
OS Can Be Installed: No
Media Type: Generic
Protocol: PCI-Express
SMART Status: Verified
Disk / Partition UUID: 83017D8A-4988-498C-9FA1-278FCFA4908F
Partition Offset: 209735680 Bytes (409640 512-Byte-Device-Blocks)

Disk Size: 4.0 TB (4000577273856 Bytes) (exactly 7813627488 512-Byte-Units)
Device Block Size: 512 Bytes

Media OS Use Only: No
Media Read-Only: No
Volume Read-Only: Not applicable (no file system)

Device Location: External
Removable Media: Fixed

Solid State: Yes
Hardware AES Support: No

This disk is an APFS Physical Store. APFS Information:
Unavailable

[jawbroken]

Yeah, that doesn't look right at all, I don't see any sign of a ZFS pool there. Mine all look something like:

Code: Select all

/dev/disk6 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *22.0 TB    disk6
   1:                        ZFS estorage                22.0 TB    disk6s1
   2: [UUID Removed]                                     8.4 MB     disk6s9


Do you have the history of commands you entered to get into this state?

[cgiard]

Just to verify, when you created the pool, did you first change the partition type to ZFS? it's technically possible to just partition a drive, leave the partition type as whatever the default is, then manually setup zfs by specifically pointing to the appropriate partitions. You could try running the following, especially since diskutil says they don't have filesystems on them.

Code: Select all

zpool import -d /dev/disk4s2 -d /dev/disk5s2 foo

[kgreene]

OK, I never manually set a partition to ZFS or anything. That seems the most likely culprit? I don't remember doing that years ago but maybe it's changed or maybe I forgot.

I wasn't trying to set a partition type at all, I assumed zfs would have done anything like that during pool creation when giving it the whole device. Maybe just saying disk5 isn't enough to have it use the entire device?

I did try mounting as you suggested which didn't work.

Basically, I first formatted both ssds as apfs so I could run some perf numbers to compare with zfs later.

Then I did a zpool create with both disks (I had originally dried to give it the UUID but had forgotten you need to use the media-* name not just the raw UUID from diskinfo so went back to using diskN as it couldn't find the UUID that way):


2024-02-18.17:20:25 zpool create -f -o ashift=12 -O normalization=formD -O compression=lz4 -O atime=off foo mirror disk6 disk7

I did a bunch of performance tests then destroyed it and created an ashift 13 to test:

% sudo zpool destroy foo
% zpool list
no pools available
% sudo zpool create -f -o ashift=13 -O normalization=formD -O compression=lz4 -O atime=off foo mirror disk6 disk7

iostat and another utility were generally showing better perform with ashift 12 so I destroyed 13 and recreated 12.

% sudo zpool create -f -o ashift=12 -O normalization=formD -O compression=lz4 -O atime=off foo mirror disk6 disk7

I did some test copying, created and destroyed a dataset, and decide to turn atime back on though I don't see how that could have messed it up.

When I rebooted the disk ids changed and nothing came up, the state I'm in now. None of the many manual attempts to mount it has worked.



My main question now, what is the partition supposed to look like? I could try to change the type to ZFS if it lets me but if it's somehow not partitioning it as expected I want to just fix that and start over.

What is the recommend way to manually set the ZFS partition type?

[cgiard]

I've forgotten the commands I used back when I created my dataset, but I think I installed gdisk (https://www.rodsbooks.com/gdisk/) or something similar from Homebrew and used that to partition the drives.

[kgreene]

I'm thinking at this point I should probably just erase the ssds, diskutil does have the zfs dataset option so I assume that's the easiest way. This is most likely just user error with not erasing the disk again and thinking that zfs would wipe it all for me.

However, I'd be interested if there was any other ideas to forcibly mount/import the zfs pool. It kind of surprises me that it worked fine before reboot with multiple create/destroys and now I can't get access to it.

Don't think I'd keep it around anyway but I'd be happy to do something risky just as a test.