Short version:
- 2.0.0 on Catalina shows the same problems as I was having on Big Sur.
- I was wrong (sorry!), raw dd data is correct so the core of the ZFS extension is working right
- Catalina (and BigSur) are having problems "using" some of the virtual disks.
Process followed:
- I run a scrub of the pool overnight (on 1.9.4).
- Grabbed the output shown below on 1.9.4.
- Exported the pool, uninstalled 1.9.4, rebooted
- Installed 2.0.0 (still on Catalina) and rebooted
- Imported the pool and grabbed the output.
- Exported the pool, reverted back to 1.9.4, and presently re-scrubbing the pool.
ZFS setup:
- I have one large pool and on that pool 3 volumes.
- I use the native zfs filesystem for most data but...
- Have 3 volumes which host JHFS+ filesystems (so timemachine, photos, etc. all work the way they prefer to work)
Observations under 2.0.0 vs 1.9.4:
- One of the volumes never shows in diskutil list under 2.0.0 (all three show under 1.9.4)
- The first three 512 byte blocks of the two volumes that do show up as /dev/disk entries are identical between 1.9.4 and 2.0.0. I looked at the last few blocks of each volume along with the first few blocks of the JHFS+ partition as well and they are correct/consistent between 1.9.4 and 2.0.0.
- One of the volumes that does have a /dev/disk entry under 2.0.0 does not show a partition table or partitions/filesystems via diskutil list and can't be mounted by OS X (yet works fine under 1.9.4) despite having the same blocks.
- Looking at that volume using gpt shows it has a valid partition (while diskutil list say it does not)
- diskutil info on that disk entry says "IOContent: None" vs under 1.9.4 where is says "IOContent: GUID_partition_scheme".
Hypothesis:
- The code that maps /dev/disk blocks to the backing ZFS volume works properly (as tools like gpt or dd which examine the /dev/disk raw blocks work.)
- Somehow something isn't being set that OS X looks for when more than one volume exists. One volume does properly show and mount, one completely fails to have any /dev/disk entry at all, and one has a /dev/disk entry that's backed by real, valid, data when accessed at the block level, but diskutil insists no partitions exist.
- This is supported by the different output the one "partially" there disk provides for a "diskutil info" under 1.9.4 vs 2.0.0.
- Under 1.9.4 the output includes: "Content (IOContent): GUID_partition_scheme."
- Under 2.0.0 the output includes: "Content (IOContent): None"
Let me know what else I can try to look at to assist in tracking down why this happens.
Under Catalina, version 1.9.4- Code: Select all
bash-3.2# uname -v
Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64
bash-3.2# kextstat | grep lund
73 1 0xffffff7f8363b000 0x4fc 0x4fc net.lundman.kernel.dependencies.33 (12.5.0) 103D4746-0E59-43E9-BCC9-8A165E0EC2B1
74 1 0xffffff7f8363c000 0x11f6000 0x11f6000 net.lundman.spl (1.9.4) F4C8FC99-7C45-3C5A-BE70-A23D3A03170D <73 8 6 5 3 1>
75 1 0xffffff7f84832000 0x3e3000 0x3e3000 net.lundman.zfs (1.9.4) 1C090B5F-A844-3606-A15B-2AA004AFAD8A <74 28 8 6 5 3 1>
The virtual disks, after zpool import, are:
- Code: Select all
bash-3.2# diskutil list
(... physical disks removed ...)
/dev/disk11 (internal, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +54.7 TB disk11
1: ZFS Dataset zfsPool 54.7 TB disk11s1
/dev/disk12 (external, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +1.6 TB disk12
1: EFI EFI 209.7 MB disk12s1
2: Apple_HFS Photos & Videos 1.6 TB disk12s2
/dev/disk13 (external, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +5.5 TB disk13
1: EFI EFI 209.7 MB disk13s1
2: Apple_HFS Network Time Machine 5.5 TB disk13s2
/dev/disk14 (external, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +1.1 TB disk14
1: EFI EFI 209.7 MB disk14s1
2: Apple_CoreStorage Local Time Machine 1.1 TB disk14s2
3: Apple_Boot Boot OS X 134.2 MB disk14s3
Each of the volumes (disk12, disk13, and disk14) show with partitions and the first few blocks do show the EFI partition and right GPT headers.
- Code: Select all
bash-3.2# dd if=/dev/disk12 conv=notrunc bs=512 count=3 | hexdump -C
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................|
000001c0 ff ff ee fe ff ff 01 00 00 00 ff ff ff bf 00 00 |................|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00000210 40 04 a3 5c 00 00 00 00 01 00 00 00 00 00 00 00 |@..\............|
00000220 ff ff ff bf 00 00 00 00 22 00 00 00 00 00 00 00 |........".......|
00000230 de ff ff bf 00 00 00 00 06 17 bc aa 3b 24 61 4a |............;$aJ|
00000240 8d cb 4f a0 e3 ad 82 80 02 00 00 00 00 00 00 00 |..O.............|
00000250 80 00 00 00 80 00 00 00 da 6c b8 ac 00 00 00 00 |.........l......|
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;|
00000410 68 bf 49 d4 61 e8 97 4e 95 42 94 47 e9 94 cf 6f |h.I.a..N.B.G...o|
00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......|
00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .|
00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.|
00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.|
00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000480 00 53 46 48 00 00 aa 11 aa 11 00 30 65 43 ec ac |.SFH.......0eC..|
00000490 81 ed 78 88 2c 31 15 4d b4 fc 76 7e 0e 5a d9 4b |..x.,1.M..v~.Z.K|
000004a0 28 40 06 00 00 00 00 00 d7 ff fb bf 00 00 00 00 |(@..............|
000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000600
bash-3.2# dd if=/dev/disk13 conv=notrunc bs=512 count=3 | hexdump -C
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
1536 bytes transferred in 0.000135 secs (11382422 bytes/sec)
000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................|
000001c0 ff ff ee fe ff ff 01 00 00 00 fe ff ff ff 00 00 |................|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00000210 da 1d 7c f4 00 00 00 00 01 00 00 00 00 00 00 00 |..|.............|
00000220 ff ff ff 7f 02 00 00 00 22 00 00 00 00 00 00 00 |........".......|
00000230 de ff ff 7f 02 00 00 00 67 17 f1 2e 4d a7 ed 49 |........g...M..I|
00000240 b5 ba 05 64 a5 b4 1d 58 02 00 00 00 00 00 00 00 |...d...X........|
00000250 80 00 00 00 80 00 00 00 b5 60 5f 3f 00 00 00 00 |.........`_?....|
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;|
00000410 5e ae 84 70 76 3d e3 41 88 bb d0 6c 08 0f e4 b1 |^..pv=.A...l....|
00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......|
00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .|
00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.|
00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.|
00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000480 00 53 46 48 00 00 aa 11 aa 11 00 30 65 43 ec ac |.SFH.......0eC..|
00000490 8d 0a 5f fe 22 15 b9 4a a0 c2 ae 2f 87 9b e9 f8 |.._."..J.../....|
000004a0 28 40 06 00 00 00 00 00 d7 ff fb 7f 02 00 00 00 |(@..............|
000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000600
bash-3.2# dd if=/dev/disk14 conv=notrunc bs=512 count=3 | hexdump -C
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................|
000001c0 ff ff ee fe ff ff 01 00 00 00 ff ff ff 7f 00 00 |................|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00000210 16 98 d6 0c 00 00 00 00 01 00 00 00 00 00 00 00 |................|
00000220 ff ff ff 7f 00 00 00 00 22 00 00 00 00 00 00 00 |........".......|
00000230 de ff ff 7f 00 00 00 00 2a 63 4f 72 eb c6 7a 48 |........*cOr..zH|
00000240 8c ec b8 a3 59 18 c7 60 02 00 00 00 00 00 00 00 |....Y..`........|
00000250 80 00 00 00 80 00 00 00 0c a3 e7 61 00 00 00 00 |...........a....|
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;|
00000410 bf 62 1c 4d 72 ce 1b 4c 9f c9 c0 18 39 c9 40 46 |.b.Mr..L....9.@F|
00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......|
00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .|
00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.|
00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.|
00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000480 72 6f 74 53 67 61 aa 11 aa 11 00 30 65 43 ec ac |rotSga.....0eC..|
00000490 f6 e1 77 e0 df dc 5f 49 89 e8 1a c9 6b 2a 67 e9 |..w..._I....k*g.|
000004a0 28 40 06 00 00 00 00 00 d7 ff fb 7f 00 00 00 00 |(@..............|
000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000500 74 6f 6f 42 00 00 aa 11 aa 11 00 30 65 43 ec ac |tooB.......0eC..|
00000510 0c b6 96 82 6b 62 10 40 aa 44 3a 59 2e 03 e7 fe |....kb.@.D:Y....|
00000520 d8 ff fb 7f 00 00 00 00 d7 ff ff 7f 00 00 00 00 |................|
00000530 00 00 00 00 00 00 00 00 42 00 6f 00 6f 00 74 00 |........B.o.o.t.|
00000540 65 00 72 00 00 00 00 00 00 00 00 00 00 00 00 00 |e.r.............|
00000550 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000600
Looking more at disk13, the 5TB volume that doesn't show a partition table under 2.0.0:
- Code: Select all
bash-3.2# bash-3.2# gpt show /dev/disk13
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 6
40 409600 1 GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B
409640 10736746416 2 GPT part - 48465300-0000-11AA-AA11-00306543ECAC
10737156056 262151
10737418207 32 Sec GPT table
10737418239 1 Sec GPT header
bash-3.2# diskutil info /dev/disk13
Device Identifier: disk13
Device Node: /dev/disk13
Whole: Yes
Part of Whole: disk13
Device / Media Name: ZVOL zfsPool/zfsTimeMachineVolume
Volume Name: Not applicable (no file system)
Mounted: Not applicable (no file system)
File System: None
Content (IOContent): GUID_partition_scheme
OS Can Be Installed: No
Media Type: Generic
Protocol: Disk Image
SMART Status: Not Supported
Disk Size: 5.5 TB (5497558138880 Bytes) (exactly 10737418240 512-Byte-Units)
Device Block Size: 512 Bytes
Read-Only Media: No
Read-Only Volume: Not applicable (no file system)
Device Location: External
Removable Media: Fixed
Solid State: Yes
Virtual: Yes
Under Catalina with 2.0.0's Catalina build:- Code: Select all
bash-3.2# uname -v
Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64
bash-3.2# kextstat | grep lund
75 1 0xffffff7f83695000 0x44c 0x44c net.lundman.kernel.dependencies.2.0.0 (12.5.0) BB7D98CE-FDB1-431A-80FB-4C7CB3F52CDB
76 1 0xffffff7f83696000 0x179f000 0x179f000 net.lundman.zfs (2.0.0) F840719D-2EE5-37C8-99DA-8822F0A117E9 <75 28 8 6 5 3 1>
The virtual disks, after zpool import, are the following.
Note: one of the zVols never showed up in the diskutil list output.- Code: Select all
bash-3.2# diskutil list
(... physical disks removed ...)
/dev/disk11 (internal, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +54.7 TB disk11
1: ZFS Dataset zfsPool 54.7 TB disk11s1
/dev/disk12 (external, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: +5.5 TB disk12
/dev/disk13 (external, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +1.6 TB disk13
1: EFI EFI 209.7 MB disk13s1
2: Apple_HFS Photos & Videos 1.6 TB disk13s2
Looking at the first few blocks I get:
- Code: Select all
bash-3.2# dd if=/dev/disk12 conv=notrunc bs=512 count=3 | hexdump -C
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................|
000001c0 ff ff ee fe ff ff 01 00 00 00 fe ff ff ff 00 00 |................|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00000210 da 1d 7c f4 00 00 00 00 01 00 00 00 00 00 00 00 |..|.............|
00000220 ff ff ff 7f 02 00 00 00 22 00 00 00 00 00 00 00 |........".......|
00000230 de ff ff 7f 02 00 00 00 67 17 f1 2e 4d a7 ed 49 |........g...M..I|
00000240 b5 ba 05 64 a5 b4 1d 58 02 00 00 00 00 00 00 00 |...d...X........|
00000250 80 00 00 00 80 00 00 00 b5 60 5f 3f 00 00 00 00 |.........`_?....|
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;|
00000410 5e ae 84 70 76 3d e3 41 88 bb d0 6c 08 0f e4 b1 |^..pv=.A...l....|
00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......|
00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .|
00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.|
00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.|
00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000480 00 53 46 48 00 00 aa 11 aa 11 00 30 65 43 ec ac |.SFH.......0eC..|
00000490 8d 0a 5f fe 22 15 b9 4a a0 c2 ae 2f 87 9b e9 f8 |.._."..J.../....|
000004a0 28 40 06 00 00 00 00 00 d7 ff fb 7f 02 00 00 00 |(@..............|
000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000600
bash-3.2# dd if=/dev/disk13 conv=notrunc bs=512 count=3 | hexdump -C
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................|
000001c0 ff ff ee fe ff ff 01 00 00 00 ff ff ff bf 00 00 |................|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00000210 40 04 a3 5c 00 00 00 00 01 00 00 00 00 00 00 00 |@..\............|
00000220 ff ff ff bf 00 00 00 00 22 00 00 00 00 00 00 00 |........".......|
00000230 de ff ff bf 00 00 00 00 06 17 bc aa 3b 24 61 4a |............;$aJ|
00000240 8d cb 4f a0 e3 ad 82 80 02 00 00 00 00 00 00 00 |..O.............|
00000250 80 00 00 00 80 00 00 00 da 6c b8 ac 00 00 00 00 |.........l......|
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;|
00000410 68 bf 49 d4 61 e8 97 4e 95 42 94 47 e9 94 cf 6f |h.I.a..N.B.G...o|
00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......|
00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .|
00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.|
00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.|
00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000480 00 53 46 48 00 00 aa 11 aa 11 00 30 65 43 ec ac |.SFH.......0eC..|
00000490 81 ed 78 88 2c 31 15 4d b4 fc 76 7e 0e 5a d9 4b |..x.,1.M..v~.Z.K|
000004a0 28 40 06 00 00 00 00 00 d7 ff fb bf 00 00 00 00 |(@..............|
000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000600
Using gpt and diskutil info on disk12 (which is the 5TB volume that corresponds to disk13 under 1.9.4) we get:
- Code: Select all
bash-3.2# gpt show /dev/disk12
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 6
40 409600 1 GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B
409640 10736746416 2 GPT part - 48465300-0000-11AA-AA11-00306543ECAC
10737156056 262151
10737418207 32 Sec GPT table
10737418239 1 Sec GPT header
bash-3.2# diskutil info /dev/disk12
Device Identifier: disk12
Device Node: /dev/disk12
Whole: Yes
Part of Whole: disk12
Device / Media Name: ZVOL zfsPool/zfsTimeMachineVolume
Volume Name: Not applicable (no file system)
Mounted: Not applicable (no file system)
File System: None
Content (IOContent): None
OS Can Be Installed: No
Media Type: Generic
Protocol: Disk Image
SMART Status: Not Supported
Disk Size: 5.5 TB (5497558138880 Bytes) (exactly 10737418240 512-Byte-Units)
Device Block Size: 512 Bytes
Read-Only Media: No
Read-Only Volume: Not applicable (no file system)
Device Location: External
Removable Media: Fixed
Solid State: Yes
Virtual: Yes