OpenZFS on OS X

by **dereed999** » Tue Nov 17, 2020 9:20 am

No more kernel panics! Yay. However, I'm still seeing unstable zVol support.

disk12 and 13 are partitioned and have HFS+ volumes on them... but, unlike disk14, they aren't showing the partition table or partitions. I'm not sure why it's happening... but this pool and volumes worked fine in 1.9.4 and prior. Is there anything I can provide to help identify what's going on?

Code: Select all: /dev/disk11 (internal, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +54.7 TB disk11 1: ZFS Dataset ⁨zfsPool⁩ 54.7 TB disk11s1 /dev/disk12 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 TB disk12 /dev/disk13 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +5.5 TB disk13 /dev/disk14 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.6 TB disk14 1: EFI ⁨EFI⁩ 209.7 MB disk14s1 2: Apple_HFS ⁨Photos & Videos⁩ 1.6 TB disk14s2

by **dereed999** » Tue Nov 17, 2020 12:40 pm

For reference, attached to the same machine, but booted off Catalina w/ v 1.9.4 jut now I have the following. So somehow the zVols aren't showing up fully under Big Sur and the 4th version of rc5.

Code: Select all: /dev/disk12 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.6 TB disk12 1: EFI EFI 209.7 MB disk12s1 2: Apple_HFS Photos & Videos 1.6 TB disk12s2 /dev/disk13 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +5.5 TB disk13 1: EFI EFI 209.7 MB disk13s1 2: Apple_HFS Network Time Machine 5.5 TB disk13s2 /dev/disk14 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 TB disk14 1: EFI EFI 209.7 MB disk14s1 2: Apple_CoreStorage Local Time Machine 1.1 TB disk14s2 3: Apple_Boot Boot OS X 134.2 MB disk14s3

If it helps, here's a zfs list (run in Catalina under 1.9.4)

Code: Select all: bash-3.2# zfs list NAME USED AVAIL REFER MOUNTPOINT zfsPool 23.0T 25.2T 15.5T /Volumes/zfsPool zfsPool/zfsLocalTimeMachineVolume 1.00T 25.3T 943G - zfsPool/zfsPhotoVolume 1.50T 25.9T 888G - zfsPool/zfsTimeMachineVolume 5.01T 25.7T 4.51T -

And a zpool get all:

Code: Select all: NAME PROPERTY VALUE SOURCE zfsPool size 87.2T - zfsPool capacity 43% - zfsPool altroot - default zfsPool health ONLINE - zfsPool guid 16160830777580825211 default zfsPool version - default zfsPool bootfs - default zfsPool delegation on default zfsPool autoreplace off default zfsPool cachefile - default zfsPool failmode wait default zfsPool listsnapshots off default zfsPool autoexpand off default zfsPool dedupratio 1.00x - zfsPool free 49.1T - zfsPool allocated 38.1T - zfsPool readonly off - zfsPool ashift 12 local zfsPool comment - default zfsPool expandsize - - zfsPool freeing 0 default zfsPool fragmentation 1% - zfsPool leaked 0 default zfsPool checkpoint - - zfsPool multihost off default zfsPool autotrim off default zfsPool feature@async_destroy enabled local zfsPool feature@empty_bpobj active local zfsPool feature@lz4_compress active local zfsPool feature@multi_vdev_crash_dump enabled local zfsPool feature@spacemap_histogram active local zfsPool feature@enabled_txg active local zfsPool feature@hole_birth active local zfsPool feature@extensible_dataset enabled local zfsPool feature@embedded_data active local zfsPool feature@bookmarks enabled local zfsPool feature@filesystem_limits enabled local zfsPool feature@large_blocks enabled local zfsPool feature@large_dnode enabled local zfsPool feature@sha512 enabled local zfsPool feature@skein enabled local zfsPool feature@edonr enabled local zfsPool feature@encryption enabled local zfsPool feature@device_removal enabled local zfsPool feature@obsolete_counts enabled local zfsPool feature@zpool_checkpoint enabled local zfsPool feature@spacemap_v2 active local zfsPool feature@allocation_classes enabled local zfsPool feature@bookmark_v2 enabled local zfsPool feature@resilver_defer enabled local

and zfs get all

Code: Select all: NAME PROPERTY VALUE SOURCE zfsPool type filesystem - zfsPool creation Fri Feb 8 14:43 2019 - zfsPool used 23.0T - zfsPool available 25.2T - zfsPool referenced 15.5T - zfsPool compressratio 1.01x - zfsPool mounted yes - zfsPool quota none default zfsPool reservation none default zfsPool recordsize 128K default zfsPool mountpoint /Volumes/zfsPool default zfsPool sharenfs off default zfsPool checksum on default zfsPool compression lz4 local zfsPool atime off local zfsPool devices on default zfsPool exec on default zfsPool setuid on default zfsPool readonly off default zfsPool zoned off default zfsPool snapdir visible local zfsPool aclmode passthrough default zfsPool aclinherit restricted default zfsPool canmount on default zfsPool xattr on default zfsPool copies 1 default zfsPool version 5 - zfsPool utf8only on - zfsPool normalization formD - zfsPool casesensitivity insensitive - zfsPool vscan off default zfsPool nbmand off default zfsPool sharesmb off default zfsPool refquota none default zfsPool refreservation none default zfsPool primarycache all default zfsPool secondarycache all default zfsPool usedbysnapshots 0 - zfsPool usedbydataset 15.5T - zfsPool usedbychildren 7.51T - zfsPool usedbyrefreservation 0 - zfsPool logbias latency default zfsPool dedup off default zfsPool mlslabel none default zfsPool sync standard default zfsPool dnodesize legacy default zfsPool refcompressratio 1.01x - zfsPool written 15.5T - zfsPool logicalused 22.0T - zfsPool logicalreferenced 15.6T - zfsPool filesystem_limit none default zfsPool snapshot_limit none default zfsPool filesystem_count none default zfsPool snapshot_count none default zfsPool snapdev hidden default zfsPool com.apple.browse on default zfsPool com.apple.ignoreowner on local zfsPool com.apple.mimic_hfs off default zfsPool com.apple.devdisk poolonly default zfsPool shareafp off default zfsPool redundant_metadata all default zfsPool overlay off default zfsPool encryption off default zfsPool keylocation none default zfsPool keyformat none default zfsPool pbkdf2iters 0 default zfsPool special_small_blocks 0 default zfsPool/zfsLocalTimeMachineVolume type volume - zfsPool/zfsLocalTimeMachineVolume creation Sun Feb 17 9:06 2019 - zfsPool/zfsLocalTimeMachineVolume used 1.00T - zfsPool/zfsLocalTimeMachineVolume available 25.3T - zfsPool/zfsLocalTimeMachineVolume referenced 943G - zfsPool/zfsLocalTimeMachineVolume compressratio 1.00x - zfsPool/zfsLocalTimeMachineVolume reservation none default zfsPool/zfsLocalTimeMachineVolume volsize 1T local zfsPool/zfsLocalTimeMachineVolume volblocksize 128K default zfsPool/zfsLocalTimeMachineVolume checksum on default zfsPool/zfsLocalTimeMachineVolume compression lz4 inherited from zfsPool zfsPool/zfsLocalTimeMachineVolume readonly off default zfsPool/zfsLocalTimeMachineVolume copies 1 default zfsPool/zfsLocalTimeMachineVolume refreservation 1.00T local zfsPool/zfsLocalTimeMachineVolume primarycache all default zfsPool/zfsLocalTimeMachineVolume secondarycache all default zfsPool/zfsLocalTimeMachineVolume usedbysnapshots 0 - zfsPool/zfsLocalTimeMachineVolume usedbydataset 943G - zfsPool/zfsLocalTimeMachineVolume usedbychildren 0 - zfsPool/zfsLocalTimeMachineVolume usedbyrefreservation 83.5G - zfsPool/zfsLocalTimeMachineVolume logbias latency default zfsPool/zfsLocalTimeMachineVolume dedup off default zfsPool/zfsLocalTimeMachineVolume mlslabel none default zfsPool/zfsLocalTimeMachineVolume sync standard default zfsPool/zfsLocalTimeMachineVolume refcompressratio 1.00x - zfsPool/zfsLocalTimeMachineVolume written 943G - zfsPool/zfsLocalTimeMachineVolume logicalused 945G - zfsPool/zfsLocalTimeMachineVolume logicalreferenced 945G - zfsPool/zfsLocalTimeMachineVolume snapshot_limit none default zfsPool/zfsLocalTimeMachineVolume snapshot_count none default zfsPool/zfsLocalTimeMachineVolume snapdev hidden default zfsPool/zfsLocalTimeMachineVolume redundant_metadata all default zfsPool/zfsLocalTimeMachineVolume encryption off default zfsPool/zfsLocalTimeMachineVolume keylocation none default zfsPool/zfsLocalTimeMachineVolume keyformat none default zfsPool/zfsLocalTimeMachineVolume pbkdf2iters 0 default zfsPool/zfsPhotoVolume type volume - zfsPool/zfsPhotoVolume creation Fri Feb 8 14:43 2019 - zfsPool/zfsPhotoVolume used 1.50T - zfsPool/zfsPhotoVolume available 25.9T - zfsPool/zfsPhotoVolume referenced 888G - zfsPool/zfsPhotoVolume compressratio 1.01x - zfsPool/zfsPhotoVolume reservation none default zfsPool/zfsPhotoVolume volsize 1.50T local zfsPool/zfsPhotoVolume volblocksize 128K default zfsPool/zfsPhotoVolume checksum on default zfsPool/zfsPhotoVolume compression lz4 inherited from zfsPool zfsPool/zfsPhotoVolume readonly off default zfsPool/zfsPhotoVolume copies 1 default zfsPool/zfsPhotoVolume refreservation 1.50T local zfsPool/zfsPhotoVolume primarycache all default zfsPool/zfsPhotoVolume secondarycache all default zfsPool/zfsPhotoVolume usedbysnapshots 0 - zfsPool/zfsPhotoVolume usedbydataset 888G - zfsPool/zfsPhotoVolume usedbychildren 0 - zfsPool/zfsPhotoVolume usedbyrefreservation 651G - zfsPool/zfsPhotoVolume logbias latency default zfsPool/zfsPhotoVolume dedup off default zfsPool/zfsPhotoVolume mlslabel none default zfsPool/zfsPhotoVolume sync standard default zfsPool/zfsPhotoVolume refcompressratio 1.01x - zfsPool/zfsPhotoVolume written 888G - zfsPool/zfsPhotoVolume logicalused 901G - zfsPool/zfsPhotoVolume logicalreferenced 901G - zfsPool/zfsPhotoVolume snapshot_limit none default zfsPool/zfsPhotoVolume snapshot_count none default zfsPool/zfsPhotoVolume snapdev hidden default zfsPool/zfsPhotoVolume redundant_metadata all default zfsPool/zfsPhotoVolume encryption off default zfsPool/zfsPhotoVolume keylocation none default zfsPool/zfsPhotoVolume keyformat none default zfsPool/zfsPhotoVolume pbkdf2iters 0 default zfsPool/zfsTimeMachineVolume type volume - zfsPool/zfsTimeMachineVolume creation Sun Feb 17 20:34 2019 - zfsPool/zfsTimeMachineVolume used 5.01T - zfsPool/zfsTimeMachineVolume available 25.7T - zfsPool/zfsTimeMachineVolume referenced 4.51T - zfsPool/zfsTimeMachineVolume compressratio 1.00x - zfsPool/zfsTimeMachineVolume reservation none default zfsPool/zfsTimeMachineVolume volsize 5T local zfsPool/zfsTimeMachineVolume volblocksize 128K default zfsPool/zfsTimeMachineVolume checksum on default zfsPool/zfsTimeMachineVolume compression lz4 inherited from zfsPool zfsPool/zfsTimeMachineVolume readonly off default zfsPool/zfsTimeMachineVolume copies 1 default zfsPool/zfsTimeMachineVolume refreservation 5.01T local zfsPool/zfsTimeMachineVolume primarycache all default zfsPool/zfsTimeMachineVolume secondarycache all default zfsPool/zfsTimeMachineVolume usedbysnapshots 0 - zfsPool/zfsTimeMachineVolume usedbydataset 4.51T - zfsPool/zfsTimeMachineVolume usedbychildren 0 - zfsPool/zfsTimeMachineVolume usedbyrefreservation 509G - zfsPool/zfsTimeMachineVolume logbias latency default zfsPool/zfsTimeMachineVolume dedup off default zfsPool/zfsTimeMachineVolume mlslabel none default zfsPool/zfsTimeMachineVolume sync standard default zfsPool/zfsTimeMachineVolume refcompressratio 1.00x - zfsPool/zfsTimeMachineVolume written 4.51T - zfsPool/zfsTimeMachineVolume logicalused 4.56T - zfsPool/zfsTimeMachineVolume logicalreferenced 4.56T - zfsPool/zfsTimeMachineVolume snapshot_limit none default zfsPool/zfsTimeMachineVolume snapshot_count none default zfsPool/zfsTimeMachineVolume snapdev hidden default zfsPool/zfsTimeMachineVolume redundant_metadata all default zfsPool/zfsTimeMachineVolume encryption off default zfsPool/zfsTimeMachineVolume keylocation none default zfsPool/zfsTimeMachineVolume keyformat none default zfsPool/zfsTimeMachineVolume pbkdf2iters 0 default

by **lundman** » Tue Nov 17, 2020 1:00 pm

Hmm ok so it created a "diskutil list" diskX for one (first?) zvol but no more after? OK I'll dig into it

by **dereed999** » Tue Nov 17, 2020 1:07 pm

It created disk entries for all the zvols... but somehow none of the volumes (except 1) showed the proper partitions. A "dd" on the first few blocks of the "/dev/disk" didn't showed data that didn't look like the disks. It's like it "showed" the virtual disks, but linked them to the wrong underlying blocks in the zpool? Thanks for looking into it!

by **lundman** » Tue Nov 17, 2020 4:58 pm

Not seeing anything immediately wrong with zvols:

Code: Select all: /dev/disk4 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 GB disk4 /dev/disk5 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 GB disk5 /dev/disk6 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 GB disk6 # dd if=binary112233.bin of=/dev/disk4 conv=notrunc bs=512 # dd if=binaryaabbcc.bin of=/dev/disk5 conv=notrunc bs=512 # dd if=binaryfaceee.bin of=/dev/disk6 conv=notrunc bs=512 # dd if=/dev/disk4 conv=notrunc bs=512 count=3 | hexdump -C 00000000 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 |................| * 00000200 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 |""""""""""""""""| * 00000400 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 |3333333333333333| * 00000600 # dd if=/dev/disk5 conv=notrunc bs=512 count=3 | hexdump -C 00000000 aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa |................| * 00000200 bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb |................| * 00000400 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc |................| * 00000600 # dd if=/dev/disk6 conv=notrunc bs=512 count=3 | hexdump -C 00000000 fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa |................| * 00000200 ce ce ce ce ce ce ce ce ce ce ce ce ce ce ce ce |................| * 00000400 ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee |................| * 00000600

by **dereed999** » Tue Nov 17, 2020 5:12 pm

Thank you for looking. Tomorrow I'll post a dd of the first few blocks from the affected zvols (/dev/disks) under Big Sur 2.0.0 rc5 build 4 and Catalina w/ 1.9.4 so you can see the difference. Is there a current 2.0.0 build for Catalina? I can always try that and see if it works or has the same issue. That might help narrow down if maybe there's a problem on Apple's side somewhere in Big Sur that didn't exist in Catalina? If so, I'll include that in my test as well if you would like?

by **lundman** » Tue Nov 17, 2020 8:25 pm

Yes, that is not a bad idea, let me see if I can spin up a Catalina-2.0 as well.

by **lundman** » Tue Nov 17, 2020 9:24 pm

I have placed a Catalina build in the same BigSur announce thread

by **dereed999** » Wed Nov 18, 2020 7:48 am

Short version:

2.0.0 on Catalina shows the same problems as I was having on Big Sur.
I was wrong (sorry!), raw dd data is correct so the core of the ZFS extension is working right
Catalina (and BigSur) are having problems "using" some of the virtual disks.

Process followed:

I run a scrub of the pool overnight (on 1.9.4).
Grabbed the output shown below on 1.9.4.
Exported the pool, uninstalled 1.9.4, rebooted
Installed 2.0.0 (still on Catalina) and rebooted
Imported the pool and grabbed the output.
Exported the pool, reverted back to 1.9.4, and presently re-scrubbing the pool.

ZFS setup:

I have one large pool and on that pool 3 volumes.
I use the native zfs filesystem for most data but...
Have 3 volumes which host JHFS+ filesystems (so timemachine, photos, etc. all work the way they prefer to work)

Observations under 2.0.0 vs 1.9.4:

One of the volumes never shows in diskutil list under 2.0.0 (all three show under 1.9.4)
The first three 512 byte blocks of the two volumes that do show up as /dev/disk entries are identical between 1.9.4 and 2.0.0. I looked at the last few blocks of each volume along with the first few blocks of the JHFS+ partition as well and they are correct/consistent between 1.9.4 and 2.0.0.
One of the volumes that does have a /dev/disk entry under 2.0.0 does not show a partition table or partitions/filesystems via diskutil list and can't be mounted by OS X (yet works fine under 1.9.4) despite having the same blocks.
Looking at that volume using gpt shows it has a valid partition (while diskutil list say it does not)
diskutil info on that disk entry says "IOContent: None" vs under 1.9.4 where is says "IOContent: GUID_partition_scheme".

Hypothesis:

The code that maps /dev/disk blocks to the backing ZFS volume works properly (as tools like gpt or dd which examine the /dev/disk raw blocks work.)
Somehow something isn't being set that OS X looks for when more than one volume exists. One volume does properly show and mount, one completely fails to have any /dev/disk entry at all, and one has a /dev/disk entry that's backed by real, valid, data when accessed at the block level, but diskutil insists no partitions exist.
- This is supported by the different output the one "partially" there disk provides for a "diskutil info" under 1.9.4 vs 2.0.0.
- Under 1.9.4 the output includes: "Content (IOContent): GUID_partition_scheme."
- Under 2.0.0 the output includes: "Content (IOContent): None"

Let me know what else I can try to look at to assist in tracking down why this happens.

Under Catalina, version 1.9.4

Code: Select all: bash-3.2# uname -v Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 bash-3.2# kextstat | grep lund 73 1 0xffffff7f8363b000 0x4fc 0x4fc net.lundman.kernel.dependencies.33 (12.5.0) 103D4746-0E59-43E9-BCC9-8A165E0EC2B1 74 1 0xffffff7f8363c000 0x11f6000 0x11f6000 net.lundman.spl (1.9.4) F4C8FC99-7C45-3C5A-BE70-A23D3A03170D <73 8 6 5 3 1> 75 1 0xffffff7f84832000 0x3e3000 0x3e3000 net.lundman.zfs (1.9.4) 1C090B5F-A844-3606-A15B-2AA004AFAD8A <74 28 8 6 5 3 1>

The virtual disks, after zpool import, are:

Code: Select all: bash-3.2# diskutil list (... physical disks removed ...) /dev/disk11 (internal, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +54.7 TB disk11 1: ZFS Dataset zfsPool 54.7 TB disk11s1 /dev/disk12 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.6 TB disk12 1: EFI EFI 209.7 MB disk12s1 2: Apple_HFS Photos & Videos 1.6 TB disk12s2 /dev/disk13 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +5.5 TB disk13 1: EFI EFI 209.7 MB disk13s1 2: Apple_HFS Network Time Machine 5.5 TB disk13s2 /dev/disk14 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 TB disk14 1: EFI EFI 209.7 MB disk14s1 2: Apple_CoreStorage Local Time Machine 1.1 TB disk14s2 3: Apple_Boot Boot OS X 134.2 MB disk14s3

Each of the volumes (disk12, disk13, and disk14) show with partitions and the first few blocks do show the EFI partition and right GPT headers.

Code: Select all: bash-3.2# dd if=/dev/disk12 conv=notrunc bs=512 count=3 | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................| 000001c0 ff ff ee fe ff ff 01 00 00 00 ff ff ff bf 00 00 |................| 000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...| 00000210 40 04 a3 5c 00 00 00 00 01 00 00 00 00 00 00 00 |@..\............| 00000220 ff ff ff bf 00 00 00 00 22 00 00 00 00 00 00 00 |........".......| 00000230 de ff ff bf 00 00 00 00 06 17 bc aa 3b 24 61 4a |............;$aJ| 00000240 8d cb 4f a0 e3 ad 82 80 02 00 00 00 00 00 00 00 |..O.............| 00000250 80 00 00 00 80 00 00 00 da 6c b8 ac 00 00 00 00 |.........l......| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;| 00000410 68 bf 49 d4 61 e8 97 4e 95 42 94 47 e9 94 cf 6f |h.I.a..N.B.G...o| 00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......| 00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .| 00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.| 00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.| 00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000480 00 53 46 48 00 00 aa 11 aa 11 00 30 65 43 ec ac |.SFH.......0eC..| 00000490 81 ed 78 88 2c 31 15 4d b4 fc 76 7e 0e 5a d9 4b |..x.,1.M..v~.Z.K| 000004a0 28 40 06 00 00 00 00 00 d7 ff fb bf 00 00 00 00 |(@..............| 000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000600 bash-3.2# dd if=/dev/disk13 conv=notrunc bs=512 count=3 | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 1536 bytes transferred in 0.000135 secs (11382422 bytes/sec) 000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................| 000001c0 ff ff ee fe ff ff 01 00 00 00 fe ff ff ff 00 00 |................| 000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...| 00000210 da 1d 7c f4 00 00 00 00 01 00 00 00 00 00 00 00 |..|.............| 00000220 ff ff ff 7f 02 00 00 00 22 00 00 00 00 00 00 00 |........".......| 00000230 de ff ff 7f 02 00 00 00 67 17 f1 2e 4d a7 ed 49 |........g...M..I| 00000240 b5 ba 05 64 a5 b4 1d 58 02 00 00 00 00 00 00 00 |...d...X........| 00000250 80 00 00 00 80 00 00 00 b5 60 5f 3f 00 00 00 00 |.........`_?....| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;| 00000410 5e ae 84 70 76 3d e3 41 88 bb d0 6c 08 0f e4 b1 |^..pv=.A...l....| 00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......| 00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .| 00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.| 00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.| 00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000480 00 53 46 48 00 00 aa 11 aa 11 00 30 65 43 ec ac |.SFH.......0eC..| 00000490 8d 0a 5f fe 22 15 b9 4a a0 c2 ae 2f 87 9b e9 f8 |.._."..J.../....| 000004a0 28 40 06 00 00 00 00 00 d7 ff fb 7f 02 00 00 00 |(@..............| 000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000600 bash-3.2# dd if=/dev/disk14 conv=notrunc bs=512 count=3 | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................| 000001c0 ff ff ee fe ff ff 01 00 00 00 ff ff ff 7f 00 00 |................| 000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...| 00000210 16 98 d6 0c 00 00 00 00 01 00 00 00 00 00 00 00 |................| 00000220 ff ff ff 7f 00 00 00 00 22 00 00 00 00 00 00 00 |........".......| 00000230 de ff ff 7f 00 00 00 00 2a 63 4f 72 eb c6 7a 48 |........*cOr..zH| 00000240 8c ec b8 a3 59 18 c7 60 02 00 00 00 00 00 00 00 |....Y..`........| 00000250 80 00 00 00 80 00 00 00 0c a3 e7 61 00 00 00 00 |...........a....| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;| 00000410 bf 62 1c 4d 72 ce 1b 4c 9f c9 c0 18 39 c9 40 46 |.b.Mr..L....9.@F| 00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......| 00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .| 00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.| 00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.| 00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000480 72 6f 74 53 67 61 aa 11 aa 11 00 30 65 43 ec ac |rotSga.....0eC..| 00000490 f6 e1 77 e0 df dc 5f 49 89 e8 1a c9 6b 2a 67 e9 |..w..._I....k*g.| 000004a0 28 40 06 00 00 00 00 00 d7 ff fb 7f 00 00 00 00 |(@..............| 000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000500 74 6f 6f 42 00 00 aa 11 aa 11 00 30 65 43 ec ac |tooB.......0eC..| 00000510 0c b6 96 82 6b 62 10 40 aa 44 3a 59 2e 03 e7 fe |....kb.@.D:Y....| 00000520 d8 ff fb 7f 00 00 00 00 d7 ff ff 7f 00 00 00 00 |................| 00000530 00 00 00 00 00 00 00 00 42 00 6f 00 6f 00 74 00 |........B.o.o.t.| 00000540 65 00 72 00 00 00 00 00 00 00 00 00 00 00 00 00 |e.r.............| 00000550 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000600

Looking more at disk13, the 5TB volume that doesn't show a partition table under 2.0.0:

Code: Select all: bash-3.2# bash-3.2# gpt show /dev/disk13 start size index contents 0 1 PMBR 1 1 Pri GPT header 2 32 Pri GPT table 34 6 40 409600 1 GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B 409640 10736746416 2 GPT part - 48465300-0000-11AA-AA11-00306543ECAC 10737156056 262151 10737418207 32 Sec GPT table 10737418239 1 Sec GPT header bash-3.2# diskutil info /dev/disk13 Device Identifier: disk13 Device Node: /dev/disk13 Whole: Yes Part of Whole: disk13 Device / Media Name: ZVOL zfsPool/zfsTimeMachineVolume Volume Name: Not applicable (no file system) Mounted: Not applicable (no file system) File System: None Content (IOContent): GUID_partition_scheme OS Can Be Installed: No Media Type: Generic Protocol: Disk Image SMART Status: Not Supported Disk Size: 5.5 TB (5497558138880 Bytes) (exactly 10737418240 512-Byte-Units) Device Block Size: 512 Bytes Read-Only Media: No Read-Only Volume: Not applicable (no file system) Device Location: External Removable Media: Fixed Solid State: Yes Virtual: Yes

Under Catalina with 2.0.0's Catalina build:

Code: Select all: bash-3.2# uname -v Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 bash-3.2# kextstat | grep lund 75 1 0xffffff7f83695000 0x44c 0x44c net.lundman.kernel.dependencies.2.0.0 (12.5.0) BB7D98CE-FDB1-431A-80FB-4C7CB3F52CDB 76 1 0xffffff7f83696000 0x179f000 0x179f000 net.lundman.zfs (2.0.0) F840719D-2EE5-37C8-99DA-8822F0A117E9 <75 28 8 6 5 3 1>

The virtual disks, after zpool import, are the following. Note: one of the zVols never showed up in the diskutil list output.

Code: Select all: bash-3.2# diskutil list (... physical disks removed ...) /dev/disk11 (internal, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +54.7 TB disk11 1: ZFS Dataset zfsPool 54.7 TB disk11s1 /dev/disk12 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +5.5 TB disk12 /dev/disk13 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.6 TB disk13 1: EFI EFI 209.7 MB disk13s1 2: Apple_HFS Photos & Videos 1.6 TB disk13s2

Looking at the first few blocks I get:

Code: Select all: bash-3.2# dd if=/dev/disk12 conv=notrunc bs=512 count=3 | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................| 000001c0 ff ff ee fe ff ff 01 00 00 00 fe ff ff ff 00 00 |................| 000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...| 00000210 da 1d 7c f4 00 00 00 00 01 00 00 00 00 00 00 00 |..|.............| 00000220 ff ff ff 7f 02 00 00 00 22 00 00 00 00 00 00 00 |........".......| 00000230 de ff ff 7f 02 00 00 00 67 17 f1 2e 4d a7 ed 49 |........g...M..I| 00000240 b5 ba 05 64 a5 b4 1d 58 02 00 00 00 00 00 00 00 |...d...X........| 00000250 80 00 00 00 80 00 00 00 b5 60 5f 3f 00 00 00 00 |.........`_?....| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;| 00000410 5e ae 84 70 76 3d e3 41 88 bb d0 6c 08 0f e4 b1 |^..pv=.A...l....| 00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......| 00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .| 00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.| 00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.| 00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000480 00 53 46 48 00 00 aa 11 aa 11 00 30 65 43 ec ac |.SFH.......0eC..| 00000490 8d 0a 5f fe 22 15 b9 4a a0 c2 ae 2f 87 9b e9 f8 |.._."..J.../....| 000004a0 28 40 06 00 00 00 00 00 d7 ff fb 7f 02 00 00 00 |(@..............| 000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000600 bash-3.2# dd if=/dev/disk13 conv=notrunc bs=512 count=3 | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fe |................| 000001c0 ff ff ee fe ff ff 01 00 00 00 ff ff ff bf 00 00 |................| 000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...| 00000210 40 04 a3 5c 00 00 00 00 01 00 00 00 00 00 00 00 |@..\............| 00000220 ff ff ff bf 00 00 00 00 22 00 00 00 00 00 00 00 |........".......| 00000230 de ff ff bf 00 00 00 00 06 17 bc aa 3b 24 61 4a |............;$aJ| 00000240 8d cb 4f a0 e3 ad 82 80 02 00 00 00 00 00 00 00 |..O.............| 00000250 80 00 00 00 80 00 00 00 da 6c b8 ac 00 00 00 00 |.........l......| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000400 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b |(s*......K...>.;| 00000410 68 bf 49 d4 61 e8 97 4e 95 42 94 47 e9 94 cf 6f |h.I.a..N.B.G...o| 00000420 28 00 00 00 00 00 00 00 27 40 06 00 00 00 00 00 |(.......'@......| 00000430 00 00 00 00 00 00 00 00 45 00 46 00 49 00 20 00 |........E.F.I. .| 00000440 53 00 79 00 73 00 74 00 65 00 6d 00 20 00 50 00 |S.y.s.t.e.m. .P.| 00000450 61 00 72 00 74 00 69 00 74 00 69 00 6f 00 6e 00 |a.r.t.i.t.i.o.n.| 00000460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000480 00 53 46 48 00 00 aa 11 aa 11 00 30 65 43 ec ac |.SFH.......0eC..| 00000490 81 ed 78 88 2c 31 15 4d b4 fc 76 7e 0e 5a d9 4b |..x.,1.M..v~.Z.K| 000004a0 28 40 06 00 00 00 00 00 d7 ff fb bf 00 00 00 00 |(@..............| 000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000600

Using gpt and diskutil info on disk12 (which is the 5TB volume that corresponds to disk13 under 1.9.4) we get:

Code: Select all: bash-3.2# gpt show /dev/disk12 start size index contents 0 1 PMBR 1 1 Pri GPT header 2 32 Pri GPT table 34 6 40 409600 1 GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B 409640 10736746416 2 GPT part - 48465300-0000-11AA-AA11-00306543ECAC 10737156056 262151 10737418207 32 Sec GPT table 10737418239 1 Sec GPT header bash-3.2# diskutil info /dev/disk12 Device Identifier: disk12 Device Node: /dev/disk12 Whole: Yes Part of Whole: disk12 Device / Media Name: ZVOL zfsPool/zfsTimeMachineVolume Volume Name: Not applicable (no file system) Mounted: Not applicable (no file system) File System: None Content (IOContent): None OS Can Be Installed: No Media Type: Generic Protocol: Disk Image SMART Status: Not Supported Disk Size: 5.5 TB (5497558138880 Bytes) (exactly 10737418240 512-Byte-Units) Device Block Size: 512 Bytes Read-Only Media: No Read-Only Volume: Not applicable (no file system) Device Location: External Removable Media: Fixed Solid State: Yes Virtual: Yes

by **dereed999** » Wed Nov 18, 2020 9:11 am

I ran another test to rule out it being my pool itself.

Under Catalina w/ 1.9.4, I:

Created a 10G file on my native OS X boot drive
Created a testing pool backed by that file
Created 3 volumes on that pool
Formatted each of the volumes as JHFS+
Everything worked as expected.

Under Big Sur w/ 2.0.0, I:

Copied that file 10G file holding the pool over to a different Big Sur machine w/ 2.0.0
Imported the pool living in that file.
One volume shows up properly.
One volume shows up but without partition table or partitions
One volume completely is missing.

Just for fun on Big Sur w/ 2.0.0, I:

Created a NEW 10G file on my native OS X boot drive
Created a testing pool backed by that file
Created 3 volumes on that pool
Same odd behavior in that:
- One volume shows up properly.
- One volume shows up but without partition table or partitions
- One volume completely is missing.

So it’s not my pool (which makes me happy!) and should be repeatable by you in your testing environment (use 3rd chunk of commands below)

Catalina commands and output w/ 1.9.4:

Code: Select all: bash-3.2# rm -f ~/Desktop/testFile bash-3.2# mkfile -n 10g ~/Desktop/testFile bash-3.2# zpool create testPool ~/Desktop/testFile bash-3.2# zfs create -V 1G testPool/vol1 bash-3.2# zfs create -V 1G testPool/vol2 bash-3.2# zfs create -V 1G testPool/vol3 bash-3.2# zfs list NAME USED AVAIL REFER MOUNTPOINT testPool 3.01G 6.19G 1.01M /Volumes/testPool testPool/vol1 1.00G 7.19G 12K - testPool/vol2 1.00G 7.19G 12K - testPool/vol3 1.00G 7.19G 12K - bash-3.2# diskutil list /dev/disk16 (internal, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +10.2 GB disk16 /dev/disk17 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 GB disk17 /dev/disk18 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 GB disk18 /dev/disk19 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 GB disk19 bash-3.2# diskutil partitionDisk /dev/disk17 GPT jhfs+ "Vol 1" 0 Started partitioning on disk17 Unmounting disk Creating the partition map Waiting for partitions to activate Formatting disk17s1 as Mac OS Extended (Journaled) with name Vol 1 Initialized /dev/rdisk17s1 as a 1024 MB case-insensitive HFS Plus volume with a 8192k journal Mounting disk Finished partitioning on disk17 /dev/disk17 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 GB disk17 1: Apple_HFS Vol 1 1.1 GB disk17s1 bash-3.2# diskutil partitionDisk /dev/disk18 GPT jhfs+ "Vol 2" 0 Started partitioning on disk18 Unmounting disk Creating the partition map Waiting for partitions to activate Formatting disk18s1 as Mac OS Extended (Journaled) with name Vol 2 Initialized /dev/rdisk18s1 as a 1024 MB case-insensitive HFS Plus volume with a 8192k journal Mounting disk Finished partitioning on disk18 /dev/disk18 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 GB disk18 1: Apple_HFS Vol 2 1.1 GB disk18s1 bash-3.2# diskutil partitionDisk /dev/disk19 GPT jhfs+ "Vol 3" 0 Started partitioning on disk19 Unmounting disk Creating the partition map Waiting for partitions to activate Formatting disk19s1 as Mac OS Extended (Journaled) with name Vol 3 Initialized /dev/rdisk19s1 as a 1024 MB case-insensitive HFS Plus volume with a 8192k journal Mounting disk Finished partitioning on disk19 /dev/disk19 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 GB disk19 1: Apple_HFS Vol 3 1.1 GB disk19s1 bash-3.2# df -h Filesystem Size Used Avail Capacity iused ifree %iused Mounted on testPool 6.2Gi 1.0Mi 6.2Gi 1% 76 12980565 0% /Volumes/testPool /dev/disk17s1 1.0Gi 33Mi 991Mi 4% 74 4294967205 0% /Volumes/Vol 1 /dev/disk18s1 1.0Gi 33Mi 991Mi 4% 74 4294967205 0% /Volumes/Vol 2 /dev/disk19s1 1.0Gi 33Mi 991Mi 4% 74 4294967205 0% /Volumes/Vol 3 bash-3.2# diskutil list /dev/disk16 (internal, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +10.2 GB disk16 /dev/disk17 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 GB disk17 1: Apple_HFS Vol 1 1.1 GB disk17s1 /dev/disk18 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 GB disk18 1: Apple_HFS Vol 2 1.1 GB disk18s1 /dev/disk19 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 GB disk19 1: Apple_HFS Vol 3 1.1 GB disk19s1 bash-3.2# zpool export testPool

Big Sur commands and output w/ 2.0.0 using existing pool file (disk#s are different as I copied the file to a different machine):

Code: Select all: bash-3.2# zpool import -d ~/Desktop testPool do_mount calling mount with fstype zfs, rpath /Volumes/testPool, fspec /dev/disk2s1, mflag 0, optptr defaults,atime,strictatime,dev,exec,rw,setuid,nomand,zfsutil, optlen 60, devdisk 0, ispool 1 bash-3.2# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT testPool 9.50G 104M 9.40G - - 0% 1% 1.00x ONLINE - bash-3.2# diskutil list /dev/disk2 (internal, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +10.2 GB disk2 1: ZFS Dataset ⁨testPool⁩ 10.2 GB disk2s1 /dev/disk3 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 GB disk3 /dev/disk4 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +1.1 GB disk4 1: Apple_HFS ⁨Vol 1⁩ 1.1 GB disk4s1

Big Sur commands and output for “from scratch” creation

Code: Select all: bash-3.2# mkfile -n 10g ~/Desktop/testFile-ALT2 bash-3.2# zpool create testPool-ALT2 ~/Desktop/testFile-ALT2 do_mount calling mount with fstype zfs, rpath /Volumes/testPool-ALT2, fspec /dev/disk2s1, mflag 0, optptr defaults,atime,strictatime,dev,exec,rw,setuid,nomand,zfsutil, optlen 60, devdisk 0, ispool 1 bash-3.2# zfs create -V 1G testPool-ALT2/vol1 bash-3.2# zfs create -V 1G testPool-ALT2/vol2 bash-3.2# zfs create -V 1G testPool-ALT2/vol3 bash-3.2# diskutil list /dev/disk0 (internal, physical): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *2.0 TB disk0 1: EFI ⁨EFI⁩ 314.6 MB disk0s1 2: Apple_APFS ⁨Container disk1⁩ 2.0 TB disk0s2 /dev/disk1 (synthesized): #: TYPE NAME SIZE IDENTIFIER 0: APFS Container Scheme - +2.0 TB disk1 Physical Store disk0s2 1: APFS Volume ⁨Macintosh HD⁩ 14.9 GB disk1s1 2: APFS Snapshot ⁨com.apple.os.update-...⁩ 14.9 GB disk1s1s1 3: APFS Volume ⁨Macintosh HD - Data⁩ 1.3 TB disk1s2 4: APFS Volume ⁨Preboot⁩ 304.0 MB disk1s3 5: APFS Volume ⁨Recovery⁩ 652.6 MB disk1s4 6: APFS Volume ⁨VM⁩ 20.5 KB disk1s5 /dev/disk2 (internal, virtual): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme +10.2 GB disk2 1: ZFS Dataset ⁨testPool-ALT2⁩ 10.2 GB disk2s1 /dev/disk3 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: +1.1 GB disk3 bash-3.2# zpool export testPool-ALT2

OpenZFS on OS X

Missing zVols on 2.0.0 Big Sur rc5 (4th)

Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Re: Missing zVols on 2.0.0 Big Sur rc5 (4th)

Who is online