Absolute noob trying to setup mirrored vdevs

[FadingIntoBlue]

2. Would be neat if my disks were renamed with their serialnumber (which i wrote down on each drive in the front of the thunderbay). In the case of one failing, i can be 100% sure which drive it was.
How can I do this? If i have to remake a pool so be it. Wouldn't be too much work.

https://openzfsonosx.org/wiki/Device_names should help

[dangriga]

Awesome! Thanks

What about GDT IDs? Is this necessary? For what purpose?

[dangriga]

Okay I messed up trying to add another mirror to my pool. now I cannot import the pool:

sudo zpool import -F
pool: tank
id: 2076556828109525209
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
config:

tank UNAVAIL insufficient replicas
mirror-0 ONLINE
media-430F38EF-905F-E841-86A0-F4F813B4BD0C ONLINE
media-69F261D0-FC13-234B-8012-41AE575BA769 ONLINE
mirror-1 ONLINE
media-FD527821-CC7B-FA46-9569-D19CF2196ECC ONLINE
media-3F2F7D1C-3EE5-064B-B7E0-6BCECC79F89A ONLINE
mirror-2 UNAVAIL insufficient replicas
disk11 UNAVAIL cannot open
disk12 UNAVAIL cannot open



the problems are these two disks:

/dev/disk4 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *14.0 TB disk4
1: Microsoft Reserved 16.8 MB disk4s1
2: Apple_APFS 14.0 TB disk4s2

/dev/disk5 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *14.0 TB disk5
1: Apple_APFS 14.0 TB disk5s1


I had formatted them as APFS but now there seem to be different partitions...
My functioning drives look like this:

/dev/disk9 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *20.0 TB disk9
1: ZFS tank 20.0 TB disk9s1
2: 6A945A3B-1DD2-11B2-99A6-080020736631 8.4 MB disk9s9


weird thing is that the new disks should be after disk9 but yet they are listed as the first two disks as disk4 and disk5
tried removing mirror-2 but without any success. Is there any way I can recover this pool of the state before trying to add the new mirror?
can i simply reformat the new drives again and then try importing the pool?

thanks in advance!



EDIT:

UFS Explorer RAID RECOVERY is luckily still recognizing the ZFS volume and all the data is accessible which gives me at least a bit of peace of mind. As i discussed this issue with various people I think one of the next steps would be to get rid of the Microsoft Partition that was unfortunately still on the drive after formatting it when I added it to the pool. Maybe after I remove it I will be able to import the pool again?

How do you guys setup a Pool backup through zfs so that it backs up the "catalogue" of the pool everyday? I need to do that in the future if I have to set up a new pool. I know I am very very stupid. Lesson learned....

[jawbroken]

would be helpful if you posted whatever commands you entered to get into this state. `zpool history` should show them

[dangriga]

Can't do that as there are no pools available...

The disks I added are definitely the issue. They used to be in a windows machine and I formatted them in mac before trying to add them to the pool. One of the two drives still had a microsoft partition on it. Tried removing it with diskutil but now there is a additional synthesized drive (disk5);

/dev/disk4 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *14.0 TB disk4
1: Apple_APFS Container disk5 16.8 MB disk4s1
2: Apple_APFS 14.0 TB disk4s2

/dev/disk5 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +16.8 MB disk5
Physical Store disk4s1

/dev/disk6 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *14.0 TB disk6
1: Apple_APFS 14.0 TB disk6s1


My plan would be to get them back to the initial mounting state before adding them to the pool. Then maybe the pool would recognize them and I could import?

At this point however I am already pretty sure I won't be able to save this pool. I have ordered new drives that I will add to the thunderbay, save the data from the old pool to them, destroy the pool and then start from scratch.

[FadingIntoBlue]

I found I got easily confused when I formatted disks using HFS+ or APFS and then tried to add them to a pool or use them as replacements. I changed to using this:

Code: Select all

sudo diskutil partitiondisk disk8 GPT 'Free Space' Empty 100%

which means the disk won't mount in macOS when you try and add the whole disk to the pool. You should still see it in a terminal listing, diskutil list if not df

Hope that helps, if not now, then in the rebuild

[dangriga]

Yeah I forgot to do that when I added the new mirror...
I just formatted the harddrives and tried to add them.

So in the furture:

Always test first with -n. If it does not work then, there is a problem, right?

For my new pool that I will have to create;

Is it smarter to have 2 pools of 4 drives with 2x20TB & 1x20TB 1x14TB mirrored or to have 1 pool with all 8 drives? (just for scrubbing this will take ages)
Is it smart to mix 14tb with 20tb harddrives even? because the 2 new hdds i tried to add were 14tb, the others are all 20tb.

If I am setting up a new pool and format the drive, how do I add the new drives as mirrors with the serialnumber as the diskname?
The drives that were part of the old pool; should I format them as APFS and then use command sudo ###diskutil partitiondisk disk8 GPT 'Free Space' Empty 100%### prior to setting up a new pool with them?

Backup:

How do I set up a backup that stores the pool file (where the whole configuration is saved) every day so in case something like this happens I can just reset the pool to the previous day?

Anything else I am missing?





EDIT:

I just formatted both 14TB in Windows to exfat, brought it back to the thunderbay, formatted it to APFS and then used your command above.

now my drives look like this:

/dev/disk6 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *14.0 TB disk6
1: EFI EFI 209.7 MB disk6s1
(free space) 14.0 TB -

/dev/disk7 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *14.0 TB disk7
1: EFI EFI 209.7 MB disk7s1
(free space) 14.0 TB -

/dev/disk8 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *20.0 TB disk8
1: ZFS tank 20.0 TB disk8s1
2: 6A945A3B-1DD2-11B2-99A6-080020736631 8.4 MB disk8s9

/dev/disk9 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *20.0 TB disk9
1: ZFS tank 20.0 TB disk9s1
2: 6A945A3B-1DD2-11B2-99A6-080020736631 8.4 MB disk9s9

/dev/disk10 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *20.0 TB disk10
1: ZFS tank 20.0 TB disk10s1
2: 6A945A3B-1DD2-11B2-99A6-080020736631 8.4 MB disk10s9

/dev/disk11 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *20.0 TB disk11
1: ZFS tank 20.0 TB disk11s1
2: 6A945A3B-1DD2-11B2-99A6-080020736631 8.4 MB disk11s9


the first two are the new, 14TB drives, the rest are the 20TB in the pool.
My pool looks like this:

pool: tank
id: 2076556828109525209
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
config:

tank UNAVAIL insufficient replicas
mirror-0 ONLINE
media-430F38EF-905F-E841-86A0-F4F813B4BD0C ONLINE
media-69F261D0-FC13-234B-8012-41AE575BA769 ONLINE
mirror-1 ONLINE
media-FD527821-CC7B-FA46-9569-D19CF2196ECC ONLINE
media-3F2F7D1C-3EE5-064B-B7E0-6BCECC79F89A ONLINE
mirror-2 UNAVAIL insufficient replicas
disk11 UNAVAIL cannot open
disk12 UNAVAIL cannot open



I had added the new two drives as disk11 and disk12. Meanwhile the disknames in diskutil changed. My new drives are now disk6 and disk7. could this be the issue that the pool cannot be imported? Can I rename them to disk11 and disk12? (Problem is that there now already is a disk11 in the pool...)

[jawbroken]

i think this is probably able to be rescued given that it sounds like you just added that mirror so there shouldn't really be any data on it, but you'll need someone with more experience than me to tell you how to remove them

[dangriga]

Where can I find an expert?
I am a fan of ZFS but to me it is quite hard to get into it. There are no real ressources for beginners like me.

[FadingIntoBlue]

Where can I find an expert?

I don't profess to be an expert, but there are a few things I can point you at to help perhaps.

This is an oldie but a goodie that I still use when I get stuck/forget. https://lildude.co.uk/zfs-cheatsheet

Using disk names like /dev/disk7 is not a good way to reference a disk while creating a pool, as you have discovered. They are not invariant, they change between boots, perhaps because other disks are connected, or for other reasons.

It is much easier to help you trouble shoot, if you show us the commands you use to produce the result you are showing us. You can use:

Code: Select all

zpool history tank

to list all the commands used with that pool ending with the most recent.

I have two suggestions for resolving your current state, though I am not sure they will work. Your mirror-2 addition to the pool either has to have the missing (mislabelled) disks replaced with the right ones, or be removed, and the disks used to create a new mirror, using invariant names, in your case by serial number would seem sensible.

I'm loath to try a replacement using this command:

Code: Select all

zpool replace /disk11 /disk6

because as you note, you already have a disk 11 in the pool, or did at least.

Show us what the output of

Code: Select all

df

is in the terminal. You should be able to map your new drives to disk numbers by their size, and the old ones by their size or the pool name.

And do show us [using the history command as above] the commands you used to get the output you have listed.

The invariant alternative would be safer So, I suggest you do the following first:

Code: Select all

$ sudo zpool export tank
$ sudo zpool import -d /var/run/disk/by-serial tank

If that shows you all the disks with their serial numbers, the fix will be fairly simple. Even without, you have a definite firm identity of your existing drives, and a replacement of the others would be possible. Effectively, what you are doing is replacing the missing drives (misnamed that is) with themselves, only with the correct names.

I'm not an expert as I noted, so if this explanation, messy as it is, is all a bit too complex for you, I'll be happy to run you through the commands I would use for a rebuild.