Want to do it right this time

New to OpenZFS on OS X (Or ZFS in general)? Ask your questions here!

Want to do it right this time

Postby GroovinBuddha » Tue Feb 16, 2021 11:03 am

Hello again,

Last time I had a failing drive I tried to swap it out using a rather primitive method and screwed myself royally with the help of some extra backup software I was running. I want to do things properly this time and could use some assistance.

I have been alerted that drive 3 (of 4) is starting to fail. I believe the procedure would be to remove the drive from the pool, swap it out, add the new drive to the pool then resilver the pool. This makes logical sense to me, I am however unclear as to the exact details. Here's what I think I need to do, could someone please provide me with some corrections and/or extra insight.

sudo zpool remove Data media-8EAACE28-9638-AF49-8E1C-227552B6DFA3

Power off my computer and swap the drive

sudo zpool add Data [new device]

sudo zpool resilver Data

Thanks in advance

Ed

PS is there a good book or three to learn this stuff?
GroovinBuddha
 
Posts: 14
Joined: Thu Jan 16, 2020 8:09 am

Re: Want to do it right this time

Postby FadingIntoBlue » Tue Feb 16, 2021 11:57 pm

You don't say what type of pool you have, mirror or raid. I've done replacements on a dual mirror pool I have several times, mainly to expand the total pool size rather than replace a failing drive. I've kept some reasonably decent notes, because I took the drive from 4x4TB drives in a a dual mirror all the way up to 4 x8TB drives, one drive at a time, over several months. My notes are below, I hope they are helpful:

PART 1: Swap the Physical Drives and identify the new drive

1. Remove one of the disks in the mirror, the one you wish to replace
2. Insert the new drive in the slot left by the withdrawal of the previous drive
3. Detach the missing drive from the mirror
4. Check your work
5. Identify the /dev/disk# of the new drive [in this case /dev/disk6, non zfs disks removed from diskutil list output]

Code: Select all
bash-3.2$ zpool status
  pool: NewPool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
   the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://zfsonlinux.org/msg/ZFS-8000-2Q
  scan: resilvered 2.31T in 36h1m with 0 errors on Mon Jul  9 04:51:38 2018
config:

   NAME                                            STATE     READ WRITE CKSUM
   NewPool                                         DEGRADED     0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-31E32D3A-142B-984C-AFBB-AB795E6D53B6  ONLINE       0     0     0
       media-A76C12E4-B84D-E34E-B27B-A591EE6A363B  ONLINE       0     0     0
     mirror-1                                      DEGRADED     0     0     0
       627905676083406759                          UNAVAIL      0     0     0  was /private/var/run/disk/by-id/media-2855F739-D0BC-8440-8A9F-0DB435F06BC3
       media-4DD878BB-BA84-024E-8C3E-A06E2C3F3E01  ONLINE       0     0     0

errors: No known data errors
bash-3.2$ sudo zpool detach NewPool media-2855F739-D0BC-8440-8A9F-0DB435F06BC3
bash-3.2$ zpool status
  pool: NewPool
 state: ONLINE
  scan: resilvered 2.31T in 36h1m with 0 errors on Mon Jul  9 04:51:38 2018
config:

   NAME                                            STATE     READ WRITE CKSUM
   NewPool                                         ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-31E32D3A-142B-984C-AFBB-AB795E6D53B6  ONLINE       0     0     0
       media-A76C12E4-B84D-E34E-B27B-A591EE6A363B  ONLINE       0     0     0
     media-4DD878BB-BA84-024E-8C3E-A06E2C3F3E01    ONLINE       0     0     0

errors: No known data errors
bash-3.2$ diskutil list
/dev/disk0, 1, 2, 3

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *6.0 TB     disk4
   1:                        ZFS NewPool                 6.0 TB     disk4s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk4s9

/dev/disk5 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *8.0 TB     disk5
   1:                        ZFS NewPool                 8.0 TB     disk5s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk5s9

/dev/disk6 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *8.0 TB     disk6

/dev/disk7 (synthesized)

/dev/disk8 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk8
   1:                        ZFS NewPool                 4.0 TB     disk8s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk8s9

/dev/disk11 (internal, virtual):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        +10.0 TB    disk11
   1:                ZFS Dataset NewPool                 10.0 TB    disk11s1


Part 2: Rebuild the Mirror

1. Create a temporary pool TempPool with the new disk
2. Export and import TempPool to get the media-####### name of the new disk
3. Attach the new disk to the disk remaining from the original mirror - you must export the temporary pool first
4. NB: Don’t export the pool you are trying to grow. This prevents you accidentally mirroring the mirror drive to the new drive

Code: Select all
bash-3.2$ sudo zpool create -n -o ashift=12 -O casesensitivity=insensitive -O normalization=formD TempPool /dev/disk6
would create ‘TempPool’ with the following layout:

   TempPool
     disk6
bash-3.2$ sudo zpool create -o ashift=12 -O casesensitivity=insensitive -O normalization=formD TempPool /dev/disk6
zmount calling mount with fstype zfs, rpath /Volumes/TempPool, fspec /dev/disk12s1, mflag 256, optptr , optlen 4096, devdisk 0, ispool 1
bash-3.2$ zpool status
  pool: NewPool
 state: ONLINE
  scan: resilvered 2.31T in 36h1m with 0 errors on Mon Jul  9 04:51:38 2018
config:

   NAME                                            STATE     READ WRITE CKSUM
   NewPool                                         ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-31E32D3A-142B-984C-AFBB-AB795E6D53B6  ONLINE       0     0     0
       media-A76C12E4-B84D-E34E-B27B-A591EE6A363B  ONLINE       0     0     0
     media-4DD878BB-BA84-024E-8C3E-A06E2C3F3E01    ONLINE       0     0     0

errors: No known data errors

  pool: TempPool
 state: ONLINE
  scan: none requested
config:

   NAME        STATE     READ WRITE CKSUM
   TempPool    ONLINE       0     0     0
     disk6     ONLINE       0     0     0

errors: No known data errors
bash-3.2$ sudo zpool export TempPool
Running process: ‘/usr/sbin/diskutil’ ‘unmount’ ‘/Volumes/TempPool’
Volume TempPool on disk12s1 unmounted
bash-3.2$ sudo zpool import -d /var/run/disk/by-id TempPool
zmount calling mount with fstype zfs, rpath /Volumes/TempPool, fspec /dev/disk12s1, mflag 256, optptr , optlen 4096, devdisk 0, ispool 1
bash-3.2$ zpool status
  pool: NewPool
 state: ONLINE
  scan: resilvered 2.31T in 36h1m with 0 errors on Mon Jul  9 04:51:38 2018
config:

   NAME                                            STATE     READ WRITE CKSUM
   NewPool                                         ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-31E32D3A-142B-984C-AFBB-AB795E6D53B6  ONLINE       0     0     0
       media-A76C12E4-B84D-E34E-B27B-A591EE6A363B  ONLINE       0     0     0
     media-4DD878BB-BA84-024E-8C3E-A06E2C3F3E01    ONLINE       0     0     0

errors: No known data errors

  pool: TempPool
 state: ONLINE
  scan: none requested
config:

   NAME                                          STATE     READ WRITE CKSUM
   TempPool                                      ONLINE       0     0     0
     media-50B0A66E-2CF5-8244-8CF9-E64D12472E39  ONLINE       0     0     0

errors: No known data errors
bash-3.2$ sudo zpool export TempPool
Running process: ‘/usr/sbin/diskutil’ ‘unmount’ ‘/Volumes/TempPool’
Volume TempPool on disk12s1 unmounted
bash-3.2$ sudo zpool attach -f NewPool media-4DD878BB-BA84-024E-8C3E-A06E2C3F3E01 media-50B0A66E-2CF5-8244-8CF9-E64D12472E39
bash-3.2$ zpool status
  pool: NewPool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
   continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Jul 16 18:07:07 2018
   576K scanned out of 5.11T at 64K/s, (scan is slow, no estimated time)
    224K resilvered, 0.00% done
config:

   NAME                                            STATE     READ WRITE CKSUM
   NewPool                                         ONLINE       0     0     0
     mirror-0                                      ONLINE       0     0     0
       media-31E32D3A-142B-984C-AFBB-AB795E6D53B6  ONLINE       0     0     0
       media-A76C12E4-B84D-E34E-B27B-A591EE6A363B  ONLINE       0     0     0
     mirror-1                                      ONLINE       0     0     0
       media-4DD878BB-BA84-024E-8C3E-A06E2C3F3E01  ONLINE       0     0     0
       media-50B0A66E-2CF5-8244-8CF9-E64D12472E39  ONLINE       0     0     0

errors: No known data errors
bash-3.2$
FadingIntoBlue
 
Posts: 55
Joined: Tue May 27, 2014 12:25 am

Re: Want to do it right this time

Postby jawbroken » Wed Feb 17, 2021 3:43 am

I think ideally you would use “zpool replace” for this, presuming you can attach the new drive and the old drive at the same time.
jawbroken
 
Posts: 17
Joined: Wed Apr 01, 2015 4:46 am

Re: Super Basic N00b

Postby GroovinBuddha » Fri Feb 19, 2021 7:12 pm

Hey guys,

Thanks for the responses, I think Jawbroken is probably on the right track. FadingIntoBlue, to answer your question, I have 4x4TB drives in a RaidZ1 config. The pool is not yet showing degraded, but DriveDX is showing that it is starting to fail and that lines up with a couple other oddities that I've noticed with my tower.

The four drives are in the four drive bays, so to have a replacement drive on at the same time I would need to connect it up to an external bay, which I do have. From what I (think I) understand ZFS shouldn't be too concerned if I do the replace function set up externally then swap the new drive to the internal bay.

So here's where I think I am:

Get my replacement drive connected with the sled

$ sudo zpool replace Data media-8EAACE28-9638-AF49-8E1C-227552B6DFA3 /dev/disk[whatever]

$ zpool status

Make sure it is all online then shut down the computer. Swap the internal failing drive with the new external one and fire it back up.

$ zpool status

Make sure it is all online with the new drive. Then:

$ zpool resilver Data

Does that look correct?

I do have a 12TB clone of the whole thing in case I screw up again, but I'm hoping I've learned at least a little since last time.

Cheers
GroovinBuddha
 
Posts: 14
Joined: Thu Jan 16, 2020 8:09 am

Re: Want to do it right this time

Postby jawbroken » Fri Feb 19, 2021 9:57 pm

I don't think anything other than the replace command is actually required. If I recall correctly, replace should take care of the formatting and resilvering. Once resilvering is finished, you can then remove the faulty drive, etc.
jawbroken
 
Posts: 17
Joined: Wed Apr 01, 2015 4:46 am

Re: Want to do it right this time

Postby GroovinBuddha » Sat Feb 20, 2021 8:26 pm

Cool, thanks.

I'll give it a shot and report the results once the new drive comes in.

Cheers
Ed
GroovinBuddha
 
Posts: 14
Joined: Thu Jan 16, 2020 8:09 am

Re: Want to do it right this time

Postby Bingo » Fri Mar 05, 2021 12:39 am

FadingIntoBlue wrote:Part 2: Rebuild the Mirror

1. Create a temporary pool TempPool with the new disk
2. Export and import TempPool to get the media-####### name of the new disk
3. Attach the new disk to the disk remaining from the original mirror - you must export the temporary pool first
4. NB: Don’t export the pool you are trying to grow. This prevents you accidentally mirroring the mirror drive to the new drive


There's no need to do all this to get the `media-*` name of the new drive. Look in `/private/var/run/disk/` to get the different drive identifiers. A simple `ls -l /private/var/run/disk/*` in terminal will list all the different identifiers available, including the `media-*` ones :) Just note the device you're looking for (from Disk Utility or elsewhere), and find it in the list.
Bingo
 
Posts: 2
Joined: Thu Mar 04, 2021 11:18 pm


Return to Absolute Beginners Section

Who is online

Users browsing this forum: No registered users and 2 guests