I started with a three disk pool of 3 X 500gb.
I swapped one out at a time with 2Tb drives, but I had two of those only, which game me 2X2Tb and 1X500Gb for still 1Tb of usable space.
I sold something, bought a 3Tb Seagate and thought I would just replace the 500Gb drive and sit pretty at 4Tb usable space. I was wrong. The 3Tb drive is of course a 4K drive. I spent last night copying all of my nearly full drive space onto the 3Tb drive. Then I destroyed that pool and rebuilt is using ashift=12. Over night my largest child (puddle/TVShare) finished copying back onto the newly rebuilt pool.
NOW it is resilvering the 3Tb drive while swapping out the 500Gb drive.
I also have another 3Tb drive that is sitting at 2.5G full. When the resilvering is done with this drive I can I think tell it something like
sudo zpool set autoexpand=on
It should then resliver again?
Whatever happens, I can rsync -r the contents of the 3Tb drive not in the pool into the pool, then replace one of the slower low speed 2Tb drives with it. This would mean I am a 3Tb drive away from 6Tb of usable RAIDZ space.
Would I gain any benefit of say creating a mirror vdev of 2X2Tb then creating a mirrored vdev that is puddle and whatever that is? Or do I just add the two 2Tb drives to puddle to create that sort of joining?
server:Volumes admin$ zpool status -v
pool: puddle
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Feb 5 05:27:02 2013
133Gi scanned out of 1.15Ti at 215Mi/s, 1h22m to go
44.4Gi resilvered, 11.37% done
config:
NAME STATE READ WRITE CKSUM
puddle ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
GPTE_4585F147-B1F7-4B23-A1C1-3804D1071611 ONLINE 0 0 0 at disk3s2
replacing-1 ONLINE 0 0 0
GPTE_8AB8BE68-92B1-4195-BAF6-AF1EE0B1D8D8 ONLINE 0 0 0 at disk4s2
GPTE_4823E6FE-6F68-4828-8494-E192FBB7CCDE ONLINE 0 0 0 at disk0s2 (resilvering)
GPTE_5DBB8F4A-9D8A-4287-BA9E-3EBDBA53255C ONLINE 0 0 0 at disk5s2