Page 1 of 1

Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 3:08 pm
by BenB
I have a Zevo installation on ML Server running a RaidZ2 pool over thunderbolt. I have recently expanded the array by replacing 2TB drives with 3TB drives (partly to get more space, partly because one of the 2TB drives was getting write errors) and Zevo has expanded the array after the last disk was installed and resilvered. The problem is that Finder is still reporting the capacity as 3.19TB while zpool list is reporting it as 9.09Ti with 8.51Ti allocated. Running a df -h command returns the same capacity as finder.
Has this happened to anyone and is there a way to get the system to see the full size of the pool?

The system configuration is:

Server:
OS 10.8.3
Mid-2011 Minimac with 8GB of RAM
Atto SAS controller in a Sonnet Thunderbolt-PCIe box
5x 3TB SATA HDDs connected with SAS-SATA fan-out cables through a Supermicro SATA backplane (1 cable per disk, no expander) in a single RaidZ2 vdev

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 5:40 pm
by ilovezfs
And is that still true even after rebooting?

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 7:29 pm
by BenB
yes, I restarted a couple of times and there was no change to the size displayed. I just exported the pool and restarted again in case there was some issue with the shutdown that wasn't exporting the pool properly but no change.
zpool list and zpool status both report 9.09Ti with 8.51Ti allocated but Finder reports 3.19TB and df -h report a size of 2.9Ti

Also, I have checked the size with a zfs calculator http://www.servethehome.com/raid-calculator/ and the size shown is consistent with a 5 disk RaidZ2 using 1.5TB drives not 2TB drives. This appears to indicate that the zpool's size also failed to update after I removed the last 1.5TB drive from it not long after it was built - I didn't notice at the time because the primary motivation for replacing the old disks was their age rather than a desire for more storage capacity and I was still relatively new to ZFS.

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 7:43 pm
by ilovezfs
(5 * 2) * (10^12 / 2^40) ≈ 9.09

So the zpool list 9.09 value is consistent with a five disk raidz-n at 2TB per disk.

For a five disk raidz-n at 3TB per disk, you should expect to see zpool list reporting

(5 * 3) * (10^12 / 2^40) ≈ 13.64

So I am wondering if you actually are using larger partitions yet. What is the output of diskutil list?

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 8:14 pm
by BenB
Right, so that's the raw storage I'm seeing there...

Code: Select all
server:~ admin$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.1 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS Macintosh HD            499.2 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                        ZFS                         3.0 TB     disk1s2
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk2
   1:                        EFI                         209.7 MB   disk2s1
   2:                        ZFS                         3.0 TB     disk2s2
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk3
   1:                        EFI                         209.7 MB   disk3s1
   2:                        ZFS                         3.0 TB     disk3s2
/dev/disk4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk4
   1:                        EFI                         209.7 MB   disk4s1
   2:                        ZFS                         3.0 TB     disk4s2
/dev/disk5
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk5
   1:                        EFI                         209.7 MB   disk5s1
   2:                        ZFS                         3.0 TB     disk5s2
/dev/disk6
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:             zfs_pool_proxy CORAID SAS             *10.0 TB    disk6
   1:       zfs_filesystem_proxy Groups                  707.7 GB   disk6s1
   2:       zfs_filesystem_proxy Media                   2.5 TB     disk6s2
   3:       zfs_filesystem_proxy Users                   319.6 GB   disk6s3

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 8:19 pm
by ilovezfs
OK, so that looks good.

What's the output of

Code: Select all
zpool get autoexpand

and

Code: Select all
zfs list -o space -t all

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 8:27 pm
by BenB
Code: Select all
server:~ admin$ zpool get autoexpand CORAID\ SAS
NAME        PROPERTY    VALUE   SOURCE
CORAID SAS  autoexpand  on      local


Code: Select all
server:~ admin$ zfs list -o space -t all
NAME                AVAIL    USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
CORAID SAS          268Gi  5.04Ti         0  2.64Ti              0     2.40Ti
CORAID SAS/Groups   268Gi   391Gi         0   391Gi              0          0
CORAID SAS/Media    268Gi  1.99Ti         0  1.99Ti              0          0
CORAID SAS/Users    268Gi  30.0Gi         0  30.0Gi              0          0

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 9:30 pm
by ilovezfs
You can see that the diskutil list (and Finder) numbers are the sum of the used and the available for each file system.

CORAID SAS/Groups: (391*(2^30) + 268*(2^30))/10^9 = 707.59586201 ≈ 707.7 GB (disk6s1)
CORAID SAS/Media: (1.99*(2^40) + 268*(2^30))/10^12 = 2.4757909481 ≈ 2.5 TB (disk6s2)
CORAID SAS/Users: (30*(2^30) + 268*(2^30))/10^9 = 319.97506355 ≈ 319.6 GB (disk6s3)
CORAID SAS (root dataset): ((5.04-2.40)*(2^40) + 268*(2^30))/10^12 = 3.1904735062 TB, as you're seeing in Finder.

So that all appears perfectly normal for 5 disk raidz-2 at 2TB per disk.

Since autoexpand=on it should automatically have recognized the increase to 3TB per disk, but apparently did not.

You can try using online -e to see if that gets it to kick in.

Code: Select all
sudo zpool online -e "CORAID SAS" disk1s2


Check if it expands right away. If not, you can try running these commands too to see if it makes any difference.

Code: Select all
sudo zpool online -e "CORAID SAS" disk2s2
sudo zpool online -e "CORAID SAS" disk3s2
sudo zpool online -e "CORAID SAS" disk4s2
sudo zpool online -e "CORAID SAS" disk5s2


If that still doesn't work, you can try toggling autoexpand.

Code: Select all
sudo zpool set autoexpand=off "CORAID SAS"
sudo zpool set autoexpand=on "CORAID SAS"

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 9:52 pm
by BenB
Yep, makes sense now I know what I'm looking at. I probably should have spotted that the number matched the raw storage....

That fixed it
After running the zpool online -e command the zpool list size updated. Following a restart finder is now showing a finder capacity of 6TB.
The only thing I had to change was the device name - for some reason my system doesn't like referring to disks that are already part of a zfs pool by the disk1s2 names, I have to use the long device names (ie GPTE_9F7600BEA-...)

I tried toggling autoexpand before I posted here, no luck with that

Thank you for your patience and assistance :)

Re: Zevo - ZFS Expand increases zpool size but not in finder

PostPosted: Mon Aug 18, 2014 9:56 pm
by ilovezfs
BenB wrote:Thank you for your patience and assistance :)

You're welcome.

BenB wrote:The only thing I had to change was the device name - for some reason my system doesn't like referring to disks that are already part of a zfs pool by the disk1s2 names, I have to use the long device names (ie GPTE_9F7600BEA-…)

Yes, that makes sense since it's ZEVO. In general with ZFS, you're supposed to use whatever names "zpool status" is reporting.

BenB wrote:I tried toggling autoexpand before I posted here, no luck with that

Yeah, so basically ZEVO autoexpand isn't working at least in certain cases.