visible sum of snapshots is less than usedbysnapshots

Moderators: jhartley, MSR734, nola

visible sum of snapshots is less than usedbysnapshots

Post by grahamperrin » Wed Mar 27, 2013 3:11 am

Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get usedbysnapshots zhandy
NAME    PROPERTY         VALUE     SOURCE
zhandy  usedbysnapshots  107Gi     -


Confusingly:

Code: Select all
macbookpro08-centrim:~ gjp22$ zfs list -t snapshot | grep zhandy | grep -v backup
zhandy@2013-03-12-224158                                   31.2Mi       -   448Gi  -
zhandy@2013-03-13-003952                                   23.2Mi       -   449Gi  -
zhandy@2013-03-13-092155                                        0       -   451Gi  -
zhandy@2013-03-13-102154                                        0       -   451Gi  -
zhandy@2013-03-17-072706                                   26.4Mi       -   446Gi  -
zhandy@2013-03-17-082706                                   20.8Mi       -   446Gi  -
zhandy@2013-03-17-092706                                   22.7Mi       -   446Gi  -
zhandy@2013-03-17-102706                                   29.6Mi       -   446Gi  -
zhandy@2013-03-19-022206                                   22.6Mi       -   448Gi  -
zhandy@2013-03-19-032206                                   7.68Mi       -   448Gi  -
zhandy@2013-03-19-075930                                    280Ki       -   448Gi  -
zhandy@2013-03-19-102926                                        0       -   448Gi  -
zhandy@2013-03-19-112927                                        0       -   448Gi  -
zhandy@2013-03-19-122927                                        0       -   448Gi  -
zhandy@2013-03-19-134210                                        0       -   448Gi  -
zhandy@2013-03-20-075025                                        0       -   447Gi  -
zhandy@2013-03-20-085024                                        0       -   447Gi  -
zhandy@2013-03-20-095024                                        0       -   447Gi  -
zhandy@2013-03-20-105024                                        0       -   447Gi  -
zhandy@2013-03-20-115024                                        0       -   447Gi  -
zhandy@2013-03-20-180101                                    433Mi       -   448Gi  -
zhandy@2013-03-23-191414                                        0       -   449Gi  -
zhandy@2013-03-23-195116                                        0       -   449Gi  -
zhandy@2013-03-23-204418                                    280Ki       -   449Gi  -
zhandy@2013-03-24-115609                                    288Ki       -   449Gi  -
zhandy@2013-03-25-113051                                        0       -   452Gi  -
zhandy@2013-03-25-123050                                        0       -   452Gi  -
zhandy@2013-03-25-133050                                        0       -   452Gi  -
zhandy@2013-03-25-143050                                        0       -   452Gi  -
zhandy@2013-03-25-153049                                        0       -   452Gi  -
zhandy@2013-03-25-163049                                        0       -   452Gi  -
zhandy@2013-03-26-065933                                    645Mi       -   440Gi  -
zhandy@2013-03-26-175344                                   12.8Mi       -   439Gi  -
zhandy@2013-03-27-020918                                    177Mi       -   439Gi  -
zhandy@2013-03-27-060937                                    408Ki       -   408Gi  -
zhandy@2013-03-27-062020                                        0       -   408Gi  -
zhandy/Pocket Time Machine@2013-03-23-191414                    0       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-23-195116                    0       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-23-204418                    0       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-24-115609                    0       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-25-235503                    0       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-26-065933                    0       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-26-175344                    0       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-27-020918                144Ki       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-27-060937                    0       -  56.6Gi  -
zhandy/Pocket Time Machine@2013-03-27-062020                    0       -  56.6Gi  -


I can't get a sum of 107Gi from that second column.

Am I misreading the output?

If this seems buggy, there's a possible consideration: the spaces in the name of the child file system (cross reference: Is space an alphanumeric character?).
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Still wrong

Post by grahamperrin » Thu Mar 28, 2013 2:52 am

This issue is the first to truly bother me since I began using ZEVO.

Renaming the child file system, to exclude spaces, has not resolved the issue.

Still there's too little free space, the file system at the root has less than 11 GB. Before the most recent snapshots, I removed some multi-gigabyte files, aiming to destroy snapshots to yield free space. I really need that yield.

Output from zdb, with a leak-free checksum of metadata:

Code: Select all
macbookpro08-centrim:~ gjp22$ sudo zdb -bcAAAd zhandy
Password:
Dataset mos [META], ID 0, cr_txg 4, 65.0M, 1116 objects
Dataset zhandy/pockettimemachine@2013-03-27-060937 [ZPL], ID 780, cr_txg 1221131, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine@2013-03-27-062020 [ZPL], ID 4097, cr_txg 1221275, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine@2013-03-25-235503 [ZPL], ID 623, cr_txg 1193066, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine@2013-03-23-191414 [ZPL], ID 491, cr_txg 1159732, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine@2013-03-23-195116 [ZPL], ID 520, cr_txg 1160177, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine@2013-03-23-204418 [ZPL], ID 551, cr_txg 1160368, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine@2013-03-27-020918 [ZPL], ID 977, cr_txg 1220037, 56.6G, 3140 objects
Dataset zhandy/pockettimemachine@2013-03-24-115609 [ZPL], ID 584, cr_txg 1170923, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine@2013-03-26-065933 [ZPL], ID 907, cr_txg 1196720, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine@2013-03-26-175344 [ZPL], ID 821, cr_txg 1197586, 56.6G, 3142 objects
Dataset zhandy/pockettimemachine [ZPL], ID 242, cr_txg 36031, 57.2G, 3453 objects
Dataset zhandy@2013-03-17-092706 [ZPL], ID 203, cr_txg 1094260, 446G, 109119 objects
Dataset zhandy@2013-03-23-195116 [ZPL], ID 525, cr_txg 1160177, 449G, 109181 objects
Dataset zhandy@2013-03-23-204418 [ZPL], ID 557, cr_txg 1160368, 449G, 109181 objects
Dataset zhandy@2013-03-17-072706 [ZPL], ID 186, cr_txg 1092827, 446G, 109119 objects
Dataset zhandy@2013-03-20-075025 [ZPL], ID 362, cr_txg 1124274, 447G, 109095 objects
Dataset zhandy@2013-03-25-163049 [ZPL], ID 779, cr_txg 1188420, 452G, 109148 objects
Dataset zhandy@2013-03-13-092155 [ZPL], ID 164, cr_txg 1042255, 451G, 108930 objects
Dataset zhandy@2013-03-19-134210 [ZPL], ID 344, cr_txg 1118655, 448G, 109183 objects
Dataset zhandy@2013-03-19-112927 [ZPL], ID 311, cr_txg 1117226, 448G, 109183 objects
Dataset zhandy@2013-03-19-075930 [ZPL], ID 282, cr_txg 1115783, 448G, 109181 objects
Dataset zhandy@2013-03-23-191414 [ZPL], ID 495, cr_txg 1159732, 449G, 109181 objects
Dataset zhandy@2013-03-13-003952 [ZPL], ID 151, cr_txg 1039829, 449G, 108677 objects
Dataset zhandy@2013-03-17-082706 [ZPL], ID 194, cr_txg 1093540, 446G, 109119 objects
Dataset zhandy@2013-03-27-060937 [ZPL], ID 795, cr_txg 1221131, 408G, 109107 objects
Dataset zhandy@2013-03-27-062020 [ZPL], ID 4110, cr_txg 1221275, 408G, 109112 objects
Dataset zhandy@2013-03-17-102706 [ZPL], ID 213, cr_txg 1094980, 446G, 109119 objects
Dataset zhandy@2013-03-26-175344 [ZPL], ID 834, cr_txg 1197586, 439G, 109158 objects
Dataset zhandy@2013-03-19-022206 [ZPL], ID 214, cr_txg 1113762, 448G, 109178 objects
Dataset zhandy@2013-03-20-115024 [ZPL], ID 444, cr_txg 1127150, 447G, 109095 objects
Dataset zhandy@2013-03-20-180101 [ZPL], ID 467, cr_txg 1130854, 448G, 109214 objects
Dataset zhandy@2013-03-26-065933 [ZPL], ID 916, cr_txg 1196720, 440G, 109163 objects
Dataset zhandy@2013-03-20-095024 [ZPL], ID 401, cr_txg 1125712, 447G, 109095 objects
Dataset zhandy@2013-03-25-113051 [ZPL], ID 624, cr_txg 1184861, 452G, 109148 objects
Dataset zhandy@2013-03-25-143050 [ZPL], ID 714, cr_txg 1186980, 452G, 109148 objects
Dataset zhandy@2013-03-25-133050 [ZPL], ID 683, cr_txg 1186260, 452G, 109148 objects
Dataset zhandy@2013-03-24-115609 [ZPL], ID 591, cr_txg 1170923, 449G, 109135 objects
Dataset zhandy@2013-03-12-224158 [ZPL], ID 24, cr_txg 1038366, 448G, 108570 objects
Dataset zhandy@2013-03-13-102154 [ZPL], ID 170, cr_txg 1042973, 451G, 108930 objects
Dataset zhandy@2013-03-20-105024 [ZPL], ID 422, cr_txg 1126430, 447G, 109095 objects
Dataset zhandy@2013-03-19-122927 [ZPL], ID 327, cr_txg 1117946, 448G, 109183 objects
Dataset zhandy@2013-03-19-032206 [ZPL], ID 252, cr_txg 1114482, 448G, 109178 objects
Dataset zhandy@2013-03-19-102926 [ZPL], ID 296, cr_txg 1116506, 448G, 109183 objects
Dataset zhandy@2013-03-25-123050 [ZPL], ID 653, cr_txg 1185540, 452G, 109148 objects
Dataset zhandy@2013-03-27-020918 [ZPL], ID 988, cr_txg 1220037, 439G, 109164 objects
Dataset zhandy@2013-03-20-085024 [ZPL], ID 381, cr_txg 1124992, 447G, 109095 objects
Dataset zhandy@2013-03-25-153049 [ZPL], ID 746, cr_txg 1187700, 452G, 109148 objects
Dataset zhandy [ZPL], ID 21, cr_txg 1, 408G, 109091 objects

Traversing all blocks to verify metadata checksums and verify nothing leaked ...

   No leaks (block sum matches space maps exactly)

   bp count:         5754798
   bp logical:    735269802496      avg: 127766
   bp physical:   610160914944      avg: 106026     compression:   1.21
   bp allocated:  614052306944      avg: 106702     compression:   1.20
   bp deduped:             0    ref>1:      0   deduplication:   1.00
   SPA allocated: 614052306944     used: 96.60%


Code: Select all
macbookpro08-centrim:~ gjp22$ zfs list -t snapshot | grep "Gi       -"
gjp22@2012-10-28-212255                     33.8Gi       -   337Gi  -
gjp22@2013-02-10-065905                     2.97Gi       -   367Gi  -
gjp22@2013-02-11-081552                     1.56Gi       -   366Gi  -
gjp22@2013-03-23-050302                     2.72Gi       -   384Gi  -
macbookpro08-centrim:~ gjp22$ zfs list -t snapshot | grep zhandy
zhandy@2013-03-12-224158                    31.2Mi       -   448Gi  -
zhandy@2013-03-13-003952                    23.2Mi       -   449Gi  -
zhandy@2013-03-13-092155                         0       -   451Gi  -
zhandy@2013-03-13-102154                         0       -   451Gi  -
zhandy@2013-03-17-072706                    26.4Mi       -   446Gi  -
zhandy@2013-03-17-082706                    20.8Mi       -   446Gi  -
zhandy@2013-03-17-092706                    22.7Mi       -   446Gi  -
zhandy@2013-03-17-102706                    29.6Mi       -   446Gi  -
zhandy@2013-03-19-022206                    22.6Mi       -   448Gi  -
zhandy@2013-03-19-032206                    7.68Mi       -   448Gi  -
zhandy@2013-03-19-075930                     280Ki       -   448Gi  -
zhandy@2013-03-19-102926                         0       -   448Gi  -
zhandy@2013-03-19-112927                         0       -   448Gi  -
zhandy@2013-03-19-122927                         0       -   448Gi  -
zhandy@2013-03-19-134210                         0       -   448Gi  -
zhandy@2013-03-20-075025                         0       -   447Gi  -
zhandy@2013-03-20-085024                         0       -   447Gi  -
zhandy@2013-03-20-095024                         0       -   447Gi  -
zhandy@2013-03-20-105024                         0       -   447Gi  -
zhandy@2013-03-20-115024                         0       -   447Gi  -
zhandy@2013-03-20-180101                     433Mi       -   448Gi  -
zhandy@2013-03-23-191414                         0       -   449Gi  -
zhandy@2013-03-23-195116                         0       -   449Gi  -
zhandy@2013-03-23-204418                     280Ki       -   449Gi  -
zhandy@2013-03-24-115609                     288Ki       -   449Gi  -
zhandy@2013-03-25-113051                         0       -   452Gi  -
zhandy@2013-03-25-123050                         0       -   452Gi  -
zhandy@2013-03-25-133050                         0       -   452Gi  -
zhandy@2013-03-25-143050                         0       -   452Gi  -
zhandy@2013-03-25-153049                         0       -   452Gi  -
zhandy@2013-03-25-163049                         0       -   452Gi  -
zhandy@2013-03-26-065933                     645Mi       -   440Gi  -
zhandy@2013-03-26-175344                    12.8Mi       -   439Gi  -
zhandy@2013-03-27-020918                     177Mi       -   439Gi  -
zhandy@2013-03-27-060937                     408Ki       -   408Gi  -
zhandy@2013-03-27-062020                     408Ki       -   408Gi  -
zhandy/pockettimemachine@2013-03-23-191414       0       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-23-195116       0       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-23-204418       0       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-24-115609       0       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-25-235503       0       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-26-065933       0       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-26-175344       0       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-27-020918   144Ki       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-27-060937       0       -  56.6Gi  -
zhandy/pockettimemachine@2013-03-27-062020       0       -  56.6Gi  -
macbookpro08-centrim:~ gjp22$


What next? This is worrying …
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

snapdir and more

Post by grahamperrin » Thu Mar 28, 2013 5:13 am

Maybe relevant: before removing the multi-gigabyte files (before the most recent snapshots), I changed the snapdir property value –

  • from visible
  • to hidden

– and almost certainly did so whilst the file system was mounted.

(Why hide it? So that Disk Inventory X could give me a good view of the volume without traversing snapshots.)

Today I reverted to visible for the snapdir, then exported and imported but still there's not the expected mass in any recent snapshot:

Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get snapdir zhandy
NAME    PROPERTY  VALUE    SOURCE
zhandy  snapdir   hidden   local
macbookpro08-centrim:~ gjp22$ zfs set snapdir=visible zhandy
macbookpro08-centrim:~ gjp22$ zfs list -t snapshot | grep "Gi       -" | grep zhandy
macbookpro08-centrim:~ gjp22$ zpool export zhandy
macbookpro08-centrim:~ gjp22$ sudo tail -n 50 /private/var/log/system.log | grep zhandy
Password:
Mar 28 08:00:28 macbookpro08-centrim kernel[0]: zfsx_unmount: '/Volumes/zhandy' (umount)
Mar 28 08:00:28 macbookpro08-centrim kernel[0]: zfsvfs_teardown: '/Volumes/zhandy' (txg_wait_synced in 113 ms)
Mar 28 08:00:34 macbookpro08-centrim kernel[0]: ZFSLabelScheme:willTerminate: this 0xffffff802dca4200 provider 0xffffff80291e4300 'zfs vdev for 'zhandy''
Mar 28 08:00:34 macbookpro08-centrim kernel[0]: ZFSLabelScheme:stop: 0xffffff802dca4200 goodbye 'zfs vdev for 'zhandy''
Mar 28 08:00:42 macbookpro08-centrim kernel[0]: ZFSLabelScheme:probe: label 'zhandy', vdev 2173720822411009046
Mar 28 08:00:42 macbookpro08-centrim kernel[0]: ZFSLabelScheme:start: 'zhandy' critical mass with 1 vdev(s) (importing)
Mar 28 08:00:42 macbookpro08-centrim kernel[0]: zfsx_kev_importpool:'zhandy' (8649809893956376149)
Mar 28 08:00:44 macbookpro08-centrim kernel[0]: zfsx_vdm_open: 'zhandy' disk5s2
Mar 28 08:00:44 macbookpro08-centrim kernel[0]: zfsx_mount: '/Volumes/zhandy'
macbookpro08-centrim:~ gjp22$ zfs list -t snapshot | grep "Gi       -" | grep zhandy


A comprehensive view of file system snapdir property values throughout the life of the pool:

Code: Select all
macbookpro08-centrim:~ gjp22$ zpool history zhandy | grep snapdir
2012-12-06.18:00:52 zpool create -o ashift=12 -O casesensitivity=insensitive -O normalization=formD -O atime=off -O snapdir=visible -O compression=gzip-9 zhandy /dev/disk2
2012-12-07.20:23:24 zfs create -o casesensitivity=insensitive -o normalization=formD -o atime=off -o snapdir=visible -o compression=gzip-9 zhandy/Pocket Time Machine
2012-12-08.16:34:14 zfs create -o casesensitivity=insensitive -o normalization=formD -o atime=off -o snapdir=visible -o compression=gzip-9 zhandy/Pocket Time Machine
2013-03-26.19:44:50 zfs set snapdir=hidden zhandy
2013-03-28.07:59:59 zfs set snapdir=visible zhandy


Browsing the list of snapshots, then opening one in Finder:

Code: Select all
macbookpro08-centrim:~ gjp22$ cd /Volumes/zhandy/.zfs
macbookpro08-centrim:.zfs gjp22$ ls -al
total 48
dr-xr-xr-x   3 root   admin   3  6 Dec 18:00 .
drwxrwxr-x+ 20 gjp22  admin  25 27 Mar 11:56 ..
dr-xr-xr-x  38 root   admin  38 28 Mar 08:00 snapshot
macbookpro08-centrim:.zfs gjp22$ cd snapshot
macbookpro08-centrim:snapshot gjp22$ ls -hl
total 0
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-12-224158
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-13-003952
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-13-092155
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-13-102154
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-17-072706
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-17-082706
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-17-092706
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-17-102706
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-19-022206
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-19-032206
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-19-075930
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-19-102926
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-19-112927
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-19-122927
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-19-134210
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-20-075025
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-20-085024
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-20-095024
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-20-105024
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-20-115024
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-20-180101
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-23-191414
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-23-195116
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-23-204418
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-24-115609
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-25-113051
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-25-123050
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-25-133050
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-25-143050
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-25-153049
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-25-163049
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-26-065933
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-26-175344
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-27-020918
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-27-060937
dr-xr-xr-x  2 root  admin     2B 28 Mar 08:00 2013-03-27-062020
macbookpro08-centrim:snapshot gjp22$ open 2013-03-27-060937
macbookpro08-centrim:snapshot gjp22$ clear


Focusing on the directory (and subdirectories) from which I removed the multi-gigabyte files:

2013-03-27-060937, 96G in the VirtualBox directory

Code: Select all
macbookpro08-centrim:snapshot gjp22$ sudo du -hs /Volumes/zhandy/.zfs/snapshot/2013-03-27-060937/Users/gjp22/Library/VirtualBox
 96G   /Volumes/zhandy/.zfs/snapshot/2013-03-27-060937/Users/gjp22/Library/VirtualBox


Back one step on the timeline, opening the previous snapshot in Finder:

Code: Select all
macbookpro08-centrim:snapshot gjp22$ open 2013-03-27-020918
macbookpro08-centrim:snapshot gjp22$ clear


2013-03-27-020918, 128G in the VirtualBox directory

Code: Select all
macbookpro08-centrim:snapshot gjp22$ sudo du -hs /Volumes/zhandy/.zfs/snapshot/2013-03-27-020918/Users/gjp22/Library/VirtualBox
128G   /Volumes/zhandy/.zfs/snapshot/2013-03-27-020918/Users/gjp22/Library/VirtualBox


There we have it:

  • 128G minus 96G = 32G

– but nothing like that 32 amongst the snapshots for the day:

Code: Select all
macbookpro08-centrim:~ gjp22$ zfs list -t snapshot | grep zhandy@2013-03-27
zhandy@2013-03-27-020918                     177Mi       -   439Gi  -
zhandy@2013-03-27-060937                     408Ki       -   408Gi  -
zhandy@2013-03-27-062020                     440Ki       -   408Gi  -


A current DiskInventory X view of the VirtualBox directory


The ~311 GB is curious, especially as the view is not of the root of the file system (and so, should not encounter snapdir stuff). However it's a relatively old app, circa 2006, so I might not expect it to work perfectly in a ZFS environment.

An error-free scrub on 2013-03-25

Code: Select all
macbookpro08-centrim:~ gjp22$ sudo zpool status -v zhandy
  pool: zhandy
 state: ONLINE
 scan: scrub repaired 0 in 13h20m with 0 errors on Mon Mar 25 11:45:50 2013
config:

   NAME                                         STATE     READ WRITE CKSUM
   zhandy                                       ONLINE       0     0     0
     GPTE_A54431D5-B46F-44A9-83B4-76802A584C6E  ONLINE       0     0     0  at disk2s2

errors: No known data errors


– and I'm almost certain that I have never seen an error for this pool.

----

I'm tempted to scrub … but as this pool is puzzling/worrying me, I have decided to stop all automated scrubs until things are understandable.

Note to self:

Code: Select all
macbookpro08-centrim:~ gjp22$ date
Thu 28 Mar 2013 10:34:44 GMT
macbookpro08-centrim:~ gjp22$ sudo launchctl list | grep scrub
-   0   com.getgreenbytes.zfs.autopoolscrubs
-   0   com.getgreenbytes.zevo.forum.satadru.zpadmin-scrub
macbookpro08-centrim:~ gjp22$ defaults read /Library/LaunchDaemons/com.getgreenbytes.zevo.forum.satadru.zpadmin-scrub
{
    Label = "com.getgreenbytes.zevo.forum.satadru.zpadmin-scrub";
    ProgramArguments =     (
        "/opt/local/bin/com.getgreenbytes.zevo.forum.satadru.zpadmin.pl",
        "-scrub"
    );
    StartCalendarInterval =     {
        Hour = 10;
        Minute = 0;
        Weekday = 1;
    };
    StartOnMount = 0;
}
macbookpro08-centrim:~ gjp22$ sudo launchctl unload -w /Library/LaunchDaemons/com.getgreenbytes.zevo.forum.satadru.zpadmin-scrub.plist
macbookpro08-centrim:~ gjp22$ defaults read /System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autopoolscrubs-weekly
{
    EnvironmentVariables =     {
        "COM_GETGREENBYTES_ZFS_NOLOAD" = 1;
    };
    KeepAlive = 0;
    Label = "com.getgreenbytes.zfs.autopoolscrubs";
    ProgramArguments =     (
        "/System/Library/Filesystems/zfs.fs/Contents/Resources/bin/zpool",
        scrub,
        automatic
    );
    RunAtLoad = 0;
    StandardOutPath = "/dev/null";
    StartInterval = 604800;
}
macbookpro08-centrim:~ gjp22$ sudo launchctl unload -w /System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autopoolscrubs-weekly.plist
macbookpro08-centrim:~ gjp22$ sudo launchctl list | grep scrub
macbookpro08-centrim:~ gjp22$
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: visible sum of snapshots is less than usedbysnapshots

Post by raattgift » Thu Mar 28, 2013 8:26 am

Differences in reported space occupancy is a common theme since the start of ZFS; it's not unique to ZEVO at all.

There are plenty of discussions about "zfs space accounting" online that you should be able to find.

You probably want to use "zfs list -o space -r -t all".

Also note what TFM says about the usedbysnapshots dataset property ("is not simply the sum of the snapshots' used properties" and that it's the "space that would be freed if all of this dataset's snapshots were destroyed"). Although TFM only offers one example of why the numbers you're asking about might diverge, there are several possible reasons.

Additionally, you should REALLLY consider avoiding the dataset compression=gzip-9 for a variety of reasons. It is likely to cause you performance problems without any significant space advantage over the standard lzjb (compression=on) except in highly unusual circumstances (recordsize-aligned data that is written once in a single-thread stream and read extremely infrequently).

If your data is really that compressible, you will always gain from using gzip-9 on the userland side rather on the filesystem side.

xar(1) is your Apple-supported friend.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: visible sum of snapshots is less than usedbysnapshots

Post by grahamperrin » Thu Mar 28, 2013 11:36 am

Compression, please see this post under Existing pool -- compression level gzip-9.

Space accounting, I think I understand. Highlights from a few months ago when I bookmarked and read Oracle's ZFS Space Accounting (Solaris ZFS Administration Guide). More recently bookmarked: What is Shared Snapshot Space? | Matt Ahrens (2012-09-28).

I'm familiar with a routine of occasionally destroying selected snapshots – snapshots greater than a particular size – to yield free space.

Now with this pool there should be an opportunity for me to delete one snapshot or more from 2013-03-27 … those snapshots were performed after I removed the massive files.

From the zfs list results above, AFAICT even if I destroy all snapshots it won't free the expected amount of space. The 32G or thereabouts.

If I'm missing (or have forgotten) something fundamental, sorry!

Code: Select all
macbookpro08-centrim:~ gjp22$ zfs list -o space -r -t all | grep -e zhandy -e AVAIL
NAME                                         AVAIL    USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zhandy                                      10.9Gi   572Gi     107Gi   408Gi              0     57.3Gi
zhandy@2013-03-12-224158                         -  31.2Mi         -       -              -          -
zhandy@2013-03-13-003952                         -  23.2Mi         -       -              -          -
zhandy@2013-03-13-092155                         -       0         -       -              -          -
zhandy@2013-03-13-102154                         -       0         -       -              -          -
zhandy@2013-03-17-072706                         -  26.4Mi         -       -              -          -
zhandy@2013-03-17-082706                         -  20.8Mi         -       -              -          -
zhandy@2013-03-17-092706                         -  22.7Mi         -       -              -          -
zhandy@2013-03-17-102706                         -  29.6Mi         -       -              -          -
zhandy@2013-03-19-022206                         -  22.6Mi         -       -              -          -
zhandy@2013-03-19-032206                         -  7.68Mi         -       -              -          -
zhandy@2013-03-19-075930                         -   280Ki         -       -              -          -
zhandy@2013-03-19-102926                         -       0         -       -              -          -
zhandy@2013-03-19-112927                         -       0         -       -              -          -
zhandy@2013-03-19-122927                         -       0         -       -              -          -
zhandy@2013-03-19-134210                         -       0         -       -              -          -
zhandy@2013-03-20-075025                         -       0         -       -              -          -
zhandy@2013-03-20-085024                         -       0         -       -              -          -
zhandy@2013-03-20-095024                         -       0         -       -              -          -
zhandy@2013-03-20-105024                         -       0         -       -              -          -
zhandy@2013-03-20-115024                         -       0         -       -              -          -
zhandy@2013-03-20-180101                         -   433Mi         -       -              -          -
zhandy@2013-03-23-191414                         -       0         -       -              -          -
zhandy@2013-03-23-195116                         -       0         -       -              -          -
zhandy@2013-03-23-204418                         -   280Ki         -       -              -          -
zhandy@2013-03-24-115609                         -   288Ki         -       -              -          -
zhandy@2013-03-25-113051                         -       0         -       -              -          -
zhandy@2013-03-25-123050                         -       0         -       -              -          -
zhandy@2013-03-25-133050                         -       0         -       -              -          -
zhandy@2013-03-25-143050                         -       0         -       -              -          -
zhandy@2013-03-25-153049                         -       0         -       -              -          -
zhandy@2013-03-25-163049                         -       0         -       -              -          -
zhandy@2013-03-26-065933                         -   645Mi         -       -              -          -
zhandy@2013-03-26-175344                         -  12.8Mi         -       -              -          -
zhandy@2013-03-27-020918                         -   177Mi         -       -              -          -
zhandy@2013-03-27-060937                         -   408Ki         -       -              -          -
zhandy@2013-03-27-062020                         -   440Ki         -       -              -          -
zhandy@2013-03-28-134617                         -       0         -       -              -          -
zhandy@2013-03-28-144616                         -       0         -       -              -          -
zhandy@2013-03-28-154616                         -       0         -       -              -          -
zhandy/pockettimemachine                    10.9Gi  57.2Gi    1.04Mi  57.2Gi              0          0
zhandy/pockettimemachine@2013-03-23-191414       -       0         -       -              -          -
zhandy/pockettimemachine@2013-03-23-195116       -       0         -       -              -          -
zhandy/pockettimemachine@2013-03-23-204418       -       0         -       -              -          -
zhandy/pockettimemachine@2013-03-24-115609       -       0         -       -              -          -
zhandy/pockettimemachine@2013-03-25-235503       -       0         -       -              -          -
zhandy/pockettimemachine@2013-03-26-065933       -       0         -       -              -          -
zhandy/pockettimemachine@2013-03-26-175344       -       0         -       -              -          -
zhandy/pockettimemachine@2013-03-27-020918       -   144Ki         -       -              -          -
zhandy/pockettimemachine@2013-03-27-060937       -       0         -       -              -          -
zhandy/pockettimemachine@2013-03-27-062020       -       0         -       -              -          -
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Additional thoughts

Post by grahamperrin » Thu Mar 28, 2013 11:46 am

For some of the large files that I chose to not remove from the affected file system, I ran commands such as this:

VBoxManage modifyhd <filename> --compact

(reference)

… and I did so for a few files, concurrently. Whilst no errors were apparent – all commands ran to 100% completion – now I wonder whether something weird occurred whilst free space was limited.

Think bubbles … whilst those commands ran, I relied on the Finder view of free space, to see that it never got close to zero. AFAIR Finder indicated more than 10 GB free throughout that period … I forgot that sometimes, Finder lies (If I find screenshots from past incidents, I might post to a separate topic).
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: visible sum of snapshots is less than usedbysnapshots

Post by raattgift » Thu Mar 28, 2013 1:36 pm

Removing large files, or compacting image files (sparseimage or sparsebundle or whatever virtualbox uses; I don't use virtualbox so don't know) will cause COW traffic against the previous snapshot. It won't free up any space in that snapshot or its ancestors.

$ zfs snapshot foo@a
$ rm foo/...
$ hdiutil compact foo/...dmg
$ zfs snapshot foo@b
$ zfs send -i foo@a foo@b | pv > /dev/null # or | wc -c

You'll see that foo@b can be quite large, and the difference between foo@a and its ancestor(s) will be unchanged.

Snapshots capture the whole block state of the dataset. Any changes to any block results in a COW operation, so deletes, erases, overwrites, etc. consume *free* space from the pool, and do not return any.

Space is only returned as free to the pool when there are no references from any snapshot whatsoever to a block.

Really, the important point is that an a snapshot cannot hold a reference into any block created (de novo or via COW) more recently than that snapshot. However, it is almost certain that it will reference data held by previous snapshots, and it will retain that referenced data when previous snapshots are destroyed.

So only blocks held uniquely by a snapshot, and not referenced by either ancestors or descendants, are released when a snapshot is destroyed.

A snapshot's "used" dataset property is precise for amount of space that may be freed only when one does a "zfs revert" to its immediate ancestor.

You might consider exploring with "zfs diff -Ft snap1 snap2"; perhaps that will help with your understanding.
raattgift Offline


 
Posts: 98
Joined: Mon Sep 24, 2012 11:18 pm

Re: visible sum of snapshots is less than usedbysnapshots

Post by grahamperrin » Thu Mar 28, 2013 9:32 pm

grahamperrin wrote:Am I misreading the output?


Yes! My mistake. Thanks to raattgift for patient steering.

(Not trouble with ZEVO. I have flagged the opening post for a move of this topic to general discussion.)

I'm familiar with deletion of a snapshot sometimes causing a change to the USED measurement for an adjacent snapshot, but it's the first time that I've seen it on this scale. Nothing wrong, it simply took me by surprise when I couldn't find a mass amongst the USED measurements.

----

For completeness, for reference only, 2013-03-29 01-23.txt at http://www.wuala.com/grahamperrin/publi ... allery&id= is a record of changes to USED measurements after each of this morning's deletions of snapshots adjacent to the oldest.

Taking care to not delete the most recent snapshot (for send and receive purposes), I'm left with:

  • just two snapshots, the oldest of which uses 77.8Gi.

I could destroy that one – I have a backup of the pool – but the ~32 GB freed by the other destructions is enough for now. (91% capacity at the pool level, in the future I should take it way below the 80% threshold.)

As expected, but not desired, Finder lied for a while.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: visible sum of snapshots is less than usedbysnapshots

Post by scasady » Fri Mar 29, 2013 8:36 am

To put it another way. The used space shown for a snapshot is that data that is unique to that snapshot, so when you delete a file if you only have one snapshot the amount deleted shows up there but If you have multiple snapshots the freed space does not show up in any of them if more than one of them refers to it. The freed space does show up in the "usedbysnapshots" amount. This results in the counter intuitive fact that deleting a snapshot can increase the total reported used by all the snapshots while the usedbysnapshots of the parent goes down. You also lose some space to bookkeeping so deleting files when you have snapshots actually reduces space available by a small amount.
scasady Offline


 
Posts: 45
Joined: Sat Sep 15, 2012 8:00 am

Re: visible sum of snapshots is less than usedbysnapshots

Post by grahamperrin » Fri Mar 29, 2013 12:39 pm

Thanks again … I ran a few highlights over the Matt Ahrens post (not all browsers can show them, but they're there).

I got the logic a little earlier … not whilst reading details from web pages, but whilst far away from the computer. idiot spammers thinking usually does it for me. I realised that most blocks of the recently removed massive files might be in the oldest snapshot.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest

cron