normalisation and restoration

Moderators: jhartley, MSR734, nola

normalisation and restoration

Post by grahamperrin » Wed Dec 05, 2012 2:16 pm

Consider these properties of a backed up snapshot:

Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get casesensitivity,normalization tall/backups/zhandy@2012-12-03-090916
NAME                                   PROPERTY         VALUE          SOURCE
tall/backups/zhandy@2012-12-03-090916  casesensitivity  insensitive    -
tall/backups/zhandy@2012-12-03-090916  normalization    none           -


Then this, to create a pool with the properties that I will require during restoration:

Code: Select all
macbookpro08-centrim:~ gjp22$ sudo zpool create -o ashift=12 -O casesensitivity=insensitive -O normalization=formD -O atime=off -O snapdir=visible -O compression=gzip-9 zhandy /dev/disk2


Finally this, run as root, to restore:

Code: Select all
sh-3.2# zfs send -v tall/backups/zhandy@2012-12-03-090916 | zfs receive -v -F zhandy


Question

Will the end result – the file system zhandy – use formD?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

review

Post by grahamperrin » Thu Dec 06, 2012 1:10 am

The restoration

Code: Select all
sh-3.2# zfs send -v tall/backups/zhandy@2012-12-03-090916 | zfs receive -v -F zhandy
sending from @ to tall/backups/zhandy@2012-12-03-090916
receiving full stream of tall/backups/zhandy@2012-12-03-090916 into zhandy@2012-12-03-090916
received 479GiB stream in 32463 seconds (15.1MiB/sec)
sh-3.2#


During restoration

Scroll to the foot:

Code: Select all
macbookpro08-centrim:~ gjp22$ zfs get all zhandy
NAME    PROPERTY              VALUE                  SOURCE
zhandy  type                  filesystem             -
zhandy  creation              Wed Dec  5 18:50 2012  -
zhandy  used                  392Gi                  -
zhandy  available             190Gi                  -
zhandy  referenced            480Ki                  -
zhandy  compressratio         1.19x                  -
zhandy  mounted               no                     -
zhandy  quota                 none                   default
zhandy  reservation           none                   default
zhandy  recordsize            128Ki                  default
zhandy  mountpoint            /Volumes/zhandy        default
zhandy  checksum              on                     default
zhandy  compression           gzip-9                 local
zhandy  atime                 off                    local
zhandy  devices               on                     default
zhandy  exec                  on                     default
zhandy  setuid                on                     default
zhandy  readonly              off                    default
zhandy  snapdir               visible                local
zhandy  canmount              on                     default
zhandy  copies                1                      default
zhandy  version               5                      -
zhandy  utf8only              on                     -
zhandy  normalization         formD                  -
zhandy  casesensitivity       insensitive            -
zhandy  refquota              none                   default
zhandy  refreservation        none                   default
zhandy  primarycache          all                    default
zhandy  secondarycache        all                    default
zhandy  usedbysnapshots       0                      -
zhandy  usedbydataset         480Ki                  -
zhandy  usedbychildren        392Gi                  -
zhandy  usedbyrefreservation  0                      -
zhandy  logbias               latency                default
zhandy  sync                  standard               default
macbookpro08-centrim:~ gjp22$ clear


End result: neither normalisation nor utf8only

Scroll to the foot, compare with what's above:

Code: Select all
macbookpro08-centrim:~ gjp22$ date
Thu  6 Dec 2012 04:01:11 GMT
macbookpro08-centrim:~ gjp22$ zfs get all zhandy
NAME    PROPERTY              VALUE                  SOURCE
zhandy  type                  filesystem             -
zhandy  creation              Wed Dec  5 18:50 2012  -
zhandy  used                  403Gi                  -
zhandy  available             180Gi                  -
zhandy  referenced            403Gi                  -
zhandy  compressratio         1.19x                  -
zhandy  mounted               yes                    -
zhandy  quota                 none                   default
zhandy  reservation           none                   default
zhandy  recordsize            128Ki                  default
zhandy  mountpoint            /Volumes/zhandy        default
zhandy  checksum              on                     default
zhandy  compression           gzip-9                 local
zhandy  atime                 off                    local
zhandy  devices               on                     default
zhandy  exec                  on                     default
zhandy  setuid                on                     default
zhandy  readonly              off                    default
zhandy  snapdir               visible                local
zhandy  canmount              on                     default
zhandy  copies                1                      default
zhandy  version               5                      -
zhandy  utf8only              off                    -
zhandy  normalization         none                   -
zhandy  casesensitivity       insensitive            -
zhandy  refquota              none                   default
zhandy  refreservation        none                   default
zhandy  primarycache          all                    default
zhandy  secondarycache        all                    default
zhandy  usedbysnapshots       6.77Mi                 -
zhandy  usedbydataset         403Gi                  -
zhandy  usedbychildren        2.98Mi                 -
zhandy  usedbyrefreservation  0                      -
zhandy  logbias               latency                default
zhandy  sync                  standard               default
macbookpro08-centrim:~ gjp22$ 


I never realised (or forgot) that the original file system was without formD.

Maybe I was careless with a zfs create subcommand a few months ago.

Next steps

I guess:

  1. again recreate the pool with appropriate properties for the file system at its root
  2. in lieu of a zfs receive subcommand, ditto(1) or rsync(1) to restore from the snapshot; I require formD.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

history, March 2012

Post by grahamperrin » Thu Dec 06, 2012 1:29 am

I never realised (or forgot) that the original file system was without formD.

Maybe I was careless with a zfs create subcommand a few months ago.


Not careless, simply ignorant at the time. In Wuala I found a 2012-10-23 view of the history of the original file system.

Critically, when history began on 14th March 2012 (emphasis added):

2012-03-14.17:29:37 zpool create -f -O compression=off -O copies=1 -O casesensitivity=insensitive -O snapdir=visible zhandy /dev/dsk/GPTE_1928482A-7FE4-482D-B692-3EC6B03159BA

To me, now, that's a peculiar combination of properties for the file system. Peculiar as in, not a combination that I recall from that time. Also when working at the command line I'm more likely to specify /dev/disk/… than /dev/dsk/…

I probably used ZEVO Setup Assistant, but something less than 1.0.3. Related:


– seven days after I created the pool, an update to ZEVO recognised that formD was an appropriate default. Months later when I realised the significance of normalisation, it never occurred to me to pay attention to properties of my everyday pools.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: review

Post by grahamperrin » Fri Dec 07, 2012 1:00 am

grahamperrin wrote:Next steps

I guess:

  1. again recreate the pool with appropriate properties for the file system at its root
  2. in lieu of a zfs receive subcommand, ditto(1) or rsync(1) to restore from the snapshot; I require formD.


Everything went well (albeit slower than zfs) with ditto, until restoration was around 98% complete. Then Billy cat trod on what I thought was a well connected USB cable.

Code: Select all
ditto: /Volumes/zhandy/chronological/10.8.2/1.3.18 (352.4)/Install OS X Mountain Lion.app/Contents/SharedSupport/InstallESD.dmg: Input/output error
^Cload: 2.66  cmd: ditto 848 uninterruptible 8.50u 726.45s
load: 2.45  cmd: ditto 848 uninterruptible 8.50u 726.45s
load: 2.45  cmd: ditto 848 uninterruptible 8.50u 726.45s
load: 2.09  cmd: ditto 848 uninterruptible 8.50u 726.45s
^C^C^C


screenshot 2012-12-07 at 05.56.08.png
screenshot 2012-12-07 at 05.56.08.png (21.71 KiB) Viewed 132 times


Hmmph!
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: normalisation and restoration

Post by grahamperrin » Sat Dec 08, 2012 2:08 am

I restarted the computer, replaced a USB cable (it seems that the small plug became unreliable – I can't blame the interruption entirely on the cat) and used Carbon Copy Cloner 3.5.2-b2 (1171) to complete the restoration.

An unexpected end result: an unusual ZEVO icon for a restored file system.

Normalisation and restoration

Back to the opening post. A result that is expected but undesirable:

Code: Select all
macbookpro08-centrim:028ba53 gjp22$ cd ~/Documents/com/github/jollyjinx/ZFS-TimeMachine/028ba53 && sudo ./zfstimemachinebackup.perl --sourcepool=zhandy --destinationpool=tall/backups/zhandy --createsnapshotonsource --recursive --verbose
Found Keytime: 86400 Valuetime: 300
Found Keytime: 604800 Valuetime: 3600
Found Keytime: 7776000 Valuetime: 86400
Found Keytime: 31471200 Valuetime: 604800
Found Keytime: 314712000 Valuetime: 2635200
Created recursive snapshot zhandy@2012-12-08-064803
Working on Sourcepool: zhandy Destinationpool:tall/backups/zhandy  Maximumtime:314712000
Could not find common snapshot between source (zhandy) and destination (tall/backups/zhandy)
Destination snapshots:
   2012-09-21-065142
   2012-09-22-195223
   2012-09-27-065807
   2012-09-28-092100
   2012-09-30-190307
   2012-10-01-210250
   2012-10-08-192418
   2012-10-10-090408
   2012-10-12-082009
   2012-10-13-210616
   2012-10-15-075739
   2012-10-16-220446
   2012-10-19-191515
   2012-10-21-203838
   2012-10-26-072610
   2012-10-27-140913
   2012-10-28-211212
   2012-10-30-212704
   2012-11-02-093135
   2012-11-03-223241
   2012-11-05-182750
   2012-11-06-190614
   2012-11-08-180628
   2012-11-10-103053
   2012-11-11-133510
   2012-11-13-080918
   2012-11-15-070650
   2012-11-22-071455
   2012-11-23-052952
   2012-11-26-003915
   2012-11-27-015606
   2012-11-28-071152
   2012-11-28-081152
   2012-11-28-125543
   2012-11-28-135543
   2012-11-28-145543
   2012-11-28-155542
   2012-11-28-165542
   2012-11-28-190044
   2012-11-30-180717
   2012-12-01-100743
   2012-12-01-141532
   2012-12-01-151531
   2012-12-01-161531
   2012-12-01-171531
   2012-12-01-181530
   2012-12-03-082859
   2012-12-03-090916
Source snapshots:
   2012-12-07-202643
   2012-12-07-212643
   2012-12-07-222642
   2012-12-07-232642
   2012-12-08-002642
   2012-12-08-064803
sending from @ to zhandy@2012-12-08-064803
cannot receive new filesystem stream: destination has snapshots (eg. tall/backups/zhandy@2012-11-22-071455)
must destroy them to overwrite it
Can't execute zfs receive -v -F "tall/backups/zhandy" at ./zfstimemachinebackup.perl line 238.
macbookpro08-centrim:028ba53 gjp22$ warning: cannot send 'zhandy@2012-12-08-064803': Broken pipe
Can't execute zfs send -v "zhandy@2012-12-08-064803" at ./zfstimemachinebackup.perl line 230.

macbookpro08-centrim:028ba53 gjp22$


So in summary:

  • it's highly desirable to use zfs for restoration (to gain at least one snapshot that is common)
  • I know of no way to zfs send from a backup where normalization=none and zfs receive where normalization=formD without losing the normalisation …
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

different file system properties at reception

Post by grahamperrin » Sat Dec 08, 2012 4:36 am

Found under Sending and Receiving ZFS Data - Oracle Solaris ZFS Administration Guide:


There's no hint of this in the man page for zfs(8) with ZEVO Community Edition 1.1.1.

I'll see whether it's possible with ZEVO then if appropriate, request an enhancement to the manual.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

option 'o' is invalid for zfs receive

Post by grahamperrin » Sat Dec 08, 2012 8:38 am

It's not possible. See viewtopic.php?p=3510#p3510
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 0 guests

cron