ZFS TimeMachine style backups update

Moderators: jhartley, MSR734, nola

observation and questions

Post by grahamperrin » Wed Nov 28, 2012 3:44 pm

shaman wrote:… expect there to be the same snapshots …


Consider your total of zero (0) used for all given snapshots of the file system:

Code: Select all
mypool1/lwrpool@2012-11-27-173743               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-173757               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174305               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174319               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174328               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174342               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174402               0       -  31.5Ki  -


Do the three below correspond with times when a script was run?

Code: Select all
mypool2/mypool1/lwrpool@2012-11-27-173757     1Ki       -  31.5Ki  -
mypool2/mypool1/lwrpool@2012-11-27-174342     1Ki       -  31.5Ki  -
mypool2/mypool1/lwrpool@2012-11-27-174402       0       -  31.5Ki  -


At each time when the script was run: was
    mypool1/lwrpool
mounted or not?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

examples

Post by grahamperrin » Wed Nov 28, 2012 3:50 pm

Probably OK here as far as I can tell:

Code: Select all
sh-3.2$ cd ~/Documents/com/github/jollyjinx/ZFS-TimeMachine/028ba53 && sudo ./zfstimemachinebackup.perl --sourcepool=zhandy --destinationpool=tall/backups/zhandy --createsnapshotonsource --recursive --verbose
Password:
Found Keytime: 86400 Valuetime: 300
Found Keytime: 604800 Valuetime: 3600
Found Keytime: 7776000 Valuetime: 86400
Found Keytime: 31471200 Valuetime: 604800
Found Keytime: 314712000 Valuetime: 2635200
Created recursive snapshot zhandy@2012-11-28-190044
Working on Sourcepool: zhandy Destinationpool:tall/backups/zhandy  Maximumtime:314712000
Last common snapshot: 2012-11-27-015606
sending from @2012-11-27-015606 to zhandy@2012-11-28-071152
receiving incremental stream of zhandy@2012-11-28-071152 into tall/backups/zhandy@2012-11-28-071152
sending from @2012-11-28-071152 to zhandy@2012-11-28-081152
received 162MiB stream in 19 seconds (8.54MiB/sec)
receiving incremental stream of zhandy@2012-11-28-081152 into tall/backups/zhandy@2012-11-28-081152
sending from @2012-11-28-081152 to zhandy@2012-11-28-125543
received 77.8MiB stream in 10 seconds (7.78MiB/sec)
receiving incremental stream of zhandy@2012-11-28-125543 into tall/backups/zhandy@2012-11-28-125543
sending from @2012-11-28-125543 to zhandy@2012-11-28-135543
sending from @2012-11-28-135543 to zhandy@2012-11-28-145543
sending from @2012-11-28-145543 to zhandy@2012-11-28-155542
sending from @2012-11-28-155542 to zhandy@2012-11-28-165542
sending from @2012-11-28-165542 to zhandy@2012-11-28-190044
received 8.36MiB stream in 2 seconds (4.18MiB/sec)
receiving incremental stream of zhandy@2012-11-28-135543 into tall/backups/zhandy@2012-11-28-135543
received 312B stream in 2 seconds (156B/sec)
receiving incremental stream of zhandy@2012-11-28-145543 into tall/backups/zhandy@2012-11-28-145543
received 312B stream in 2 seconds (156B/sec)
receiving incremental stream of zhandy@2012-11-28-155542 into tall/backups/zhandy@2012-11-28-155542
received 312B stream in 2 seconds (156B/sec)
receiving incremental stream of zhandy@2012-11-28-165542 into tall/backups/zhandy@2012-11-28-165542
received 312B stream in 2 seconds (156B/sec)
receiving incremental stream of zhandy@2012-11-28-190044 into tall/backups/zhandy@2012-11-28-190044
received 108MiB stream in 11 seconds (9.85MiB/sec)
Will keep snapshot:  2012-11-27-015606=1353981366 Backup in bucket: $backupbucket{147600}=2012-11-27-015606
Will keep snapshot:  2012-11-26-003915=1353890355 Backup in bucket: $backupbucket{237600}=2012-11-26-003915
Will keep snapshot:  2012-11-23-052952=1353648592 Backup in bucket: $backupbucket{478800}=2012-11-23-052952
Will keep snapshot:  2012-11-23-042953=1353644993 Backup in bucket: $backupbucket{482400}=2012-11-23-042953
Will keep snapshot:  2012-11-23-002851=1353630531 Backup in bucket: $backupbucket{496800}=2012-11-23-002851
Will keep snapshot:  2012-11-22-232852=1353626932 Backup in bucket: $backupbucket{500400}=2012-11-22-232852
Will keep snapshot:  2012-11-22-222852=1353623332 Backup in bucket: $backupbucket{504000}=2012-11-22-222852
Will keep snapshot:  2012-11-22-212852=1353619732 Backup in bucket: $backupbucket{507600}=2012-11-22-212852
Will keep snapshot:  2012-11-22-202852=1353616132 Backup in bucket: $backupbucket{511200}=2012-11-22-202852
Will keep snapshot:  2012-11-22-192853=1353612533 Backup in bucket: $backupbucket{514800}=2012-11-22-192853
Will keep snapshot:  2012-11-22-071455=1353568495 Backup in bucket: $backupbucket{558000}=2012-11-22-071455
Will keep snapshot:  2012-11-21-202149=1353529309 Backup in bucket: $backupbucket{597600}=2012-11-21-202149
Will keep snapshot:  2012-11-21-183127=1353522687 Backup in bucket: $backupbucket{604800}=2012-11-21-183127
Will remove snapshot:2012-11-21-081851=1353485931 Backup in bucket: $backupbucket{604800}=2012-11-21-183127
Will remove snapshot:2012-11-21-021521=1353464121 Backup in bucket: $backupbucket{604800}=2012-11-21-183127
Will remove snapshot:2012-11-20-222858=1353450538 Backup in bucket: $backupbucket{604800}=2012-11-21-183127
Will keep snapshot:  2012-11-15-070650=1352963210 Backup in bucket: $backupbucket{1123200}=2012-11-15-070650
Will keep snapshot:  2012-11-13-080918=1352794158 Backup in bucket: $backupbucket{1296000}=2012-11-13-080918
Will keep snapshot:  2012-11-11-133510=1352640910 Backup in bucket: $backupbucket{1468800}=2012-11-11-133510
Will keep snapshot:  2012-11-10-103053=1352543453 Backup in bucket: $backupbucket{1555200}=2012-11-10-103053
Will keep snapshot:  2012-11-08-180628=1352397988 Backup in bucket: $backupbucket{1728000}=2012-11-08-180628
Will keep snapshot:  2012-11-06-190614=1352228774 Backup in bucket: $backupbucket{1814400}=2012-11-06-190614
Will keep snapshot:  2012-11-05-182750=1352140070 Backup in bucket: $backupbucket{1987200}=2012-11-05-182750
Will keep snapshot:  2012-11-03-223241=1351981961 Backup in bucket: $backupbucket{2073600}=2012-11-03-223241
Will keep snapshot:  2012-11-02-093135=1351848695 Backup in bucket: $backupbucket{2246400}=2012-11-02-093135
Will keep snapshot:  2012-10-30-212704=1351632424 Backup in bucket: $backupbucket{2419200}=2012-10-30-212704
Will keep snapshot:  2012-10-28-211212=1351458732 Backup in bucket: $backupbucket{2592000}=2012-10-28-211212
Will keep snapshot:  2012-10-27-140913=1351343353 Backup in bucket: $backupbucket{2764800}=2012-10-27-140913
Will keep snapshot:  2012-10-26-072610=1351232770 Backup in bucket: $backupbucket{2851200}=2012-10-26-072610
Will keep snapshot:  2012-10-21-203838=1350848318 Backup in bucket: $backupbucket{3196800}=2012-10-21-203838
Will keep snapshot:  2012-10-19-191515=1350670515 Backup in bucket: $backupbucket{3456000}=2012-10-19-191515
Will keep snapshot:  2012-10-16-220446=1350421486 Backup in bucket: $backupbucket{3628800}=2012-10-16-220446
Will keep snapshot:  2012-10-15-075739=1350284259 Backup in bucket: $backupbucket{3801600}=2012-10-15-075739
Will keep snapshot:  2012-10-13-210616=1350158776 Backup in bucket: $backupbucket{3888000}=2012-10-13-210616
Will keep snapshot:  2012-10-12-082009=1350026409 Backup in bucket: $backupbucket{4060800}=2012-10-12-082009
Will keep snapshot:  2012-10-10-090408=1349856248 Backup in bucket: $backupbucket{4233600}=2012-10-10-090408
Will keep snapshot:  2012-10-08-192418=1349720658 Backup in bucket: $backupbucket{4406400}=2012-10-08-192418
Will keep snapshot:  2012-10-01-210250=1349121770 Backup in bucket: $backupbucket{4924800}=2012-10-01-210250
Will keep snapshot:  2012-09-30-190307=1349028187 Backup in bucket: $backupbucket{5097600}=2012-09-30-190307
Will keep snapshot:  2012-09-28-092100=1348820460 Backup in bucket: $backupbucket{5270400}=2012-09-28-092100
Will keep snapshot:  2012-09-27-065807=1348725487 Backup in bucket: $backupbucket{5356800}=2012-09-27-065807
Will keep snapshot:  2012-09-22-195223=1348339943 Backup in bucket: $backupbucket{5788800}=2012-09-22-195223
Will keep snapshot:  2012-09-21-065142=1348206702 Backup in bucket: $backupbucket{5875200}=2012-09-21-065142
Working on Sourcepool: zhandy/Pocket Time Machine Destinationpool:tall/backups/zhandy/Pocket Time Machine  Maximumtime:314712000
Last common snapshot: 2012-11-27-015606
sending from @2012-11-27-015606 to zhandy/Pocket Time Machine@2012-11-28-190044
receiving incremental stream of zhandy/Pocket Time Machine@2012-11-28-190044 into tall/backups/zhandy/Pocket Time Machine@2012-11-28-190044
received 429KiB stream in 1 seconds (429KiB/sec)
Will keep snapshot:  2012-11-27-015606=1353981366 Backup in bucket: $backupbucket{147600}=2012-11-27-015606
Will keep snapshot:  2012-11-26-003915=1353890355 Backup in bucket: $backupbucket{237600}=2012-11-26-003915
Will keep snapshot:  2012-11-25-202317=1353874997 Backup in bucket: $backupbucket{252000}=2012-11-25-202317
Will keep snapshot:  2012-11-08-173319=1352395999 Backup in bucket: $backupbucket{1728000}=2012-11-08-173319
Will keep snapshot:  2012-11-05-182242=1352139762 Backup in bucket: $backupbucket{1987200}=2012-11-05-182242
Will keep snapshot:  2012-09-23-153424=1348410864 Backup in bucket: $backupbucket{5702400}=2012-09-23-153424
snapshot not in YYYY-MM-DD-HHMMSS format: 2012-09-23-0330 - ignoring
Will keep snapshot:  2012-09-18-210251=1347998571 Backup in bucket: $backupbucket{6048000}=2012-09-18-210251
sh-3.2$ clear


Code: Select all
sh-3.2$ cd ~/Documents/com/github/jollyjinx/ZFS-TimeMachine/028ba53 && sudo ./zfstimemachinebackup.perl --sourcepool=gjp22 --destinationpool=tall/backups/gjp22 --createsnapshotonsource --recursive --verbose
Found Keytime: 86400 Valuetime: 300
Found Keytime: 604800 Valuetime: 3600
Found Keytime: 7776000 Valuetime: 86400
Found Keytime: 31471200 Valuetime: 604800
Found Keytime: 314712000 Valuetime: 2635200
Created recursive snapshot gjp22@2012-11-28-190208
Working on Sourcepool: gjp22 Destinationpool:tall/backups/gjp22  Maximumtime:314712000
Last common snapshot: 2012-11-27-013703
sending from @2012-11-27-013703 to gjp22@2012-11-27-030216
receiving incremental stream of gjp22@2012-11-27-030216 into tall/backups/gjp22@2012-11-27-030216
sending from @2012-11-27-030216 to gjp22@2012-11-27-040216
received 228MiB stream in 72 seconds (3.17MiB/sec)
receiving incremental stream of gjp22@2012-11-27-040216 into tall/backups/gjp22@2012-11-27-040216
sending from @2012-11-27-040216 to gjp22@2012-11-27-050216
received 436MiB stream in 69 seconds (6.32MiB/sec)
receiving incremental stream of gjp22@2012-11-27-050216 into tall/backups/gjp22@2012-11-27-050216
sending from @2012-11-27-050216 to gjp22@2012-11-27-065032
received 273MiB stream in 44 seconds (6.22MiB/sec)
receiving incremental stream of gjp22@2012-11-27-065032 into tall/backups/gjp22@2012-11-27-065032
sending from @2012-11-27-065032 to gjp22@2012-11-27-075032
received 654MiB stream in 147 seconds (4.45MiB/sec)
receiving incremental stream of gjp22@2012-11-27-075032 into tall/backups/gjp22@2012-11-27-075032
sending from @2012-11-27-075032 to gjp22@2012-11-27-111521
received 325MiB stream in 72 seconds (4.51MiB/sec)
receiving incremental stream of gjp22@2012-11-27-111521 into tall/backups/gjp22@2012-11-27-111521
sending from @2012-11-27-111521 to gjp22@2012-11-27-121521
received 414MiB stream in 114 seconds (3.63MiB/sec)
receiving incremental stream of gjp22@2012-11-27-121521 into tall/backups/gjp22@2012-11-27-121521
sending from @2012-11-27-121521 to gjp22@2012-11-27-131521
received 184MiB stream in 49 seconds (3.75MiB/sec)
receiving incremental stream of gjp22@2012-11-27-131521 into tall/backups/gjp22@2012-11-27-131521
sending from @2012-11-27-131521 to gjp22@2012-11-27-141520
received 162MiB stream in 37 seconds (4.39MiB/sec)
receiving incremental stream of gjp22@2012-11-27-141520 into tall/backups/gjp22@2012-11-27-141520
sending from @2012-11-27-141520 to gjp22@2012-11-27-151520
received 254MiB stream in 106 seconds (2.39MiB/sec)
receiving incremental stream of gjp22@2012-11-27-151520 into tall/backups/gjp22@2012-11-27-151520
sending from @2012-11-27-151520 to gjp22@2012-11-27-161520
received 265MiB stream in 54 seconds (4.90MiB/sec)
receiving incremental stream of gjp22@2012-11-27-161520 into tall/backups/gjp22@2012-11-27-161520
sending from @2012-11-27-161520 to gjp22@2012-11-27-181932
received 192MiB stream in 47 seconds (4.08MiB/sec)
receiving incremental stream of gjp22@2012-11-27-181932 into tall/backups/gjp22@2012-11-27-181932
sending from @2012-11-27-181932 to gjp22@2012-11-27-191931
received 240MiB stream in 48 seconds (5.00MiB/sec)
receiving incremental stream of gjp22@2012-11-27-191931 into tall/backups/gjp22@2012-11-27-191931
sending from @2012-11-27-191931 to gjp22@2012-11-27-201931
received 117MiB stream in 29 seconds (4.05MiB/sec)
receiving incremental stream of gjp22@2012-11-27-201931 into tall/backups/gjp22@2012-11-27-201931
sending from @2012-11-27-201931 to gjp22@2012-11-27-211931
received 154MiB stream in 38 seconds (4.05MiB/sec)
receiving incremental stream of gjp22@2012-11-27-211931 into tall/backups/gjp22@2012-11-27-211931
sending from @2012-11-27-211931 to gjp22@2012-11-27-221931
received 193MiB stream in 50 seconds (3.86MiB/sec)
receiving incremental stream of gjp22@2012-11-27-221931 into tall/backups/gjp22@2012-11-27-221931
sending from @2012-11-27-221931 to gjp22@2012-11-27-231931
received 190MiB stream in 56 seconds (3.39MiB/sec)
receiving incremental stream of gjp22@2012-11-27-231931 into tall/backups/gjp22@2012-11-27-231931
sending from @2012-11-27-231931 to gjp22@2012-11-28-001930
received 226MiB stream in 64 seconds (3.53MiB/sec)
receiving incremental stream of gjp22@2012-11-28-001930 into tall/backups/gjp22@2012-11-28-001930
sending from @2012-11-28-001930 to gjp22@2012-11-28-012032
received 263MiB stream in 60 seconds (4.38MiB/sec)
receiving incremental stream of gjp22@2012-11-28-012032 into tall/backups/gjp22@2012-11-28-012032
sending from @2012-11-28-012032 to gjp22@2012-11-28-022032
received 472MiB stream in 67 seconds (7.05MiB/sec)
receiving incremental stream of gjp22@2012-11-28-022032 into tall/backups/gjp22@2012-11-28-022032
sending from @2012-11-28-022032 to gjp22@2012-11-28-032032
received 118MiB stream in 31 seconds (3.81MiB/sec)
receiving incremental stream of gjp22@2012-11-28-032032 into tall/backups/gjp22@2012-11-28-032032
sending from @2012-11-28-032032 to gjp22@2012-11-28-052222
received 153MiB stream in 38 seconds (4.02MiB/sec)
receiving incremental stream of gjp22@2012-11-28-052222 into tall/backups/gjp22@2012-11-28-052222
sending from @2012-11-28-052222 to gjp22@2012-11-28-110821
received 247MiB stream in 783 seconds (323KiB/sec)
receiving incremental stream of gjp22@2012-11-28-110821 into tall/backups/gjp22@2012-11-28-110821
sending from @2012-11-28-110821 to gjp22@2012-11-28-183943
received 975MiB stream in 194 seconds (5.03MiB/sec)
receiving incremental stream of gjp22@2012-11-28-183943 into tall/backups/gjp22@2012-11-28-183943
sending from @2012-11-28-183943 to gjp22@2012-11-28-190208
received 941MiB stream in 208 seconds (4.52MiB/sec)
receiving incremental stream of gjp22@2012-11-28-190208 into tall/backups/gjp22@2012-11-28-190208
received 94.3MiB stream in 32 seconds (2.95MiB/sec)
Will keep snapshot:  2012-11-27-013703=1353980223 Backup in bucket: $backupbucket{151200}=2012-11-27-013703
Will remove snapshot:2012-11-27-012556=1353979556 Backup in bucket: $backupbucket{151200}=2012-11-27-013703
Will keep snapshot:  2012-11-27-002556=1353975956 Backup in bucket: $backupbucket{154800}=2012-11-27-002556
Will keep snapshot:  2012-11-26-232557=1353972357 Backup in bucket: $backupbucket{158400}=2012-11-26-232557
Will keep snapshot:  2012-11-26-222556=1353968756 Backup in bucket: $backupbucket{162000}=2012-11-26-222556
Will keep snapshot:  2012-11-26-212558=1353965158 Backup in bucket: $backupbucket{165600}=2012-11-26-212558
Will keep snapshot:  2012-11-26-190701=1353956821 Backup in bucket: $backupbucket{172800}=2012-11-26-190701
Will keep snapshot:  2012-11-26-165107=1353948667 Backup in bucket: $backupbucket{180000}=2012-11-26-165107
Will keep snapshot:  2012-11-26-155108=1353945068 Backup in bucket: $backupbucket{183600}=2012-11-26-155108
Will keep snapshot:  2012-11-26-133851=1353937131 Backup in bucket: $backupbucket{194400}=2012-11-26-133851
Will keep snapshot:  2012-11-26-123851=1353933531 Backup in bucket: $backupbucket{198000}=2012-11-26-123851
Will keep snapshot:  2012-11-26-113852=1353929932 Backup in bucket: $backupbucket{201600}=2012-11-26-113852
Will keep snapshot:  2012-11-26-103851=1353926331 Backup in bucket: $backupbucket{205200}=2012-11-26-103851
Will keep snapshot:  2012-11-26-084313=1353919393 Backup in bucket: $backupbucket{212400}=2012-11-26-084313
Will keep snapshot:  2012-11-26-074314=1353915794 Backup in bucket: $backupbucket{216000}=2012-11-26-074314
Will keep snapshot:  2012-11-26-064314=1353912194 Backup in bucket: $backupbucket{219600}=2012-11-26-064314
Will keep snapshot:  2012-11-26-042243=1353903763 Backup in bucket: $backupbucket{226800}=2012-11-26-042243
Will keep snapshot:  2012-11-26-032243=1353900163 Backup in bucket: $backupbucket{230400}=2012-11-26-032243
Will keep snapshot:  2012-11-26-022242=1353896562 Backup in bucket: $backupbucket{234000}=2012-11-26-022242
Will keep snapshot:  2012-11-26-012242=1353892962 Backup in bucket: $backupbucket{237600}=2012-11-26-012242
Will remove snapshot:2012-11-26-011402=1353892442 Backup in bucket: $backupbucket{237600}=2012-11-26-012242
Will keep snapshot:  2012-11-26-002139=1353889299 Backup in bucket: $backupbucket{241200}=2012-11-26-002139
Will keep snapshot:  2012-11-25-232915=1353886155 Backup in bucket: $backupbucket{244800}=2012-11-25-232915
Will keep snapshot:  2012-11-25-222557=1353882357 Backup in bucket: $backupbucket{248400}=2012-11-25-222557
Will keep snapshot:  2012-11-25-213515=1353879315 Backup in bucket: $backupbucket{252000}=2012-11-25-213515
Will keep snapshot:  2012-11-25-202140=1353874900 Backup in bucket: $backupbucket{255600}=2012-11-25-202140
Will keep snapshot:  2012-11-25-192141=1353871301 Backup in bucket: $backupbucket{259200}=2012-11-25-192141
Will keep snapshot:  2012-11-25-182141=1353867701 Backup in bucket: $backupbucket{262800}=2012-11-25-182141
Will keep snapshot:  2012-11-25-172141=1353864101 Backup in bucket: $backupbucket{266400}=2012-11-25-172141
Will keep snapshot:  2012-11-25-162141=1353860501 Backup in bucket: $backupbucket{270000}=2012-11-25-162141
Will keep snapshot:  2012-11-25-152141=1353856901 Backup in bucket: $backupbucket{273600}=2012-11-25-152141
Will keep snapshot:  2012-11-25-142142=1353853302 Backup in bucket: $backupbucket{277200}=2012-11-25-142142
Will keep snapshot:  2012-11-25-132142=1353849702 Backup in bucket: $backupbucket{280800}=2012-11-25-132142
Will keep snapshot:  2012-11-25-122142=1353846102 Backup in bucket: $backupbucket{284400}=2012-11-25-122142
Will keep snapshot:  2012-11-25-112142=1353842502 Backup in bucket: $backupbucket{288000}=2012-11-25-112142
Will keep snapshot:  2012-11-25-091839=1353835119 Backup in bucket: $backupbucket{295200}=2012-11-25-091839
Will keep snapshot:  2012-11-25-080547=1353830747 Backup in bucket: $backupbucket{298800}=2012-11-25-080547
Will keep snapshot:  2012-11-25-045845=1353819525 Backup in bucket: $backupbucket{309600}=2012-11-25-045845
Will keep snapshot:  2012-11-25-035845=1353815925 Backup in bucket: $backupbucket{313200}=2012-11-25-035845
Will keep snapshot:  2012-11-25-025845=1353812325 Backup in bucket: $backupbucket{316800}=2012-11-25-025845
Will keep snapshot:  2012-11-25-015845=1353808725 Backup in bucket: $backupbucket{320400}=2012-11-25-015845
Will keep snapshot:  2012-11-25-005846=1353805126 Backup in bucket: $backupbucket{324000}=2012-11-25-005846
Will keep snapshot:  2012-11-24-235846=1353801526 Backup in bucket: $backupbucket{327600}=2012-11-24-235846
Will keep snapshot:  2012-11-24-225846=1353797926 Backup in bucket: $backupbucket{331200}=2012-11-24-225846
Will keep snapshot:  2012-11-24-215847=1353794327 Backup in bucket: $backupbucket{334800}=2012-11-24-215847
Will keep snapshot:  2012-11-24-185847=1353783527 Backup in bucket: $backupbucket{345600}=2012-11-24-185847
Will keep snapshot:  2012-11-24-175848=1353779928 Backup in bucket: $backupbucket{349200}=2012-11-24-175848
Will keep snapshot:  2012-11-24-165848=1353776328 Backup in bucket: $backupbucket{352800}=2012-11-24-165848
Will keep snapshot:  2012-11-24-155848=1353772728 Backup in bucket: $backupbucket{356400}=2012-11-24-155848
Will keep snapshot:  2012-11-24-145848=1353769128 Backup in bucket: $backupbucket{360000}=2012-11-24-145848
Will keep snapshot:  2012-11-24-125849=1353761929 Backup in bucket: $backupbucket{367200}=2012-11-24-125849
Will keep snapshot:  2012-11-24-105849=1353754729 Backup in bucket: $backupbucket{374400}=2012-11-24-105849
Will keep snapshot:  2012-11-24-095849=1353751129 Backup in bucket: $backupbucket{378000}=2012-11-24-095849
Will keep snapshot:  2012-11-24-085848=1353747528 Backup in bucket: $backupbucket{381600}=2012-11-24-085848
Will keep snapshot:  2012-11-24-054339=1353735819 Backup in bucket: $backupbucket{396000}=2012-11-24-054339
Will keep snapshot:  2012-11-24-030049=1353726049 Backup in bucket: $backupbucket{403200}=2012-11-24-030049
Will keep snapshot:  2012-11-24-020049=1353722449 Backup in bucket: $backupbucket{406800}=2012-11-24-020049
Will keep snapshot:  2012-11-24-010049=1353718849 Backup in bucket: $backupbucket{410400}=2012-11-24-010049
Will keep snapshot:  2012-11-23-235349=1353714829 Backup in bucket: $backupbucket{414000}=2012-11-23-235349
Will keep snapshot:  2012-11-23-225350=1353711230 Backup in bucket: $backupbucket{417600}=2012-11-23-225350
Will keep snapshot:  2012-11-23-215350=1353707630 Backup in bucket: $backupbucket{421200}=2012-11-23-215350
Will keep snapshot:  2012-11-23-205350=1353704030 Backup in bucket: $backupbucket{424800}=2012-11-23-205350
Will keep snapshot:  2012-11-23-195350=1353700430 Backup in bucket: $backupbucket{428400}=2012-11-23-195350
Will keep snapshot:  2012-11-23-185349=1353696829 Backup in bucket: $backupbucket{432000}=2012-11-23-185349
Will keep snapshot:  2012-11-23-175349=1353693229 Backup in bucket: $backupbucket{435600}=2012-11-23-175349
Will keep snapshot:  2012-11-23-155408=1353686048 Backup in bucket: $backupbucket{442800}=2012-11-23-155408
Will keep snapshot:  2012-11-23-145408=1353682448 Backup in bucket: $backupbucket{446400}=2012-11-23-145408
Will keep snapshot:  2012-11-23-135408=1353678848 Backup in bucket: $backupbucket{450000}=2012-11-23-135408
Will keep snapshot:  2012-11-23-125408=1353675248 Backup in bucket: $backupbucket{453600}=2012-11-23-125408
Will keep snapshot:  2012-11-22-164338=1353602618 Backup in bucket: $backupbucket{529200}=2012-11-22-164338
Will keep snapshot:  2012-11-22-154338=1353599018 Backup in bucket: $backupbucket{532800}=2012-11-22-154338
Will keep snapshot:  2012-11-22-114123=1353584483 Backup in bucket: $backupbucket{547200}=2012-11-22-114123
Will keep snapshot:  2012-11-22-072421=1353569061 Backup in bucket: $backupbucket{561600}=2012-11-22-072421
Will keep snapshot:  2012-11-22-062421=1353565461 Backup in bucket: $backupbucket{565200}=2012-11-22-062421
Will keep snapshot:  2012-11-22-052421=1353561861 Backup in bucket: $backupbucket{568800}=2012-11-22-052421
Will keep snapshot:  2012-11-22-042422=1353558262 Backup in bucket: $backupbucket{572400}=2012-11-22-042422
Will keep snapshot:  2012-11-22-032422=1353554662 Backup in bucket: $backupbucket{576000}=2012-11-22-032422
Will keep snapshot:  2012-11-22-022422=1353551062 Backup in bucket: $backupbucket{579600}=2012-11-22-022422
Will keep snapshot:  2012-11-22-012422=1353547462 Backup in bucket: $backupbucket{583200}=2012-11-22-012422
Will keep snapshot:  2012-11-22-002422=1353543862 Backup in bucket: $backupbucket{586800}=2012-11-22-002422
Will keep snapshot:  2012-11-21-232423=1353540263 Backup in bucket: $backupbucket{590400}=2012-11-21-232423
Will keep snapshot:  2012-11-21-222423=1353536663 Backup in bucket: $backupbucket{594000}=2012-11-21-222423
Will keep snapshot:  2012-11-21-212423=1353533063 Backup in bucket: $backupbucket{597600}=2012-11-21-212423
Will keep snapshot:  2012-11-21-202423=1353529463 Backup in bucket: $backupbucket{601200}=2012-11-21-202423
Will keep snapshot:  2012-11-21-192424=1353525864 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-183119=1353522679 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-171212=1353517932 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-161212=1353514332 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-151212=1353510732 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-131213=1353503533 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-075247=1353484367 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-021507=1353464107 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-011151=1353460311 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-21-001151=1353456711 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-20-231152=1353453112 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-20-221152=1353449512 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-20-211152=1353445912 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will remove snapshot:2012-11-20-201151=1353442311 Backup in bucket: $backupbucket{604800}=2012-11-21-192424
Will keep snapshot:  2012-11-20-165721=1353430641 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-155721=1353427041 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-145721=1353423441 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-135722=1353419842 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-125722=1353416242 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-115722=1353412642 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-105721=1353409041 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-080432=1353398672 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-070432=1353395072 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-060433=1353391473 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-050433=1353387873 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-040433=1353384273 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-030434=1353380674 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-020434=1353377074 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will remove snapshot:2012-11-20-010434=1353373474 Backup in bucket: $backupbucket{691200}=2012-11-20-165721
Will keep snapshot:  2012-11-19-010440=1353287080 Backup in bucket: $backupbucket{777600}=2012-11-19-010440
Will keep snapshot:  2012-11-18-000446=1353197086 Backup in bucket: $backupbucket{864000}=2012-11-18-000446
Will keep snapshot:  2012-11-17-000452=1353110692 Backup in bucket: $backupbucket{950400}=2012-11-17-000452
Will keep snapshot:  2012-11-16-000458=1353024298 Backup in bucket: $backupbucket{1036800}=2012-11-16-000458
Working on Sourcepool: gjp22/intrigue Destinationpool:tall/backups/gjp22/intrigue  Maximumtime:314712000
Last common snapshot: 2012-11-27-013703
sending from @2012-11-27-013703 to gjp22/intrigue@2012-11-28-190208
receiving incremental stream of gjp22/intrigue@2012-11-28-190208 into tall/backups/gjp22/intrigue@2012-11-28-190208
received 253KiB stream in 2 seconds (127KiB/sec)
Will keep snapshot:  2012-11-27-013703=1353980223 Backup in bucket: $backupbucket{151200}=2012-11-27-013703
Will keep snapshot:  2012-11-26-011402=1353892442 Backup in bucket: $backupbucket{237600}=2012-11-26-011402
Will keep snapshot:  2012-11-26-001238=1353888758 Backup in bucket: $backupbucket{241200}=2012-11-26-001238
sh-3.2$


Code: Select all
sh-3.2$ zfs list -t snapshot | grep intrigue
gjp22/intrigue@2012-11-20-220005                            184Ki       -   680Ki  -
gjp22/intrigue@2012-11-25-201117                                0       -   728Ki  -
gjp22/intrigue@2012-11-25-212500                                0       -   728Ki  -
gjp22/intrigue@2012-11-25-213515                                0       -   728Ki  -
gjp22/intrigue@2012-11-25-222557                                0       -   728Ki  -
gjp22/intrigue@2012-11-25-232915                                0       -   728Ki  -
gjp22/intrigue@2012-11-26-001238                                0       -   728Ki  -
gjp22/intrigue@2012-11-26-011402                                0       -   728Ki  -
gjp22/intrigue@2012-11-27-013703                            184Ki       -   744Ki  -
gjp22/intrigue@2012-11-28-190208                                0       -   760Ki  -
tall/backups/gjp22/intrigue@2012-11-26-001238                 8Ki       -   712Ki  -
tall/backups/gjp22/intrigue@2012-11-26-011402                 8Ki       -   712Ki  -
tall/backups/gjp22/intrigue@2012-11-27-013703               176Ki       -   728Ki  -
tall/backups/gjp22/intrigue@2012-11-28-190208                   0       -   748Ki  -
sh-3.2$


Code: Select all
sh-3.2$ sudo zdb -hvvvvv tall | grep intrigue
2012-11-26.00:37:08 zfs receive -v -F tall/backups/gjp22/intrigue
2012-11-26.01:15:22 zfs receive -v -F tall/backups/gjp22/intrigue
2012-11-27.01:55:48 zfs receive -v -F tall/backups/gjp22/intrigue
2012-11-28.19:45:23 zfs receive -v -F tall/backups/gjp22/intrigue
sh-3.2$
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: ZFS TimeMachine style backups update

Post by shuman » Fri Nov 30, 2012 12:37 pm

Sorry I disappeared from this topic for so long, but I think it's working (mostly) now. What I mean by mostly is, I have one large FS under the root pool (zdata/Videos) where the stream just appears to die; perl scripts still running, but no actual data transfer or error messages that I know of. Unfortunately this was the first FS it was sending. I actually think it's probably more to do with transferring via USB2.0 --> USB2.0, not sure though. Anyway, starting with smaller filesystems and transferring individually, to start with, now allows me to execute the zfs TM on all of zdata/Users without issue. This leaves me with two questions:

1) How can I ensure that I got an exact replica of zdata/Users sent to my backup pool? Are there commands I can run to do this analysis?
2) I still need to get my zdata/Videos FS transferred to the backup. Are there monitoring tools I should be running during this process to help ID the issue?

I know this is somewhat veering OT, so please feel free to move if necessary. And once again, very impressive scripting! I just wish I REALLY understood what it was doing.

-Shuman
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

verification of replication

Post by grahamperrin » Fri Nov 30, 2012 2:24 pm

shuman wrote:… 1) How can I ensure that I got an exact replica of zdata/Users sent to my backup pool? …


At least two things come to mind:


Related, in Ask Different: How should a user of OS X verify whether backed up data comprises everything required for lossless restoration?
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

spin off for troubleshooting zfs send | zfs receive

Post by grahamperrin » Fri Nov 30, 2012 2:38 pm

shuman wrote:… 2) I still need to get my zdata/Videos FS transferred to the backup. Are there monitoring tools I should be running during this process to help ID the issue?… 


Some of the approaches to diagnosis will be without Patrick's scripts, so (yes/maybe) a separate topic could be good … especially if it transpires that problems are unrelated to the scripts. Feel free to spin off, and edit your earlier post to include a link. Thanks.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: ZFS TimeMachine style backups update

Post by jollyjinx » Mon Dec 03, 2012 6:14 pm

shuman wrote:I've been trying this script out on some tmp pools just to see how it works and try to understand it. It's very cool, bye the way. I'm having a couple issues I thought maybe someone could help with.

Destination snapshots missing on child filesystems:
Code: Select all
NAME                                         USED   AVAIL   REFER  MOUNTPOINT
mypool1@2012-11-27-173405                       0       -  32.5Ki  -
mypool1@2012-11-27-173429                       0       -  32.5Ki  -
mypool1@2012-11-27-173743                       0       -  32.5Ki  -
mypool1@2012-11-27-173757                       0       -  32.5Ki  -
mypool1@2012-11-27-174305                       0       -  32.5Ki  -
mypool1@2012-11-27-174319                       0       -  32.5Ki  -
mypool1@2012-11-27-174328                       0       -  32.5Ki  -
mypool1@2012-11-27-174342                       0       -  32.5Ki  -
mypool1@2012-11-27-174402                       0       -  32.5Ki  -
mypool1/lwrpool@2012-11-27-173743               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-173757               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174305               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174319               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174328               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174342               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174402               0       -  31.5Ki  -
mypool2/mypool1@2012-11-27-173757             1Ki       -  32.5Ki  -
mypool2/mypool1@2012-11-27-174342             1Ki       -  32.5Ki  -
mypool2/mypool1@2012-11-27-174402               0       -  32.5Ki  -
mypool2/mypool1/lwrpool@2012-11-27-173757     1Ki       -  31.5Ki  -
mypool2/mypool1/lwrpool@2012-11-27-174342     1Ki       -  31.5Ki  -
mypool2/mypool1/lwrpool@2012-11-27-174402       0       -  31.5Ki  -

I would expect there to be the same snapshots at mypool2/mypool1/lwrpool as there are on mypool2/mypool 3 vs. 10

I used the following command:
Code: Select all
./zfstimemachinebackup.perl --sourcepool=mypool1 --destinationpool=mypool2/mypool1 --recursive --snapshotstokeeponsource=5 --createsnapshotonsource


I'm also getting a lot of "dataset is busy" messages:
Code: Select all
Could not destroy snapshot: zfs destroy "mypool1@2012-11-27-173405"cannot destroy 'mypool1@2012-11-27-173429': dataset is busy

This is on a brand new filesystem with no actual files. I'm not sure why it's busy. It shouldn't be doing anything.

Anyway, great work! Any help is greatly appreciated.


I have not seen any dataset busy error messages. It's not the same amount of snapshots as they are less than 5 minutes away and therefore fall into the same 5 minute bucket and will be deleted - you should have seen deletion messages for that.
jollyjinx Offline


 
Posts: 60
Joined: Sun Sep 16, 2012 12:40 pm
Location: Munich - Germany

Re: ZFS TimeMachine style backups update

Post by shuman » Tue Dec 04, 2012 9:45 am

I've moved to using script on an actual pool. The only issue I'm having now is that it appears some of the zfs processes are just staling. Perl script seems to still be running there just isn't any actual transfer of data. The FS causing the problem is a very large FS at zdata/Videos. I've since used the script with the other FSs omitting the "Videos" FS and they seem to be working OK. I'm currently (as we speak) in the process of sending only the zdata/Videos FS with your script. I'm at 771Gi of 935Gi. I think it might actually transfer this time. I believe the problem I'm having is actually with zfs or the cheap 4 bay USB 2 enclosure.

I would be interested in seeing the plist file you created to get launchd to execute the script every 5 minutes.
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

Re: ZFS TimeMachine style backups update

Post by shuman » Tue Dec 04, 2012 12:12 pm

It transferred OK and I was just able to complete running the script recursively from the very top level FS, zdata.
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

Re: ZFS TimeMachine style backups update

Post by si-ghan-bi » Wed Dec 12, 2012 9:56 am

Another option for Time Machine-style snapshots:
https://github.com/aXon/ZFS-snap

What about the other thread?
viewtopic.php?f=4&t=41
aren't they duplicate?
si-ghan-bi Offline


 
Posts: 145
Joined: Sat Sep 15, 2012 5:55 am

Re: ZFS TimeMachine style backups update

Post by jollyjinx » Wed Dec 12, 2012 11:29 am

si-ghan-bi wrote:Another option for Time Machine-style snapshots:
https://github.com/aXon/ZFS-snap

What about the other thread?
viewtopic.php?f=4&t=41
aren't they duplicate?



The other script creates just snapshots it does not send the snapshots to another disk/host.
jollyjinx Offline


 
Posts: 60
Joined: Sun Sep 16, 2012 12:40 pm
Location: Munich - Germany

PreviousNext

Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 1 guest

cron