ZFS TimeMachine style backups update

Moderators: jhartley, MSR734, nola

ZFS TimeMachine style backups update

Post by jollyjinx » Tue Nov 20, 2012 4:07 pm

I've updated my TimeMachine style backup script. ( https://github.com/jollyjinx/ZFS-TimeMachine )

You can now recursively snapshot and send filesystems.
You can specify the hash how long snapshots are kept on the destination on the command line like:

Code: Select all
   --keepbackupshash=24h=>5min,7d=>1h,90d=>1d,1y=>1w,10y=>1month


which means:

Code: Select all
   24h=>5mi   for snapshots younger than keep not more than one per 5 minutes
   7d=>1h      for snapshots younger than 7 days keep not more than one snapshot per 1 hour
   ....



You can specify regexes for to match a maximum time a speficied filesystem is kept on the remote:

Code: Select all
   --maximumtimeperfilesystemhash=.*=>10yrs,.+/(Dropbox|Downloads|Caches|Mail Downloads|Saved Application State|Logs)$=>1month   


which means:


Code: Select all
   .*=>10yrs   keep everything 10 years by default - after that snapshots are removed
   .+/(Dropbox|Downloads|Caches|Mail Downloads|Saved Application State|Logs)$=>1month
            remove snapshots older than one month for directories ending with the regex.



Have fun - Patrick
jollyjinx Offline


 
Posts: 60
Joined: Sun Sep 16, 2012 12:40 pm
Location: Munich - Germany

Re: ZFS TimeMachine style backups update

Post by grahamperrin » Tue Nov 20, 2012 4:39 pm

Cool, thanks! I'll test very soon. A minor point: maybe change the date at line 3 … or some other approach to future identification of versions of the script.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: ZFS TimeMachine style backups update

Post by jollyjinx » Tue Nov 20, 2012 4:43 pm

well it was the inital creation date of the script. I've removed it :-)
jollyjinx Offline


 
Posts: 60
Joined: Sun Sep 16, 2012 12:40 pm
Location: Munich - Germany

unexpected files ('1month' and others) alongside

Post by grahamperrin » Tue Nov 20, 2012 5:09 pm

Issue moved to GitHub: unexpected files ('1month' and others) alongside zfstimemachinebackup.perl

– closed by me 2012-11-25. Thanks to Patrick.
Last edited by grahamperrin on Sun Nov 25, 2012 3:40 pm, edited 2 times in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Option recursive does not take an argument

Post by grahamperrin » Tue Nov 20, 2012 5:34 pm

At https://github.com/jollyjinx/ZFS-TimeMa ... afac#L0R51

Set --recursive=1 if you want to send the pools and all sub pools recursively.


– but then (see below) Option recursive does not take an argument

If I'm doing something wrong, sorry …

Code: Select all
sh-3.2$ cd ~/Documents/com/github/jollyjinx/ZFS-TimeMachine/current && sudo ./zfstimemachinebackup.perl --sourcepool=zhandy --destinationpool=tall/backups/zhandy --snapshotstokeeponsource=100 --createsnapshotonsource --recursive=1
Password:
Option recursive does not take an argument
Found Keytime: 86400 Valuetime: 300
Found Keytime: 604800 Valuetime: 3600
Found Keytime: 7776000 Valuetime: 86400
Found Keytime: 31471200 Valuetime: 604800
Found Keytime: 314712000 Valuetime: 2635200
Found Filesystem: .* Valuetime: 314712000
Found Filesystem: .+/(Dropbox|Downloads|Caches|Mail Downloads|Saved Application State|Logs)$ Valuetime: 2635200
Working on Sourcepool: zhandy Destinationpool:tall/backups/zhandy  Maximumtime:314712000
Last common snapshot:   2012-11-15-070650
sending from @2012-11-15-070650 to zhandy@2012-11-20-222858
receiving incremental stream of zhandy@2012-11-20-222858 into tall/backups/zhandy@2012-11-20-222858
received 96.8MiB stream in 42 seconds (2.31MiB/sec)
Snapshots to delete on source: ,2012-09-30-180307
sh-3.2$ ls -l
total 72
-rw-r--r--  1 gjp22  staff      0 20 Nov 22:00 1d,1y=
-rw-r--r--  1 gjp22  staff      0 20 Nov 22:00 1h,90d=
-rw-r--r--  1 gjp22  staff    140 20 Nov 22:00 1month
-rw-r--r--  1 gjp22  staff      0 20 Nov 22:00 1w,10y=
-rw-r--r--  1 gjp22  staff      0 20 Nov 22:00 5min,7d=
drwxr-xr-x@ 2 gjp22  staff      5 20 Nov 21:00 JNX
-rw-r--r--@ 1 gjp22  staff  10233 20 Nov 21:00 README.md
-rwxr-xr-x@ 1 gjp22  staff   1071 20 Nov 21:00 autoscrub.perl
-rwxr-xr-x@ 1 gjp22  staff   1633 20 Nov 21:00 checkbackup.perl
-rwxr-xr-x@ 1 gjp22  staff  12351 20 Nov 21:50 zfstimemachinebackup.perl
sh-3.2$ cat 1month
Last common snapshot:   2012-11-15-070637
receiving incremental stream of gjp22@2012-11-15-073447 into tall/backups/gjp22@2012-11-15-073447
sh-3.2$
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: ZFS TimeMachine style backups update

Post by jollyjinx » Tue Nov 20, 2012 6:24 pm

The documentation was wrong. --recursive is a flag so it does not need an argument. I've fixed it.
jollyjinx Offline


 
Posts: 60
Joined: Sun Sep 16, 2012 12:40 pm
Location: Munich - Germany

thanks

Post by grahamperrin » Tue Nov 20, 2012 9:50 pm

Thanks (and sorry – I should have posted to GitHub).
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

terminology

Post by grahamperrin » Mon Nov 26, 2012 6:17 am

Patrick, please: are the following expressions specific to your scripts, or more generally understood in relation to ZFS?

  • backupbucket
  • Keytime
  • Maximumtime
  • Valuetime
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: ZFS TimeMachine style backups update

Post by shuman » Tue Nov 27, 2012 7:04 pm

I've been trying this script out on some tmp pools just to see how it works and try to understand it. It's very cool, bye the way. I'm having a couple issues I thought maybe someone could help with.

Destination snapshots missing on child filesystems:
Code: Select all
NAME                                         USED   AVAIL   REFER  MOUNTPOINT
mypool1@2012-11-27-173405                       0       -  32.5Ki  -
mypool1@2012-11-27-173429                       0       -  32.5Ki  -
mypool1@2012-11-27-173743                       0       -  32.5Ki  -
mypool1@2012-11-27-173757                       0       -  32.5Ki  -
mypool1@2012-11-27-174305                       0       -  32.5Ki  -
mypool1@2012-11-27-174319                       0       -  32.5Ki  -
mypool1@2012-11-27-174328                       0       -  32.5Ki  -
mypool1@2012-11-27-174342                       0       -  32.5Ki  -
mypool1@2012-11-27-174402                       0       -  32.5Ki  -
mypool1/lwrpool@2012-11-27-173743               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-173757               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174305               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174319               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174328               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174342               0       -  31.5Ki  -
mypool1/lwrpool@2012-11-27-174402               0       -  31.5Ki  -
mypool2/mypool1@2012-11-27-173757             1Ki       -  32.5Ki  -
mypool2/mypool1@2012-11-27-174342             1Ki       -  32.5Ki  -
mypool2/mypool1@2012-11-27-174402               0       -  32.5Ki  -
mypool2/mypool1/lwrpool@2012-11-27-173757     1Ki       -  31.5Ki  -
mypool2/mypool1/lwrpool@2012-11-27-174342     1Ki       -  31.5Ki  -
mypool2/mypool1/lwrpool@2012-11-27-174402       0       -  31.5Ki  -

I would expect there to be the same snapshots at mypool2/mypool1/lwrpool as there are on mypool2/mypool 3 vs. 10

I used the following command:
Code: Select all
./zfstimemachinebackup.perl --sourcepool=mypool1 --destinationpool=mypool2/mypool1 --recursive --snapshotstokeeponsource=5 --createsnapshotonsource


I'm also getting a lot of "dataset is busy" messages:
Code: Select all
Could not destroy snapshot: zfs destroy "mypool1@2012-11-27-173405"cannot destroy 'mypool1@2012-11-27-173429': dataset is busy

This is on a brand new filesystem with no actual files. I'm not sure why it's busy. It shouldn't be doing anything.

Anyway, great work! Any help is greatly appreciated.
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

unexplained business

Post by grahamperrin » Wed Nov 28, 2012 12:28 am

Wonder whether unexplained business is contributory to the issue with snapshots.

I have a vague memory of seeing unexplained business without use of Patrick's scripts so (maybe) move to a separate topic? Unless this is immediately recognisable to Patrick.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 0 guests

cron