automatic snapshots and scrubs

Moderators: jhartley, MSR734, nola

automatic snapshots and scrubs

Post by jsb204 » Mon Sep 17, 2012 8:36 pm

Just for everyone's information, I noticed that there are launchd plists contained within the ZEVO installer package, but they do not appear to actually get installed. They are for automatic snapshots and scrubs:

Code: Select all
zfs_autopoolscrubs-monthly.plist
zfs_autopoolscrubs-semimonthly.plist
zfs_autopoolscrubs-weekly.plist
zfs_autosnapshots-daily.plist
zfs_autosnapshots-hourly.plist
zfs_autosnapshots-weekly.plist


Intriguingly, they invoke "zfs snapshot" with the undocumented modifier, "automatic," that creates snapshots of all pools (but not child filesystems) and gives the snapshots timestamps, e.g. mypool@2012-09-17-180742.

Code: Select all
   <key>ProgramArguments</key>
   <array>
      <string>/System/Library/Filesystems/zfs.fs/Contents/Resources/bin/zfs</string>
      <string>snapshot</string>
      <string>automatic</string>
   </array>


Are there any other undocumented features that people might want to know about, for example Time Machine-like snapshot rotation?
jsb204 Offline


 
Posts: 16
Joined: Sat Sep 15, 2012 11:32 am

the six launch daemons for scrub and snapshot

Post by grahamperrin » Mon Sep 17, 2012 9:45 pm

I find all six installed, with some already loaded:

Code: Select all
macbookpro08-centrim:~ gjp22$ sudo launchctl load -w /System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autopoolscrubs-weekly.plist
Password:
com.getgreenbytes.zfs.autopoolscrubs: Already loaded
macbookpro08-centrim:~ gjp22$ sudo launchctl load -w /System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autosnapshots-daily.plist
macbookpro08-centrim:~ gjp22$ sudo launchctl load -w /System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autosnapshots-weekly.plist
com.getgreenbytes.zfs.autosnapshots: Already loaded
macbookpro08-centrim:~ gjp22$ sudo launchctl load -w /System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autosnapshots-hourly.plist
com.getgreenbytes.zfs.autosnapshots: Already loaded


– but maybe too soon to tell whether they'll be effective in 1.1.

Code: Select all
macbookpro08-centrim:~ gjp22$ defaults read /System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autosnapshots-hourly
{
    EnvironmentVariables =     {
        "COM_GETGREENBYTES_ZFS_NOLOAD" = 1;
    };
    KeepAlive = 0;
    Label = "com.getgreenbytes.zfs.autosnapshots";
    ProgramArguments =     (
        "/System/Library/Filesystems/zfs.fs/Contents/Resources/bin/zfs",
        snapshot,
        automatic
    );
    RunAtLoad = 0;
    StandardOutPath = "/dev/null";
    StartInterval = 3600;
}


– I have had no such snapshots.

Instead I have experimented with things such as this, for my dataset zhandy and its child:

Code: Select all
/usr/sbin/zfs snapshot -r zhandy@`date +"%F-%H%M%S"`


At the command line it has the desired effect, but for some reason (probably a basic mistake by me) as an hourly agent, it's not effective:

Code: Select all
macbookpro08-centrim:~ gjp22$ defaults read ~/Library/LaunchAgents/net.redirectme.perrin.zfs.snapshot.zhandy
{
    Label = "net.redirectme.perrin.zfs.snapshot.zhandy";
    ProgramArguments =     (
        "/usr/sbin/zfs",
        snapshot,
        "-r",
        "zhandy@`date",
        "+\\"%F-%H%M%S\\"`"
    );
    StartInterval = 3600;
}
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: the six launch daemons for scrub and snapshot

Post by jsb204 » Tue Sep 18, 2012 1:54 am

grahamperrin wrote:I find all six installed, with some already loaded:


Thanks, I somehow missed that they were installed.

Like you, I see no evidence of automatic snapshots being actually made.
jsb204 Offline


 
Posts: 16
Joined: Sat Sep 15, 2012 11:32 am

limited success with zfs_autosnapshots-hourly.plist

Post by grahamperrin » Thu Sep 20, 2012 3:45 am

Limited success with zfs_autosnapshots-hourly.plist

I experimented in a few ways then found hourly snapshots of multiple file systems.

At least one limitation:

  • I could not achieve recursion – during my test periods, no child file system gained a snapshot.

That partial success was not rocket science, but:

  • for 1.1 currently I should view this as neither officially supported nor thoroughly tested
  • I was careless/lazy with my note-taking during test periods
  • what I did might have been a little beyond the spirit of the ZEVO EULA (v91112)

– so for now, I'll refrain from sharing the details – sorry.

My preferred alternative: local/remote backups Time Machine style, automatic scrub
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: automatic snapshots and scrubs

Post by si-ghan-bi » Fri Sep 28, 2012 4:52 pm

The script works by keeping TM-style snapshots only on the destination, right? Is it possible to make periodic snapshot and manage them in the same wa, without sending the diffs remotely?
si-ghan-bi Offline


 
Posts: 145
Joined: Sat Sep 15, 2012 5:55 am

zfs_autosnapshots-hourly.plist and beyond

Post by grahamperrin » Sat Sep 29, 2012 11:15 pm

I experiment with only one of the scripts,
zfs_autosnapshots-hourly.plist

si-ghan-bi wrote:… TM-style snapshots only on the destination, right? …


From what I see, no; no destination.

A snapshot in ZFS on any Mac is comparable to a Apple Mobile Time Machine snapshot in OS X on a laptop.

A snapshot of a dataset is – by default with ZEVO – written to a subdirectory of a directory that is hidden from Finder. That hidden .zfs directory is at the root of the file system. Currently in the knowledge base we have an example for a dataset named builds.

A fictional example: have just one external hard disk drive manufactured by Acme Corporation. Give that drive entirely to ZEVO, name the one and only volume 'cake'. Automate, or manually create, snapshots of cake. Those snapshots are stored on:

  • the Acme hardware.

Next, what we can do with snapshots …

Destinations

In ZFS terminology, a snapshot may be sent and received; received at a destination.

The destination may be local. (Reception could be at a different external hard disk drive.)

The destination may be remote.

For ZFS enthusiasts there's Oracle's Sending and Receiving ZFS Data - Oracle Solaris ZFS Administration Guide.

For people who are new to ZFS, a guide such as that can be daunting. Scripts such as zfstimemachinebackup.perl can make things easier. Recommended reading:

grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: automatic snapshots and scrubs

Post by si-ghan-bi » Sun Sep 30, 2012 8:43 am

Actually I was referring to the scripts you linked. If I go to the github website I can read:

the script creates a snapshot on the source pool every time it is called.
then it figures out the last snapshot on the destination pool that matches to one on the source pool.
it sends the snapshot from the source to the destination.
removes old snapshots on the source - it keeps just n-snapshots.
removes old snapshots on the destination - time machine fashion : 5min/last day, 1 hour last week, 1 day last 3 months, 1 week thereafter


That is why I wrote "TM-style only on the destination": on the source only n-snapshots are kept.

I would like a script to perform regular snapshots but also automatically delete old ones, while keeping one per hours, then one per day, then one per week.
si-ghan-bi Offline


 
Posts: 145
Joined: Sat Sep 15, 2012 5:55 am

Re: automatic snapshots and scrubs

Post by grahamperrin » Sun Sep 30, 2012 9:00 am

Sorry – I thought you were asking about the GreenBytes-provided items in the opening post, or zfs_autosnapshots-hourly.plist in particular.

There's a separate topic for the third party scripts, would you mind posting there? Thanks.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Integral to ZEVO: limited automatic snapshots and scrubs

Post by grahamperrin » Thu Oct 03, 2013 1:13 am

https://groups.google.com/d/msg/maczfs- ... CtOc6Ze54J observes, amongst the features of ZEVO Community Edition 1.1.1:

    1. …
    2. automatically creates snapshots daily … 
    3. autoscrubs weekly …

With what's integral to ZEVO:

  • automatic snapshots can be daily, hourly or weekly
  • automatic scrubs can be weekly, semi-monthly or monthly.

Some readers may wonder why:


A summary of what's installed:

Code: Select all
sh-3.2$ ls /System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_auto*
/System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autopoolscrubs-monthly.plist
/System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autopoolscrubs-semimonthly.plist
/System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autopoolscrubs-weekly.plist
/System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autosnapshots-daily.plist
/System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autosnapshots-hourly.plist
/System/Library/Filesystems/zfs.fs/Contents/Resources/launchd/zfs_autosnapshots-weekly.plist


Limitations to what's integral to ZEVO CE 1.1.1

From what I can tell, after a year or so of testing the integral approach to automated snapshots:

  1. pools benefit only if automatic import occurs relatively soon after boot
  2. child file systems do not benefit (see above).

Point (a) in simple terms:

  • aim for disks to be present/connected before or during startup of the operating system.

Technically: I guess, you should allow ZEVO to perform its dynamic import before the operating system loads the daemon for the snapshots.

Automated scrubs

Not tested by me. I don't imagine any issue. ylluminate might like to comment.

Side notes

As Apple Core Storage is not designed to allow automated imports, so ZFS pools that use Core Storage for encryption can not benefit from ZEVO's automated snapshots. (For me this is not an issue. My home directory gains snapshots from regular use of ZFS TimeMachine (jollyjinx).)

There's another for the released Community Edition of ZEVO being not full-featured (less appealing than the beta) for automated snapshots, but I'll prefer to keep this quiet … at least until after we discover what's to happen to ZEVO for Mavericks.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Correction and apology

Post by grahamperrin » Sat Oct 05, 2013 1:02 am

… As Apple Core Storage is not designed to allow automated imports, so ZFS pools that use Core Storage for encryption can not benefit from ZEVO's automated snapshots. …


Please ignore that statement, it's wrong. Sorry.

(I can't recall where I got that idea. I realised that there are more than a few hourly snapshots of my Core Storage encrypted ZEVO home directory.)
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest

cron