Page 1 of 1

crashplan backups on ZFS volumes

PostPosted: Sun Nov 26, 2017 8:29 pm
by calbear88
Hi, I am using a backup program called Crashplan to backup my data.

Crashplan relies on spotlight to detect new or modified files. If I create a new file on my boot drive (formatted in APFS), and I run crashplan, it will detect the new file and back it up.

If a create a new file on my ZFS formatted volume, crashplan doesn't seem to detect the new file. I have to run a "verify selection" option in crashplan in order for it detect the file and back it up. During this "verify selection" process, crashplan scans the hard drive to look for any file changes (takes about 2 hours for 2TB).

I confirmed that spotlight is set to index to ZFS volume

There seems to a known issue with spotlight indexes on zfs volume that are deleted at reboot reported at viewtopic.php?f=26&t=2826

Perhaps this is a similar issue with spotlight having problems indexing ZFS volumes?

here's some info on how crashplan and spotlight work

https://support.code42.com/CrashPlan/4/ ... _Spotlight

"The CrashPlan app real-time backup feature cannot detect any new or changed files in your file selection if Spotlight does not report the changes to the files to CrashPlan."

Re: crashplan backups on ZFS volumes

PostPosted: Mon Nov 27, 2017 2:51 am
by lundman
I wonder how it queries spotlight to get the items changed? Perhaps we are not setting the correct date/time somewhere

Re: crashplan backups on ZFS volumes

PostPosted: Fri Dec 01, 2017 8:00 pm
by calbear88
HI, I asked Crashplan support about how their software detects real time file changes using spotlight, and why it doesn't work on ZFS volumes. Here's the reply I received

"As configured, Spotlight generates indexes of the files on your computer. CrashPlan reads these indexes for reference when determining what has changed and what has not. While we are not able to go into technical detail as to how this indexed information is organized by Spotlight, this is what the real-time scanning of CrashPlan relies on when seeing files and changes.

In regards to the drive in question with ZFS as its file system, we would not be able to determine how your Spotlight service works with ZFS. Further research is stating that Spotlight has traditionally not worked on ZFS, although we are uncertain if this has changed. We would otherwise not be able to assist with troubleshooting Spotlight on ZFS, or clarify why CrashPlan would not see changes if Spotlight was in fact working.

It is otherwise important to mention that if the drive in question is a network shared location, not local to the device, then real-time scanning would certainly not work.

In short, there is not any more-specific method that CrashPlan takes when using Spotlight, versus other applications that reference it. If you are noticing that this real-time scanning does not work, your best area of investigation is Spotlight itself."

Re: crashplan backups on ZFS volumes

PostPosted: Sun Dec 03, 2017 3:45 pm
by lundman
Yeah nobody really knows how Spotlight works, its got some deep dark secrets inside it. So it sounds like CrashPlan reads the Spotlight index databases directly, which is quite interesting. We do have a bunch of mmap/pagein/out fixes coming, which is cleaning up some issues in that general area - so it would be interesting to see how it runs with those.

Re: crashplan backups on ZFS volumes

PostPosted: Wed Feb 21, 2018 7:02 am
by cactustweeter
That's too bad. I back up with CrashPlan. I was hoping to put my macOS High Sierra home directory on an external ZFS pool, keeping macOS High Sierra boot drive internal, and back it up with CrashPlan. It sounds like that's not a realistic setup if I want to stay with CrashPlan.

Re: crashplan backups on ZFS volumes

PostPosted: Sun Feb 25, 2018 4:28 am
by calbear88
I set my crashplan backup settings to verify the selection every 1 day. Although crashplan won't detect realtime changes, the once a day backup is ok for my needs.

Re: crashplan backups on ZFS volumes

PostPosted: Sun Feb 25, 2018 7:09 am
by cactustweeter
I set my crashplan backup settings to verify the selection every 1 day.

How big is your CrashPlan backup?

Re: crashplan backups on ZFS volumes

PostPosted: Tue Feb 27, 2018 9:01 am
by calbear88
cactustweeter wrote:
I set my crashplan backup settings to verify the selection every 1 day.

How big is your CrashPlan backup?

2.5tb

Re: crashplan backups on ZFS volumes

PostPosted: Tue Oct 02, 2018 8:34 pm
by calbear88
Since upgrading to ZFS 1.7.4 and OS X Mojave, spotlight hasn't been working, and as a result, Crashplan has been unable to backup files that are added or changed on my ZFS volume. For now, I'm saving important files that I want to make sure are backed up on my APFS volume