Unable to mount snapshot

All your general support questions for OpenZFS on OS X.

Unable to mount snapshot

Postby slu » Mon Mar 27, 2017 2:59 pm

Hello,

I'm using OpenZFS on OSX Version 1.6.1 running on my Mac mini Server Late 2012 with macOS Sierra Version 10.12.3

The ZFS and Pool configuration is stable and running without any crashes.

But I have no idea why it is not possible to mount snapshots always successfully, which have been created few seconds ago.
Often such messages occur while running, e.g.:
Code: Select all
# sudo zfs mount z_pool/Other@Other
cannot mount '/Volumes/z_pool/Other/.zfs/snapshot/Other': directory is not empty
# echo $?
255


Sometimes a different message occur:
Code: Select all
cannot mount 'z_pool/Other@ Other': No such file or directory


It is necessary to destroy the fresh created snapshot followed by umount corresponding ZFS-Filesystem to mount this back again and to retry creation of new snapshot to successfully mount snapshot.

In less cases it is enough to destroy the snapshot followed by creation of new snapshot to successfully mount snapshot.

Is there any known issue for OpenZFS code or is that a general problem of OS X itself.

Did anybody have the same problem and general resolved this by self.

I'm using snapshots to backup my pool data using Rsync of mounted snapshots to a different ZFS-Pool. In general it works perfect.

But at the moment it is hard to automated this because of the issue to mount snapshots. With each run at least one or more snapshots cannot mounted successfully.

My pool has 10 ZFS-Filesystems all with the same ZFS settings. I'm not using compression or dedup. The parameter snapdir has been set to visible.

I'm thankful about any hint and help.

Greetings and many thanks in advance for your support
slu
slu
 
Posts: 4
Joined: Mon Mar 27, 2017 1:57 pm

Re: Unable to mount snapshot

Postby Brendon » Thu Mar 30, 2017 11:00 pm

Hi,

I'm not 100% sure whether you can mount snapshots at the moment - I know you can't automount them by cding into the .zfs directory.

I access snapshot content by cloning the snapshot then destroying when finished at the moment.

Cheers
Brendon
Brendon
 
Posts: 285
Joined: Thu Mar 06, 2014 12:51 pm

Re: Unable to mount snapshot

Postby slu » Fri Mar 31, 2017 1:46 pm

Hello Brendon,

thanks a lot for your answer.


With reference to https://openzfsonosx.org/wiki/FAQ ..

Q) How can I access the .zfs snapshot directories?
A) You need to set snapdir visible and manually mount a snapshot.
Code: Select all
$ sudo zfs set snapdir=visible tank/bob
$ sudo zfs mount tank/bob@yesterday
$ ls -l /tank/bob/.zfs/snapshot/yesterday/


Q) Is .zfs snapdir auto-mounting supported?
A) No, not at this time. You must manually "zfs mount" snapshots manually to see them in the snapdir.

Q) OK, I manually mounted my snapshot but still cannot see it in Finder. What gives?
A) Currently mounted snapshots are only visible from Terminal, not from Finder.
Code: Select all
$ ls -l /tank/bob/.zfs/snapshot/yesterday/



So, for my opinion ..
1) you are right, auto-mounting snapdir is not support
2) but to mount snapshots manually should work


My pool has 10 ZFS-Filesystems all with the same ZFS settings. I'm not using compression or dedup. The parameter snapdir has been set to visible.

Therefore I don't understand why it is not possible to mount some snapshots after creating snapshots for each of this 10 filesystems.

It is not reproducible which filesystem/snapshot is always a problem. Sometimes only for one or two filesystems it is not possible to mount the fresh created snapshots. Sometimes for three or for filesystems. But never for the same filesystems/snapshots at the next run.

For the most filesystems/snapshots it works and looks like this:
Code: Select all
$ sudo zfs snapshot z_pool/data@data
$ sudo zfs mount z_pool/data@data
ZFS: snapshot mountpoint '/Volumes/z_pool/data/.zfs/snapshot/data'
$ ls -latr /Volumes/z_pool/data/.zfs/snapshot/data
total 300412
..
..
-rw-r--r--   1 root  wheel     239735  2 Mär 22:41 .VolumeIcon.icns
drwx------   5 root  wheel          5  2 Mär 22:41 .Spotlight-V100
d-wx-wx-wt   2 root  wheel          2  2 Mär 22:41 .Trashes
drwxr-xr-x@ 21 root  wheel         21  3 Mär 07:05 .
drwx------  41 root  wheel         41 31 Mär 22:32 .fseventsd
dr-xr-xr-x   3 root  wheel          3 31 Mär 22:32 ..


Depending on the run for some filesystems/snapshots it looks like this:
Code: Select all
$ sudo zfs snapshot z_pool/data@data
$ sudo zfs mount z_pool/data@data
cannot mount '/Volumes/z_pool/data/.zfs/snapshot/data': directory is not empty
$ sudo zfs destroy z_pool/data@data
$ sudo zfs snapshot z_pool/data@data
$ sudo zfs mount z_pool/data@data
ZFS: snapshot mountpoint '/Volumes/z_pool/data/.zfs/snapshot/data'
$ ls -latr /Volumes/z_pool/data/.zfs/snapshot/data
total 300412
..
..
-rw-r--r--   1 root  wheel     239735  2 Mär 22:41 .VolumeIcon.icns
drwx------   5 root  wheel          5  2 Mär 22:41 .Spotlight-V100
d-wx-wx-wt   2 root  wheel          2  2 Mär 22:41 .Trashes
drwxr-xr-x@ 21 root  wheel         21  3 Mär 07:05 .
drwx------  41 root  wheel         41 31 Mär 22:32 .fseventsd
dr-xr-xr-x   3 root  wheel          3 31 Mär 22:32 ..


From time to time it is necessary to umount the filesystem to resolve the issue of unable to mount snapshot:
Code: Select all
$ sudo zfs snapshot z_pool/data@data
$ sudo zfs mount z_pool/data@data
cannot mount '/Volumes/z_pool/data/.zfs/snapshot/data': directory is not empty
$ sudo zfs destroy z_pool/data@data
$ sudo zfs snapshot z_pool/data@data
$ sudo zfs mount z_pool/data@data
cannot mount '/Volumes/z_pool/data/.zfs/snapshot/data': directory is not empty
$ sudo zfs destroy z_pool/data@data
$ sudo zfs umount z_pool/data
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/z_pool/data'
Unmount successful for /Volumes/z_pool/data
$ sudo zfs mount z_pool/data
$ sudo zfs snapshot z_pool/data@data
$ sudo zfs mount z_pool/data@data
ZFS: snapshot mountpoint '/Volumes/z_pool/data/.zfs/snapshot/data'
$ ls -latr /Volumes/z_pool/data/.zfs/snapshot/data
total 300412
..
..
-rw-r--r--   1 root  wheel     239735  2 Mär 22:41 .VolumeIcon.icns
drwx------   5 root  wheel          5  2 Mär 22:41 .Spotlight-V100
d-wx-wx-wt   2 root  wheel          2  2 Mär 22:41 .Trashes
drwxr-xr-x@ 21 root  wheel         21  3 Mär 07:05 .
drwx------  41 root  wheel         41 31 Mär 22:32 .fseventsd
dr-xr-xr-x   3 root  wheel          3 31 Mär 22:32 ..


That is strange, and make it impossible to automate my preferred backup method to use snapshots to rsync mounted snapshot data to a different pool.
Because cloning snapshot takes double capacity into the zfs pool and needs much more time while creating.

At the moment I see no other choice a to try to find solution for backup using send/receive functionality.

Nevertheless it would be great to know the root cause of the strange behavior.

Maybe one of the "OpenZFS on OS X" developer have any idea. I would be very grateful.

Thanks and greetings
Sebastian
slu
 
Posts: 4
Joined: Mon Mar 27, 2017 1:57 pm

Re: Unable to mount snapshot

Postby slu » Fri Mar 31, 2017 3:18 pm

Hello Brendon,

using send/receive is the solution .. using rsync needs to mount the snapshot, but using send/receive not.

For my Linux clients I'm using rsync which is working perfect. But with a second ZFS pool for backup it is not necessary to use rsync.

1) Creating a initial replica (receive option -u avoids automatically mounting the filesystem after receiving, option -v generates informations for the stream)
Code: Select all
$ sudo zfs send z_pool/data@data | sudo zfs receive -uv z_backup/data
receiving full stream of z_pool/data@data into z_backup/data@data
received 154MB stream in 1 seconds (154MB/sec)


2) very important is to set the readonly parameter for the backup filesystem, because mounting the filesystem and accessing before setting the readonly parameter is causing a filesystem modification to call for an issue for the next send/receive run to transfer the differences
Code: Select all
$ sudo zfs set readonly=on z_backup/data


3) to transfer the differences after creation a new snapshot
Code: Select all
$ sudo zfs send -i z_pool/data@data z_pool/data@data1 | sudo zfs receive -uv z_backup/data
receiving incremental stream of z_pool/data@data1 into z_backup/data@data1
received 31,4MB stream in 1 seconds (31,4MB/sec)


4) have a look for the existing snapshots and filesystems on source and target pool
Code: Select all
$ sudo zfs list -t all|grep data
z_backup/data                             180M  2,72T   180M  /Volumes/z_backup/data
z_backup/data@data                         148K      -   149M  /Volumes/z_backup/data/.zfs/snapshot/data
z_backup/data@data1                           0      -   180M  /Volumes/z_backup/data/.zfs/snapshot/data1
z_pool/data                               180M  5,91T   180M  /Volumes/z_pool/data
z_pool/data@data                           221K      -   149M  /Volumes/z_pool/data/.zfs/snapshot/data
z_pool/data@data1                          157K      -   180M  /Volumes/z_pool/data/.zfs/snapshot/data1


5) then delete the first snapshot for the filesystem on source and target pool
Code: Select all
$ sudo zfs destroy z_backup/data@data
$ sudo zfs destroy z_pool/data@data
$ sudo zfs list -t all|grep data
z_backup/data                             180M  2,72T   180M  /Volumes/z_backup/data
z_backup/data@data1                           0      -   180M  /Volumes/z_backup/data/.zfs/snapshot/data1
z_pool/data                               180M  5,91T   180M  /Volumes/z_pool/data
z_pool/data@data1                          163K      -   180M  /Volumes/z_pool/data/.zfs/snapshot/data1


6) create a new snapshot for the next backup run
Code: Select all
$ sudo zfs snapshot z_pool/data@data2
$ sudo zfs list -t all|grep data
z_backup/data                             180M  2,72T   180M  /Volumes/z_backup/data
z_backup/data@data1                           0      -   180M  /Volumes/z_backup/data/.zfs/snapshot/data1
z_pool/data                               371M  5,90T   371M  /Volumes/z_pool/data
z_pool/data@data1                          209K      -   180M  /Volumes/z_pool/data/.zfs/snapshot/data1
z_pool/data@data2                             0      -   371M  /Volumes/z_pool/data/.zfs/snapshot/data2


7) and transfer the differences again
Code: Select all
$ sudo zfs send -i z_pool/data@data1 z_pool/data@data2 | sudo zfs receive -uv z_backup/data
receiving incremental stream of z_pool/data@data2 into z_backup/data@data2
received 403MB stream in 2 seconds (202MB/sec)
$ sudo zfs list -t all|grep data
z_backup/data                             366M  2,72T   366M  /Volumes/z_backup/data
z_backup/data@data1                        140K      -   180M  /Volumes/z_backup/data/.zfs/snapshot/data1
z_backup/data@data2                           0      -   366M  /Volumes/z_backup/data/.zfs/snapshot/data2
z_pool/data                               371M  5,90T   371M  /Volumes/z_pool/data
z_pool/data@data1                          209K      -   180M  /Volumes/z_pool/data/.zfs/snapshot/data1
z_pool/data@data2                             0      -   371M  /Volumes/z_pool/data/.zfs/snapshot/data2


8) then delete the oldest snapshot for the filesystem on source and target pool
Code: Select all
$ sudo zfs destroy z_backup/data@data1
$ sudo zfs destroy z_pool/data@data1


.. and so on .. cool stuff ..

Thanks and greetings
Sebastian
slu
 
Posts: 4
Joined: Mon Mar 27, 2017 1:57 pm

Re: Unable to mount snapshot

Postby Brendon » Fri Mar 31, 2017 6:33 pm

Hi,

I wasnt advocating rsync, sorry for the confusion.

I was just saying I access snapshots on the rare occasions that I need to by zfs cloning them, until such time as snapshot automount works.

Cheers
Brendon
Brendon
 
Posts: 285
Joined: Thu Mar 06, 2014 12:51 pm

Re: Unable to mount snapshot

Postby moriz » Tue Apr 04, 2017 8:13 am

Hi,

Regarding the issue, are all the filesystems actually mounted?

I have a similar issue in that,

zpoolA > filesystem1 > filesystem2 > filesystem3

all appear to be there (at first glance), but filesystem2 is actually just a directory, and filesystem2 didn't actually mount,
and nevertheless, filesystem3 did mount but just under a directory called "filesystem2"

On a Mac is it easy to see in the Finder, as it'll have a folder icon instead of a ZFS icon, if you walk down the pool's filesystems.

And that might be why I'm having trouble mounting some snapshots because their parent filesystem isn't actually mounted.

Anyway this is just a quick comment, I haven't tried to look closely at it yet.
I don't know why some of the mount points appear to get left over as directories and then prevent the actual filesystem mounting there, perhaps because they are not "empty".
moriz
 
Posts: 3
Joined: Mon Aug 17, 2015 12:35 am

Re: Unable to mount snapshot

Postby slu » Wed Apr 12, 2017 2:34 am

Hello Moriz,

sorry, for my late answer.

The filesystem structure is flat:

z_pool > filesystem1
z_pool > filesystem2
..
z_pool > filesystem10

And sure, in case of this error message occurs, the directory is not empty.
Code: Select all
$ sudo zfs mount z_pool/data@data
cannot mount '/Volumes/z_pool/data/.zfs/snapshot/data': directory is not empty


Thanks and greetings
Sebastian
slu
 
Posts: 4
Joined: Mon Mar 27, 2017 1:57 pm

Re: Unable to mount snapshot

Postby hamiid » Tue Nov 28, 2017 1:17 pm

I am also experiencing the same problem on macOS Sierra.
Have to do multiple manual unmounts before I can mount the snapshots and browse them from within terminal.
There are also (un)related issues where directory names:

    - report sizes much larger than dataset itself
    - disappear randomly when listing them in terminal right after a successful mount
hamiid
 
Posts: 1
Joined: Tue Nov 28, 2017 1:08 pm


Return to General Help

Who is online

Users browsing this forum: No registered users and 1 guest

cron