Big problem with raidz mount point after 10.11 upgrade

All your general support questions for OpenZFS on OS X.

Big problem with raidz mount point after 10.11 upgrade

Postby nekton34 » Fri Nov 20, 2015 5:13 am

Guys I could really do with some help/advice here or I am losing my job.
Two raidz pools each of 4TB x 4 called master and backup have been attached to a macmini running 10.10 from two USB raid cases and working fine until yesterday.
Master was backed-up to backup each night using Chronosync. Then somebody in the office mistakenly filled backup to the brink with a set of HD videos and it was so full that Chronsync could not backup to it and we could not delete any files. The best thing seemed to be to recreate the raidz pool on it and let tonight's Chronosyc run. Then someone (me—bloody fool) mentioned that maybe we should update it to 10.11 and I went ahead and exported master, switched it off and ran the 10.11 upgrade. It went normally. At the final powering-up, of course master did not mount because of the new SIP in 10.11 so I switched SIP off (Recovery Mod—csrutil disable), ran the latest 1.4.5 zfs installer and rebooted. Then TSHTF—no master mounting. I tried import master and get the message that the disk is busy (the LEDs on the front of the disks are flashing like crazy for sure as if some processing is running); zpool status shows that all disks 1 through 4 are ONLINE and there are no data errors, and zpool list shows master as having a size of 14.5 TB with 5.85 ALLOC and 6.65TB FREE at 40% capacity, so the raidz disk seems fine and the amount of data is about right (see enclosed screenshot). It seems like the data is still on the raidz and healthy BUT try as I might I cannot get master to mount on the macmini desktop. I have to tread lightly now and not make a stupid misyake because this is now the only disk with all the company data as the backup disk still does not have a copy.
I have read all the Oracle documents on mountpoints and cannot see my way out for the trees.
First, what command can I use to check the existing mountpoint of the now unmounted raid, and second, how can I set a new mountpoint to get master mounted on the desktop again?
Be gentle and clear like talking to a baby, it's been a REALLY long day.
The office it now closed but I do have remote access into screensharing and can work in terminal on the disk or collect other screenshots to post as requested.
I'll buy anyone in Tokyo a few pints for some help here. Lundman do you drink?
Robert
Screen Shot 2015-11-20 at 9.56.38 PM.png
Screen Shot 2015-11-20 at 9.56.38 PM.png (72.01 KiB) Viewed 9484 times
nekton34
 
Posts: 34
Joined: Tue Feb 10, 2015 2:05 am
Location: Tokyo, Japan

Re: Big problem with raidz mount point after 10.11 upgrade

Postby ilovezfs » Fri Nov 20, 2015 5:52 am

ilovezfs
 
Posts: 232
Joined: Thu Mar 06, 2014 7:58 am

Re: Big problem with raidz mount point after 10.11 upgrade

Postby ilovezfs » Sat Nov 21, 2015 1:16 am

I haven't seen you in IRC yet. Is the situation resolved?

My guess is you have userland/kernel-extension disagreement right now.

The problem is unlikely related to mountpoints specifically, but to answer your question, to see mountpoints the commands are

Code: Select all
$ zfs list


Code: Select all
$ zfs get mountpoint


Code: Select all
$ zfs mount


Code: Select all
$ mount

In any case, can we see the output of this script? I'm looking for multiple installations.

Code: Select all
#!/bin/bash

pkgutil --pkgs | grep openzfs

ls -ld /Library/OpenZFSonOSX
ls -ld /Library/Extensions/spl.kext
ls -ld /System/Library/Extensions/spl.kext
ls -ld /usr/src/spl-*
ls -ld /Library/Extensions/zfs.kext
ls -ld /System/Library/Extensions/zfs.kext
ls -ld /usr/src/zfs-*
ls -ld /usr/lib/modules-load.d
ls -ld /usr/lib/systemd
ls -ld /usr/local/etc/init.d
ls -ld /etc/init.d
ls -ld /usr/local/etc/zfs
ls -ld /usr/local/include/libspl
ls -ld /usr/local/include/libzfs
ls -ld /usr/include/libspl
ls -ld /usr/include/libzfs
ls -ld /usr/local/lib/dracut
ls -ld /usr/lib/dracut
ls -ld /usr/local/lib/udev
ls -ld /usr/lib/udev
ls -ld /usr/local/share/zfs
ls -ld /usr/share/zfs
ls -ld /usr/local/libexec/zfs
ls -ld /usr/libexec/zfs

ls -l /usr/local/share/man/man1/zhack.1
ls -l /usr/local/share/man/man1/zpios.1
ls -l /usr/local/share/man/man1/ztest.1
ls -l /usr/local/share/man/man5/vdev_id.conf.5
ls -l /usr/local/share/man/man5/zfs-module-parameters.5
ls -l /usr/local/share/man/man5/zpool-features.5
ls -l /usr/local/share/man/man8/fsck.zfs.8
ls -l /usr/local/share/man/man8/mount.zfs.8
ls -l /usr/local/share/man/man8/vdev_id.8
ls -l /usr/local/share/man/man8/zdb.8
ls -l /usr/local/share/man/man8/zfs.8
ls -l /usr/local/share/man/man8/zinject.8
ls -l /usr/local/share/man/man8/zpool.8
ls -l /usr/local/share/man/man8/zstreamdump.8
ls -l /usr/share/man/man1/zhack.1
ls -l /usr/share/man/man1/zpios.1
ls -l /usr/share/man/man1/ztest.1
ls -l /usr/share/man/man5/vdev_id.conf.5
ls -l /usr/share/man/man5/zfs-module-parameters.5
ls -l /usr/share/man/man5/zpool-features.5
ls -l /usr/share/man/man8/fsck.zfs.8
ls -l /usr/share/man/man8/mount.zfs.8
ls -l /usr/share/man/man8/vdev_id.8
ls -l /usr/share/man/man8/zdb.8
ls -l /usr/share/man/man8/zed.8
ls -l /usr/share/man/man8/zfs.8
ls -l /usr/share/man/man8/zinject.8
ls -l /usr/share/man/man8/zpool.8
ls -l /usr/share/man/man8/zstreamdump.8

ls -l /usr/local/lib/libnvpair.1.dylib
ls -l /usr/local/lib/libnvpair.a
ls -l /usr/local/lib/libnvpair.dylib
ls -l /usr/local/lib/libnvpair.la
ls -l /usr/local/lib/libuutil.1.dylib
ls -l /usr/local/lib/libuutil.a
ls -l /usr/local/lib/libuutil.dylib
ls -l /usr/local/lib/libuutil.la
ls -l /usr/local/lib/libzfs.2.dylib
ls -l /usr/local/lib/libzfs.a
ls -l /usr/local/lib/libzfs.dylib
ls -l /usr/local/lib/libzfs.la
ls -l /usr/local/lib/libzfs_core.1.dylib
ls -l /usr/local/lib/libzfs_core.a
ls -l /usr/local/lib/libzfs_core.dylib
ls -l /usr/local/lib/libzfs_core.la
ls -l /usr/local/lib/libzpool.1.dylib
ls -l /usr/local/lib/libzpool.a
ls -l /usr/local/lib/libzpool.dylib
ls -l /usr/local/lib/libzpool.la
ls -l /usr/lib/libnvpair.1.dylib
ls -l /usr/lib/libnvpair.a
ls -l /usr/lib/libnvpair.dylib
ls -l /usr/lib/libnvpair.la
ls -l /usr/lib/libuutil.1.dylib
ls -l /usr/lib/libuutil.a
ls -l /usr/lib/libuutil.dylib
ls -l /usr/lib/libuutil.la
ls -l /usr/lib/libzfs.2.dylib
ls -l /usr/lib/libzfs.a
ls -l /usr/lib/libzfs.dylib
ls -l /usr/lib/libzfs.la
ls -l /usr/lib/libzfs_core.1.dylib
ls -l /usr/lib/libzfs_core.a
ls -l /usr/lib/libzfs_core.dylib
ls -l /usr/lib/libzfs_core.la
ls -l /usr/lib/libzpool.1.dylib
ls -l /usr/lib/libzpool.a
ls -l /usr/lib/libzpool.dylib
ls -l /usr/lib/libzpool.la

ls -l /usr/local/bin/arcstat.pl
ls -l /usr/local/bin/InvariantDisks
ls -l /usr/local/bin/zdb
ls -l /usr/local/bin/zdb_static
ls -l /usr/local/bin/zed
ls -l /usr/local/bin/zfs
ls -l /usr/local/bin/zhack
ls -l /usr/local/bin/zinject
ls -l /usr/local/bin/zpios
ls -l /usr/local/bin/zpool
ls -l /usr/local/bin/zstreamdump
ls -l /usr/local/bin/ztest
ls -l /usr/local/bin/ztest_static
ls -l /usr/local/sbin/InvariantDisks
ls -l /usr/local/sbin/zdb
ls -l /usr/local/sbin/zdb_static
ls -l /usr/local/sbin/zed
ls -l /usr/local/sbin/zfs
ls -l /usr/local/sbin/zhack
ls -l /usr/local/sbin/zinject
ls -l /usr/local/sbin/zpios
ls -l /usr/local/sbin/zpool
ls -l /usr/local/sbin/zstreamdump
ls -l /usr/local/sbin/ztest
ls -l /usr/local/sbin/ztest_static
ls -l /usr/bin/arcstat.pl
ls -l /usr/sbin/InvariantDisks
ls -l /usr/sbin/zdb
ls -l /usr/sbin/zdb_static
ls -l /usr/sbin/zed
ls -l /usr/sbin/zfs
ls -l /usr/sbin/zhack
ls -l /usr/sbin/zinject
ls -l /usr/sbin/zpios
ls -l /usr/sbin/zpool
ls -l /usr/sbin/zstreamdump
ls -l /usr/sbin/ztest
ls -l /usr/sbin/ztest_static

ls -l /sbin/mount.zfs
ls -l /sbin/mount_zfs
ls -l /sbin/umount_zfs
ls -l /usr/local/bin/mount.zfs
ls -l /usr/local/bin/mount_zfs
ls -l /usr/local/bin/umount_zfs
ls -l /usr/local/sbin/mount.zfs
ls -l /usr/local/sbin/mount_zfs
ls -l /usr/local/sbin/umount_zfs

ls -l /etc/zfs/zpool.cache
ls -l /etc/zfs/zpool.cache.tmp

ls -l /etc/zfs/vdev_id.conf.alias.example
ls -l /etc/zfs/vdev_id.conf.multipath.example
ls -l /etc/zfs/vdev_id.conf.sas_direct.example
ls -l /etc/zfs/vdev_id.conf.sas_switch.example
ls -l /etc/zfs/zsysctl.conf.example

ls -l /etc/zfs/zed.d/checksum-email.sh
ls -l /etc/zfs/zed.d/data-email.sh
ls -l /etc/zfs/zed.d/generic-email.sh
ls -l /etc/zfs/zed.d/io-email.sh
ls -l /etc/zfs/zed.d/resilver.finish-email.sh
ls -l /etc/zfs/zed.d/scrub.finish-email.sh

ls -l /etc/zfs/zed.d/all-debug.sh
ls -l /etc/zfs/zed.d/all-syslog.sh
ls -l /etc/zfs/zed.d/checksum-notify.sh
ls -l /etc/zfs/zed.d/checksum-spare.sh
ls -l /etc/zfs/zed.d/config.remove.sh
ls -l /etc/zfs/zed.d/config.sync.sh
ls -l /etc/zfs/zed.d/data-notify.sh
ls -l /etc/zfs/zed.d/generic-notify.sh
ls -l /etc/zfs/zed.d/io-notify.sh
ls -l /etc/zfs/zed.d/io-spare.sh
ls -l /etc/zfs/zed.d/resilver.finish-notify.sh
ls -l /etc/zfs/zed.d/scrub.finish-notify.sh
ls -l /etc/zfs/zed.d/snapshot.mount.sh
ls -l /etc/zfs/zed.d/zed.rc
ls -l /etc/zfs/zed.d/zpool.destroy.sh
ls -l /etc/zfs/zed.d/zpool.import.sh
ls -l /etc/zfs/zed.d/zvol.create.sh
ls -l /etc/zfs/zed.d/zvol.remove.sh

ls -l /Library/LaunchDaemons/org.openzfsonosx.InvariantDisks.plist
ls -l /Library/LaunchDaemons/org.openzfsonosx.zconfigd.plist
ls -l /Library/LaunchDaemons/org.openzfsonosx.zed.plist
ls -l /Library/LaunchDaemons/org.openzfsonosx.zed.service.plist
ls -l /Library/LaunchDaemons/org.openzfsonosx.zpool-autoimport.plist
ls -l /Library/LaunchDaemons/org.openzfsonosx.zpool-import-all.plist

ls -ld /Library/Filesystems/zfs.fs
ls -ld /System/Library/Filesystems/zfs.fs
ilovezfs
 
Posts: 232
Joined: Thu Mar 06, 2014 7:58 am

Re: Big problem with raidz mount point after 10.11 upgrade

Postby JasonBelec » Sat Nov 21, 2015 12:30 pm

your already getting some of the best help available, but I can offer some aid once a response to the above is posted. Currently I have run every version of OS X including current betas with all OpenZFS OS X updates and so far no issues at all. If your pool came on line, sounds like the ZFS files are just going through a check. However for more info would help offer advice.
JasonBelec
 
Posts: 32
Joined: Mon Oct 26, 2015 1:07 pm

Re: Big problem with raidz mount point after 10.11 upgrade

Postby nekton34 » Sat Nov 21, 2015 4:40 pm

Thanks for your advice and help Guys.
Sorry for the blank day—the problem is that it is a 3-day weekend here in Tokyo and I have no access to the file server until Tuesday and nothing to talk about until then. I thought I'd be able to ssh or remote-access in but I am getting refused at the present! Some large refrigeration room maintenance work was ongoing when I left late on Friday night and I wonder if they tripped the power and nobody is there to restart, etc.
I tried the suggested mount commands while there on Friday and all I could get was the "disk is busy" message.
I'll be back on here at about 10 am Japan Standard Time Tuesday—I am praying that the massive amount of disk access was a sign that the raidz was doing some scrubbing type process when I left.
nekton34
 
Posts: 34
Joined: Tue Feb 10, 2015 2:05 am
Location: Tokyo, Japan

Re: Big problem with raidz mount point after 10.11 upgrade

Postby nekton34 » Sun Nov 22, 2015 3:51 pm

Good news.
I managed to get remote access this morning and it seems the master raidz is ONLINE and mounted on the desktop and the files are all there.
Huge sigh of relief—I guess the raidz was going thru some form of check before it could mount up.
Is there a terminal command that can show the running status of a check? Everything I tried on Friday just returned "disk is busy message'.
Back in tomorrow to put the backup raidz online and get a good backup of master before doing anything.
Thanks for the suggestions when I was in panic mode.
nekton34
 
Posts: 34
Joined: Tue Feb 10, 2015 2:05 am
Location: Tokyo, Japan

Re: Big problem with raidz mount point after 10.11 upgrade

Postby Brendon » Sun Nov 22, 2015 5:41 pm

It was probably doing an unlinked drain, which must complete before the filesystems can be mounted. This can take a long time, having said that I thought the need for it was much reduced with recent builds.

Anyway, glad it's all up and running.

Cheers
Brendon
Brendon
 
Posts: 286
Joined: Thu Mar 06, 2014 12:51 pm

Re: Big problem with raidz mount point after 10.11 upgrade

Postby lundman » Mon Nov 23, 2015 4:39 pm

Sorry, long weekend, and lots of drinking :)

The pool is full and can't delete. This is expected of a COW filesystem, and if you have snapshots, deleting a file does not free space! As it is still in the snapshot. In this situation, the right thing is to delete one of the snapshots, that will give you some breathing room.

As for the pool healthy, but didn't mount. I would guess that it was possibly processing the "unlinked drain list". Which can be large if a lot of things have been deleted recently. If you look in /var/log/system.log you can potentially see any ZFS messages, including the progress reporting of unlinked drain.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Big problem with raidz mount point after 10.11 upgrade

Postby JasonBelec » Mon Nov 23, 2015 6:08 pm

yeah, what they said.... You should also set pool quota/limit to prevent running out of total space and being unable to nuke stuff. Just a suggestion.
JasonBelec
 
Posts: 32
Joined: Mon Oct 26, 2015 1:07 pm

Re: Big problem with raidz mount point after 10.11 upgrade

Postby nekton34 » Wed Nov 25, 2015 10:15 pm

Thanks for the advice about setting a disk quota—did that to leave 500 GB free and prevent the disk full issues again.
nekton34
 
Posts: 34
Joined: Tue Feb 10, 2015 2:05 am
Location: Tokyo, Japan

Next

Return to General Help

Who is online

Users browsing this forum: No registered users and 25 guests