Export *always* hangs, only reboot helps

[rattlehead]

Hello developers,
I have a problem with the two or three latest versions for quite a while now (on a 2012 Macbook Retina with regularly updated OS X, currently 10.11.5):

Code: Select all

sudo zpool export <poolname>

hangs after successfully unmounting all filesystems -- every time. I cannot even kill the process, just close the terminal window (with a warning, because of a running process), but the process itself then is still hanging in the background: When I try to e.g. mount the filesystems again, this attempt fails too. The only thing that helps in my experience is rebooting the Macbook, which is quite annoying, because I reboot very rarely and my mindset is like "if I connect this external drive now, I cannot detach it until the next reboot! Hm, I better not this time." which kinda thwarts the whole idea of an external drive...
I had hoped for a fix with new versions, but it does not seem to be a common problem.

I'd love to give you more technical details, like reports/logs, but I don't remember how to enable verbose logging, where to find the logs etc.

Help would be highly appreciated,
rattlehead

[Brendon]

Hi,

Sounds like something rather unique is happening to you. https://openzfsonosx.org/wiki/Getting_involved contains some instructions on how to gather a spindump, which you should do when you have a stuck export. Please join the IRC channel and bring your spindump for further support.

- Brendon

[rattlehead]

Please join the IRC channel and bring your spindump for further support.

Will do, thanx for the quick reply.

[Brendon]

@rattlehead

We have looked at the dump you provided, and we can see where it is stopped. It is not clear _why_ its stopped at all.

Can you please describe your setup a bit (hardware, dataset layout etc etc) and also what is the system doing when you export?

Cheers
Brendon

[rattlehead]

Ok, here we go:
I) System setup
- Macbook Pro retina 2012 16GB RAM
- Usually connected is a Pegasus R8 (via Thunderbolt) with native RAID6 and HFS+, usually having constant but low activity
- Then from time to time I connect (via Thunderbolt) a Pegasus R6 with 6 x 8TB and a ZPOOL configuration like this:

Code: Select all

Fri May 22 11:40:28 CEST 2015
  pool: zfsraid
 state: ONLINE
  scan: resilvered 116K in 0h0m with 0 errors on Fri May 22 11:40:20 2015
config:

    NAME        STATE     READ WRITE CKSUM
    zfsraid     ONLINE       0     0     0   
      raidz2-0  ONLINE       0     0     0   
        disk9   ONLINE       0     0     0   
        disk2   ONLINE       0     0     0   
        disk7   ONLINE       0     0     0   
        disk6   ONLINE       0     0     0   
        disk10  ONLINE       0     0     0   
        disk8   ONLINE       0     0     0   

errors: No known data errors

- no duplication, atime switched off, 4k blocksize set, no compression, UTF-8, formD as character normalization
- I have two filesystems
1. zfsraid configured as specified above
2. zfsraid/logs with pretty much the same configuration, only difference is compression being switched on
- Usually, there is one other user logged in to the same machine locally (with GUI session) but if I remember correctly, the problem persisted even before I used to have that.

II) When I try to export the pool after having used the volume for a bit (played videos from there, moved files/folders around, etc.), then it could so far always unmount /Volumes/zfsraid/logs, but sometimes fails to unmount /Volumes/zfsraid, even if lsof does not show any processes accessing the volume (but I experienced that with other non-ZFS volumes, too. So I don't think this is an issue that you can fix).
Most of the times, the unmounting of /Volumes/zfsraid also succeeds, but the process does not return to the console, and I can wait for the "export successful" message literally forever (I have waited a couple of days one time, until I disconnected and turned the case off the hard way). In those cases, I usually close all my programs and processes and try a clean shutdown/reboot, which most often succeeds, sometimes in the past not, in which case I turned off the computer via the Power button. After the reboot (and given a few moments), the volumes are mounted without importing them (so they still are imported from the last session(?)). Often with much activity on the drives of the zpool. I usually do the export, after the drive activity comes to a hold (not having created any activity on the drives myself since the reboot). These times, the exports are usually successful.

Cheers
rattlehead

[Brendon]

Are you able to compile our code?

Cheers
Brendon

[rattlehead]

I don't know how to do that.

[rattlehead]

What should I do now?

[Jimserac]

Same problem here. What worked for me was to permanently shut off
Spotlight, from the command line, this way: (El Capitan)

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

After doing that, sudo zpool export works every time. Slight delay, maybe 5 seconds, and then its done.

Note that just dragging the icon into System Preferences, Spotlight, Privacy, did NOT work.

Only shutting off Spotlight worked for me.

[Brendon]

@jimserac

Thanks for the suggestion. However I would just like to note that this should not be required, and in fact I don't do it (I have no trouble with export).

We have some supporting log data from this user and we know where it hangs, just dont quite understand why. There is a good chance that its not a software issue, or if it is then a fairly unique one. Lundman is preparing and instrumented build to help this guy diagnose the problem as far as I know.

- Brendon