Spotlight Issues on macOS Sonoma

[Haravikk]

Is anybody else experiencing problems with Spotlight for ZFS datasets on macOS Sonoma or Sequoia?

I've had to disable Spotlight on some datasets to prevent Spotlight from apparently failing during indexing, causing it to stop returning all file related results (only extra search results from the web, dictionary etc. are shown). But this isn't a good long term option as some of my datasets really need to be searchable.

What seems to be happening is that Spotlight is indexing the dataset(s), but right at the end it discards the dataset's .Spotlight-V100 database and starts again, this can be seen by the presence of a bunch of files being moved to .Spotlight-V100/Store-V2/DeadFiles which seems to contain the old database as it's being deleted and a new one created.

I've already tried the usual Spotlight steps of running:

Code: Select all

mdutil -i off /path/to/dataset
mdutil -X /path/to/dataset
mdutil -i on /path/to/dataset

After the second command I tend to verify that the .Spotlight-V100 folder has been fully deleted using a suitable rm command (as always with rm, take care to ensure it's only deleting what it needs to, I like to use ls first, then change to rm once I'm happy it's correct). I do this because mdutil -X doesn't seem to delete the entire folder structure, but I usually prefer to force Spotlight to recreate the whole thing fresh to eliminate any possible permissions issues or whatever.

However this doesn't fix the problem, Spotlight just indexes to what appears to be completion and then decides to throw away all of its work and start again, after several rounds of this the Spotlight server will go "offline" and Spotlight will stop returning any results at all (mdutil -as will show all volumes as "no index" or "spotlight server disabled") until it eventually restarts and the problems starts all over.

I never had any problems like this under Catalina, so it seems that Apple has somehow made Spotlight a lot less resilient in one of the newer macOS versions. Has anyone else experienced anything like this, and have you discovered any ways to fix it? Spotlight is an area of macOS that I have no experience debugging – I haven't been able to find any logs to point to any kind of obvious cause, or indicate an actual crash that I could look into.

[jawbroken]

Do you have any properties set that might be confusing it? Like mimicking HFS or APFS or something to do with extended attributes? I'm afraid I don't have any direct experience here (I've always deliberately disabled Spotlight on my fileserver), so I'm just guessing.

[Haravikk]

Hmm, it currently has com.apple.mimic=hfs so I'll give it a try without when I get a chance, but I have other datasets with this set (it's set at pool root) without issue which is why it's such a strange problem to try and track down. I have xattr=sa for extended attribute support, but again that's at root so it's applied to other indexed datasets.

You say you have Spotlight disabled for your file server, are you using something else instead for searching or do you not need to search it?

[jawbroken]

It's a fileserver that hosts a few other services so I don't really need to search it often, and I can do that from the command line (e.g. with find). I just have two very large pools, so I decided not to let Spotlight churn away on them forever. I think Spotlight was also triggering the crash on the early Apple Silicon builds of OpenZFS (by increasing read/write traffic) that has since been fixed.

edit: The reason my first guess is mimicking HFS/APFS or extended attributes is that it seems plausible that it might think the index is broken and re-index if it's supposedly a normal HFS drive but it doesn't support an operation it is supposed to, or returns slightly weird data.

[Haravikk]

So I did some experimenting; it does seem that com.apple.mimic=hfs was in fact the problem in this case for some reason, though I'm no closer to figuring out why (as other datasets with =hfs are working just fine). So the following resolved the issue in this case:

Code: Select all

mdutil -i off /path/to/volume
mdutil -X /path/to/volume
zfs set com.apple.mimic=off pool/dataset
mdutil -i on /path/to/volume

Also, while Spotlight was disabled I discovered that it's still possible to create smart folders, and these will use a non-Spotlight search, which persists even once Spotlight is re-enabled. I had no idea that this was possible.

Of course if Spotlight is enabled then any newly created smart folders will use it, and they'll be useless if you are having any Spotlight related problems, and they also stop working if Spotlight is disabled, so this non-Spotlight smart folder behaviour seems to be tied to the folder at time of creation. The opposite behaviour is also true – a non-Spotlight smart folder seems to use a non-Spotlight search even once Spotlight has been re-enabled, so it will be slower but may give results that Spotlight can't.

This is also true of general searching of a volume (or sub-directory) with Spotlight disabled, so that at least means that searching a Spotlight disabled ZFS volume is still possible in parts of the macOS UI. But you won't be able to search via Spotlight general search, only by Finder search of the Spotlight disabled dataset, or a smart folder created from such a search.

I guess I'll probably try to move towards disabling com.apple.mimic on most of my datasets, see which ones I actually still need it for, it's just weird that this is even a problem, and only for certain datasets (with no clear reason why), as previously setting com.apple.mimic=hfs was safe to do and generally meant that things worked better overall.

[jawbroken]

i wouldn't say that pretending to be another filesystem was ever actually safe to do, because it always risks behavioural differences causing serious issues. it can be necessary for compatibility, though

[doctordeej]

@Haravikk @jawbroken thank you both for posting about this.

I have today registered on OpenZFSonOSX because this is the only place I can find anyone who is complaining about the problem I am experiencing..

However, my problem is on a vanilla (ie non ZFS enhanced) Sonoma MacBook Pro M1 (2021)

When I restart or, as I have done today, update to the latest Sonomo version, I can search for "Ter" and get "Terminal" as a suggested search result.
Spotlight then reindexes and this - along with any file searches - just disappears. I"m left with Mail and Safari recommendations (oh, and Tips - very useful).

I have tried all the recommended "SUDO mdutil" hacks but to no avail.
It just tells me that "Spotlight Server is Disabled"

What I haven't done is try to look inside the .Spotlight-V100 folder, since doing so appears tricky on the startup disk (Since Catalina there are separate System and Data volumes, which is confusing me ...)

Going a bit mad here ...

Do you have any thoughts on whether this might be a deeper problem with Spotlight on Sonoma?


PS - I'm a newbie here, but have been using Macs for ever (My first an SE! and I am sufficiently long-in-the-tooth to have owned an Apple II before that).

Happy to try any diagnostics you can recommend.

Thanks in advance.

DJ

[doctordeej]

OK. This somewhat dramatic series of interventions managed to solve my problem.

The issue appears to be a corruption of the .Spotlight-V100 folder - I have no idea if this is relevant to any other readers here, but I hope it's useful

[list=]Disable csutile
enable root user
bootout MDS daemon
login a root
delete .Spotlight-V100 folder
logout of root
restart
re-enable csutils
[/list]

My gratitude to the authors here: https://discussions.apple.com/thread/255188805

DJ

[Haravikk]

So after struggling with this for a while I'm yet to find a persistent fix of any kind.

Whatever is wrong with Spotlight, it seems to keep corrupting its database – sometimes it will build it fine and work for a while, but the issue eventually resurfaces and I have to disable Spotlight on the dataset and discard the Spotlight folder (mdutil -i off, mdutil -X).

Fortunately with Spotlight disabled it's still possible to search using the Finder etc., it just uses the older, slower method.

However, I'm struggling to find a way to force macOS to permanently use that search method. Disabling Spotlight with mdutil -i off is only temporary for ZFS datasets, the next time one is mounted it will be re-enabled. However, adding the .metadata_never_index file to the dataset root disables all searching, including non-Spotlight searching, which is bizarre (since that one doesn't require metadata indexing).

All I can conclude is that some update to Spotlight since Catalina has caused it to corrupt itself on ZFS datasets, and I don't see a way to fix it. Changing the com.apple.mimic property has no effect, all options exhibit the same problem.

The only temporary solution I've found so far is to run a script that triggers mdutil -i off <mount_point> after the dataset is mounted, since this disables Spotlight while leaving searching enabled, but it's not an ideal solution. I guess I could try adding support for a property into my dataset mounting scripts that does it sooner?

[Sharko]

I, too, have had many problems with Spotlight on my ZFS datasets over the years; when I was running my user home folder on a ZFS dataset it was especially annoying. It wasn't just that it wouldn't find files - Spotlight also lost the ability to search for and launch apps from the keyboard. I eventually gave up on expecting it to work, and I developed a two-pronged solution: for the purpose of finding files I switched to Thomas Tempelmann's Find Any File utility (found at https://apps.tempel.org/FindAnyFile/). FAF works well for my purposes, though it is not as fast as Spotlight.

My other primary use for Spotlight was launching apps, which I used to initiate with CMD-spacebar; I decided to assign a keyboard shortcut of Option-spacebar to bring up LaunchPad, which is a feature of OS X that hardly anybody ever uses. When you invoke LaunchPad you are immediately presented with a full-screen display of all your apps, and you can start typing the first letters of the app you are interested in. The display filters down as you type, and when you have typed enough to make the selection unambiguous you can just hit Enter to launch the app.