It's time to set aside my SharePoint 2013 farm and get into the way-back machine to troubleshoot the Indexer for MOSS SharePoint 2007 Shared Services Provider (SSP)...
(As I mentioned in a previous post on Search Terminology, I try avoiding the term "Indexer" because it has a muddied connotation in SharePoint 2010 & 2013. However, given this relates to SP2007, consider this post one of those exceptions)
Isolate failures that occur during a crawl (e.g. problems with the content being crawled)
First, there are several generic scenarios that vary greatly in potential root causes, but are worth noting some basic strategies:
For the cases when the Indexer is thoroughly misbehaving (e.g. hung crawls, Services Instances not provisioning properly, unable to load the Search Settings and/or Content Sources, etc), the following troubleshooting steps may help...
Check the Windows SharePoint Services Administration service
On the Indexer server, make sure the Windows SharePoint Services Administration service is running with an account that has local Admin permissions (by default, this service runs as the Local System account). That service is what the SharePoint Timer service invokes when provisioning services, sites, etc. Thus, if the Administration service isn’t running, then the search service instance (e.g. "the bits") and/or the search related web services may not get properly provisioned. Once running, manually run any pending administrative tasks using: stsadm -o execadmsvcjobs
Verify connectivity to the Search Admin Web Services from the SSP
Various errors will occur if the SSP cannot access all of the Search Admin web services. You may also see errors with queries if the Web Front Ends cannot access these web services. To determine if there is a problem accessing the Search Admin web service (running on the Indexer server) from the server hosting the SSP Admin site, try the following:
The crawl portion of search is a state machine and components sometimes get out of sync. Because the state of the crawl is stored in the Search databases, the search processes can be recycled. When restarted, these will simply regain state from the DB. Also, as noted above, there are cases where provisioning components may get hung up, so recycling the Timer and Administration service could also help:
From services.msc on the Indexer, then recycle:
Check for Hung Crawls and Propagation Issues
If a crawl appears hung, check the CrawlID, Status, and SubStatus of the crawl using the MSSCrawlHistory table in the Search database using a query such as the following (replace the date/time to specify the time before the problematic crawl started). This query effectively gets all crawls that started after this time that have not completed successfully:
SELECT * FROM MSSCrawlHistory WITH (nolock) WHERE Status <> 11 and StartTime > '2012-08-22 00:00:00'
From this, use the CrawlID to check the item counts with the two queries such as the following (replace the example CrawlID 12345 below). If you run these over several intervals a few minutes apart each run, the crawl is likely hung:
SELECT count(*) FROM MSSCrawlQueue WITH (nolock) WHERE CrawlId = 12345
SELECT count(*) FROM MSSCrawlUrl WITH (nolock) WHERE CrawlId = 12345
If neither count changes over a period of time, verify the problem does not relate to propagation.
Check Other Common Show Stoppers
Determine if the Search service appears correctly provisioned on the Indexer server
If you find any items that seem off, try recycling the services (as above) and then review the Index/registry paths again to verify if they are present.
Invasive Option 1: Build/migrate to a new SSP
At this point, it may make sense to create a new SSP and begin crawling content while continuing to work on the problematic SSP. With this, you could be that much further ahead if the efforts troubleshooting garner no result and you find yourself needing to create a new SSP.
Invasive Option 2: Reset the Index
Invasive Option 3: Re-provision Search and/or migrate the Indexer role to another server
If creating a new SSP is not a good option (e.g. you have too many managed properties, scopes, etc), then you can attempt to re-provision search services using the following:
Check the RegistryBlob for Corruption
If the problem persists or things seem to be incompletely provisioned, then the SSP may be corrupted. In several occasions, I found this caused by the RegistryBlob stored in the SSP database getting corrupted. The RegistryBlob is propagated locally to the Indexer - look for a file name RegistryBlob.reg in the path for the Index (check the SSP configuration settings to confirm this path for the Indexer). If this file doesn't exist or contains corrupted characters, then chances are good that the value in the database is also corrupted. If you find a corrupted RegistryBlob, it's worth opening a support case, but - given the nature of corruption - there is no guarantee that this can be resolved.
If the problem still persists after all of this, unfortunately, creating a new SSP is then going to be the only real option…