SSP App Pool failure in IIS 7
Recently ran into a peculiar issue with MOSS installed on Server 2008/IIS 7 wherein the SSP app pool would stop after the app pool was recycled and you tried browsing to the SSP site. The resulting error code was 503: Service Unavailable. Checking the app logs on the server yielded very little of use: event ID 5139, citing listener channel failures for http. Five failures, and the Windows Process Activation Service (WAS) shut down the app pool.
The build configuration:
• MOSS 2007 64-bit slipstream installed on fresh 64-bit Server 2008 machines
• Project Server 2007 also installed (64-bit)
• App server #1 (of 2) running Shared Services, Search, Indexing
• App Server #2 running Central Admin and the Project Application Service
• Three WFEs (in the DMZ)
The Shared Service site would run fine at first – for some time – after the install. Then, after as little as 8 hours and as long as 3 days, WAS shut down the app pool again. Restarting the app pool was successful, but the site would not display in the browser, and attempts to browse to it shut down the app pool again. We reset IIS, rebooted, you name it - no change in the behavior.
This was seen initially with Central Admin running on the same server as the SSP. When that happened, Central Admin also showed the same behavior. The two app pools run under different identities, but both showed the failure. After CA was moved to the second app server, it ran just fine, while app server 1 with Shared Services continued to fail. Checked all manner of blogs and TechNet, but couldn't find the root cause. I suspected something in IIS7, but never did find anything there, despite some vagaries in the installed configuration of that server compared to the rest of the farm.
The culprit turned out to be SharePoint Search. The error stemmed from IPv6 entries being added into the server’s Hosts file by Sharepoint Search – one for the server, and one for each web app (note: the actual addresses below are obfuscated ):
fe##::####:####:####:#### prod-server-app1
# Added by Office SharePoint Server Search (6/24/2008 6:02 PM).
127.0.0.1 localhost
fe##::####:####:####:#### home.customer.com
# Added by Office SharePoint Server Search (6/24/2008 6:02 PM).
fe##::####:####:####:#### my.customer.com
# Added by Office SharePoint Server Search (6/24/2008 6:02 PM).
If we recycled the app pool, these got added. Once the app pool was recycled, if we browsed to the SSP site, the app pool stopped. If we recycled the app pool, then removed or commented these added lines before browsing to the SSP site, everything worked as it should.
This actually resulted from our configuration which specified a dedicated crawl server. Once we split the crawl across all servers, the lines were no longer added to the Hosts file.
Still looking for a way to stop SharePoint Search from adding these to the Hosts file on app pool recycle so that we can use a dedicated crawl server. Simply disabling IPv6 on the server (it's not being used at this customer) was ineffective.
Side note/tip: the TechNet documentation on disabling IPv6 calls for a DWORD entry in the registry, with a value of 0xFF. We were unable to actually type "0xFF" in the field (wouldn't accept the "x"), but typing just "FF" gave the desired value. Something I didn't know.
