Ron Grzywacz's Blog

Over the last couple of months I’ve been helping my customer work through MOSS Crawling Performance issues. The sad part of this task is, there isn’t an easy way to do it, because there are so many moving parts and variables. A couple of months ago, some good blog posts were put up on the Enterprise Search Blog, under the tag “Perf and Scale”. These gave some good insight as to what specifically needs to be monitored, but still required a lot of good old fashioned knowledge with tools like Windows Performance Monitor AKA Perfmon. A fellow PFE, Clint Huffman, has developed a tool that helps to automate the analysis of Perfmon log files which generates a nice "results" report. That tool, Performance Analysis of Logs (PAL), can be found on CodePlex as a free download. Realizing the gap for analyzing crawl performance data, and knowing the capabilities PAL provides, I decided to put both together.

The result is a PAL threshold file and a Perfmon Template file that lets you capture and analysis MOSS crawling performance. The next release of PAL should include these two files, but for those of you wanting to get a head start, I’ve attached them here. The threshold file (xml) needs to go in the PAL directory, and the perfmon template (htm) can go into the templates folder inside PAL directory. Once you have the proper files in place, you’ll need to fire up perfmon and start capturing data on your index server. Once that’s done, you’ll need to fire up PAL. On the counter log tab, point to the perfmon data file to analyze. Go to the Threshold file tab, and select the “Microsoft Office SharePoint Server – Search” option. Then answer each of the questions. This is important, because each answer factors into the analysis calculations. Once your done with the questions, proceed to the next tab. There you will have an option for the analysis interval. Choosing the right analysis interval will help you to better understand the data and what opportunities for improvement you have. Let me explain.

In an ideal scenario, you’ll have the crawler running constantly. You’d do this to keep your search index as fresh as possible. This is the typical scenario for customers with large amounts of data. With large amounts of data, there are usually multiple content sources. Each of these content sources plays into the crawl schedule, with usually  limited number of content sources being crawled at once. Now, how does this play into our analysis interval? Ideally we’ll have a sampling of perfmon data that covers our whole crawl schedule, where each content source is crawled at least once. We’ll need knowledge of the crawl schedule as well. For this example, we’ll say that we have 20 content sources, and every hour a new content source crawl kicks off. We’d then want to set our analysis interval to 1 hour. This will cause our performance intervals to mesh with our crawl schedules. We can correlate suggestions in the PAL report with the good and bad time periods of the crawl schedule. We can then perform suggested actions and begin the analysis task again.


Demo Screenshots. Please note, these may vary slightly from release names/titles, as they were taken during development.

This post is about a case I worked on back in April, which was a real doozy, to say the least. My customer was trying to set up SSO for Excel Services. In short, they wanted to create a couple of shared data connections, that used one credential to access a data source. Different domain groups and users would then have access to the data connections to utilize in Excel. Seems straight forward enough, right? ;) 

The picture got a little trickier, because the customer has two domains, a user domain and a resource domain, with a one way trust in place. To simplify this piece, we set up a test lab that had a two way trust between the domains. For some reason, we still were getting what appeared to be access denied errors, when users would try to pull data from the shared data source via Excel. Knowing that we granted the user permissions to the data connection, and the service account trying to make a connection to the actual data had permission, we had to assume authentication was failing somewhere in the chain. To figure this out, we turned to the ULS logs. We set SSO and all the Excel components to verbose and started digging. We found the following critical error in the ULS logs.

“User DOMAIN\ServiceAccount-UserName failed to retrieve credentials using an access token for user DOMAIN\USERNAME, for enterprise application definition AdventureWorks.  The error returned was 0x80070005. For more information, see the Microsoft SharePoint Products and Technologies Software Development Kit (SDK).”

Doing some further investigation, I spotted the following message.

“AuthzInitializeContextFromSid() failed for S-1-5-21-913004495-3688694002-1748028306-21984”

I then hit the web to translate this message into English. :) I found the following hit.
http://support.microsoft.com/kb/842423

Although this issue didn’t relate to MOSS, it did relate to SSRS, which does similar activities to what some MOSS components do. Reading through the KB, things started to come together. In order to look up user domain group memberships, users, in this instance, the service account, needs rights to the TGGAU attribute of a user account. Whether or not this functionality is enabled depends on how your AD infrastructure was installed and configured.

In order to have SSO and Excel function correctly together, the service account needs to be able to look up what groups a user is a member of, to see if they’re in the group that has access to a shared data connection. If you don’t have the permissions set up the way they need to be, follow the directions in the KB article linked above. In short, assuming a 2003 domain functional level, there’s a domain group, Windows Authorization Access Group, that has the permissions necessary to do user domain group membership lookups. You will need to add your service account to this group. After that, you should be able to fully leverage this powerful and complex technology. I hope this saves someone out there some time, because it took some time to figure this one out.

Below I’ve pasted come entries from ULS logs that contain this issue, in hopes it helps people find this post.

“StoredCredentialsProvider.GetCredentialsFromStore: SsoProvider.GetCredentialsUsingTicket threw a SingleSignOnException. LastErrorCode: -2147024891, Exception: Microsoft.SharePoint.Portal.SingleSignon.SingleSignonException: A call into SPS Single Sign-on failed. The error code returned was '-2147024891'.   
at Microsoft.SharePoint.Portal.SingleSignon.Credentials.GetCredentialsUsingTicket(UInt32 ui32Flags, String strApplicationName, String strTicket, String[]& Credentials)   
at Microsoft.SharePoint.Portal.SingleSignon.SpsSsoProvider.GetCredentialsUsingTicket(String Ticket, String AppID)   
at Microsoft.Office.Excel.Server.CalculationServer.StoredCredentialsProvider.GetCredentialsFromStore(Request request, ConnectionInfo connInfo).”

“SSO has failed with the following error code -2147024891, exception: A call into SPS Single Sign-on failed. The error code returned was '-2147024891'. [Session: 21.3JktrWHF9yqzbHMMHYA+W90.5.en-US5.en-US73.+0300#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060 User: DOMAIN\Username]”

“Unable to establish a connection using credentials retrieved from SSO. This could be because the Unattended Service Account is not configured or because the credentials retrieved from SSO are not valid. [Session: 21.3JktrWHF9yqzbHMMHYA+W90.5.en-US5.en-US73.+0300#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060 User: DOMAIN\Username]”

“IsImpersonateMemberOfGroup() failed”

I know there are plenty of folks out there who have had to SUFFER to try and work through an installation/integration of MOSS2007 with SSRS2008. I recently ran across a blog posting with a FANTASTIC description of how the process works, in an environment that has 3 machines. Do yourself a favor and read this blog post if you need to do this setup. It will answer a lot of questions.

http://mosshowto.blogspot.com/2009/03/reporting-services-sharepoint-multiple.html

When configuring Inter-Farm Shared Services, as per the following article, there is a slight variation. If you’re running MOSS Standard, in the child farm, you won’t see the option to select an SSP for the local Excel Services, since the farm cannot provide Excel Services. If you eventually enable the Enterprise SKU, you will need to go back and set this up.

I recently had my laptop refreshed. My oh so terrible personal space heater/noise maker Toshiba M5 was replaced with a lovely new Dell Latitude E6500. From time to time I need to run a VM on the laptop when I’m on the road. I installed VPC2007 SP1 and fired up a VM. (I’m running Vista x64 SP1) I noticed that things seemed a tad slower than before. I realized Hardware Assisted Virtualization was not enabled, and I didn’t have the ability to turn to it. The settings were greyed out, because VPC was things I didn’t have a machine that supported this configuration, while I indeed did. I made sure to have the settings enabled in the BIOS. This was the part that was causing the issue. In the BIOS virtualization settings, I have 3 options for enabling virtualization, and I had turned on all 3. Turning off the third option, “Trusted Execution”, resolved the issue. I hope this helps some other folks out there.

I recently ran across a scenario where SSO stopped working in a parent/child SSP farm set up. In this instance, there was one farm. A new parent SSP farm was created. SSO was configured there. The child farm was connected to the new parent farm. All the content stayed in the child farm, except for My Sites, which are now owned by the parent SSP. All web parts utilizing SSO in the child farm stopped working, while web parts using SSO on My Sites functioned. The issue is caused by the caching of SSO credentials in the SharePoint configuration cache. Resetting the configuration cache on machines in the child farm corrected this issue. To reset the configuration cache, do the following steps.

1. Stop the OWSTIMER services via services.msc
2. Navigate to the following directory
• Server 2003 location: Drive:\Documents and Settings\All Users\Application Data\Microsoft\SharePoint\Config\GUID and delete all the XML files from the directory.
• Server 2008 location: Drive:\ProgramData\Microsoft\SharePoint\Config\GUID and delete all the XML files from the directory.
3. Delete all the xml files
4. Edit cache.ini and reset the number in the file to 1
5. Start the OWSTIMER service on the server and ensure XML files to begin to reappear in the directory

It may take a minute or two for timer jobs to pull settings from the configuration cache as well, so have little bit of patience.

For those of you looking to double check your custom SharePoint code for proper object disposal, Microsoft has now officially released a tool to help you do this. Check out Paul Andrew's Blog Post for full details. The tool is available for download here. In my opinion, this tool should be used by every developer writing custom SharePoint code.

I recently ran into an issue when applying the MOSS August CU. This issue is not necessarily isolated to this patch. When the patch ran, the B2B upgrade process was failing in the site collection action upgrade sequence. The specific step was 12.1.6301.0, which adds the new Search Federation webparts to a site's webpart gallery. In the error logs, we saw the following:\

NewWebParts] [12.1.6301.0] [INFO] [1/10/2009 8:15:40 AM]: Add the new S2 federation webparts to the web part gallery of sites in which the portal layouts feature is active
[NewWebParts] [12.1.6301.0] [DEBUG] [1/10/2009 8:15:40 AM]: Begin Upgrade()
....
....
[SiteSequence] [ERROR] [1/10/2009 8:15:40 AM]: Action 12.1.6301.0 of Microsoft.SharePoint.Portal.Upgrade.SiteSequence failed.
[SiteSequence] [ERROR] [1/10/2009 8:15:40 AM]: Unknown server error number: d
[SiteSequence] [ERROR] [1/10/2009 8:15:40 AM]:    at Microsoft.SharePoint.Library.SPRequestInternalClass.PutFile(String bstrUrl, String bstrWebRelativeUrl, Object varFile, PutFileOpt PutFileOpt, String bstrCreatedBy, String bstrModifiedBy, Int32 iCreatedByID, Int32 iModifiedByID, Object varTimeCreated, Object varTimeLastModified, Object varProperties, String bstrCheckinComment, UInt32& pdwVirusCheckStatus, String& pVirusCheckMessage)
   at Microsoft.SharePoint.Library.SPRequest.PutFile(String bstrUrl, String bstrWebRelativeUrl, Object varFile, PutFileOpt PutFileOpt, String bstrCreatedBy, String bstrModifiedBy, Int32 iCreatedByID, Int32 iModifiedByID, Object varTimeCreated, Object varTimeLastModified, Object varProperties, String bstrCheckinComment, UInt32& pdwVirusCheckStatus, String& pVirusCheckMessage)
[SiteSequence] [ERROR] [1/10/2009 8:15:40 AM]: Unknown server error number: d
[SiteSequence] [ERROR] [1/10/2009 8:15:40 AM]:    at Microsoft.SharePoint.Library.SPRequest.PutFile(String bstrUrl, String bstrWebRelativeUrl, Object varFile, PutFileOpt PutFileOpt, String bstrCreatedBy, String bstrModifiedBy, Int32 iCreatedByID, Int32 iModifiedByID, Object varTimeCreated, Object varTimeLastModified, Object varProperties, String bstrCheckinComment, UInt32& pdwVirusCheckStatus, String& pVirusCheckMessage)
   at Microsoft.SharePoint.SPFileCollection.AddInternal(String urlOfFile, Object file, PutFileOpt fileOpt, String createdBy, String modifiedBy, Int32 createdByID, Int32 modifiedByID, DateTime timeCreated, DateTime timeLastModified, Object varProperties, String checkInComment, SPVirusCheckStatus& virusCheckStatus, String& virusCheckMessage)
   at Microsoft.SharePoint.SPFileCollection.Add(String urlOfFile, Byte[] file, Boolean overwrite, String checkInComment, Boolean checkRequiredFields, SPVirusCheckStatus& virusCheckStatus, String& virusCheckMessage)
   at Microsoft.SharePoint.SPFileCollection.Add(String urlOfFile, Byte[] file, Boolean overwrite, String checkInComment, Boolean checkRequiredFields)
   at Microsoft.SharePoint.SPFileCollection.Add(String urlOfFile, Byte[] file, Boolean overwrite)
   at Microsoft.SharePoint.Portal.Upgrade.GenericWebPartGalleryUpgrade.AddWebPart(SPFileCollection rootWebFiles, String dwpWebPath, String dwpFilePath, String dwpGroup)
   at Microsoft.SharePoint.Portal.Upgrade.GenericWebPartGalleryUpgrade.UpdateWebParts(SPSite site, String[] deleteDwps, String[] newDwps, String wpGroup, String strPath)
   at Microsoft.SharePoint.Portal.Upgrade.NewWebParts.Upgrade()
   at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade()

The "unknown server error number: d" is caused by an issue with ForeFront. Disabling ForeFront via Central Admin and rerunning PSconfig, or detaching/reattaching the content databases will clear up this issue. Don't forget to turn ForeFront scanning back on when you're done.

I'd been running SCVVM Beta 2. It was throwing some errors and I didn't need the functionality anymore, so I uninstalled it. After I did that, when I'd try to connect to VMs from Hyper-V manager, I was getting the following error.

"A connection will not be made because credentials may not be sent to the remote computer. For assistance, contact your system administrator."

The following directions will fix it.

http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/ce6a9d04-45bf-4d77-869e-bbf784cbefd1/

Adobe has FINALLY released a 64 bit PDF iFilter. For everyone out there running 64 bit MOSS, without a proper 64 bit PDF iFilter, please go get this for yourself. I haven't seen any performance data, but it should increase your indexing performance of PDFs. Details are here.  Instructions for installing on MOSS can be found on the linked page.

UPDATE: Some internal testing was done with this Adobe iFilter. The performance still leaves something to be desired. In a head to head comparison, the FoxIT iFilter still runs signicantly faster. The iFilter Blog has some data on this, which can be found here. It's likely that Adobe is continuing to use the 32 bit iFilter and the wrapper from the previous work around.

In my opinion, part of being a good SharePoint admin is to pay attention to the size of your content databases and try to keep them to a specified maximum size. (Usually for DR reasons) In order to do this, you should limit the number of site collections per content database. Naturally, site collection sizes will vary, so setting a generic maximum number might not be ideal. There are two ways to control the number of site collections in a content database. The first is to set the status of the content database to Offline. While this is a novel idea, doing this can cause some issues. One of the biggest I've seen is issues with different backup products. A better way to limit the number is to change the maximum number of sites for the content database to the number of sites in the database when the database reaches the desired size. You may need to lower the number of sites that will generate a warning, but there's no control on what this number actually is. Happy Admin'ing!

I've had the question from customers come up, about why there's differences in their crawl times. It usually relates to the number of changes that need to be processed with each crawl. SharePoint SEE Russ Maxwell has just recently posted some good stuff about this. Check out his posting. Let's hope he keeps this kind of stuff coming on a regular basis.

http://blogs.msdn.com/russmax/archive/2008/11/17/how-to-determine-the-number-of-changes-an-incremental-crawl-will-process-prior-to-initiating-the-crawl.aspx

I've recently been doing some work analyzing audit logs for a customer. Exporting the audit logs via the Site Collection's Administration pages reports fails due to the amount of data I'm pulling back, so I've turned to the Object Model to get the data out. There's a small issue here, because the data that gets exported is poorly formed for recycle bin entries. The slash is missing in the closing <recycle> tag. In order to work around this, you're going to need to modify the entries with recycle tags to fix the trailing tag.

<AuditEntry><SiteId>ea85895e-d475-4744-9c50-7212963d9d98</SiteId><ItemId>9d14e64b-9d5c-411d-9218-58da76694e6a</ItemId><ItemType>Document</ItemType><UserId>AM\starservice</UserId><DocLocation>sites/star/medrequestform/Lebanon/Fulfillment_T2/200811/LBN-0065903.doc</DocLocation><LocationType>Url</LocationType><Occurred>11/1/2008 8:54:48 AM</Occurred><Event>Delete</Event><EventSource>SharePoint</EventSource><EventData><Version><AllVersions/></Version><Recycle>1<Recycle></EventData></AuditEntry>

The errors you'll likely see are "XML parse Error" and Error Code: -1072896659, Reason : End tag 'EventData' does not match the start tag 'Recycle'

A new issue has been discovered that interferes with crawling content in specific situations. If you're indexer is set to crawl itself, via the Central Admin setting, or manual hosts modification, you might be running into this issue. You will see a combination of errors in your event logs, ULS logs and crawl logs that match the following. In order to fix this, please follow the steps in the following KB article.

http://support.microsoft.com/kb/896861

Crawl Log:
Event Type: Warning
Event Source: Office Server Search
Event Category: Gatherer
Event ID: 2436
Date:  10/22/2008
Time:  10:02:55 AM
User:  N/A
Computer: MOSS2007
Description:
The start address <http://portal> cannot be crawled.

Context: Application 'SharedServices1', Catalog 'Portal_Content'

Details:
Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content.   (0x80041205)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

ULS Logs: (verbose)
10/22/2008 10:59:04.23  mssdmn.exe (0x1420)                      0x151C Search Server Common           MS Search Indexing             7hoo Verbose  GetVirtualServerPolicy is about to call WS for virtual server, strVirtualServerUrl http://portal
10/22/2008 10:59:04.23  mssdmn.exe (0x1420)                      0x151C Search Server Common           PHSts                          0 Monitorable ***** Couldn't retrieve server http://portal policy, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:548 
10/22/2008 10:59:04.23  mssdmn.exe (0x1420)                      0x151C Search Server Common           PHSts                          0 Monitorable ****** Server portal security initialization failed, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:317 
10/22/2008 10:59:04.23  mssdmn.exe (0x1420)                      0x151C Search Server Common           PHSts                          0 Monitorable CSTS3Accessor::InitURLType: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:1542 
10/22/2008 10:59:04.23  mssdmn.exe (0x1420)                      0x151C Search Server Common           PHSts                          0 Monitorable CSTS3Accessor::InitURLType fails, Url http://portal, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:184 
10/22/2008 10:59:04.23  mssdmn.exe (0x1420)                      0x151C Search Server Common           PHSts                          0 Monitorable CSTS3Accessor::Init fails, Url http://portal, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:243 
10/22/2008 10:59:04.23  mssdmn.exe (0x1420)                      0x151C Search Server Common           PHSts                          0 Monitorable CSTS3Handler::CreateAccessorExB: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:261 

Security Logs:
Logon Failure:
Reason: An error occurred during logon
User Name: mosscrawl
Domain: SPI
Logon Type: 3
Logon Process: Èù8
Authentication Package: NTLM
Workstation Name: MOSS2007
Status code: 0xC000006D
Substatus code: 0x0
Caller User Name: -
Caller Domain: -
Caller Logon ID: -
Caller Process ID: -
Transited Services: -
Source Network Address: XX.XX.XX.XX
Source Port: 1417

If you haven't had the pleasure of patching WSS 3.0 or MOSS2007, you're missing a real fun experience. If you are one of the folks who've had to go through the pain, there's some good news for you. If you're particularly adept at reading KB articles, you might have noticed this little tidbit in 953749.

"Improvements to the time that is required to update and upgrade Windows SharePoint Services sites."

Starting with the release of the Infrastructure Updates, we've changed the patching process a bit. Each time a patch was applied, we followed all of the upgrade sequences to perform a build to build upgrade to get an install to the latest version. One of these sequences is the site collection upgrade sequence. This sequence went out and touched all site collections, even if it was to just update the build number. With the IU, we no longer do this. We'll only call this upgrade sequence if there are changes that need to be made to the schema that affect site collections. (Touching every site collection was a very time intensive process) This is the good news.

However, this is only part of the picture. While this may show some improvement, it's not the end all of fixes. If you're only performing major service pack patching, this is highly unlikely to save you any time, as there are likely to be object level schema changes. The real benefit will be for applying QFE and CU updates. It also needs to be said that SharePoint (WSS and MOSS) patches are cumulative, so if you think you're only applying a recent QFE/CU, you are indeed getting all the fixes between your currently installed build and the the build of what you're patching to.

As always, different environments will perform differently. In order to get accurate estimates as to how long patching will take for you, you're going to need to test with a copy of production data in a test environment.

Patching is a very confusing process for most customers. Please do not hesitate to contact Premier Support for guidance. That's what we're here for.