Part 1 - Scott, why do you always ask me to run SPSreport when I call in for help?

Scott Fellman — Sat, 30 Jun 2007 15:44:00 GMT

At Microsoft, our support process goes basically like this: Our customers research an issue and don't find any answer using all resources available (Knowledge Base, TechNet, MSDN, www.live.com search, WSS Technical Library, resource kit, etc...) that helps them resolve said issue. They then call in to support, give their name, phone number and other information to a Customer Service Rep, and finally get transferred to a Support Engineer with the incident number. Note: our SharePoint SEs do not operate from a script! They have freedom to troubleshoot an issue in any logical (or illogical) way that they choose. But I digress...

At this point, the customer describes the issue that they are seeing and hopefully then the engineer engages in scoping with the customer. Scoping is the narrowing of focus to single, indivisible issue. An example of where scoping during a call might go like this:

MS Customer: Our SharePoint search is broken
MS Engineer: Can you give me more detail on how search is broken? What symptoms do you see?
MS Customer: Our CEO types a keyword that I know exists in a document in our executive SharePoint document library into the search box at the top of the page and the document is not returned in the search results.
MS Engineer: Great, thanks for that description, that helps me narrow our scope a bit, but I need to ask you a couple of further questions
MS Customer: I am really in a hurry, can we start troubleshooting now? My boss is all over me to get this issue resolved.
MS Engineer: I understand that you are in a hurry, no one calls support to tell us what a great job the software is doing - that information usually goes to our sales team. Now; question one - what is the document type that is being searched?
MS Customer: It’s a PDF document called triple_secret_business_stuff.pdf. The size of the file is slightly over 100 meg.
MS Engineer: Great, can you tell me the URL of the document library hosting the doc lib?
MS customer: Yes, it's http://foo.bar.internal/sites/exec/forms/allitems.aspx

Now we have completely scoped the issue: An executive can't return search results from a document library using a keyword that is known to exist within that document. Specifically the document is a large PDF housed in what appears to be a simple WSS document library.

From the above description, we have a scope, but what should the troubleshooting look like from here? Well, in order to understand this, we need to know a few things:

How search works in SharePoint (architecture, components, flow between the components)
Special conditions that apply in this case that may differ from Microsoft/SharePoint default settings.
How permissions work in SharePoint sites

And we may need to know other things as well:

The topology of the farm where the issue is happening
Hardware/OS specifics of the machines in the farm (x64 or 32 bit, teamed NICs, etc...)
Version and build of SharePoint
Binaries for the various modules that are required to make search work properly
Events in the application event log or other logs that may help diagnose the point of failure
Special SharePoint diagnostic logging output (by default, c:\program files\common files\Microsoft shared\web server extensions\12\logs)

So how best to gather this data? There are several options. Back in the day (around 10 years ago), our troubleshooting options were pretty limited; screenshots over email or verbal description of each required piece of information. Not too fast these, and our example customer above would be feeling less and less joy with each passing question or email since his CEO is asking for status updates every 23.62 seconds. More recently, remoting technologies like Terminal Services and Microsoft Office LiveMeeting have made the data collection process quicker, but in a case like our example, the level of complexity, and therefore the number of things that may need to be reviewed is huge.

Enter the request for SPSreport from each machine in the farm as made by your friendly Microsoft Support Engineer. SPSreport is a free download from http://www.codeplex.com/spsreport. From the description of the tool on that site:

The SPS Reporting Tool is utilized to gather detailed information regarding a systems current configuration. The data collected will assist the Microsoft Support Professional with fault isolation.

In part two of this web log, I will set up a farm with four SharePoint machines, a remote SQL 2005 server, an Executive team site and a large PDF. I will break the farm so that the symptom, "PDF not showing in search results" happens. I will break the farm in multiple ways, so that reviewing the entire troubleshooting/research process may take more than a single blog to cover completely. By tying the same symptom to various root causes, I hope to show the value of our Support Engineer's request for SPSreports; who knows - maybe by the end of this series our customers will do their research AND run the report for each machine before calling in for support.

See you soon!

Outlook without religion is lame : Scoping

Part 1 - Scott, why do you always ask me to run SPSreport when I call in for help?