Microsoft SQL Server Support Blog : Miscellaneous

Tools of the Trade: Part IV – WinDbg & AdPlus (I)

sqlblog — Tue, 04 Sep 2007 17:50:00 GMT

In this series of posts, we’re going to take a look at some of the tools we at Microsoft Developer Support -- Database commonly use, how to use them, and the types of issues they can be used to resolve.

In this article, Enamul Khaleque starts down the road of using WinDbg and AdPlus. Future articles will move beyond identification, and discuss step-by-step approach of using WinDbg to solve wide array of issue including but not limited to: memory leaks, high & low CPU hangs, heap corruption, race conditions and crashes/exceptions in both managed and unmanaged environments.

The Purpose

You may be wondering why I am talking about WinDbg and AdPlus while you already have Visual Studio debugger. If you ask your network/system admin to install Visual Studio in a production environment, the odds of hearing a big NO is 99.99% if not 100%. That’s because Visual Studio is an extensive development environment and not suited for a production environment. Fortunately, we can capture runtime state in a file (called a memory dump or simply dump) using light-weight tools such as AdPlus and later analyze them using WinDbg. For this article, I will assume you have no prior experience in WinDbg or AdPlus.

Installation

WinDbg and AdPlus are part of Debugging Tools for Windows. Both 32-bit and 64-bit versions are downloadable from following URL:

http://www.microsoft.com/whdc/devtools/debugging/default.mspx

AdPlus Basics

Imagine that after upgrading your application in production, you are now seeing significant response delay. You are not able to reproduce this behavior in a Dev, Test or QA environment, so your only option is to capture the application state while the latency is occurring at runtime. With AdPlus, you can capture a “hang mode” memory dump to determine what the process is waiting on.

Instead of seeing delays, let’s say this application is crashing in the production and you can’t determine why. With AdPlus, you can also capture “crash mode” memory dump to determine what causes the process to crash.

Typically in a hang or a memory leaks scenario, you take multiple dumps in few minutes interval. This helps to understand and analyze the state of the application over a period of time. One thing you need to keep in mind that a dump file contains only the state of the process at the time the dump was taken, so multiple dumps are useful when we want to see the process state over a time span.

Live debugging should be your first option since dump is limited to show conditions only of what already happened. Also you can’t set conditions in dump as you can do in live debugging. You capture dump when application is running in an environment live debugging is not at all possible. Here is a list of typical scenarios when we want to capture a dump file:

· Application hangs with high or low CPU cycle

· Application is leaking memory

· Application is crashing or throwing exceptions

· Heap corruption

The ADPlus.vbs script itself does not require any specific configuration to use it, however you might be prompted to change your default script interpreter from WScript.exe to CScript.exe. Allowing ADPlus.vbs to set CScript as the default script interpreter is strongly recommended. If you continue to use WScript as your default script engine, then you will see a separate window when you run a command against ADPlus.vbs. Optionally, you can place CScript at the beginning of your command line, but you must include the ‘.vbs’ extension on ADPlus.vbs.

AdPlus creates dump files. These dumps contain stack information about the process(s) you are monitoring. This dump file and stack information relies on symbols (.pdb) to resolve that stack information. If a symbol path is not specified in the command line, it will use the _NT_SYMBOL_PATH environment variable if one is set, otherwise stack information in the dump will be limited. Here is how you can set the symbol path at the machine level.

Set _NT_SYMBOL_PATH=SRV*http://msdl.microsoft.com/download/symbols

To use AdPlus, you must specify a series of command-line options or arguments to the script. At a minimum, AdPlus requires two command-line options: one that specifies the mode of operation, and one that specifies a target process to operate against.

C:\> Adplus.vbs –hang –p 123 –o C:\Dumps

First, a run mode is required when passing arguments; here we used –hang. The run mode has two options –hang and –crash. The second mandatory parameter is the process to monitor. Here we are using the –p switch to tell ADPlus.vbs to capture a hang dump of a process with the process ID of 123. When AdPlus is running in hang mode, AdPlus must be started after the process hangs or is consuming high CPU utilization.

C:\> Adplus.vbs –crash –p 123 –o C:\Dumps

Crash mode is supported in a Terminal Server session on Windows XP and later versions of Windows. Hang mode will work inside a Terminal Server session on any platform. When AdPlus is running in crash mode, AdPlus must be started before the process crashes or becomes unstable.

When ADPlus.vbs is running in crash mode, a debugger remains attached to each process that is specified on the command line for the lifetime of that process. To manually detach the debugger from the process, you must maximize the debugger window and press CTRL+C to break into the debugger.

WinDbg Basics

WinDbg is the ultimate debugger both for User mode and Kernel mode debugging. WinDbg is used mostly for analyzing dumps, yet you can capture dumps with this just like AdPlus.

In our next article on this series, we will see step-by-step examples of capturing dumps with AdPlus and then analyzing the dump with WinDbg.

What’s Next

In future posts, we will deal with this vast subject with one scenario at a time including but not limited to memory leak, High & low CPU hang, Heap corruption, Race condition and crash/exception both managed and unmanaged environments. We will also see how to leverage WinDbg with extension DLLs to make our debugging more pleasant.

References

HOWTO: Use AdPlus to troubleshoot "hangs" and "crashes" http://support.microsoft.com/default.aspx?scid=kb;en-us;Q286350

Posted By: Enamul Khaleque

SQL Server Support FAQ

sqlblog — Thu, 16 Aug 2007 07:15:00 GMT

How often should I expect to hear from my Support Engineer?

This will vary based on the issue and its impact to your business. If you have specific preferences for frequency of contact, time of day, or method (prefer phone vs. e-mail), please discuss this with your support engineer.

My Support Engineer is not available and I need help now.

If you need immediate assistance and your engineer is unavailable, please contact the engineer’s manager. Manager contact information is in the engineer’s signature block and in the initial e-mail.

If you are unable to reach the manager, you can call the Microsoft Support line (800-936-5800), provide your case number, and request to speak with the next available support engineer. You will be routed to the phone queue for that specialty and connected to an engineer. The Support Engineer receiving the call may need time to review the case history and steps taken to date, depending on the duration and complexity of the issue.

My support issue is not progressing well.

Each Support Engineer should provide a management contact in his/her signature block and in the initial e-mail. If you have:

· Schedule conflicts with your assigned engineer’s availability.

· Communications issues.

· Need a faster resolution.

· Or are otherwise dissatisfied with the support provided, please engage the engineer’s manager.

We strive to deliver outstanding customer service, and we appreciate you letting us know if we are not meeting your expectations.

I need a different engineer.

As mentioned in the preceding section, “My support issue is not progressing well”, you may contact the engineer’s manager and request a different engineer.

My Support Engineer has suggested that we archive the case. What does this mean?

If there is going to be an extended delay before you can implement a change or collect requested data, we may ask to archive your case. This means that the case will be closed on our end. You can reopen the case if you encounter the same error on the same instance/server again, or if you have the data needed to continue troubleshooting.

In order to reopen the case, contact the support engineer or his/her manager directly. You may be asked to provide feedback on support. Please assess the support based on the service received to date, with the understanding that Microsoft does not consider an archived case resolved.

We thought that our issue was solved, but the problem has recurred. Can I re-open my support case?

As long as the problem and the server are the same as in the case, you can re-open the case. To do this, first try to reach the previous engineer or his/her manager. If unable to reach them, contact the Microsoft Support line (800-936-5800), provide your case number, and request that Microsoft re-open the case.

Note that in some cases, the same symptoms can be caused by multiple issues. For example, if a server experienced slow performance and the disk I/O subsystem was fixed and performance restored, and later performance degraded again, this could be due to out of date statistics or other issues. In these ambiguous cases, you may be asked to open a new case.

What do the Microsoft Case Severities mean?

A – Critical – 1 hour initial response goal

· High-impact problem in which production, operations, or development is proceeding but is severely impacted, or where production and/or profitability will be severely impacted within days.

B – Urgent – 4 hour initial response goal

· High-impact problem where production is proceeding, but in a significantly impaired fashion.

· Time sensitive issue important to long-term productivity that is not causing an immediate work stoppage.

C – Important - 24 hour initial response goal

· Important issue which does not have significant current productivity impact for the customer.

D – Monitor

· This severity is used for support issues that are waiting, usually to confirm a provided solution or when waiting for an intermittent problem to recur, in order to collect needed data.

The Support Engineer for my support issue said that I need to open an additional support incident. Why should I have to open another incident?

Microsoft defines an incident or case as:

· A single support issue and the commercially reasonable efforts to resolve it

· A problem that cannot be broken down into subordinate problems

NOTE: The Customer and the Support Engineer must agree on the case resolution. Also be aware that Microsoft does not charge for code defects, documentation errors, and product “wishes”.

As documented on Microsoft’s public website at http://support.microsoft.com/?LN=en-us&sd=tech&scid=gp%3Ben-us%3Bofferprophone&x=7&y=16#faq607

EXAMPLE: A support issue is opened for slow performance of an application with a SQL Server back-end. Troubleshooting identifies a specific stored procedure as the problem, and updates to statistics and an increase in covering indexes resolves the issue.

However, the same server is also experiencing slow performance for merge replication. This is considered a different issue, which is unrelated to the original issue for which the support case was opened.

I’m interested in a higher level of service and a closer relationship with Microsoft.

Microsoft offers a managed support relationship known as Premier Support, designed to meet the needs of an enterprise customer. For an overview of the service elements, please see http://www.microsoft.com/services/microsoftservices/srv_prem.mspx or call 1-800-936-3500 to request contact by a Premier Support representative.

Advisory vs. Break-Fix

Microsoft provides support services for customers with problems encountered implementing or using Microsoft products. These are commonly known as “break-fix issues”. Occasionally customers need short-term (less than 40 hours) consultative assistance, as opposed to the standard break-fix service.

This short-term consultative assistance is known as Advisory service. This is available to Microsoft’s Premier customers as an element in their support contracts. It is also available to other customers on an hourly charge basis.

For more information on Advisory Services, please see the Advisory Services section at http://support.microsoft.com/default.aspx?scid=fh;EN-US;OfferProPhone or go to the Advisory Services page, http://support.microsoft.com/gp/advisoryservice .

Hope this helps!

Posted By: Microsoft SQL Server Support Team

Tools of the Trade: Part III – Kerberos (I)

sqlblog — Mon, 16 Apr 2007 21:57:00 GMT

In this article, Adam Saxton starts down the long road of double-hop troubleshooting by providing a gentle introduction to Kerberos, and how to recognize Kerberos+SQL related issues. Future articles will move beyond identification, and discuss the tools and processes we use to troubleshoot such issues.

Kerberos

We receive numerous calls where the problem falls under Kerberos. This will present itself when trying to establish a connection to SQL Server (or a Linked Server). I’m going to discuss what Kerberos is, how to recognize when there is a problem with it, and go over some of the items that need to be looked at in order to resolve a Kerberos issue with SQL Server. As there is a lot of information on this topic, I’ll be splitting it up over a few blog posts.

What is Kerberos?

Kerberos is a secure method for authenticating a request for a service in a computer network, and is used to verify a user or host identity. Kerberos lets a user request an encrypted "ticket" from an authentication process that can then be used to request a particular service from a server. The user's password does not have to pass through the network.

Put simply, it is a more secure form of Windows Authentication versus NTLM. Also, understand that Kerberos is not a Windows specific technology. Windows will always try to use Kerberos first over TCP. If that doesn’t work, it will drop down to NTLM. The exception to this is if you are trying to access a resource that is local. NTLM will always be used when accessing local resources. Also note that Kerberos cannot currently be used over Named Pipes or Shared Memory.

How does Kerberos work?

There are a lot of details into how Kerberos works. I’m not going to get that deep into it within this blog series. I’m going to relate this to how it works with SQL. To do that, I’m going to use the classic example of a Double-hop with a Web Server. We use the work Double-hop to explain that the IIS Server is using a resource that is located on a different server. In this case the first “hop” is from the web browser client to the IIS Server (ASP or ASPX Page); the second hop is to the SQL Server on a different machine. The server on the second hop is also referred to as the Backend Server.

In order to successfully establish a connection using Kerberos, a SQL ServicePrincipalName (SPN) needs to exist within Active Directory. I will talk more about SPN’s and where they are located in a later post as the topic is complex enough to deserve its own. I would say about 80-90% of calls we receive relating to Kerberos involve either missing SPN or duplicate SPN’s.

When you log onto your machine, you will get what we call a Kerberos Ticket. To be more specific, you get a Ticket-Granting Ticket (TGT). You use the TGT as a master ticket to access all Kerberos services on a network. A Kerberos ticket includes all the user credentials or computer credentials in an encrypted format. These credentials are used to identify a specific user or a specific computer on a network for access to Kerberos services. When you access a Kerberos service, that service uses the ticket that you received to authenticate you. After the service verifies your identity and authenticates you, the service issues a service ticket. This is where the SPN’s come into play. Think of the SPN as a pointer for Kerberos so it knows where to go. That’s why, if it’s missing or there are duplicates, it doesn’t know what to do.

How to recognize a Kerberos error with SQL Server:

There are a few errors that we see where we can make an educated guess that a connectivity issue is Kerberos related.

Cannot Generate SSPI Context

Login failed for user ‘(null)’

Login failed for user ‘NT AUTHORITY\ANONYMOUS’

Those are the three biggest errors we see in relation to Kerberos with SQL Server. These errors occur when the credentials from the first hop cannot be passed to the second hop. Usually, this is because we are failing over to NTLM or our SPN configuration is not correct. There are also other settings within Active Directory that come into play as well as certain machine configuration based on how services are setup. For now, let’s focus on the basics, so I’ll get more into those setting and machine configurations in a later post.

One of the last things I’d like to mention is that I’ve seen customers get the following error confused with a Kerberos issue:

Login failed for user ‘SomeUser’

The SomeUser could be whatever user (SQL or Windows) that you are trying to connect with. If you see the actual username in the error that means the credentials actually got to the SQL Server. The issue at that point lies in the SQL Security configuration itself (either the server as a whole, or the database you are trying to connect to). This is not a Kerberos related issue, but instead is a much simpler login failure.

Resources:

Kerberos in Windows 2000

Kerberos in Windows 2003

How to use Kerberos authentication in SQL Server

Posted By: Adam Saxton

Tools of the Trade: Part II – The Repro

sqlblog — Mon, 16 Apr 2007 21:52:00 GMT

This post might seem like a bit of a stretch. For one thing, it’s not about a SQL related tool. That’s not such a big deal – there are plenty of tools that we use that have applications beyond SQL. You can expect more posts in this series about tools that have applications far beyond troubleshooting databases and database connectivity.

However, this isn’t about a specific, pre-existing tool either. Call it a meta-tool, I guess. Pseudo-tool. Something along those lines. One way or the other, even if it isn’t already built for you, a reproduction can be one of the most invaluable and time-saving tools you’ll use to troubleshoot your most difficult issues.

The Repro

In the Product Support Services world at Microsoft, we refer to a simplified, stripped down series of steps that lead to a specific behavior as a Repro. This may simply be a step-by-step to recreate an error, or a simple application that recreates an unusual behavior. When we are working on an issue that is buried in a hulking, production application, we often like to tackle it is by attempting to recreate the issue outside of the in vivo environment. There are a lot of benefits to investing some time in developing a Repro.

It’s probably pretty obvious why you would not want to troubleshoot in a production environment. I personally like to avoid debugging anything that generates revenue – it just seems to ease the tension. There’s more to it, though – not only do you avoid tinkering with your company’s livelihood, a Repro also gives you the advantage of portability. Whatever technology you’re working with, it is much, much easier to get the assistance of Subject Matter Experts if you can actually send them something to play with.

Another big benefit of the Repro is that it helps you to focus on the issue. This is particularly helpful when you need to use other tools to analyze the behavior with which you’re dealing. Whatever kind of tracing you’re doing, you are bound to pick-up extraneous information if you’re running the trace on a production application. Unless you’re very fond of wading through gigantic logs looking for clues, a Repro really helps to avoid getting lost in superfluous data.

Finally, a repro can save you a huge amount of time. It can get so irritating having to click through the same series of screens and options, over and over again, as you troubleshoot an issue in a full-scale application. Being able to make a change to code or the environment, and quickly test dramatically improves the speed of the troubleshooting process. It also has the added benefit of keeping debugging from getting boring.

That’s my case for investing the time in creating a Repro. The next question is: what is the best way to build a Repro? Well, that’s going to depend greatly on the issue and the environment.

One thing that I like to do is work from the inside out. In other words, if I know that a specific line of code is throwing an exception, I’ll create an application with the smallest amount of code I can use to execute that line. If that doesn’t reproduce the behavior, I’ll add more code from the application around it, building up the Repro until it gets the behavior I’m looking for.

Another thing that I’ve found is important is to make sure that I attempt my Repro in an environment that’s as similar to production as possible. I do not recommend attempting to reproduce an issue occurring in a data-access application running on Windows Server 2003 on your grandma’s Mac. It’ll be much easier to create a repro on something that’s as close to the original scenario as possible and then move it to other environments to see if the behavior is any different.

One of the Technical Leads on our team has a good tip for reproducing issues that occur intermittently. For example, say you find that every-so-often your application gets an error when attempting to connect to a data-store. In this situation, it’s probably worth a try to take the connection you’re trying to make and wrap it in a tight loop. It’ll be much easier to launch this and catch the failure than it will be to keep running and re-running a small repro that makes the connection a single time and exits. If you are troubleshooting an issue occurring in a web-application, you may try a tool like Application Center Test to execute your repro many times simultaneously (see: this link for a discussion of using ACT).

Finally, I really recommend carefully documenting your repro. Again, it makes it much easier to get the help of others, and it will also prevent you from forgetting what you had done on Friday after a long weekend. Even if you really cannot find a way to reproduce the issue in a stripped down environment, documenting the steps an end-user takes when the issue occurs can be valuable for helping to clarify the problem statement and focus your efforts.

Posted By: Justin Trobec

We upgraded our servers to Windows 2003 SP1, and now our data-driven application fails with strange errors!

sqlblog — Wed, 15 Nov 2006 16:55:00 GMT

Problem Description

After upgrading an application server or database server to Windows 2003 SP1, the application encounters errors originating from data-access components. Typically we'll see this when one server in the application server/database server scenario is updated and the second server is left on a previous version of the OS (particularly Windows 2000 Server).

There can be a variety of errors involved, the most obvious is along the lines of:

[Microsoft][ODBC SQL Server Driver][SQL Server]The operation could not be performed because the OLE DB provider 'MSDASQL' was unable to begin a distributed transaction.

However, we've often seen much less obvious errors as well

"Operation is not allowed when the object is closed."

The particular error message can be largely dependent on how thoroughly the component that executes the data-access code examines the results of method calls.

Possible Solutions

Most of the times when customers call in with this scenario, the solution has been to simply modify the security settings for MSDTC on the Windows 2003 SP1 machine.

Windows 2003 SP1 and Windows XP SP2 add new security enhancements to MSDTC, including support for secure communication between two machines participating in a distributed transaction.

The catch is that by default, MSDTC on Windows 2003 SP1 will attempt to authenticate the other machine with these new features. Consequently, if the other machine is not Windows 2003 SP1 or later, it will not be able to participate in a distributed transaction with the updated server.

In order to disable MSDTC authentication on the Windows 2003 SP1 server:

1) Open Control Panel -> Administrative Tools

2) Open the Component Services msc.

3) Expand the Console Root\Component Services\Computers node.

4) Right Click on My Computer and select properties

5) Select the MSDTC tab, and click the Security Configuration… button.

6) Make sure the following values are checked:

a. Network DTC Access

b. Allow Inbound

c. Allow Outbound

d. No Authentication Required

e. If you’re using JDBC or Oracle, you’ll also want Enable XA Transactions checked

7) Click OK until all the dialog boxes are closed, and exit the Component Services tool.

8) While not strictly required, it may be worth recycling your application process, or even rebooting to make sure the changes are in effect.

9) Retest your application.

Further Information

If you need more information regarding what the different settings in the Component Services MSDTC dialog mean, this blog post does a good job of explaining them:

Florin Lazar's WebLog -- XP SP2 and Transactions

A common question in regards to these settings is - does turning off Mutual Authentication mean we’re not secure?

The answer is that you’re no less secure than you were before installing Windows 2003 SP1, so it should not be a problem for most organizations. On the other hand, if you can use Mutual Authentication (which would require updating all servers involved in distributed transactions to Windows 2003 SP1), then it’s a good idea to do so.

Hope this helps!

Posted By: Justin Trobec

Installing MS06-061 on NT4

sqlblog — Sat, 11 Nov 2006 02:28:00 GMT

When installing MS06-061 for MSXML4 SP2, you may receive the following error when trying to run msxml4-KB925672-enu.exe:

The specified program requires a newer version of Windows.

NOTE - Before installing this update, please make sure you have already installed SP2 for MSXML4.

In order to install the update properly, you need to extract the MSI file from the EXE.

To do that, run the following at a command prompt on a Windows 2000 machine or higher:

msxml4-KB925672-enu.exe /extract:"c:\temp"

This will extract msxml.msi. You can then install msxml.msi.

Posted By: Adam Saxton