Thou shalt not PANIC

Under the hood with live debugging

2009-03-18T21:47:42Z

The team i belong to (Portugal Adcs) has a blog called Deviations (http://blogs.msdn.com/deviations) and from time to time i will post some stuff over there (when that happens i will put a link here).

My latest post there was about live debugging using Windbg and you can check it out at

http://blogs.msdn.com/deviations/archive/2009/03/18/under-the-hood-with-live-debugging.aspx

Have fun
Bruno

WCF: Getting MessageSecurityException while calling webservice

2009-03-03T18:37:21Z

Time flies and it looked like it was yesterday that i wrote my last post. Last couple of months have been very busy, so blogging was put on hold but everytime i got a case that i believe i should talk about it, i kept it. So expect a burst on posts.

Yesterday, I and a co-worker were troubleshooting an strange issue that didn´t occur in dev but in staging (if i get a cent every time i ear this) a MessageSecurityException was being thrown stating unauthorized access. If we allowed Anonymous Access then no exception was being thrown. Below is the scenario.

Scenario

Single Machine environment
Windows 2003 and IIS6
Both WebApp and WebService on same pool and running under an account with privileges
WebApp calling locally WebService
Integrated Authentication (Anonymous not allowed)

Details

So, we decided to take a memory dump on System.ServiceModel.Security.MessageSecurityException using DebugDiag.(http://www.microsoft.com/downloadS/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en)

Lets look at the faulting thread

0:025> !clrstack

OS Thread Id: 0x37c (25)

ESP EIP

1c44ec54 77e4bee7 [HelperMethodFrame: 1c44ec54]

1c44ecf8 1ba39e6d System.ServiceModel.Channels.HttpChannelUtilities.ValidateAuthentication

1c44ed1c 1af83ec3 System.ServiceModel.Channels.HttpChannelUtilities.ValidateRequestReplyResponse(System.Net.HttpWebRequest, System.Net.HttpWebResponse, System.ServiceModel.Channels.HttpChannelFactory, System.Net.WebException)

1c44ed50 1af83e4e System.ServiceModel.Channels.HttpChannelFactory+HttpRequestChannel+HttpChannelRequest.WaitForReply(System.TimeSpan)

1c44ed90 1af82f24 System.ServiceModel.Channels.RequestChannel.Request(System.ServiceModel.Channels.Message, System.TimeSpan)

1c44ee04 1af82da7 System.ServiceModel.Dispatcher.RequestChannelBinder.Request(System.ServiceModel.Channels.Message, System.TimeSpan)

1c44ee18 1af95cb2 System.ServiceModel.Channels.ServiceChannel.Call(System.String, Boolean, System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.Object[], System.Object[], System.TimeSpan)

1c44ef48 1af95af3 System.ServiceModel.Channels.ServiceChannel.Call(System.String, Boolean, System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.Object[], System.Object[])

1c44ef68 1af95a03 System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage, System.ServiceModel.Dispatcher.ProxyOperationRuntime)

1c44ef8c 1af978e5 System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage)

1c44efd0 792c128d System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)

1c44f270 79e71e04 [TPMethodFrame: 1c44f270]

Now, lets look at exception details

0:025> !pe

Exception object: 0e92a348

Exception type: System.ServiceModel.Security.MessageSecurityException

Message: The HTTP request is unauthorized with client authentication scheme 'Ntlm'. The authentication header received from the server was 'Negotiate,NTLM'.

InnerException: System.Net.WebException, use !PrintException 0e929790 to see more

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80131501

025> !pe 0e929790

Exception object: 0e929790

Exception type: System.Net.WebException

Message: The remote server returned an error: (401) Unauthorized.

InnerException: <none>

StackTrace (generated):

SP IP Function

1C44ED10 7AA9918E System_ni!System.Net.HttpWebRequest.GetResponse()+0x51123e

1C44ED50 1AF83D97 System_ServiceModel_ni!System.ServiceModel.Channels.HttpChannelFactory+HttpRequestChannel+HttpChannelRequest.WaitForReply(System.TimeSpan)+0x37

StackTraceString: <none>

HResult: 80131509

Also in IIS Logs we could see 401.1 unauthorized.

In our case the solution was to install security update 957097. More details, info on workaround or download update at

http://support.microsoft.com/default.aspx?scid=kb;EN-US;896861 (You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or IIS 6)

After doing this we decided to look at dev environment, and surprise, this update was installed.

I know you all have eared this before, so here it goes again: it is very important to keep environments (dev, stage, production) the most similar to the extent possible.

Have fun.

Bruno

Process Explorer to the rescue

2008-10-16T17:10:00Z

Monday was a day to forget. I arrived at the office around 8:30 am, started my machine (like always), inserted my credentials, the desktop shows up and suddenly it gets unresponsive. First thing I do is CTRL+ALT+DEL, start Task Manager and it also gets hanged. The only commands that were responsive was log off and switch user (very useful right? J ).

My first mistake was not to use the same approach I always use in customers (do what I say not what I do J ) and started trying the try…error approach. The only thing this did was to waste my time with successive reboots, remove profile, add profile and other stuff. In my machine I have two profiles: the local admin one and another that I use to connect to Microsoft. The problems were in this second profile. The first one worked without any problems.

So I decided to stop for a minute and follow the typical methodology I always follow. So since my suspicions were on a driver or some program I had installed over the weekend (beta versions, don´t get me wrong I love beta versions because I believe they are a great way to help products getting better and better).

Since Switch User was working, the first step was to reproduce the problem. So I logged in with the profile that was having problems and then I switched to my local admin account. Then I started Process Explorer (aka ProcExp, you can download it from http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx ). If you don´t know this tool, you can think of it as task manager on steroids and is a very valuable troubleshooting tool.

After opening ProcExp you will see a list of all processes running on your system and also the account that started the process (this allowed me to see processes that were running in my problematic profile).

To see the username, you have to right click on one of the columns and “Select Columns” > “Username”. Then if you click order by Username, you will notice that each group will have a different color. You can see this in the image below (i´ve removed the usernames from my machine, that´s why you see a blank column under Username)

Process Explorer allows you to see threads and call stacks for each process, so my approach was to look at processes that were executing under DOMAIN\USER and dig into each one of them trying to find strange behaviors. You might ask, what is a strange behavior? I would say, it depends (wow, I’ve never heard that answer before). In my case I decided to start looking at processes that were consuming more CPU and those with names that I was not familiar with.

So how do you look at threads and call stacks? Right click on a process and select Properties. Inside Properties you have a tab called Threads. Click on it.

Here you can try to identify some thread that does not look good, or it might be that they are all hang (these are just some examples, and the more experience you have the easier it would be for you) . Let´s imagine I had one thread that was consuming almost all CPU and then i could look at the stack (select your thread and click on STACK)

So I did this for a couple of processes until I identified the bad boy. Then I disabled it, switched user and “voilá” all my problems were gone (and part of my working day).

Paulo, after all it was not the beta version you gave me that got me into troubles J

Happy debugging!!!

Bruno

Tools @ Opening huge log files

2008-10-16T13:48:00Z

Part of my work is looking at huge log files and this can sometimes be a big pain to open in any text editor. I was talking with a coworker about this, and he showed EditPad Lite, a free text editor (for non-commercial use, you also have the non-free version) and I was very impressed how fast it was to open huge files and scrolling up and down without the slightest feeling of hang.

So, I suggest you give a look at it. You can get it at http://www.editpadpro.com/editpadlite.html

Bruno

Debug Jscript with Internet Explorer 8

2008-10-09T12:47:00Z

My team has started a blog. Since we all have different skills, it will be more generalist and there you can get information on MOSS, Biztalk, Troubleshooting, Dev, Silverlight and other stuff.

Every time I post there I will put here a link to the post (I will talk mostly about troubleshooting).

My first post there is about Developer Tools that come with Internet Explorer 8 and how they can help us troubleshooting Jscript errors (I know, they are a big pain for us all developers J ).

The post can be seen at http://blogs.msdn.com/deviations/archive/2008/10/09/debug-jscript-with-internet-explorer-8.aspx.

About the team blog, you can check it at http://blogs.msdn.com/deviations.

Bruno

Production Troubleshooting Flowchart

2008-09-19T12:29:00Z

Usually (probably always I hope) when troubleshooting production environments we don´t have Visual Studio or other “nice looking” tools to help us finding problems. But we have one of my favorite tools: Windbg.

The first time someone showed me Windbg and all the stuff around production debugging like memory dump, … I was a little (maybe not just a little) scared on this new world that had just opened in front of me. So my next question was: where do I start from? And probably like almost everyone that starts exploring this “new world” there are excellent blogs like http://blogs.msdn.com/tess (If Broken it is, fix it you should), http://blogs.msdn.com/dougste (Notes from a dark corner), http://blogs.msdn.com/tom (ASP.NET Debugging) and many others.

One of my biggest problems at first was to understand (and remember) what and when I should collect information that could help me to troubleshoot the issue. The blogs above have a lot of information about taking the first steps. To simplify this task I´ve created a little graphical memo that would help me. Now don´t take this flowchart as the “holy grail” but I believe it can help (at least until you get the basics in your mind).

Right click on the image to save it and it will be more “readable”.

Have fun

Bruno

Windbg: Using .shell to search text

2008-09-06T21:02:00Z

To me one of the most useful commands when using windbg is .shell. According to Debugging Tools For Windows documentation

“The .shell command launches a shell process and redirects its output to the debugger, or to a specified file.”

So, why would I find that interesting and useful in my day to day work to launch a shell process ? Maybe to impress friends (that know nothing about debugging) with strange commands or make it sound very complicated. Well, actually I use .shell in one of the most simple tasks we all do everyday and that´s finding text. Now you might be thinking, so why don´t you use Ctrl+F and find what you want ? (We will talk about this in a couple of minutes)

So, has I was saying, I use it a lot to find text inside memory dumps and this saves me time. Since .shell launches a shell process, the key here is to use the old FIND command from DOS to help us. FIND allows to search for text inside a file. If you open a command line and do FIND /? You will see something like below.

Searches for a text string in a file or files.

FIND [/V] [/C] [/N] [/I] [/OFF[LINE]] "string" [[drive:][path]filename[ ...]]

/V Displays all lines NOT containing the specified string.

/C Displays only the count of lines containing the string.

/N Displays line numbers with the displayed lines.

/I Ignores the case of characters when searching for the string.

/OFF[LINE] Do not skip files with offline attribute set.

"string" Specifies the text string to find.

[drive:][path]filename

Specifies a file or files to search.

If a path is not specified, FIND searches the text typed at the prompt

or piped from another command.

The sample below is an example (a very simple one) of finding a specific string in the call stack. I use it a lot to find specific values inside objects properties but i´m sure you will find other useful uses for this command.

0:000:x86> .shell -ci "~*kb" FIND /I "BaseCachedThreadroutine"

0c8ffdb0 75721c6b 00512fe0 0c8ffdc8 76c9e3f3 rpcrt4!BaseCachedThreadRoutine+0x9e

0effff60 75721c6b 00512fe0 0effff78 76c9e3f3 rpcrt4!BaseCachedThreadRoutine.shell: Process exited

The argument –ci specifies that the output of the command “~*kb” is to be used as input for FIND command. There are some more options that you can look at Debugging Tools For Windows help.

Till next time. Have Fun!!!

Bruno

ClassicASP: XMLHTTP and ServerXMLHTTP

2008-09-06T11:40:00Z

From time to time i get some memory dumps related to classic ASP. Last month I got some and the symptom was slow performance or event total hang with all requests just sitting around.

Typically when i´m looking at a classic ASP memory dump I like to use the extension that comes with DebugDiag called IISInfo.dll that helps me getting a nice view on requests executing. Below are the commands available with this extension (I won´t get into details but they are pretty much self explanatory).

IISInfo.dll - DebugDiag/WinDBG hybrid extension for IIS and ASP information.

-----------------------------------------------------------------------------------------------

Usage:

!clientconns - Active client connections

!asprequests - ASP request information about all executing ASP requests

!asprequest [<CHitObj>] - ASP request information executing on current thread or specified optional CHitObj address

!templates - All cached templates

!aspstack - Script call stack for the ASP page running on the current thread

!asppages - ASP page running on all threads

!asptemplate <CTemplate> - Detailed information and include heirarchy for the specified template

!templatecode <CTemplate> - Compiled template code or expanded code for the specified template

!includecode <FileMapKey> - Compiled template code or expanded code for the specified include file

!aspapps - Loaded ASP applications

!aspapp <CAppln> - Detailed information for the specified ASP application

!appvars <CAppln> - Variables stored in the specified ASP Application collection

!sessions <CAppln> - Active sessions in the specified ASP application

!session <CSession> - Detailed information about the specified ASP session

!sessvars <CSession> - Variables stored in the specified ASP Session collection

!help - Shows this help

In this specific case, after looking at requests executing all were getting struck on Xmlhttp send

Function Scope Line Of Code

----------------------------------------------------------------------------------------

GetRequest oXmlhttpServer.send [PATH @ 123]

Global Scope GetRequest(parameters)

And call stack showed something like this

NTDLL!NtWaitForSingleObject+0xb

KERNEL32!WaitForSingleObjectEx+0x71

KERNEL32!WaitForSingleObject+0xf

WININET!URL_CONTAINER::LockContainer+0x23

WININET!URL_CONTAINER::GetHeaderData+0x10

WININET!GetCurrentSettingsVersion+0x29

WININET!InternetSettingsChanged+0x10

WININET!InternetConnectA+0x93

URLMON!CINet::INetAsyncConnect+0x135

URLMON!CINet::INetAsyncOpen+0xde

URLMON!CINet::INetAsyncStart+0x15

URLMON!CINet::Start+0x1d9

URLMON!COInetProt::Start+0x62

URLMON!CTransaction::Start+0x3ac

msxml4!URLMONRequest::send+0xc9

msxml4!XMLHttp::send+0x3a

msxml4!XMLHttp::_invoke+0xf2

msxml4!_dispatchImpl::InvokeHelper+0x10b

msxml4!_dispatchImpl::Invoke+0x3d

msxml4!__dispatch::Invoke+0x2b

The source code was showing that a XMLHTTP object was being created and then a request was made. So what was the problem about creating a XMLHTTP object ? Well, sending XMLHTTP requests from server is not supported and can lead to poor performance or even worse. Our solution was to use the SERVERXMLHTTP.

Below are some links of interest around XMLHTTP vs SERVERXMLHTTP.

“Active Server Pages (ASP) script or Internet Server API (ISAPI) code that attempts to use XMLHttpRequest (Microsoft.XMLHTTP) functionality of the Microsoft XML engine (MSXML) to send XML requests to another Web server may function incorrectly or perform poorly.”

http://support.microsoft.com/kb/237906/en-us (PRB: Loading Remote XML or Sending XML HTTP Requests from Server Is Not Supported)

“Using ServerXMLHTTP or WinHTTP objects to make recursive Hypertext Transfer Protocol (HTTP) requests to the same Internet Information Server (IIS) server is not recommended. More specifically, the calling Active Server Page (ASP) should not send requests to an ASP in the same virtual directory or to another virtual directory in the same pool or process. This can result in poor performance due to thread starvation.”

http://support.microsoft.com/default.aspx?scid=kb;EN-US;316451 (INFO: Do Not Send ServerXMLHTTP or WinHTTP Requests to the Same Server)

XMLHTTP and ServerXMLHTTP?

As the name suggests, ServerXMLHTTP is recommended for server applications and XMLHTTP is recommended for client applications. XMLHTTP has some advantages such as caching and auto-discovery of proxy settings support. It can be used on Windows 95 and Windows 98 platforms, and it is well-suited for single-user desktop applications.”

http://support.microsoft.com/kb/290761 (Frequently asked questions about ServerXMLHTTP)

Have fun!!!

Bruno

Windbg: Get image loaded in memory

2008-09-06T10:54:00Z

I´m starting a new section called Tips with the goal of showing commands that are very useful when doing memory dump analysis. The first one is on how to get a image loaded into memory to file.

When i´m looking at a managed memory dump almost always there is the need to look inside the code so that I can get a better view on how the application i´m troubleshooting works. Since usually I don´t have access to source code the path to follow is to get the image loaded in memory and save it to disk. Then I use (and I bet that you also know this tool) Lutz fantastic tool .NET Reflector (now acquired by RedGate http://www.red-gate.com/products/reflector/ ).

If you are using sos/psscor extension then there is available the command !SaveModule that allows you to get this file. The syntax is pretty simple help provided by executing !Help !SaveModule is pretty self explanatory.

!SaveModule <Base address> <Filename>

This command allows you to take a image loaded in memory and write it to a file. This is especially useful if you are debugging a full memory dump, and don't have the original DLLs or EXEs. This is most often used to save a managed binary to a file, so you can disassemble the code and browse types with ILDASM.

The base address of an image can be found with the "LM" debugger command:

0:000> lm

start end module name

00400000 00408000 image00400000 (deferred)

10200000 102ac000 MSVCR80D (deferred)

5a000000 5a0b1000 mscoree (deferred)

5a140000 5a29e000 mscorjit (deferred)

5b660000 5c440000 mscorlib_dll (deferred)

5d1d0000 5e13c000 mscorwks (deferred)

...

If I wanted to save a copy of mscorwks.dll, I could run:

0:000> !SaveModule 5d1d0000 c:\pub\out.tmp

4 sections in file

section 0 - VA=1000, VASize=e82da9, FileAddr=400, FileSize=e82e00

section 1 - VA=e84000, VASize=24d24, FileAddr=e83200, FileSize=ec00

section 2 - VA=ea9000, VASize=5a8, FileAddr=e91e00, FileSize=600

section 3 - VA=eaa000, VASize=c183c, FileAddr=e92400, FileSize=c1a00

The diagnostic output indicates that the operation was successful. If

c:\pub\out.tmp already exists, it will be overwritten.

!SaveModule <Base address> <Filename>

Now if by any chance you can´t use sos/psscor or you want to get a image from a unmanaged memory dump you can use .WriteMem command to get the job done. All you have to do is get the base address and size of the asembly

The .writemem command writes a section of memory to a file. If you execute .hh .writemem on windbg you will get

Syntax:

.writemem FileName Range

Parameters:

FileName

Specifies the name of the file to be created. You can specify a full path and file name, or just the file name. If the file name contains spaces, FileName should be enclosed in quotation marks. If no path is specified, the current directory is used.

Range

Specifies the memory range to be written to the file

Bruno

SQL Server High CPU caused by trace filter

2008-09-06T10:41:00Z

Consider the scenario where you take several memory dumps from SQL Server because you are getting very high CPU without apparent reason. After looking at this memory dumps you notice that almost every thread has a call stack similar to the one below then you might be facing the issue described at http://support.microsoft.com/kb/953496 (FIX: CPU utilization is high when you run a trace that contains a text filter in SQL Server 2005).

374 Id: 11e0.1a10 Suspend: 0 Teb: 7fe37000 Unfrozen

ChildEBP RetAddr Args to Child

79bcef70 7c827bab 77ea4914 00007318 000093b4 ntdll!KiFastSystemCallRet

79bcef74 77ea4914 00007318 000093b4 00000000 ntdll!ZwSignalAndWaitForSingleObject+0xc

79bcefe8 010021a1 00007318 000093b4 ffffffff kernel32!SignalObjectAndWait+0xaf

79bcf00c 01002931 4805a040 4805a040 7293a0e8 sqlservr!SOS_Scheduler::Switch+0x81

79bcf180 010029ad 42bbc0e8 7293a0e8 0b9720a4 sqlservr!SOS_Scheduler::SwitchContext+0x2f3

79bcf198 01001c57 4805a040 00000102 7293a0e8 sqlservr!SOS_Scheduler::SuspendNonPreemptive+0xb9

79bcf1b4 01001edc 7293a0e8 ffffffff 0b972060 sqlservr!SOS_Scheduler::Suspend+0x2e

79bcf1d8 0134e44d 00001a10 ffffffff 0b9720a4 sqlservr!SOS_Event::Wait+0x12a

79bcf224 01276e37 7293a0e8 4805a040 0b9720a4 sqlservr!SOS_UnfairMutexPair::LongWait+0x110

79bcf248 010055d2 7293a0e8 0b9720a4 00000000 sqlservr!SOS_UnfairMutexPair::AcquirePair+0x41

79bcf268 0173a4c1 0b972040 0000263a 0128c3d4 sqlservr!CMemThread::Alloc+0x42

79bcf2ec 0173a889 0b989540 2c9d31f8 0000000a sqlservr!CTraceFilter::FComparePattern+0xea

79bcf318 0147baa2 0b989540 61b44040 2c9d3860 sqlservr!CTraceFilter::FCompare+0x3d

79bcf35c 0145d9d6 61b4b07c 79bcf7f8 79bc45b9 sqlservr!CFilterGroupCollection::FPassesFilters+0x1f7

79bcf780 0145e29d 0000000a 79bcf7f8 79bcf7a8 sqlservr!CTrace::WriteRecord+0x1cc

79bcf824 01742709 0000000a 2c9d2278 00000000 sqlservr!CTraceController::ProduceRecord+0x501

79bcf86c 01b869d6 79bcf8e8 79bc4af1 2c9d2278 sqlservr!PostResetConn+0xc0

79bcf8c8 01b86fc1 79bc4b61 2c9d2278 79bcf9a8 sqlservr!CAutoPostResetConn::~CAutoPostResetConn+0x7a

79bcf958 01b87479 2c9d2278 79bcfb40 79bc4821 sqlservr!FCleanSessionForReuseKeepCtxt+0xfc

79bcfa18 010da3e4 2c9d2ab8 2c9d2a18 0f287eb0 sqlservr!ResetConnAndRedoLogin+0x15e

I can´t post the details about the next step (finding the trace and filters) because I used private symbols but after you see this call stack I suggest that you try to find what traces are running in your environment at the time the issue occurred.

Have fun!!!

Bruno

SOS/PSSCOR & .effmach

2008-09-01T18:07:00Z

Recently in one of our internal forums there was this thread around debugging different architectures (IA64, AMD64,x86) and the right environment to analyze the memory dump.

If you have a IA64 memory dump then you will have to use a IA64 architecture to analyze the memory dump.

If you have a 64 bit process memory dump (don´t confuse 64 bit process with 64 bit environment) then you will have to use a 64 bit environment to analyze the memory dump.

If you have a 32 bit process memory dump taken on a 32 bit environment then you can use a 32 bit or 64 bit debugger to analyze the memory dump.

What i want to talk about is when you take a 32 bit process memory dump in a 64 bit environment. When you do this you can actually use a 32 or 64 bit environment to do the analysis but if you are going to use a 32 bit environment there are some steps you need to be aware of.

When you look (using a 32 bit debugger) at a memory dump of a 32 bit process taken in a 64 bit environment you might find that almost all your threads look like the one below

75 Id: 1028.848 Suspend: 0 Teb: 00000000`7ee74000 Unfrozen

RetAddr : Args to Child : Call Site

00000000`75863b06 : 00000023`77d1090d 00000000`00000023 00000000`00000202 00000000`1c23f7fc : wow64cpu!CpupSyscallStub+0x9

00000000`756eab46 : 00000000`77b50000 00000000`1715fd20 00000000`6262595c 00000000`1715f620 : wow64cpu!Thunk0ArgReloadState+0x1a

00000000`756ea14c : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : wow64!RunCpuSimulation+0xa

00000000`77bb73db : 00000000`00000000 00000000`00000000 00000000`7efdf000 00000000`00000000 : wow64!Wow64LdrpInitialize+0x4b4

00000000`77b785ce : 00000000`1715f620 00000000`00000000 00000000`7efdf000 00000000`00000000 : ntdll!_LdrpInitialize+0x3edeb

00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

What you are seeing is the subsystem the runs the 32 bit processes in 64 bit environment (more information on windows-on-windows at http://msdn.microsoft.com/en-us/library/aa384249(VS.85).aspx ). In order to see what you really want you can use .effmach to switch to a x86 mode. Below is an example of how to do this

0:000> .load wow64exts (loading the wow extension)

0:000> .effmach x86 (switch to x86 mode)

Effective machine: x86 compatible (x86)

After doing this if you issue a K command you will see the stack you want. You will also notice the x86 added to the prompt.

0:000:x86> k

ChildEBP RetAddr

0033f978 75f2dcea ntdll_77cf0000!ZwWaitForMultipleObjects+0x15

0033fa14 76dd8f76 kernel32!WaitForMultipleObjectsEx+0x11d

0033fa68 71588717 user32!RealMsgWaitForMultipleObjectsEx+0x14d

0033fa98 716b9071 MSO!MsgWaitForMultipleObjectsAlertable+0x3a

0033fabc 2fd5e803 MSO!MsoPWMsgWaitForMultipleObjectsEx+0x4f

This is very beautiful but (there is always one J) if you are troubleshooting a .NET issue and want to use SOS/PSSCOR extension you are going to have problems because SOS/PSSCOR doesn´t work with .effmach. (Sorry)

Have fun!!!

Bruno

Tools @ Performance Analysis of Logs

2008-08-01T20:01:00Z

I´ve decided to start a new section in my blog around tools I use that help me a lot in my daily work. But first I want to give a big thanks to everyone that has projects under CodePlex because there are a lot of excellent projects there and for the time you guys take from yourself in helping the community free of charge J.

Today I will talk about PAL (Performance Analysis of Logs). This wonderful tool can be downloaded at http://www.codeplex.com/PAL .

There are a lot of situations where we collect performance counters to help in troubleshooting issues and all of you that use them know the pain that is to look at all those counters collected overtime and then having to relate them to try to find the right path. PAL gives an excellent first view of this information in one very nice report. The tool is actually very simple to run (almost NEXT … NEXT … NEXT). The report that it generates is something like the picture below (in my sample I will show you one counter)

It is divided into three sections:

1. Brief description of the counter and thresholds

2. Graph showing the behavior

3. Details for each counter instance and it will show green, yellow and red colors in each line of this table according to the defined thresholds

Have fun.

Bruno

Switch to the right context (COM+ Fault)

2008-07-29T12:29:00Z

If you are dealing with a memory dump (unmanaged) a very fast way to get an initial look is to use DebugDiag Analysis. It will provide you a nice report and it might help you. Sometimes in DebugDiag reports you will see a rebuilt stack trace like the one below (COM+)

Thread 16 - System ID 2912

Entry point comsvcs!STAThread::STAThreadWorker

Create time 7/14/2008 5:22:49 AM

Time spent in user mode 0 Days 00:02:04.015

Time spent in kernel mode 0 Days 00:00:40.640

This thread is blocked by an unhandled exception which caused a COM+ FailFast to occur.

Function Source

NTDLL!NtWaitForSingleObject+b

NTDLL!RtlpWaitForCriticalSection+9e

NTDLL!RtlEnterCriticalSection+46

ADVAPI32!MapPredefinedHandle+39

ADVAPI32!RegOpenKeyExW+c5

ADVAPI32!RegOpenKeyW+73

comsvcs!FF_DumpProcess+3e

comsvcs!FailFast+40

comsvcs!ComSvcsExceptionFilter+9c

(…)

Recovered stack for thread 16

Function Arg 1 Arg 2 Arg 3 Source

NTDLL!RtlRaiseStatus+24 c0000024 00000360 77f87fdd

NTDLL!RtlpUnWaitCriticalSection+25 7c32d5a0 7c2f488e 7c32d5a0

NTDLL!RtlLeaveCriticalSection+1d 7c32d5a0 01fee8b4 80000000

ADVAPI32!MapPredefinedHandle+96 80000000 01fee888 091428e8

ADVAPI32!RegOpenKeyExW+c5 80000000 01fee8b4 00000000

oledb32!CError::CacheErrLookUpObject+99 01feec54 024d3098 027c2dd8

oledb32!CImpIErrorRecords::AddErrorRecord+cb 045da5b8 024d3070 10000000

sqloledb!CError::PostHResult+65 80040e21 027c2dc8 01feedd0

And you will see a recovered stack for this thread with a lot more useful information. Today I will show you how you can get this information using Windbg so that you could dig deep in this memory dump.

The first parameter in ComSvcsExceptionFilter it´s a pointer to EXCEPTION_POINTERS struct.

Below in yellow is the address of this first parameter

0:016> kb

ChildEBP RetAddr Args to Child

01fee0f8 77f8f295 000018f8 00000000 00000000 NTDLL!NtWaitForSingleObject+0xb

01fee16c 77f87f26 7c32d500 7c2f4854 7c32d5a0 NTDLL!RtlpWaitForCriticalSection+0x9e

01fee174 7c2f4854 7c32d5a0 78822ba0 80000002 NTDLL!RtlEnterCriticalSection+0x46

01fee188 7c2f4a4d 80000002 01fee1ac 01fee3fe ADVAPI32!MapPredefinedHandle+0x39

01fee1b0 7c2f4c36 80000002 78822ba0 00000000 ADVAPI32!RegOpenKeyExW+0xc5

01fee1c8 78822cc0 80000002 78822ba0 01fee1e4 ADVAPI32!RegOpenKeyW+0x73

01fee3fc 788231c6 7876041c 01feee40 00000000 comsvcs!FF_DumpProcess+0x3e

01fee400 7876041c 01feee40 00000000 78745548 comsvcs!FailFast+0x40

01fee410 7878f576 01fee440 7878f4c0 000ee008 comsvcs!ComSvcsExceptionFilter+0x9c

If you dump this address, the first two elements are pointers to EXCEPTION_RECORD and to CONTEXT

0:016> dc 01fee440

01fee440 01fee50c 01fee520

You can then use .exr and .cxr to dump respectively the exception and the context.

0:016> .cxr 01fee520

eax=01fee7ec ebx=00000000 ecx=01010101 edx=ffffffff esi=7c32d5a0 edi=00000000

eip=77fac57c esp=01fee7ec ebp=01fee83c iopl=0 nv up ei pl zr na pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246

NTDLL!RtlRaiseStatus+0x24:

77fac57c c9 leave

After switching to this context if you look at the stack you will see a more useful one and equal to the one that appears in DebugDiag. Then just keep digging until you find the root cause of your problem.

0:016> kb

*** Stack trace for last set context - .thread/.cxr resets it

ChildEBP RetAddr Args to Child

01fee83c 77f8f3a5 c0000024 00000360 77f87fdd NTDLL!RtlRaiseStatus+0x24

01fee848 77f87fdd 7c32d5a0 7c2f488e 7c32d5a0 NTDLL!RtlpUnWaitCriticalSection+0x25

01fee850 7c2f488e 7c32d5a0 01fee8b4 80000000 NTDLL!RtlLeaveCriticalSection+0x1d

01fee864 7c2f4a4d 80000000 01fee888 091428e8 ADVAPI32!MapPredefinedHandle+0x96

01fee88c 027544c0 80000000 01fee8b4 00000000 ADVAPI32!RegOpenKeyExW+0xc5

01feec3c 027545ec 01feec54 024d3098 027c2dd8 oledb32!CError::CacheErrLookUpObject+0x99

01feecd0 027c4a3b 045da5b8 024d3070 10000000 oledb32!CImpIErrorRecords::AddErrorRecord+0xcb

01feed14 027c4b44 80040e21 027c2dc8 01feedd0 sqloledb!CError::PostHResult+0x65

01feed64 02751c0f 026ef374 00000001 01feedb8 sqloledb!CImpIDBProperties::SetProperties+0x3e9

Bruno

OutOfMemory & CompiledAssembly

2008-07-18T14:10:00Z

Problems seem to occur in batches. Last week the hot topic was OutOfMemory exceptions.

I got a memory dump that a customer took around the time they were getting OutOfMemory exceptions. I asked them to collect some performance counters so that we could figure out what this leak was all about.

The performance counters we got were

.NET CLR Loading\Current Assemblies

Process\Private Bytes

Process\Virtual Bytes

.NET CLR Memory\# Bytes in all heaps

.NET CLR Memory\# Total committed bytes

.NET CLR Memory\# Total reserved bytes

The picture below shows the output.

The line that keeps increasing matches “Current Assemblies”. Now this is strange since usually after a while this should stop increasing. So it appears we are having assembly leaking.

Let’s try to find what these assemblies are.

0:000> !DumpDynamicAssemblies

Domain: xxxxxxx

-------------------

Assembly: 0x15ba5ab0 [-fxq0b2q] Dynamic Module: 0x15bbc7c0 loaded at: 0x170a1000 Size: 0xf000((null))

Assembly: 0x15bb52f0 [m_o9jngq] Dynamic Module: 0x224830 loaded at: 0x17421000 Size: 0x26000((null))

Assembly: 0x15bb5868 [dgttiya4] Dynamic Module: 0x224ed0 loaded at: 0x17451000 Size: 0x1200((null))

Assembly: 0x1748ee30 [tfgs3roi] Dynamic Module: 0x1748b7a0 loaded at: 0x17e11000 Size: 0x1b000((null))

Assembly: 0x15bc54f0 [xw0tr4my] Dynamic Module: 0x15bc5928 loaded at: 0x1e7b1000 Size: 0x1fd000((null))

At the time this memory dump was taken we had almost 2000 dynamic assemblies in memory. Digging into some of these assemblies ...

0:000> !savemodule 0x1748b7a0 c:\leak.dll

Successfully saved file: c:\leak.dll

And opening them with Lutz Reflector we find a pattern

public class _DinAssembly

{

public bool _method1()

{

return true;

}

After talking with dev teams and doing some search in source code we found that this assembly is being generated in source code.

(…)

results.CompiledAssembly.CreateInstance

(…)

If you look at

http://msdn.microsoft.com/en-us/library/system.codedom.compiler.compilerresults.compiledassembly.aspx

you can see

Note:

(…) After calling the get accessor, the compiled assembly cannot be deleted until the current AppDomain is unloaded.

So some possible solutions are:

· Try some caching mechanism (if it´s possible)

· Load this assemblies into a separated domain so that you could unload it

· Do you really need to generate this assembly?

Bruno

BLOBCache, Web Garden and DisallowOverlappingRotation. Are they best friends ?

2008-07-02T11:22:00Z

In my opinion BLOBCache is a must have in most MOSS environments. It is known that BLOBCache and Web Garden are not the best of friends, and according to Optimizing Office SharePoint Server for WAN environments don´t use Web Garden if you have BLOBCache in your environment. The main reason for this is that only one process can acquire the lock to this cache and if the process serving the request is not the one who acquired the lock then you can imagine you won´t get the result you were expecting.

So far so good, but there is another setting that can also get you into trouble even if you don´t have a Web Garden scenario.

IIS 6.0 has a metabase property called DisallowOverlappingRotation and according to documentation (DisallowOverlappingRotation Metabase Property)

“The DisallowOverlappingRotation property specifies whether or not the World Wide Web Publishing Service (WWW Service) should start up another worker process to replace the existing worker process while it is shutting down."

The default value for this property is FALSE (see table below) which means that WWW Service is allowed to start a new worker process when the old one is shutting down.


Attribute Name	Attribute Value
XML Data Type	Boolean
WMI Data Type	Boolean
ADSI Data Type	Boolean
ABO Data Type	Boolean
ABO Metabase Identifier	MD_APPPOOL_DISALLOW_OVERLAPPING_ROTATION
Attributes	INHERIT
Default Value	false
MetaFlagsEx	CACHE_PROPERTY_MODIFIED
User Type	IIS_MD_UT_SERVER
ID	9015

Now you can imagine there is a frame in time where you have multi-instance because both processes still exist (this is like Web Garden but just for a short period of time). When this new process starts it will try to acquire the lock to the cache but what happens if the process that is shutting down hasn´t yet released this lock? The new process won´t be able to acquire and it won´t be able to use this cache to improve the performance of your website. You will notice a decrease in performance until you do an IISReset.

Also in documentation about this property we can read the following

“The value of this property should be set to true if the worker process loads any application code that does not support multi-instance.”

My advice is that if your environment makes use of BLOBCache also set IIS 6.0 property DisallowOverlappingRotation to TRUE.

Bruno