Thou shalt not PANIC

Under the hood with live debugging

The team i belong to (Portugal Adcs) has a blog called Deviations (http://blogs.msdn.com/deviations) and from time to time i will post some stuff over there (when that happens i will put a link here).

My latest post there was about live debugging using Windbg and you can check it out at

http://blogs.msdn.com/deviations/archive/2009/03/18/under-the-hood-with-live-debugging.aspx

Have fun
Bruno

WCF: Getting MessageSecurityException while calling webservice

Time flies and it looked like it was yesterday that i wrote my last post. Last couple of months have been very busy, so blogging was put on hold but everytime i got a case that i believe i should talk about it, i kept it. So expect a burst on posts.

Yesterday, I and a co-worker were troubleshooting an strange issue that didn´t occur in dev but in staging (if i get a cent every time i ear this) a MessageSecurityException was being thrown stating unauthorized access. If we allowed Anonymous Access then no exception was being thrown. Below is the scenario.

 Scenario

• Single Machine environment
• Windows 2003 and IIS6
• Both WebApp and WebService on same pool and running under an account with privileges
• WebApp calling locally WebService
• Integrated Authentication (Anonymous not allowed)

Details

So, we decided to take a memory dump on System.ServiceModel.Security.MessageSecurityException using DebugDiag.(http://www.microsoft.com/downloadS/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en)

Lets look at the faulting thread

Now, lets look at exception details

Also in IIS Logs we could see 401.1 unauthorized.

In our case the solution was to install security update 957097. More details, info on workaround or download update at

http://support.microsoft.com/default.aspx?scid=kb;EN-US;896861 (You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or IIS 6)

After doing this we decided to look at dev environment, and surprise, this update was installed.

I know you all have eared this before, so here it goes again: it is very important to keep environments (dev, stage, production) the most similar to the extent possible.

Have fun.

Bruno

Process Explorer to the rescue

Monday was a day to forget. I arrived at the office around 8:30 am, started my machine (like always), inserted my credentials, the desktop shows up and suddenly it gets unresponsive. First thing I do is CTRL+ALT+DEL, start Task Manager and it also gets hanged. The only commands that were responsive was log off and switch user (very useful right? J ).

My first mistake was not to use the same approach I always use in customers (do what I say not what I do J ) and started trying the try…error approach. The only thing this did was to waste my time with successive reboots, remove profile, add profile and other stuff. In my machine I have two profiles: the local admin one and another that I use to connect to Microsoft. The problems were in this second profile. The first one worked without any problems.

So I decided to stop for a minute and follow the typical methodology I always follow. So since my suspicions were on a driver or some program I had installed over the weekend (beta versions, don´t get me wrong I love beta versions because I believe they are a great way to help products getting better and better).

Since Switch User was working, the first step was to reproduce the problem. So I logged in with the profile that was having problems and then I switched to my local admin account. Then I started Process Explorer (aka ProcExp, you can download it from http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx ). If you don´t know this tool, you can think of it as task manager on steroids and is a very valuable troubleshooting tool.

After opening ProcExp you will see a list of all processes running on your system and also the account that started the process (this allowed me to see processes that were running in my problematic profile).

To see the username, you have to right click on one of the columns and “Select Columns” > “Username”. Then if you click order by Username, you will notice that each group will have a different color. You can see this in the image below (i´ve removed the usernames from my machine, that´s why you see a blank column under Username)

Process Explorer allows you to see threads and call stacks for each process, so my approach was to look at processes that were executing under DOMAIN\USER and dig into each one of them trying to find strange behaviors. You might ask, what is a strange behavior? I would say, it depends (wow, I’ve never heard that answer before). In my case I decided to start looking at processes that were consuming more CPU and those with names that I was not familiar with.

So how do you look at threads and call stacks? Right click on a process and select Properties. Inside Properties you have a tab called Threads. Click on it.

Here you can try to identify some thread that does not look good, or it might be that they are all hang (these are just some examples, and the more experience you have the easier it would be for you) . Let´s imagine I had one thread that was consuming almost all CPU and then i could look at the stack (select your thread and click on STACK)

So I did this for a couple of processes until I identified the bad boy. Then I disabled it, switched user and “voilá” all my problems were gone (and part of my working day).

Paulo, after all it was not the beta version you gave me that got me into troubles J

Happy debugging!!!

Bruno

Tools @ Opening huge log files

Part of my work is looking at huge log files and this can sometimes be a big pain to open in any text editor. I was talking with a coworker about this, and he showed EditPad Lite, a free text editor (for non-commercial use, you also have the non-free version) and I was very impressed how fast it was to open huge files and scrolling up and down without the slightest feeling of hang.

So, I suggest you give a look at it. You can get it at http://www.editpadpro.com/editpadlite.html

Bruno

Debug Jscript with Internet Explorer 8

My team has started a blog. Since we all have different skills, it will be more generalist and there you can get information on MOSS, Biztalk, Troubleshooting, Dev, Silverlight and other stuff.

Every time I post there I will put here a link to the post (I will talk mostly about troubleshooting).

My first post there is about Developer Tools that come with Internet Explorer 8 and how they can help us troubleshooting Jscript errors (I know, they are a big pain for us all developers J ).

The post can be seen at http://blogs.msdn.com/deviations/archive/2008/10/09/debug-jscript-with-internet-explorer-8.aspx.

About the team blog, you can check it at http://blogs.msdn.com/deviations.

Bruno

Production Troubleshooting Flowchart

Usually (probably always I hope) when troubleshooting production environments we don´t have Visual Studio or other “nice looking” tools to help us finding problems. But we have one of my favorite tools: Windbg.

The first time someone showed me Windbg and all the stuff around production debugging like memory dump, … I was a little (maybe not just a little) scared on this new world that had just opened in front of me. So my next question was: where do I start from? And probably like almost everyone that starts exploring this “new world” there are excellent blogs like http://blogs.msdn.com/tess (If Broken it is, fix it you should), http://blogs.msdn.com/dougste (Notes from a dark corner), http://blogs.msdn.com/tom (ASP.NET Debugging) and many others.

One of my biggest problems at first was to understand (and remember) what and when I should collect information that could help me to troubleshoot the issue. The blogs above have a lot of information about taking the first steps. To simplify this task I´ve created a little graphical memo that would help me. Now don´t take this flowchart as the “holy grail” but I believe it can help (at least until you get the basics in your mind).

Right click on the image to save it and it will be more “readable”.

Have fun

Bruno

Windbg: Using .shell to search text

To me one of the most useful commands when using windbg is .shell. According to Debugging Tools For Windows documentation

“The .shell command launches a shell process and redirects its output to the debugger, or to a specified file.”

So, why would I find that interesting and useful in my day to day work to launch a shell process ? Maybe to impress friends (that know nothing about debugging) with strange commands or make it sound very complicated. Well, actually I use .shell in one of the most simple tasks we all do everyday and that´s finding text. Now you might be thinking, so why don´t you use Ctrl+F and find what you want ? (We will talk about this in a couple of minutes)

So, has I was saying, I use it a lot to find text inside memory dumps and this saves me time. Since .shell launches a shell process, the key here is to use the old FIND command from DOS to help us. FIND allows to search for text inside a file. If you open a command line and do FIND /? You will see something like below.

The sample below is an example (a very simple one) of finding a specific string in the call stack. I use it a lot to find specific values inside objects properties but i´m sure you will find other useful uses for this command.

The argument –ci specifies that the output of the command “~*kb” is to be used as input for FIND command. There are some more options that you can look at Debugging Tools For Windows help.

Till next time. Have Fun!!!

Bruno

ClassicASP: XMLHTTP and ServerXMLHTTP

From time to time i get some memory dumps related to classic ASP. Last month I got some and the symptom was slow performance or event total hang with all requests just sitting around.

Typically when i´m looking at a classic ASP memory dump I like to use the extension that comes with DebugDiag called IISInfo.dll that helps me getting a nice view on requests executing. Below are the commands available with this extension (I won´t get into details but they are pretty much self explanatory).

In this specific case, after looking at requests executing all were getting struck on Xmlhttp send

The source code was showing that a XMLHTTP object was being created and then a request was made. So what was the problem about creating a XMLHTTP object ? Well, sending XMLHTTP requests from server is not supported and can lead to poor performance or even worse.  Our solution was to use the SERVERXMLHTTP.

Below are some links of interest around XMLHTTP vs SERVERXMLHTTP.

“Active Server Pages (ASP) script or Internet Server API (ISAPI) code that attempts to use XMLHttpRequest (Microsoft.XMLHTTP) functionality of the Microsoft XML engine (MSXML) to send XML requests to another Web server may function incorrectly or perform poorly.”

http://support.microsoft.com/kb/237906/en-us  (PRB: Loading Remote XML or Sending XML HTTP Requests from Server Is Not Supported)

 “Using ServerXMLHTTP or WinHTTP objects to make recursive Hypertext Transfer Protocol (HTTP) requests to the same Internet Information Server (IIS) server is not recommended. More specifically, the calling Active Server Page (ASP) should not send requests to an ASP in the same virtual directory or to another virtual directory in the same pool or process. This can result in poor performance due to thread starvation.”

http://support.microsoft.com/default.aspx?scid=kb;EN-US;316451 (INFO: Do Not Send ServerXMLHTTP or WinHTTP Requests to the Same Server)

XMLHTTP and ServerXMLHTTP?

As the name suggests, ServerXMLHTTP is recommended for server applications and XMLHTTP is recommended for client applications. XMLHTTP has some advantages such as caching and auto-discovery of proxy settings support. It can be used on Windows 95 and Windows 98 platforms, and it is well-suited for single-user desktop applications.”

http://support.microsoft.com/kb/290761 (Frequently asked questions about ServerXMLHTTP)

Have fun!!!

Bruno

Windbg: Get image loaded in memory

I´m starting a new section called Tips with the goal of showing commands that are very useful when doing memory dump analysis. The first one is on how to get a image loaded into memory to file.

When i´m looking at a managed memory dump almost always there is the need to look inside the code so that I can get a better view on how the application i´m troubleshooting works. Since usually I don´t have access to source code the path to follow is to get the image loaded in memory and save it to disk. Then I use (and I bet that you also know this tool) Lutz fantastic tool .NET Reflector   (now acquired by RedGate http://www.red-gate.com/products/reflector/ ).

If you are using sos/psscor extension then there is available the command !SaveModule that allows you to get this file. The syntax is pretty simple help provided by executing !Help !SaveModule is pretty self explanatory.

Now if by any chance you can´t use sos/psscor or you want to get a image from a unmanaged memory dump you can use .WriteMem command to get the job done. All you have to do is get the base address and size of the asembly

The .writemem command writes a section of memory to a file. If you execute .hh .writemem on windbg you will get

Bruno

SQL Server High CPU caused by trace filter

Consider the scenario where you take several memory dumps from SQL Server because you are getting very high CPU without apparent reason. After looking at this memory dumps you notice that almost every thread has a call stack similar to the one below then you might be facing the issue described at http://support.microsoft.com/kb/953496  (FIX: CPU utilization is high when you run a trace that contains a text filter in SQL Server 2005).

I can´t post the details about the next step (finding the trace and filters) because I used private symbols but after you see this call stack I suggest that you try to find what traces are running in your environment at the time the issue occurred.

Have fun!!!

Bruno

SOS/PSSCOR & .effmach

Recently in one of our internal forums there was this thread around debugging different architectures (IA64, AMD64,x86) and the right environment to analyze the memory dump.  

If you have a IA64 memory dump then you will have to use a IA64 architecture to analyze the memory dump.

If you have a 64 bit process memory dump (don´t confuse 64 bit process with 64 bit environment) then you will have to use a 64 bit environment to analyze the memory dump.

If you have a 32 bit process memory dump taken on a 32 bit environment then you can use a 32 bit or 64 bit debugger to analyze the memory dump.

What i want to talk about is when you take a 32 bit process memory dump in a 64 bit environment. When you do this you can actually use a 32 or 64 bit environment to do the analysis but if you are going to use a 32 bit environment there are some steps you need to be aware of.

When you look (using a 32 bit debugger) at  a memory dump of a 32 bit process taken in a 64 bit environment you might find that almost all your threads look like the one below

What you are seeing is the subsystem the runs the 32 bit processes in 64 bit environment (more information on windows-on-windows at http://msdn.microsoft.com/en-us/library/aa384249(VS.85).aspx ). In order to see what you really want you can use .effmach to switch to a x86 mode. Below is an example of how to do this

After doing this if you issue a K command you will see the stack you want. You will also notice the x86 added to the prompt.

This is very beautiful but (there is always one J) if you are troubleshooting a .NET issue and want to use SOS/PSSCOR extension you are going to have problems because SOS/PSSCOR doesn´t work with .effmach. (Sorry)

Have fun!!!

Bruno

Tools @ Performance Analysis of Logs

I´ve decided to start a new section in my blog around tools I use that help me a lot in my daily work. But first I want to give a big thanks to everyone that has projects under CodePlex because there are a lot of excellent projects there and for the time you guys take from yourself in helping the community free of charge J.

Today I will talk about PAL (Performance Analysis of Logs). This wonderful tool can be downloaded at http://www.codeplex.com/PAL .

There are a lot of situations where we collect performance counters to help in troubleshooting issues and all of you that use them know the pain that is to look at all those counters collected overtime and then having to relate them to try to find the right path. PAL gives an excellent first view of this information in one very nice report. The tool is actually very simple to run (almost NEXT … NEXT … NEXT).  The report that it generates is something like the picture below (in my sample I will show you one counter)

It is divided into three sections:

1.    Brief description of the counter and thresholds

2.    Graph showing the behavior

3.    Details for each counter instance and it will show green, yellow and red colors in each line of this table according to the defined thresholds

Have fun.

Bruno

Switch to the right context (COM+ Fault)

If you are dealing with a memory dump (unmanaged) a very fast way to get an initial look is to use DebugDiag Analysis. It will provide you a nice report and it might help you. Sometimes in DebugDiag reports you will see a rebuilt stack trace like the one below (COM+)

And you will see a recovered stack for this thread with a lot more useful information. Today I will show you how you can get this information using Windbg so that you could dig deep in this memory dump.

The first parameter in ComSvcsExceptionFilter it´s a pointer to EXCEPTION_POINTERS struct.

Below in yellow is the address of this first parameter

If you dump this address, the first two elements are pointers to EXCEPTION_RECORD and to CONTEXT

You can then use .exr and .cxr to dump respectively the exception and the context.

After switching to this context if you look at the stack you will see a more useful one and equal to the one that appears in DebugDiag. Then just keep digging until you find the root cause of your problem.

Bruno

OutOfMemory & CompiledAssembly

Problems seem to occur in batches. Last week the hot topic was OutOfMemory exceptions.

I got a memory dump that a customer took around the time they were getting OutOfMemory exceptions. I asked them to collect some performance counters so that we could figure out what this leak was all about.

The performance counters we got were

The picture below shows the output.

The line that keeps increasing matches “Current Assemblies”. Now this is strange since usually after a while this should stop increasing. So it appears we are having assembly leaking.

Let’s try to find what these assemblies are.

At the time this memory dump was taken we had almost 2000 dynamic assemblies in memory. Digging into some of these assemblies ...

And opening them with Lutz Reflector we find a pattern

After talking with dev teams and doing some search in source code we found that this assembly is being generated in source code.

If you look at

http://msdn.microsoft.com/en-us/library/system.codedom.compiler.compilerresults.compiledassembly.aspx

you can see

So some possible solutions are:

·         Try some caching mechanism (if it´s possible)

·         Load this assemblies into a separated domain so that you could unload it

·         Do you really need to generate this assembly?

Bruno

BLOBCache, Web Garden and DisallowOverlappingRotation. Are they best friends ?

In my opinion BLOBCache is a must have in most MOSS environments. It is known that BLOBCache and Web Garden are not the best of friends, and according to Optimizing Office SharePoint Server for WAN environments don´t use Web Garden if you have BLOBCache in your environment. The main reason for this is that only one process can acquire the lock to this cache and if the process serving the request is not the one who acquired the lock then you can imagine you won´t get the result you were expecting.

So far so good, but there is another setting that can also get you into trouble even if you don´t have a Web Garden scenario.

IIS 6.0 has a metabase property called DisallowOverlappingRotation and according to documentation (DisallowOverlappingRotation Metabase Property)

“The DisallowOverlappingRotation property specifies whether or not the World Wide Web Publishing Service (WWW Service) should start up another worker process to replace the existing worker process while it is shutting down."

The default value for this property is FALSE (see table below) which means that WWW Service is allowed to start a new worker process when the old one is shutting down.

Now you can imagine there is a frame in time where you have multi-instance because both processes still exist (this is like Web Garden but just for a short period of time). When this new process starts it will try to acquire the lock to the cache but what happens if the process that is shutting down hasn´t yet released this lock? The new process won´t be able to acquire and it won´t be able to use this cache to improve the performance of your website. You will notice a decrease in performance until you do an IISReset.

Also in documentation about this property we can read the following

“The value of this property should be set to true if the worker process loads any application code that does not support multi-instance.”

My advice is that if your environment makes use of BLOBCache also set IIS 6.0 property DisallowOverlappingRotation to TRUE.

Bruno