Debugging Toolbox

[Recommended Books] Advanced .NET Debugging book – Preview

2009-05-16T03:22:00Z

For the second time I have had the privilege of being one of the reviewers for another debugging book. This time I’m referring to Mario Hewardt’s new book: Advanced .NET Debugging.

It was a great learning process. After months of reviewing chapters and source code and giving suggestions, I feel my .NET debugging knowledge is not the same. It is much better now!

Let me tell you why I’m so thrilled with this book.

First of all, this book discusses in depth several tools used by our product teams when isolating internal bugs from .NET application and also the tools used by CSS (Customer Service and Support). In case you ask, they are all public tools!

Also the book gives a deep and clear explanation about the CLR (Common Language Runtime). In my opinion, that is one of the most important parts of the book. If you are starting with .NET Debugging, you may be familiar with some of the SOS commands. That’s the easy part. However, if you want to improve your skills as a debugger you must go deeper and understand the CLR internals, like method table, method descriptor, memory management, etc. The chapter about CLR explains the internals in a superb way, and it’s my favorite!

If you’re new to debugging, you’re going to like the chapter about Basic Debugging Tasks. It’s didactic enough for beginners, demonstrating how to use the most used commands and how to get useful information from basic commands.

Another part of the book talks about more complex scenarios and technologies. It’s a deep explanation! You’re going to learn a lot about Managed Heap, Garbage Collection, Thread Synchronization and Interoperability. Based on my experiences helping customers from different countries, I can tell you the number one application problem from .NET applications: memory issues. I know it’s ironic since GC is supposed to manage memory, so let me explain: GC does the job very well. The problem is mixed code: managed and native. The GC only has control of the managed code. The book explains it very well, and there are lots of sample codes reproducing different issues scattered among the chapters. This is cool because you can use the sample code to apply the knowledge while debugging specific scenarios.

Like Mario’s previous book, this one is also very pragmatic. You won’t see those situations that are unlikely to happen on a production server, actually the opposite: you’ll see issues that we face on a daily basis with the approaches, techniques and tools we use when debugging these issues. Very helpful!

"Fortunately, Mario's book combines just the right amount of explanation of the runtime to help you understand the thought process and the techniques he uses to solve problems with many practical and clever tricks learned from debugging real world applications. So if you want to quickly get up to speed debugging your CLR applications, this is definitely the book to read. It covers many aspects of debugging managed applications—in particular, it gives insights into areas that are hard to diagnose such as thread synchronization problems. The illustration of debugging techniques is largely done with examples which makes it easier to follow." (Excerpted from the Foreword by Patrick Dussud, Technical Fellow, CLR, .NET FX, Microsoft Corporation.)

Here is the index of the book:

Part I – Overview

Part I introduces the reader to the basic concepts required to efficiently debug .NET applications using the Debugging Tools for Windows package.

1. Introduction to the Tools

This chapter contains a brief introduction to the tools used throughout the book, including basic usage scenarios, download locations, and installation instructions. Amongst the tools covered are:

Debugging Tools for Windows

.NET 2.0 redist/SDK

SOS

SOSEX

CLR Profiler

Performance Counters

Reflector for .NET

PowerDBG

Managed Debugging Assistants

2. CLR Fundamentals

This chapter discusses the core fundamentals of the CLR. The chapter begins with a high level overview of the major runtime components and subsequently drills down into the details and covers topics such as:

· CLR and the Windows Loader. Discusses how .NET applications are natively supported using the PE file format.

· Application domains (system, shared, default, custom) as well as loader heaps etc.

· Assembly overview

· Assembly manifest

· Type metadata

· And more…

3. Basic Debugging Tasks

Chapter 3 gives hands on examples of the basic .NET debugging tasks such as:

· Thread management

· Display stack traces

· Display local variables

· Breakpoint management

· Dumping heap based objects

· .NET exception model and how to analyze in the debuggers

· Basics of postmortem debugging

· Remote debugging

Part II – Applied Debugging

Par t II is a practical approach to debugging common .NET problems encountered in the development process. Each chapter is dedicated to a specific CLR component and/or technology and begins by detailing the inner workings of that component. Subsequently, a number of common problems related to the specific technologies are discussed and how the debuggers and tools can be used for root cause analysis.

4. Assembly Loader

The complexity of .NET applications can range from simple command line applications to complex multi-process/multi machine server applications with a large number of assemblies living in harmony. In order to efficiently debug problems in .NET applications one much be careful to understand the dependencies of .NET assemblies. This chapter takes a look at how the CLR assembly loader does its work and common problems surrounding that area.

5. Managed Heap and Garbage Collection

While .NET developers can enjoy the luxury of automatic memory management, care must still be taken to avoid costly mistakes. The highly sophisticated CLR garbage collector is an automatic memory manager that allows developers to focus less on memory management and more on application logic. Even though the CLR manages memory for the developer, care must be taken to avoid pitfalls that can wreak havoc in your applications. In this chapter we look at how the garbage collector works, how to peek into the internals of the garbage collector and some common programming mistakes related to automatic garbage collection (non memory resources, memory fragmentation, finalizer problems etc).

6. Synchronization

A multithreaded environment allows for a great deal of flexibility and efficiency. With this flexibility comes a lot of complexity in the form of thread management. To avoid costly mistakes in your application, care must be taken to ensure that threads perform their work in an orchestrated fashion. This chapter introduces the synchronization primitives available in .NET and discusses how the debuggers and tools can be used to debug common thread synchronization problems. Scenarios such as deadlocks, lock contentions, sync blocks, thin locks and thread pool problems are discussed.

7. Interoperability

.NET relies heavily on underlying Windows components. In order to invoke the native Windows components the CLR exposes two primary methods of interoperability:

1. Platform Invocation

2. COM Interoperability

Since the.NET and Win32 programming models are often very different, idiosyncrasies often lead to hard to track down problems. In this chapter we look at some very common mistakes done when working in the Interoperability layer and how to use the debuggers and tools to troubleshoot the problems.

Part III – Advanced Topics

8. Postmortem Debugging

Quite often it’s not feasible to expect full access to a failing machine so that a problem can be debugged. Bugs that surface on production machines on customer sites are rarely available for debugging. This chapter outlines the mechanisms for debugging a problem without access to the physical machine. Topics discussed include the basics of crash dumps, generating crash dumps, analyzing crash dumps etc.

9. Power Tools

Introduces the reader to power tools such as Debugging via Powershell , IISDiag and more.

10. CLR 4.0

This chapter is reserved for any substantial enhancements coming out while the book is being authored/published.

Index

[PowerShell Script] Statistics from .NET Applications

2009-04-15T05:52:00Z

This script is more a template to show you how to use PowerDbg.

I must say the idea is from my teammate Aaron Barth!

This script collects information from all threads running managed code and gives the user statistics by threads like:

- CLR stack.

- Managed objects from the stack.

- ASP.NET page.

- What the thread is doing.

- Exceptions by threads.

- Threads running ASP.NET pages.

Contrary to what you may think, this script is very simple. It’s very easy to customize or improve it. For example, you may want to display the ASP.NET pages or queries/stored procedures by threads. Or you may want to create a report using HTML. The output from this script is text, simple and straight. Again, the idea is to show you how to use the cmdlets from PowerDbg in order to create scripts that automate the debugging session.

Screenshots:

Source code for PowerShellScriptASPXStatistics.ps1:

########################################################################################################

# Script: PowerDbgScriptASPXStatistics

# Parameters: None.

# Purpose: Shows statistics from threads running ASP.NET pages.

# Attention! This script was not tested on Win64.

# Changes History: 04/15/2009 – Fixed bug when there’s just one managed thread.

# Roberto Alexis Farah

# All my functions are provided "AS IS" with no warranties, and confer no rights.

########################################################################################################

set-psdebug -strict

$ErrorActionPreference = "stop"

trap {"Error message: $_"}

write-Host "Scanning all threads and extracting the CLR stack..." -foreground Green -background Black

# First, let's scan all threads and identify those running managed code.

Send-PowerDbgCommand "~* e !clrstack"

Parse-PowerDbgCLRSTACK

# Get all the stacks running managed code.

$clrStack = Convert-PowerDbgCSVToHashTable

write-Host "Done!" -foreground Green -background Black

# Sorts the keys by Thread Number and save them into an array.

$arrayOfThreads = @($clrStack.keys | Sort-Object {[int] $_})

# Let's consider the situation where the dump has no thread running managed code.

if($arrayOfThreads.Count -eq 0)

{

write-Host "There are not threads running managed code!" -foreground Red -background Black

return

}

write-Host "Scanning all threads and extracting the managed objects..." -foreground Green -background Black

Send-PowerDbgCommand "~* e !dso"

Parse-PowerDbgDSO

$dso = Convert-PowerDbgCSVToHashTable

write-Host "Done!" -foreground Green -background Black

write-Host "Collecting information about each thread..." -foreground Green -background Black

Send-PowerDbgCommand "!Threads"

Parse-PowerDbgTHREADS

$threads = Convert-PowerDbgCSVToHashTable

write-Host "Done!" -foreground Green -background Black

write-Host "Collecting information from threads running ASP.NET..." -foreground Green -background Black

Send-PowerDbgCommand "!ASPXPages"

Parse-PowerDbgASPXPAGES

$aspxPages = Convert-PowerDbgCSVToHashTable

write-Host "Done!" -foreground Green -background Black

write-Host "Scanning all threads and preparing statistics..." -foreground Green -background Black

# Scans all threads running managed code.

for($i = 0; $i -lt $arrayOfThreads.Length; $i++)

{

# Make sure the content is not null.

if($arrayOfThreads[$i] -eq "")

{

continue; # Invalid, get next element.

}

write-Progress -activity "Thread Statistics" -status "Thread number $arrayOfThreads[$i]" -percentComplete ($i / $arrayOfThreads.length * 100)

write-Host "==============================================================" -foreground Green -background Black

write-Host "`nThread number: " -foreground Green -background Black -nonewline

write-Host $arrayOfThreads[$i] -foreground Red -background Black

write-Host "`nCLR stack:`n" -foreground Green -background Black

[string] $temp = $clrstack[$arrayOfThreads[$i]]

$temp = $temp.Replace($global:g_frameDelimiter, "`n")

$temp = $temp.Replace(";", ",")

write-Host $temp -foreground Red -background Black

write-Host "`nManaged objects from the stack:`n" -foreground Green -background Black

$temp = $dso[$arrayOfThreads[$i]]

$temp = $temp.Replace($global:g_frameDelimiter, "`n")

$temp = $temp.Replace(";", ",")

write-Host $temp -foreground Red -background Black

write-Host "`nThread Number ID OSID ThreadOBJ State GC Context Domain Count APT Exception`n" -foreground Green -background Black

write-Host " " $arrayOfThreads[$i] " " $threads[$arrayOfThreads[$i]] -foreground Red -background Black

$threadNum = $arrayOfThreads[$i]

# Change context to the current thread being analyzed.

Send-PowerDbgCommand "~ $threadNum s"

# Get exception.

Send-PowerDbgCommand "!PrintException"

Parse-PowerDbgPRINTEXCEPTION

$exception = $null

$exception = Convert-PowerDbgCSVToHashTable

# Makes sure there is an exception coming from that thread.

if($exception["Message:"] -ne $null)

{

write-Host "`nException object:" -foreground Green -background Black

write-Host $exception["Exception object:"] -foreground Red -background Black

write-Host "Exception type:" -foreground Green -background Black

write-Host $exception["Exception type:"] -foreground Red -background Black

write-Host "Message:" -foreground Green -background Black

write-Host $exception["Message:"] -foreground Red -background Black

write-Host "Inner Exception:" -foreground Green -background Black

write-Host $exception["InnerException:"] -foreground Red -background Black

write-Host "HRESULT:" -foreground Green -background Black

write-Host $exception["HResult:"] -foreground Red -background Black

}

# User must press any key to continue after 5 threads were displayed.

if((($i + 1) % 5) -eq 0)

{

write-Host "`n####### Press any key to see 5 more threads... #######"

$keyboard = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")

}

write-Host "Done!" -foreground Green -background Black

write-Host "`nASP.NET Pages:`n" -foreground Green -background Black

write-Host "HttpContext Timeout Completed Running ThreadId ReturnCode Verb RequestPath QueryString" -foreground Green -background Black

foreach($item in $aspxPages.keys)

{

write-Host $item " " $aspxPages[$item] -foreground Red -background Black

}

[PowerShell Script] PowerDbg v5.2—Using PowerShell to Control WinDbg

2009-04-15T05:43:00Z

This version has two improvements and some scripts were changed to be compatible with this new version:

- Starting on version 5.2 the log used to save the command output, enabling the communication between PowerShell and WinDbg is not used anymore. Now all communication uses memory so it’s faster than before.

Note: the file POWERDBG-PARSED.LOG continues to exist since it’s useful for troubleshooting purposes and can be used as input for Excel or LogParser.

- Installation is easier now. Just one file has everything.

I’ve created a simple PowerDbg script that gives ASPX information. It’s going to be blogged after this article and should be used as reference.

Note: It was difficult for me to find some free time and work on PowerDbg. I tested it using all scripts (modified two of them) and using different dumps. However, it may have bugs. If you hunt a bug let me know, please!

DOWNLOAD POWERDBG

Download PowerDbg

POWERDBG FILES

WinDbg.PSM1 ß Starting with this version this is the only file.

INSTALLATION

WinDbg.PSM1

Goes to %\WindowsPowerShell\Modules\WinDbg

Note: PowerDbg assumes the folder c:\debuggers as the default installation folder. This is true for the default installation of our private debugger version (Microsoft) but not for the public version, so, please, change this variable to reflect your installation:

param($cdbPath = "C:\debuggers\cdb.exe")

In order to know the exact location, use this command from PowerShell:

$profile

REQUIREMENT

PowerShell v2.0

USAGE

First, make sure you can run scripts:

set-executionpolicy remotesigned

From the WinDbg window type:

.server tcp:port=10456,server=ServerName ß ServerName is your server name.

The command above enables a port communication with the WinDbg instance as the server. You can use other port numbers.

From the PowerShell window you must initialize the communication:

Import-module WinDbg ß Imports our module WinDbg.PSM1

Connect-Windbg "tcp:Port=10456,Server=SERVER" ß Connects session to WinDbg instance.

Or:

Connect-Dbg "tcp:Port=10456,Server=SERVER" ß Connects session to WinDbg

Note: Don’t forget to load symbols and your extensions!

At this point you’re ready to use PowerDbg or PowerDbg scripts.

Example:

Analyze-PowerDbgThreads ß Cmdlet.

.\PowerDbgScriptExceptions ß Script.

Example 2:

Send-PowerDbgCommand "~* kpn 1000"

Parse-PowerDbgK

$ht = @{}

$ht = Convert-PowerDbgCSVToHashTable

Let's display the stack for thread 0:

# Replace the internal frame delimiter by new line and displays the stack.

write-host $ht["0"].Replace($global:g_frameDelimiter, "`n")

Example 3:

Send-PowerDbgCommand "!DumpObj 027a4c3c"

Parse-PowerDbgDUMPOBJ

$ht = Convert-PowerDbgCSVToHashTable

$ht["Name:"] <-- Displays the content of the field Name:

$ht["MethodTable:"] <-- Displays the content of the field MethodTable:

NEW FOR POWERDBG v5.2

PowerDbg now uses just one file that has the content of the previous two files and is not the Profile file anymore.

Easier to install and maintain.

The information from WinDbg to PowerDbg used a text file, now it’s all memory based.

It’s faster now than before and eliminates timing issues that used to happen when the file was being saved and the next PowerDbg command executed.

As a collateral benefit you don’t need to use Start-Sleep in your scripts anymore to avoid timing issues.

Note: The CSV file POWERDBG-PARSED.LOG is still used. It facilitates the troubleshooting and can be used as input for Excel or LogParser.

NEW FOR POWERDBG v5.1

Load-PowerDbgSymbols <$symbolPath>

Load symbols.

Usage:

Load-PowerDbgSymbols “SRV*c:\PUBLICSYMBOLS*http://msdl.microsoft.com/download/symbols"

Parse-PowerDbgASPXPAGES

Maps the output from the !ASPXPages command and saves it into the CSV file POWERDBG-PARSED.LOG

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

For this version the fields being considered are:

Key: HttpContext

Value: Timeout+Completed+Running+ThreadId+ReturnCode+Verb+RequestPath+QueryString

Parse-PowerDbgCLRSTACK

Maps the output from the !clrstack command or ~* e !clrstack and saves it into the CSV file POWERDBG-PARSED.LOG

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

Attention! The key is the thread number, and the value is the call stack separated by $global:g_frameDelimiter.

Commas "," are replaced for ";" to avoid confusion with the comma used by the CSV file.

If you use this cmdlet to parse the output from ~* e !clrstack, the threads not running managed code are automatically ignored.

Parse-PowerDbgTHREADS

Maps the output from the !threads command and saves it into the CSV file POWERDBG-PARSED.LOG

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

The following fields are extracted:

Thread Number - Key

ID+OSID+ThreadOBJ+State+GC+Context+Domain+Count+APT+Exception - Value

Parse-PowerDbgDSO

Maps the output from the !dso or ~* e !dso command and saves it into the CSV file POWERDBG-PARSED.LOG

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

The Thread Number is the key, and the stack is the value, like the way that Parse-PowerDbgK or Parse-PowerDbgCLRSTACK operates.

Attention! Commas are replaced by ";" and $global:g_FrameDelimiter is used to separate frames.

CMDLETS FROM POWERDBG

Send-PowerDbgCommand <$command>

This was the most complex cmdlet, but now it’s just a wrapper for Invoke-WinDbgCommand.

SendPowerDbgCommand sends commands to WinDbg.

Parse-PowerDbgDT [$useFieldNames]

Parses the output from the dt command and saves it into POWERDBG-PARSED.LOG using a CSV file format.

If $useFieldNames has a value, the cmdlet stores fields from struct/classes and values. Otherwise it stores offsets and values.

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

Convert-PowerDbgCSVToHashTable

Converts the output from the Parse-PowerDbg* cmdlets to a Hash Table.

Send-PowerDbgDML <$hyperLinkDML> <$commandDML>

Creates a DML command and sends it to WinDbg.

DML stands for Debug Markup Language. Using DML you can create hyperlinks that execute commands.

Parse-PowerDbgNAME2EE

Maps the output from the !name2ee and saves it into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Parse-PowerDbgDUMPMD

Maps the output from !dumpmd command and saves it into the CSV file POWERDBG-PARSED.LOG.

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Parse-PowerDbgDUMPMODULE

Maps the output from !DumpModule command and saves it into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Parse-PowerDbgDUMPLMI

Maps the output from !lmi command and saves it into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Has-PowerDbgCOMMANDSUCCEEDED

Returns $true if the last command succeeded or $false if not.

Send-PowerDbgComment

Sends a comment, a string in bold, to the WinDbg window.

Parse-PowerDbgVERTARGET

Maps the output from vertarget command, either the Kernel Time or the User Time.

The output is saved into the CSV file POWERDBG-PARSED.LOG.

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Parse-PowerDbgRUNAWAY

Maps the output of !runaway 1 or !runaway 2 and stores the results into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Attention! If you need to know the top threads consuming CPU time, use Convert-PowerDbgRUNAWAYtoArray. The items will be in the same exact order of the original command.

Convert-PowerDbgRUNAWAYtoArray

Returns an array of two dimensions corresponding to the output of !runaway 1 or !runaway 2.

Parse-PowerDbgK

Maps the output of k command and its variations like kv, kbn, kpn, etc.

The output is saved into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Attention! This cmdlet doesn’t work with kPn. Another thing, it replaces “,” with “;” to avoid conflict with the CSV delimiter.

Parse-PowerDbgSymbolsFromK

Maps just the symbols from k command and variations, saving the content into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Attention! This cmdlet doesn’t work with kPn. Another thing, it replaces “,” with “;” to avoid conflict with the CSV delimiter.

Parse-PowerDbgLM1M

Maps just the output from lm1m and stores it into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Classify-PowerDbgThreads

Returns one array where the index is the thread number and the value is one of these values:

0 UNKNOWN_SYMBOL

1 WAITING_FOR_CRITICAL_SECTION

2 DOING_IO

3 WAITING

4 GC_THREAD

5 WAIT_UNTIL_GC_COMPLETE

6 SUSPEND_FOR_GC

7 WAIT_FOR_FINALIZE

8 TRYING_MANAGED_LOCK

9 DATA_FROM_WINSOCK

It’s very easy to add more symbols and constants to get a more granular analysis. Look at the source code for details.

Analyze-PowerDbgThreads

Analyzes and shows what each thread is doing and its corresponding CPU time, sorted by User Mode time.

This cmdlet is very useful for scenarios like hangs, high CPU, and crashes.

Attention! This command requires thread information if debugging a dump file.

Parse-PowerDbgPRINTEXCEPTION

Maps the output from !PrintException command and saves it into the CSV file POWERDBG-PARSED.LOG.

The following fields are considered while others are ignored:

Exception object:

Exception type;

Message:

InnerException:

HRESULT:

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Parse-PowerDbgDD-L1

Maps the output from dd <address> L1 or dd poi(<address>) L1 and saves the results into the CSV file POWERDBG-PARSED.LOG.

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Parse-PowerDbgGCHANDLELEAKS

Maps the output from !GCHandleLeaks command and saves it into the CSV file POWERDBG-PARSED.LOG.

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Parse-PowerDbgDUMPOBJ

Maps the output from !DumpObj command and saves it into the CSV file POWERDBG-PARSED.LOG.

The assembly path and file name are saved using the key name “Assembly:.”

If the object is invalid, the ‘Name:’ field will have the string “Invalid Object.” You may want to check this string to make sure you have valid data.

The keys are the fields or Method Table, and values are the corresponding value.

Convert-PowerDbgCSVToHashTable converts the output into a Hash Table.

Attention! This version maps the fields below “Fields:” using MethodTable as key and Value as value. The problem with this approach is that the same MethodTable may appear more than once. If it happens, the last or most recent MethodTable and value will be considered.

Based on users’ feedback this approach may be changed in the near future.

POWERDBG SCRIPTS

PowerDbgScriptDumpDict.PS1

Extracts the key/value pair from a dictionary.

PowerDbgScriptExceptions.PS1

Displays the call stacks that have inner or hidden exceptions.

PowerDbgScriptGCHandleLeaksChart.PS1

Displays statistics and a chart from the top 20 objects leaking.

PowerDbgScriptHighCPU.PS1

Displays all threads consuming high CPU using a specific time as a threshold.

PowerDbgScriptSaveModule.PS1

Saves all modules that have a specific method. You provide the method name, and it gives you the corresponding modules.

[PowerShell Script] PowerDbg v5.1—Using PowerShell to Control WinDbg

2009-03-19T18:28:00Z

So, here we go again. This is a minor version with a few new cmdlets. These new cmdlets are those that we use most of the time.

DOWNLOAD POWERDBG

Download PowerDbg

POWERDBG FILES

WinDbg.PSM1 ß Contains cmdlets used to communicate with WinDbg.

Microsoft.PowerShell_Profile.PS1 ß Contains cmdlets that parse command output. Uses WinDbg.PSM1 under the hood.

INSTALLATION

Goes to %\Documents\WindowsPowerShell\Modules\WinDbg

Microsoft.PowerShell_Profile.PS1

Goes to %\Documents\windowspowershell

In order to know the exact location, use this command from PowerShell:

$profile

REQUIREMENT

PowerShell v2.0

USAGE

First, make sure you can run scripts:

set-executionpolicy remotesigned

From the WinDbg window type:

.server tcp:port=10456,server=ServerName ß ServerName is your server name.

The command above enables a port communication with the WinDbg instance as the server. You can use other port numbers.

From the PowerShell window you must initialize the communication:

Import-module WinDbg ß Importing our module WinDbg.PSM1

Connect-Windbg "tcp:Port=10456,Server=SERVER" ß Connects session to WinDbg instance.

Note: Don’t forget to load symbols and your extensions!

At this point you’re ready to use PowerDbg or PowerDbg scripts.

Example:

Analyze-PowerDbgThreads ß Cmdlet.

.\PowerDbgScriptExceptions ß Script.

Example 2:

Send-PowerDbgCommand "~* kpn 1000"

Parse-PowerDbgK

$ht = @{}

$ht = Convert-PowerDbgCSVToHashTable

Let's display the stack for thread 0:

# Replace the internal frame delimiter by new line and displays the stack.

write-host $ht["0"].Replace($global:g_frameDelimiter, "`n")

Example 3:

Send-PowerDbgCommand "!DumpObj 027a4c3c"

Parse-PowerDbgDUMPOBJ

$ht = Convert-PowerDbgCSVToHashTable

$ht["Name:"] <-- Displays the content of the field Name:

$ht["MethodTable:"] <-- Displays the content of the field MethodTable:

For more examples see the source code from PowerDbg scripts.

NEW FOR POWERDBG v5.1

Load-PowerDbgSymbols <$symbolPath>

Load symbols.

Usage:

Load-PowerDbgSymbols “SRV*c:\PUBLICSYMBOLS*http://msdl.microsoft.com/download/symbols"

Parse-PowerDbgASPXPAGES

Maps the output from the !ASPXPages command and saves it into the CSV file POWERDBG-PARSED.LOG

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

For this version the fields being considered are:

Key: HttpContext

Value: Timeout+Completed+Running+ThreadId+ReturnCode+Verb+RequestPath+QueryString

Parse-PowerDbgCLRSTACK

Maps the output from the !clrstack command or ~* e !clrstack and saves it into the CSV file POWERDBG-PARSED.LOG

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

Attention! The key is the thread number, and the value is the call stack separated by $global:g_frameDelimiter.

Commas "," are replaced for ";" to avoid confusion with the comma used by the CSV file.

If you use this cmdlet to parse the output from ~* e !clrstack, the threads not running managed code are automatically ignored.

Parse-PowerDbgTHREADS

Maps the output from the !threads command and saves it into the CSV file POWERDBG-PARSED.LOG

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

The following fields are extracted:

Thread Number - Key

ID+OSID+ThreadOBJ+State+GC+Context+Domain+Count+APT+Exception - Value

Parse-PowerDbgDSO

Maps the output from the !dso or ~* e !dso command and saves it into the CSV file POWERDBG-PARSED.LOG

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

The Thread Number is the key, and the stack is the value, like the way that Parse-PowerDbgK or Parse-PowerDbgCLRSTACK operates.

Attention! Commas are replaced by ";" and $global:g_FrameDelimiter is used to separate frames.

CMDLETS FROM POWERDBG

Send-PowerDbgCommand <command>

This was the most complex cmdlet, but now it’s just a wrapper for Invoke-WinDbgCommand.

SendPowerDbgCommand sends commands to WinDbg.

Parse-PowerDbgDT [$useFieldNames]

Parses the output from the dt command and saves it into POWERDBG-PARSED.LOG using a CSV file format.

If $useFieldNames has a value, the cmdlet stores fields from struct/classes and values. Otherwise it stores offsets and values.

To convert the CSV file to a Hash Table use Convert-PowerDbgCSVToHashTable.

Convert-PowerDbgCSVToHashTable

Convert the output from the Parse-PowerDbg* cmdlets to a Hash Table.

Send-PowerDbgDML <$hyperLinkDML> <$commandDML>

Creates a DML command and sends it to WinDbg.

DML stands for Debug Markup Language. Using DML you can create hyperlinks that execute commands.

Parse-PowerDbgNAME2EE

Maps the output from the !name2ee and saves it into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Parse-PowerDbgDUMPMD

Maps the output from !dumpmd command and saves it into the CSV file POWERDBG-PARSED.LOG.

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Parse-PowerDbgDUMPMODULE

Maps the output from !DumpModule command and saves it into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Parse-PowerDbgLMI

Maps the output from lmi command and saves it into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Has-PowerDbgCOMMANDSUCCEEDED

Returns $true if the last command succeeded or $false if not.

Send-PowerDbgComment

Sends a comment, a string in bold, to the WinDbg window.

Parse-PowerDbgVERTARGET

Maps the output from vertarget command, either the Kernel Time or the User Time.

The output is saved into the CSV file POWERDBG-PARSED.LOG.

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Parse-PowerDbgRUNAWAY

Maps the output of !runaway 1 or !runaway 2 and stores the results into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Attention! If you need to know the top threads consuming CPU time, use Convert-PowerDbgRUNAWAYtoArray. The items will be in the same exact order of the original command.

Convert-PowerDbgRUNAWAYtoArray

Returns an array of two dimensions corresponding to the output of !runaway 1 or !runaway 2.

Parse-PowerDbgK

Maps the output of k command and its variations like kv, kbn, kpn, etc.

The output is saved into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Attention! This cmdlet doesn't work with kPn. Another thing, it replaces "," with $global:g_frameDelimiter to avoid conflict with the CSV delimiter.

Parse-PowerDbgSymbolsFromK

Maps just the symbols from k command and variations, saving the content into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Attention! This cmdlet doesn't work with kPn. Another thing, it replaces "," with $global:g_frameDelimiter to avoid conflict with the CSV delimiter.

Parse-PowerDbgLM1M

Maps just the output from lm1m and stores it into the CSV file POWERDBG-PARSED.LOG

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Classify-PowerDbgThreads

Returns one array where the index is the thread number and the value is one of these values:

0 UNKNOWN_SYMBOL

1 WAITING_FOR_CRITICAL_SECTION

2 DOING_IO

3 WAITING

4 GC_THREAD

5 WAIT_UNTIL_GC_COMPLETE

6 SUSPEND_FOR_GC

7 WAIT_FOR_FINALIZE

8 TRYING_MANAGED_LOCK

9 DATA_FROM_WINSOCK

It’s very easy to add more symbols and constants to get a more granular analysis. Look at the source code for details.

Analyze-PowerDbgThreads

Analyzes and shows what each thread is doing and its corresponding CPU time, sorted by User Mode time.

This cmdlet is very useful for scenarios like hangs, high CPU, and crashes.

Attention! This command requires thread information if debugging a dump file.

Parse-PowerDbgPRINTEXCEPTION

Maps the output from !PrintException command and saves it into the CSV file POWERDBG-PARSED.LOG.

The following fields are considered while others are ignored:

Exception object:

Exception type;

Message:

InnerException:

HRESULT:

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Parse-PowerDbgDD-L1

Maps the output from dd <address> L1 or dd poi(<address>) L1 and saves the results into the CSV file POWERDBG-PARSED.LOG.

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Parse-PowerDbgGCHANDLELEAKS

Maps the output from !GCHandleLeaks command and saves it into the CSV file POWERDBG-PARSED.LOG.

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Parse-PowerDbgDUMPOBJ

Maps the output from !DumpObj command and saves it indo the CSV file POWERDBG-PARSED.LOG.

The assembly path and file name are saved using the key name ‘Assembly:’.

If the object is invalid the ‘Name:’ field will have the string “Invalid Object.” You may want to check this string to make sure you’ve got valid data.

The keys are the fields or Method Table, and values are the corresponding value.

Convert-PowerDbgCSVtoHashTable converts the output into a Hash Table.

Based on users’ feedback this approach may be changed in the near future.

POWERDBG SCRIPTS

PowerDbgScriptDumpDict.PS1

Extracts the key/value pair from a Dictionary.

PowerDbgScriptExceptions.PS1

Displays the call stacks that have inner or hidden exceptions.

PowerDbgScriptGCHandleLeaksChart.PS1

It displays statistics and a chart from the top 20 objects leaking.

PowerDbgScriptHighCPU.PS1

It displays all threads consuming high CPU using a specific time as a threshold.

PowerDbgScriptSaveModule.PS1

It saves all modules that have a specific method. You provide the method name, and it gives you the corresponding modules.

Special Command—Peeking Memory Addresses Using !address

2009-03-17T22:10:00Z

Let’s say that you get a memory address and you want to know if it’s from the heap, the stack, or someplace else. Or yet, let’s say you have a .NET application consuming lots of memory, and you want to get a better understanding of this memory consumption.

The !address command is helpful in both situations mentioned above and probably others not mentioned in this article.

Usage:

!address <address> ß gives you information about the address type.

!address –summary ß displays all addresses and a summary at the end.

To interpret the results follow this table:

RegionUsageIsVAD	The "busy" region. This region includes all virtual allocation blocks, the SBH heap, memory from custom allocators, and all other regions of the address space that fall into no other classification.
RegionUsageFree	The available memory in the target's virtual address space. This memory includes all memory that has not been committed or reserved.
RegionUsageImage	The memory region that mapped images of binaries use.
RegionUsageStack	The memory region that is used for the stacks owned by the threads in the target process.
RegionUsageTeb	The memory region that is used for the thread environment blocks (TEBs) for all threads in the target process.
RegionUsageHeap	The memory region that is used for the heaps that the target process owns.
RegionUsagePageHeap	The memory region that is used for the full-page heap that the target process owns.
RegionUsagePeb	The memory region that is used for the process environment block (PEB) of the target process.
RegionUsageProcessParametrs	The memory region that is used for the startup parameters of the target process.
RegionUsageEnvironmentBlock	The memory region that is used for the environment block of the target process.

Examples:

!address <address>

!address –summary

ProcessParameters 003e1860 in range 003e0000 004b0000

Environment 003e0808 in range 003e0000 004b0000

-------------------- Usage SUMMARY --------------------------

TotSize ( KB) Pct(Tots) Pct(Busy) Usage

3b09000 ( 60452) : 02.88% 41.28% : RegionUsageIsVAD ß Location where .NET allocates heaps.

770f1000 ( 1950660) : 93.02% 00.00% : RegionUsageFree ß Not committed memory or not reserved memory.

46ff000 ( 72700) : 03.47% 49.65% : RegionUsageImage

6ff000 ( 7164) : 00.34% 04.89% : RegionUsageStack ß Used for stacks.

7000 ( 28) : 00.00% 00.02% : RegionUsageTeb ß Used for Thread Environment Block.

5f0000 ( 6080) : 00.29% 04.15% : RegionUsageHeap ß Heaps from native code allocations.

0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap

1000 ( 4) : 00.00% 00.00% : RegionUsagePeb

0 ( 0) : 00.00% 00.00% : RegionUsageProcessParametrs

0 ( 0) : 00.00% 00.00% : RegionUsageEnvironmentBlock

Tot: 7fff0000 (2097088 KB) Busy: 08eff000 (146428 KB)

-------------------- Type SUMMARY --------------------------

TotSize ( KB) Pct(Tots) Usage

770f1000 ( 1950660) : 93.02% : <free>

46ff000 ( 72700) : 03.47% : MEM_IMAGE

181f000 ( 24700) : 01.18% : MEM_MAPPED

2fe1000 ( 49028) : 02.34% : MEM_PRIVATE

-------------------- State SUMMARY --------------------------

TotSize ( KB) Pct(Tots) Usage

5d48000 ( 95520) : 04.55% : MEM_COMMIT

770f1000 ( 1950660) : 93.02% : MEM_FREE

31b7000 ( 50908) : 02.43% : MEM_RESERVE

Largest free region: Base 04950000 - Size 5ee50000 (1554752 KB)

Special Command—Parsing Strings, Files, and Commands Output Using .foreach

2009-03-12T00:41:00Z

This is by far one of the most powerful WinDbg commands. Even if you don’t create scripts, you’ll benefit from this command. It’s powerful because it’s flexible. You can use it for a huge variety of operations.

The .foreach token parses the output of one or more debugger commands and uses each value in this output as the input to one or more additional commands.

Syntax

.foreach [Options] ( Variable  { InCommands } ) { OutCommands }

.foreach [Options] /s ( Variable  "InString" ) { OutCommands }

.foreach [Options] /f ( Variable  "InFile" ) { OutCommands }

Options:

Can be any combination of the following options:

/pS InitialSkipNumber

Causes some initial tokens to be skipped. InitialSkipNumber specifies the number of output tokens that will not be passed to the specified OutCommands.

/ps SkipNumber

InString

Used with /s. Specifies a string that will be parsed; the resulting tokens will be passed to OutCommands.

InFile

Used with /f. Specifies a text file that will be parsed; the resulting tokens will be passed to OutCommands. The file name InFile must be enclosed in quotation marks.

I know it looks complicated. To help you understand the variations, I present several different examples ranging from the simplest to the most complex.

Dumping virtual memory addresses that have the “nt” string:

.foreach(obj {s -[1]a 0 0FFFFFFF "nt"}){da /c 0n100 ${obj}}

Using !DumpObj for each object address from !DumpStackObjects command. Both commands are from the extension SOS.DLL:

.foreach /pS 1 /ps 2 (obj {.shell -i - -ci "!dso" FIND "System."}){!do ${obj}}

In bold below is the part skipped using /pS 1 (skips 1 token) and in red are the parts skipped using /ps 2 (causes 2 tokens to be skipped repeatedly). The output is from !DumpStackObjects:

OS Thread Id: 0x1d3c (0)

ESP/REG Object Name

ebx 01d9eb80 System.Windows.Forms.Application+ThreadContext

esi 01e56a84 System.Collections.Hashtable+HashtableEnumerator

002df0a4 01e434e4 System.Windows.Forms.NativeMethods+MSG[]

002df0a8 01d9eb80 System.Windows.Forms.Application+ThreadContext

002df0b0 01d9ee10 System.Windows.Forms.Application+ComponentManager

002df0fc 01f63884 System.Windows.Forms.ApplicationContext

002df104 01f63884 System.Windows.Forms.ApplicationContext

002df128 01d9eb80 System.Windows.Forms.Application+ThreadContext

002df178 01f63884 System.Windows.Forms.ApplicationContext

002df17c 01e62b28 System.Windows.Forms.Application+ThreadContext

002df18c 01df1130 System.Windows.Forms.ApplicationContext

002df194 01f505a0 System.ComponentModel.EventHandlerList

002df198 01f63898 System.EventHandler

002df1a0 01f63898 System.EventHandler

002df1a4 01f63898 System.EventHandler

002df1a8 01e9e33c NetWiz.frmNetWiz

002df1ac 01f63884 System.Windows.Forms.ApplicationContext

002df1b0 01e62b28 System.Windows.Forms.Application+ThreadContext

002df1bc 01f63884 System.Windows.Forms.ApplicationContext

002df1c4 01e9e33c NetWiz.frmNetWiz

002df1d0 01d9ef48 System.IO.FileStream

002df1d4 01d9e960 System.Object[] (System.Diagnostics.Process[])

002df1d8 01d85a10 System.Diagnostics.Process

Not impressed yet? Ok, take a look at the code below. I’m going to run commands from a string:

$$ Creates an alias with several different commands.

as ${/v:CommandString} kb r !clrstack

$$ Extracts the commands from the string and executes them.

.foreach /s (obj "CommandString") {${obj}}

Same thing, but gets the commands from a text file:

.foreach /f (obj "C:\downloads\test.txt") {${obj}}

Content from text file:

kpn !clrstack !dso r

As I said, this is a very powerful command! I bet you’re going to find other ways to leverage it!

Special Command—Parsing Commands Using .shell

2009-02-19T20:51:00Z

Finally I’m writing about this command. I love it! It’s so powerful! .shell command launches a shell process and redirects its output to the debugger or to a specified file.

Usage:

.shell [Options] [ShellCommand]
.shell -i InFile [-o OutFile [-e ErrFile]] [Options] ShellCommand

According to the WinDbg help, options can be:

-ci "Commands"

Processes the specified debugger commands and then passes their output as an input file to the process being launched. Commands can be any number of debugger commands, separated by semicolons, and enclosed in quotation marks.

-x

Causes any process being spawned to be completely detached from the debugger. This allows you to create processes which will continue running even after the debugging session ends.

The way I use it is:

.shell –i - -ci “command” FIND “string”

Why is that? I use it to extract specific information from a command.

The FIND command is not part of .shell. It’s a DOS command. Remember, .shell gives you access to the shell.

FIND

Searches for a text string in a file or files.

FIND [/V] [/C] [/N] [/I] "string" [[drive:][path]filename[ ...]]

/V	Displays all lines NOT containing the specified string.
/C	Displays only the count of lines containing the string.
/N	Displays line numbers with the displayed lines.
/I	Ignores the case of characters when searching for the string.
"string"	Specifies the text string to find.
[drive:][path]filename	Specifies a file or files to search.

If a pathname is not specified, FIND searches the text typed at the prompt or piped from another command.

So the combination of .shell and FIND is very powerful and it’s the most common way to use this command.

Example #1 – Searching for a specific managed object:

.shell –i - -ci “!dso” FIND “Sherlock.FoundBlocking”

Example #2 – Searching for a specific mnemonic from the disassembled code:

.shell –i - -ci “uf USER32!InternalCallWinProc” FIND “cmp”

Example #3 – Dumping part of the raw call stack and searching for USER32 calls:

.shell –i - -ci “dps @$csp L300” FIND “USER32”