Daniel Pearson [MSFT]

Off to Perth

danpear — Mon, 06 Aug 2007 10:34:00 GMT

I'll be in Perth shortly after Tech Ed presenting to the Perth Infrastructure Group, oink, oink! I'll be delivering the Windows Debugging Demystified session that I've presented previously in Sydney, Brisbane and Canberra. If you're around on the 16th of August then feel free to attend. Don't forget to print out the slides to write notes on.

Venue: EXCOM, Level 2, 23 Barrack Street
Date: Thursday, 16th August
Time: 5:15 PM for a 5:30 PM start
Close: Session generally closes by 7:30 PM
Cost: Free!

Attending Tech Ed on the Gold Coast

danpear — Sat, 21 Jul 2007 09:04:00 GMT

So Rocky and Chuck have managed to organize a few spots at Tech Ed so I can share the joys of debugging with the world. If you're there then swing by my sessions on Thursday and Friday. I'll also be around for the Ask The Experts sessions on Wednesday evening so feel free to drop by then too. Let me know if there's any interesting debugging scenarios you wish to discuss or if you just want to hear more about how Microsoft delivers technical support and we can schedule some 1:1 time throughout the conference.

On the road again

danpear — Tue, 22 May 2007 11:42:00 GMT

I'll be in Canberra next month presenting to the Canberra Windows User Group. I'll be delivering the Windows Debugging Demystified session that I've presented previously in Sydney and Brisbane. If you're around on the 12th of June then feel free to attend. Microsoft will cover the pizza! Don't forget to print out the slides to write notes on.

Venue: Microsoft, Level 2, 44 Sydney Avenue, Barton
Date: Tuesday, 12th June
Time: 5:30 PM for a 6 PM start
Close: Session generally closes by 7:30 PM
Cost: Free!

Debugging Tools for Windows Updated

danpear — Fri, 27 Apr 2007 05:29:00 GMT

It's been a while since our last update but the latest version, 6.7.5.0, of the Debugging Tools for Windows is now available for download from microsoft.com. There's a ton of new features and improvements so get clicking!

New Features

Add %Y{l} format specifier to take a ULONG64 argument and display a source line.
Add $ntdll[w|n]sym built-in aliases.

New command-line options

Windbg: -lsrcpath to set local source path.
All: -log[a|o]u to generate Unicode log files.

Changes to default configuration

When loading modules from a user-mode minidump provide available misc and CV record info from dump. This can allow symbols to be loaded in some cases when PE image file is not available.
.reload with specific module name now defers loading. Force-loading can be done with /f as usual.
SDK content is now installed by default.
Samples now explicitly default to not building Vista-only.
.remote pipe names may no longer start with ‘/’.
Change symbol lookup order: root\file, root\extension\file, root\symbols\file. Note that you cannot depend on lookup order remaining unchanged.

!analyze enhancements
New and updated commands: .trap, lmDv, .allow_exec_cmds, .pcmd, .dml_file, gu, .fnent, .pagein, dt, bs, bsc, bm
New options for commands: .foreach, .reload, .dump, .dumpcab, x, uf, ln, .call, .open
New and updated extensions: !chksym, !cpuid, !dml_proc, !address, !chkimage, !vm, !error
Source Server

CVS sample scripts.
Updated documentation.

Additional Symbol Server support

Allow SymSrvNoProxy to apply to WinInet functionality.
New option to force WinInet or WinHttp regardless of whether Symbol Server is running as a service.

ExtCpp extension improvements

Allow empty strings as default values.
Add Unicode output methods.

Better Debugging

Improved handling of IA64 trap frames.
Improved handling of DML content.
Minidump generation improvements.
Improved handling of CABs.
Improved handling of out-of-memory conditions.
Increased symbol functionality.
Bugfixes for EXDI targets.

New and updated Tools

Kdsrv now automatically uses IPv4 or IPv6.
Updates for AutoDump Plus.
Convertstore: converts a two tier symbol store to a three tier symbol store.

Windows Debugging Demystified

danpear — Thu, 26 Apr 2007 05:34:00 GMT

I'll be presenting a session on Windows debugging to the Brisbane Infrastructure Group on Tuesday, 1st May in Brisbane. The focus of the talk will be on bugchecks and the why, how and what to do about them. I've attached the PowerPoint slides I'll be using so feel free to print them out as they're great for taking notes on. If you're in Brisbane that evening then why not swing by. The cost is free and pizza will be provided.

Venue: Microsoft, Level 9, 1 Eagle Street, Waterfront Place
Date: Tuesday, 1st May
Time: 5:30 PM for a 6 PM start
Close: Session generally closes by 8 PM
Cost: Free!

The lifts close at 6 PM and please RSVP ASAP so they know how much pizza to order.

What error?

danpear — Thu, 05 Apr 2007 18:12:00 GMT

A large percentage of Windows APIs return zero to indicate success and non-zero for failure. Most of the MSDN documentation for each API included with the operating system contains a Return Value section. The Return Value section documents what return values can be expected.

Windows allows for developers to access an additional return code via the Last Error mechanism. A last error code is maintained for every thread in the system. It allows for API developers to return further information to the calling application such as why an API failed.

The last error code is stored in the thread environment block (TEB). A TEB is a user mode structure that contains various information about each thread.

There's a few different ways to read the TEB and the last error code contained within it. With a debugger there's the choice of several extensions or the TEB can be read directly. An application can use the GetLastError API.

Windows makes use of the FS segment register to store information about the current thread. The address of the TEB for the current thread is stored at FS:[18].

0:000> dd fs:18 l 1
003b:00000018  7ffdf000

The TEB can either be displayed by using the !teb extension or the dt command. The !teb extension will show a formatted view whereas the dt command will show the full structure.

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        0007fb00
    StackBase:            00080000
    StackLimit:           0006f000
    ...

0:000> dt _TEB 7ffdf000
ntdll!_TEB
   +0x000 NtTib            : _NT_TIB
   +0x01c EnvironmentPointer : (null) 
   +0x020 ClientId         : _CLIENT_ID
   ...

The last error code is a ULONG value. It's stored in the LastErrorValue field at an offset of 52 (0x34) bytes from the start of the TEB.

0:000> dt _TEB 7ffdf000 LastErrorValue
ntdll!_TEB
   +0x034 LastErrorValue : 0

The GetLastError API uses the same information when it returns the last error code. It reads the location of the TEB from the FS register and then returns a ULONG value from 52 bytes into that address range.

7c839325 mov     eax,fs:[00000018]
7c83932b mov     eax,[eax+0x34]
7c83932e ret

In the debugger it can be entered as one command.

0:000> dd poi(fs:18)+34 l 1
7ffdf034  00000000

There's one final debugger extension and that is !gle. The !gle extension returns both the LastErrorValue and LastStatusValue values. !gle goes one step further and checks several known DLLs in an attempt to locate the error code in any of their message tables. If a match is found then !gle displays both the error code and the error message.

0:000> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed ...
LastStatusValue: (NTSTATUS) 0 - STATUS_WAIT_0

The LastErrorValue field is also writable. DLLs can call SetLastError to set the error code. Ever since Windows XP, SetLastError has an undocumented trick up it's sleeve. The following information applies to both Windows XP and Windows Server 2003.

Hiding inside of kernel32's address space is a global variable called g_dwLastErrorToBreakOn. It turns out that SetLastError checks the value of this variable and if it's non-zero, calls DbgBreakPoint if the two conditions match. The downside is that SetLastError is only called from within KERNEL32.DLL. Other calls to SetLastError are redirected to a function located in NTDLL.DLL, RtlSetLastWin32Error.

Still g_dwLastErrorToBreakOn can be extremely useful when trying to track down when an error occurs in Kernel32 APIs and the last error value is set. Of course you can set a breakpoint on SetLastError and check which value is passed but it can be somewhat slow. Using the global variable incurs no overhead as it's always checked.

In Windows Vista, SetLastError in Kernel32 jumps directly to RtlSetLastWin32Error in NTDLL and checks the global, g_dwLastErrorToBreakOn, which is now part of NTDLL.DLL. This allows the last error mechanism for all DLLs to be easily debugged.

The variable needs to be edited directly to enter the value to break on. In this case the error is ERROR_ACCESS_DENIED. For Windows Vista, the global ntdll!g_dwLastErrorToBreakOn should be set instead.

0:000> ed kernel32!g_dwLastErrorToBreakOn 5

Once the breakpoint is triggered, the application breaks into the debugger and a full stack trace can be read.

0:013> k
01c7d524 ntdll!DbgBreakPoint
01c7d534 kernel32!SetLastError+0x24
01c7d544 kernel32!BaseSetLastNTError+0x17
01c7d828 kernel32!FindFirstFileExW+0x4c5
01c7d84c shell32!SHFindFirstFile+0x2a
01c7daa0 shell32!SHFindFirstFileRetry+0x5b
01c7dcf0 shell32!CFileSysEnum::Init+0x14b
...

All error codes that are returned by the operating system are listed in winerror.h. The MSDN Library also maintains a list of system error codes. Application defined error codes will not be listed.

You and I, Watson, we have done our part

danpear — Fri, 30 Mar 2007 18:49:00 GMT

On his site, Raymond Chen discussed earlier the origins of Dr. Watson. In fact the original name for the tool wasn't Dr. Watson at all, it was Sherlock. This can still be seen in the 16-bit version of the application. There's three pieces of evidence that give it away. The main windows procedure, SherlockWndProc, the main dialog procedure, SherlockDialog, and the template name given to the dialog box, SherDiag. The icons were a little different then too. The initial icon was a smoking pipe which was later changed to a doctor's bag with a stethoscope. The final change came when the icon with the doctor and a stethoscope together was introduced. When the port to Windows NT was made, all references to Sherlock were dropped. Sadly Dr. Watson was removed from Windows Vista to make way for Windows Error Reporting.

Narrowing down CPU usage

danpear — Wed, 28 Mar 2007 07:52:00 GMT

Accounting for CPU usage isn't always a straight forward task. One of the easiest methods is to use Task Manager. Task Manager provides a CPU column which breaks down each processes CPU usage into a percentage. Clicking on the CPU column header will sort the data from highest to lowest. The offending process should be easy to spot. It'll usually be sitting at the top of the list happily chewing up CPU cycles.

So what if the offending process is listed as System? Then it gets a little trickier. There is no System file on disk. System is a collection of internal worker and device driver threads. It's impossible to know which system or device driver thread could be causing the issue without digging deeper.

We can use several tools and techniques to pinpoint the offending code. Which one to use depends on the type of problem and the tools we have access to. There's always a risk using 3rd party tools and it's common that systems have strict change control so I tend to stick to tools that are supplied with the operating system. Performance Monitor is an extremely powerful tool and will give us most of what we need. With support from a few development tools we can break it down further.

We need to identify the instance of the system thread that is utilizing the CPU. It's a good idea when working with Performance Monitor to create logs of all your monitoring because it provides the greatest flexibility when analyzing the results and they can also be used later as a baseline. Logs are also easily shareable with support staff.

The object we're interested in is the Thread object. It's best to use objects and not counters. Selecting the object will record data for every thread in the system. I'd rather collect more data than is needed than less. Once the data has been collected then we're ready to analyze it.

Start by adding an instance for each of the System threads to the graph. The counter to select is Thread » % Processor Time. This will give us an overview of what is going on. It should be straight forward spotting which thread it is that is causing the problem. If you get stuck then try using the Highlight tool, CTRL+H, while cycling through each thread.

Performance Monitor uses an instance count to track all threads. It shouldn't be confused with a thread ID. Add the Thread » Start Address counter for the instance identified earlier. It's easier to switch to report view to read the results. If you use a graph then the results with be in decimal and you'll have to convert them manually using a calculator.

\\VIPER
    Thread                          48
        Start Address       0xFA0067F0

The start address is the virtual address of the thread. With the start address it's possible to identify the driver and also the function. We just need a way to tie the two together. The Process and Thread Status tool is the best choice. It ships as part of the Windows Support Tools which is included on the Windows XP CD. It's important that the tool be run whilst the performance data is being collected.

C:\Program Files\Support Tools>pstat

Process Status shows each driver and the virtual address at which it is loaded. To locate the driver it's simply a matter of finding which address range the start address lies in. Start by checking the second column.

  ModuleName Load Addr   Code    Data   Paged LinkDate
------------------------------------------------------------
ntoskrnl.exe 804D7000  478976   93440 1243264 Wed Aug 04 ...
     hal.dll 806EC000   29312    6400   25088 Wed Aug 04 ...
 toaster.sys FA006000       0       0       0
   ntdll.dll 7C900000  503808   20480       0 Wed Aug 04 ...
------------------------------------------------------------
       Total          5013792  507136 4408544

We now know the start address and the device driver. If we have access to the symbols for the device driver we can locate the function. If you're using a kernel debugger then the ln command works a treat. It displays the symbols either at or near a given address. The debuggers are available for download in the Debugging Tools for Windows package.

kd> ln fa0067f0 (fa0067f0)
   toaster!ToasterPollingThread   |  ...
Exact matches:
    toaster!ToasterPollingThread = <no type information>

If we were using private symbols then source line information would be available too. It's then just a case of reading through the source. "Use the source, Luke".

With a few easily available tools we've been able to supply more than enough information to get an understanding of what is going wrong. Armed with that information the developer should be able to correct the issue.

We're hiring!

danpear — Tue, 27 Mar 2007 14:05:00 GMT

We currently have two positions available for support engineers in our Global Technical Support Center based in Sydney. Both are in the Platforms team which covers Windows technologies. We're looking for candidates with strong Active Directory or networking skills. If you feel like a challenge then get clicking!

Testing ... one, one, zero, zero, zero, zero

danpear — Tue, 27 Mar 2007 05:39:00 GMT

My name is Daniel Pearson and I'm a Lead in support for Windows. I spend most of my day with my head either in a debugger or reading through code. I'm a huge fan of using debuggers to troubleshoot and I enjoy using them to resolve complicated issues. I've always been involved with Windows, and by Windows I mean Windows NT, since the start of my career, right through to Digital Equipment Corporation and then on to Microsoft. I’ve had the opportunity to work on various other technologies in Microsoft too. I was in sustained engineering for Mobile Information Server for a while and have had the luck of working for Microsoft in 5 different countries so far!

I decided to start writing as way to reach out to developers and IT professionals. As software and systems grow, so does the complexity of troubleshooting them. It's one thing to be able to step through code, line by line, on a development system tucked away in an office, it's another having to debug a live system that is still required to service thousands of users to determine why an application is terminating due to heap corruption, or some other serious bug, and prevent it from ever happening again. Hopefully I can share experiences that will give you the skills so you'll never need to contact support.

I want to use this to create an environment that is rich with ideas and information. I have a lot of ideas of what I want to do already so you should be seeing a lot of changes. Let me know if there's something that you want to see covered or want more on.

If you haven't yet seen or used the Debugging Tools for Windows, then I suggest getting familar with them. We're going to use them a lot!