John Eldridge : Windows CE Debugging

MEDC 2006

2006-03-30T03:23:00Z

Just FYI, I plan to be at MEDC in Vegas for pretty much the entire event. If you'd like to speak with me in person I will be staffing at the "Ask The Experts" sessions as well as participating in some panel sessions.

Have a look at http://www.medc2006.com for details on the developer conference, and see you in Vegas!

-John.

Comments on my site

2006-03-30T03:15:00Z

Ug. Sorry, folks. I just noticed that some comments to my blog posts weren't being published automatically. I was not intentionally filtering. Hopefully I have it set correctly now.

Virtual Memory and Thread Stacks

2006-01-31T08:44:00Z

I know. It's been nearly a year since I posted anything. Ouch. Bad John.

What inspires a new post? Virtual memory. One of my favorite subjects, but unfortunately, also a common stumbling block on Windows CE, especially on feature-rich Pocket PCs. I've been debugging several stress cases recently that were impacted by virtual memory pressures and I thought I should remind folks about one common source of unnecessary pain.

When you build applications within Platform Builder or the Windows Mobile Adaptation Kit, the default thread stack is 64kB of reserved virtual memory (the physical RAM is committed one page at a time). This is a pretty reasonable limit for most things on an embedded device. However, when you build your applications in Visual Studio, the default thread stack is 1MB of reserved virtual memory. This is often much more than needed and consumes precious virtual memory real estate. Windows CE processes are limited to 32 MB of virtual address space, unless you employ specific techniques to allocate larger blocks outside the process space. If your application only has one or two threads this may not be a big deal, but if your app is heavily multi-threaded, you can very quickly run out of virtual address space on a CE-based device.

The better option is to set the default stack within the VS options back to 64k (see photo) and then allocate larger stacks only if necessary. The way to create a thread with a larger than default stack on Windows CE is to call CreateThread with the STACK_SIZE_PARAM_IS_A_RESERVATION set in dwCreationFlags and dwStackSize set to how large you would like your stack to be. That value will be rounded up to the next 64kB boundary. The physical RAM will still be committed just a page at a time as needed.

I'd personally like to change the VS default to a much smaller value, but that could introduce unexpected backwards-compatibility issues for folks that couldn't be caught at build time. I'm curious to hear feedback though of whether anyone targeting CE-based devices in Visual Studio is relying on the 1MB default? Or if you are changing the stack sizes already? Or just don't care since your apps only have a couple threads anyway?

Platform Builder Debug Symbols : Are They Correct??

2005-02-22T15:05:00Z

PB Symbols

On Platform Builder versions prior to 5.0, one thing I find myself doing in a lot of debugging cases is verifying that debugger symbols match what is actually running on a device. In most cases the debugger can detect incorrect symbols and will notify the user in two ways.

The first notification comes when the module loads. There will be a notification in the debug output window that says something like “no matching symbols found”. For instance, here’s an example of loading OSSVCS.DLL when the PDB is missing from the release directory.

Debug Output Window

Loaded '<path>\OSSVCS.DLL', no matching symbolic information found.

The second way to see status is to look in the modules and symbols window in Platform Builder. It will show the symbol load status and where the symbols have been loaded from.

Modules and Symbols Window

oleaut32.dll 0x037D0000 - 0x037FFFFF 0x01F3E000 - 0x01F3EED0 Loaded

ossvcs.dll 0x033F0000 - 0x0341BFFF 0x01EFB000 - 0x01EFB9B0 Unloaded

Unfortunately, there are cases where this isn’t enough. The Windows CE tools format modules in the MODULES section of the BIB to strip out any information that is not necessary to run the module. In doing so, the original file format is not maintained and it’s possible for the code resident on the device to not match the symbols that the debugger loads. The most common case of this is when you receive a .BIN file from someone without the matching symbols. The second common case can happen if you regenerate a module, updating it in the release directory, without updating it on the device. In this case the DLL and PDB in the release directory match each other, but don’t match what’s running on the device.

Mismatched symbols can lead to a poor or misleading debugging experience. One method I use to manually verify the symbols is to look at the start of a few functions from the module I’m working with. Pick a function and find out what address it’s loaded at by typing the module and function name into the watch window, as follows…

Watch Window

{,,ossvcs.dll} PlaySoundScript

This should resolve to a specific address like this…

Watch Window

{,,ossvcs.dll} PlaySoundScript 0x033f895c PlaySoundScript(…)

If it doesn’t, you may not have symbols at all, in which case you may need to regenerate or get the symbols from whoever built the module originally.

Now that you have the function address, click on the result and drag the contents to the disassembly window. The assembly code for this function should appear. This is the code that is being read from the device at this address. If the symbols match, then you should see something like this…

Disassembly Window

PlaySoundScript:

033F895C E92D4010 stmdb sp!, {r4, lr}

033F8960 E24DD004 sub sp, sp, #4

$M36115:

033F8964 E3A04000 mov r4, #0

…

How can you tell if this is correct? Well, you need to know generally what a function entry looks like; let me explain that here. I’m going to show example from ARM assembly, but other processors have similar entry methods. The compiler generates a prolog on function entry. A prolog is a handful of machine instructions that prepare the processor to execute a new function. In most cases this is simply to push a set of registers onto the stack and to decrement the stack pointer by enough to allow room for local function variables to live on the stack if necessary.

For instance, the most common first instruction on ARM processor for a function entry point is a ‘store multiple’ instruction, here saving R4 and LR onto the stack.

Disassembly Window

PlaySoundScript:

033F895C E92D4010 stmdb sp!, {r4, lr}

Now here’s an example of a case of symbol mismatch where the debugger could not detect the mismatch on module load.

Disassembly Window

SetPolicy:

0340863C E8BD8030 ldmia sp!, {r4, r5, pc}

03408640 033F19A4 ???

$M30427:

03408644 033F2A00 ???

03408648 E92D40F0 stmdb sp!, {r4 - r7, lr}

0340864C E24DD01C sub sp, sp, #0x1C

In this case, I had updated the module in the release directory, but was running an older binary image in the device. Here you see that there is not ‘store multiple’ instruction at the beginning. Instead the ‘store multiple’ is a few assembly instructions down because of the symbol mismatch.

Mismatched symbols can affect your debugging experience, even if you are not specifically looking at an issue that directly involves the mismatched module. For instance, here’s a call stack for a thread from a Windows Mobile device with correct symbols:

Callstack Window

0x0c0bfdbc NK!SC_WaitForMultiple() schedule.c line 4434 + 20 bytes

0x0c0bfde8 NK!UB_WaitForMultiple() schedule.c line 4483

0x0c0bfe00 COREDLL!xxx_WaitForMultipleObjects() tkfuncs.c line 101 + 52 bytes

0x0c0bfe18 GWES!MsgQueue::MsgWaitForMultipleObjectsEx_I() msgque.cpp line 5296 + 16 bytes

0x0c0bfe64 COREDLL!xxx_MsgWaitForMultipleObjectsEx() twinuser.cpp line 1205

0x0c0bfe70 OSSVCS!SSUpdate::Thread() ssupdate.cpp line 1444

0x0c0bfed4 COREDLL!ThreadBaseFunc() apis.c line 419

Here’s that same exact call stack if I manually unload the symbols and put wrong symbols for OSSVCS.DLL into the release directory.

Callstack Window

0x0c0bfdbc NK!SC_WaitForMultiple() schedule.c line 4434 + 20 bytes

0x0c0bfde8 NK!UB_WaitForMultiple() schedule.c line 4483

0x0c0bfe00 COREDLL!xxx_WaitForMultipleObjects() tkfuncs.c line 101 + 52 bytes

0x0c0bfe18 GWES!MsgQueue::MsgWaitForMultipleObjectsEx_I() msgque.cpp line 5296 + 16 bytes

0x0c0bfe64 COREDLL!xxx_MsgWaitForMultipleObjectsEx() twinuser.cpp line 1205

0x0c0bfe70 OSSVCS!SSUpdate::s_WndProc() ssupdate.cpp line 1543 + 4 bytes

0x0c0bfe70 NK!PrefetchAbort + 336 bytes

0x0c0bfe88 NK!PrefetchAbort + 336 bytes

0x0c0bfea0 NK!PrefetchAbort + 336 bytes

0x0c0bfeb8 NK!PrefetchAbort + 336 bytes

0x0c0bfed0 NK!PrefetchAbort + 336 bytes

0x0c0bfee8 NK!PrefetchAbort + 336 bytes

…

The debugger tries to unwind the call stack. When it unwinds to a function in OSSVCS.DLL, it looks for the prolog in order to fix up the stack. But because the prolog isn’t at the right address, the debugger miscalculates where to find the next function on the callstack and ends up showing garbage.

The bottom line: It is critical to have the correct symbols to effectively use the debugger.

Controlling PB docking windows

2005-01-31T17:20:00Z

Minor tip here, but I know I was thrilled the first time I learned it. The docking windows view used to really annoy me because of the way they'd decide to position themselves as I'd drag the window. Well, you can just hold CTRL key while dragging to leave it unattached but placement is in your control.

Finding functions in watch window

2005-01-29T10:20:00Z

One of the elements of the Windows CE debugger that I use everyday, but have found to be not that well-known, is the ability to look at global variables and functions in any named DLL or EXE. Doesn't have to be in the currently stopped location. For instance, break into the debugger, and add a watch in the watch window named...

{,,coredll.dll} LocalAlloc

This says to resolve the name "LocalAlloc" in the context of the module "COREDLL.DLL". You should see something like...

0x03f9a4f4 LocalAlloc(unsigned int, unsigned int)

Which means that the LocalAlloc() function lives at 0x03f9a4f4. That alone has limited usefulness. But now the really cool part... click on the result (Value Column) and drag it into the disassembly window. Now the disassembly window shows the assembly for the LocalAlloc function.

Want the source instead? Now right-click in the disassembly window and choose "Go To Source". If you didn't compile the code yourself for the function you are looking at, it may ask you to input the path. I actually use this trick a lot just to let the debugger tell me where source code lives for certain functions that I'm not familiar with.

Did I mention that you don't get this functionality without implementing KITL? :-)

Debugging in 1024x768... ouch.

2005-01-29T08:27:00Z

So, I've recently been doing a lot of debugging with Platform Builder on my laptop and with mobile customers who debug on laptops. Sadly, the beautifully light, portable laptops are often XGA resolution. Not a big deal, until you try to see Debug Output, Watch Window, Disassembly, Target Control, Registers, Callstack, etc all at the same time. Then it gets really tight.

Well, the thing that strikes me the most is going to another person's laptop which is using the out-of-the-box PB font settings in XGA resolution. PB defaults to a nice "safe" font selection of, like, Terminal or Fixedsys or something hideous. Ouch. Can barely fit one window, let alone 6. But thankfully it's quick and easy to change under Tools | Options | Format (tab). I run my laptop PB at 7-point Lucida Console for all windows. With ClearType enabled, it's very usable and beats swapping windows to the foreground all the time.

That said, while on the road, I seriously miss my dual-monitor setup back home. In my opinion, upping monitor real-estate is a simple, easy way to increase debugging productivity.

Windows CE Debugging Tips : A random outlet

2005-01-29T08:08:00Z

Welcome. I've been thinking recently about all the random little tips and tricks I've learned over the years about debugging in Windows CE. Well, I'm pretty bad about writing documentation that requires a good flow or consideration of a specific audience. So here I am, embarking on a experiment to see if I can do any better in just publishing my thoughts in free form blog format. That way, I can give a little bit here and there as the mood strikes me without too much stress on format. With any luck someone will find the bits and pieces interesting and/or helpful. If it ends up being useful to you, please let me know.

A little about myself first... my name is John Eldridge. I've been working for Microsoft since 1995, the whole time on Windows CE. I've spent most of my time in the lower layers, writing device drivers and kernel code. Currently I work mostly with OEMs doing debugging. For some strange reason, I find debugging software issues as one of the most rewarding elements of my job. Go figure.

My motto (also this site's subtitle and my sign in name) is "KITL First. Ask Questions Later." If you are a Windows CE developer using Platform Builder and you don't know what KITL is or what it has to offer... go do it. Now. Don't even finish reading this. It will be well worth your investment. Seriously.

Okay, you're still here, so you must know about KITL. I expect most of my future entries will assume that you have internalized its power. "Learn it. Love it. Live it." :-)

Anyway, here's to "ENTRY 1".