Sue Loh's blog : Debugging, Kernel & misc. File System issues

Windows CE eHow-tos and Tutorials! WOW!!

sloh — Wed, 25 May 2005 21:49:00 GMT

I saw a blip of this at MEDC, and just checked it out myself. How cool! It's like a DevCon for all to share!

http://msdn.microsoft.com/embedded/getstart/basics/tutorialsce/default.aspx

Most of it is pretty basic stuff, but if you're new to Windows CE, you should check some of these out! (If you are new to performance tools like Remote Kernel Tracker which I've been posting about, take a look at the Microsoft Windows CE 5.0 Advanced Lab that's on that site.)

Sue

Using RAM over 512MB

sloh — Wed, 25 May 2005 18:23:00 GMT

Q: I can use only 512MB of RAM from my device. How can I address the remaining RAM ? On one of your posts you said that there is a way to go with more that 512 MB RAM. Can you help me on this issue?

A: Windows CE can't address more than 512MB. There's no way to make the OS use it the way it uses other RAM. OEMAddressTable, OEMGetExtensionDRAM and OEMEnumExtensionDRAM are all limited to 512MB.

The only thing you can do is write applications that specifically know about the extra memory. They can use VirtualAlloc with address 0 to reserve some virtual address space, and then use VirtualCopy with the PAGE_PHYSICAL to map the physical memory above 512MB into the virtual address space. In this manner the application can use the additional RAM. But the memory won't be used by the system the way other memory is. It won't go into the pool that's used for general operation; other applications won't ever get that memory. Your system could use up all of the first 512MB and then "run out of memory" because it can't use the other 512MB.

If your device needs a lot of resources like this, you might want to look into Windows XP Embedded instead.

Q: Are there any plans to address this problem in future releases of Windows CE?

A: I cannot comment on what will or will not be in future releases. I can only say, so far I don't think it is a high priority.

How does the OS use the registry?

sloh — Mon, 11 Apr 2005 20:07:00 GMT

Q: As an embedded programmer I am new to all windows development. And as such I am ignorant of the registry. It seems to play a crucial part in the initialization of the OS (device drivers, services, etc) Do you know of a good source of information about how the OS (not individual applications) uses the registry?

A: Sorry, I don't have any good sources of information to point you at. The registry is sort of a cross between a database and a file system -- the data is stored hierarchically like a file system, in small values with different types. The OS mostly uses the registry for reading in configuration information that you (or applications) can change. There isn't one big document for all registry information that is used by the system, but I know our documentation team has been making an effort to document all the settings on an area-by-area basis. For example, http://msdn.microsoft.com/library/en-us/wcedata5/html/wce50oriFileSystemsDataStoreRegistrySamples.asp

Most parts of the OS have default registry settings that they use, some of which may be controlled by relatively high-level flags you can set like PRJ_* and IMG*. http://msdn.microsoft.com/library/en-us/wceosdev5/html/wce50conPRJEnvironmentVariables.asp

The OS should be able to boot and run based on these variables, without you having to tweak many (any?) individual settings. Then you can tweak things to modify OS behavior as needed. The hard question I can't answer is what settings you might want to tweak. I guess you will have to find those out as you go.

Sorry I'm not more help than that,

Sue

GetTickCount – Truth and Fiction

sloh — Wed, 06 Apr 2005 02:16:00 GMT

I was surprised and dismayed to read a recent article in Embedded Systems Programming (http://www.embedded.com/showArticle.jhtml?articleID=159902113) that gets so many things wrong about the GetTickCount API and portrays Windows CE so negatively.

Apparently I’m behind on my reading, since the same author also wrote a previous article on the same subject, to which Mike Hall wrote a rebuttal on his own blog: http://blogs.msdn.com/mikehall/archive/2005/01/28/362498.aspx Reading the more recent article makes me want to write my own rebuttal.

GetTickCount is a pretty simple API from the caller’s perspective. Every millisecond the tick count increments, and you can use GetTickCount to retrieve the number of milliseconds since boot. Since GetTickCount returns only a 32-bit number, after about 49 days, the counter wraps. This is documented behavior, and to properly use GetTickCount you have to understand that. Typically GetTickCount is used to time the duration between two events, in which case you’re generally safe if you subtract two values; subtraction is safe in the presence of rollover. (eg. If you get a tick count of 0xFFFFFF00 before the rollover, and a tick count of 0x200 after the rollover, subtraction gives you get a difference of 0x300 as expected.) The only time subtraction can get you into hot water is if there’s a chance the time delta will exceed 49 days, because you may end up needing a difference that’s larger than you can represent in 32 bits. In which case GetTickCount is the wrong API for you. I guess in that case you would probably need to implement something using GetSystemTime and SystemTimeToFileTime. Applications that use GetTickCount get into trouble if they are subtracting over such a long time period, or if they are using something besides subtraction with the tick count. For example,

if (GetTickCount() > MyTickValue) { ... }

will also get you into trouble. If I were to guess, that’s where I’d say most applications probably go wrong using this API.

To help catch such errors in applications and drivers, Windows CE does a little thing on debug builds – it initializes the tick count such that it rolls over 3 minutes after boot. The author talks about our 3-minute rollover as if it’s indicative of a problem in the OS, when really it’s just a meager attempt to help catch bugs in applications. It’s not much help really, if you ask me, but it might help catch a bug or two. I’d love to improve on it, but it’s tough to arrange for the timer to roll over at a really useful time for testing your application. For example you might think we could create an IOCTL you could call, to set the timer at run-time. But making the timer jump at run-time could mess things up. Suddenly drivers and applications would think 47 days have passed, maybe network drivers time out, all your appointments fire, who knows… I’m making things up since I don’t really know how networking or appointments are implemented, but you get the idea. If you really want to be careful about testing your application or driver that uses GetTickCount, probably the best thing to do is create a wrapper that your code uses to call GetTickCount, and arrange for your wrapper to manipulate the times. I’d love to see suggestions, and if you can come up with something good we can do in the OS, hey, maybe we will take your advice.

So now I’ll describe how GetTickCount is implemented internally. First off, the function is technically owned completely by the OEM, though we provide as many implementations as we can manage, so that OEMs can use those. The implementation varies per CPU and per OAL, but in general, there’s a 32-bit counter, CurMSec, that is incremented once per millisecond with a timer interrupt. That millisecond timer interrupt is also used for other things, like scheduling threads.

To conserve power, when the kernel has no threads to schedule, the system goes into an idle state (implemented by the OEM’s OEMIdle function) where the timer interrupt is extended to a longer period. That allows the CPU to spend a longer time in a low-power state. The idle period is ended when an interrupt fires (the extended timer interrupt or some other interrupt). When the system leaves the idle state, OEMIdle updates CurMSec with the amount of time that the system was idle, using whatever timer the hardware has.

All the magic is really in the OEM’s implementation of OEMIdle. If you want some code to look at, see the CE 5.0 help article.

Now, back to the article. Here are my responses:

One of the things that floors me about this article is that it claims that GetTickCount counts downward, not upward. That is just plain wrong. I don’t know what gave the author that impression but the counter starts out at 0 and goes up. If you see any documentation or code that claims otherwise, please tell me and we’ll get it corrected.
The author also asks whether the counter “sticks” at the same value after rollover, which it doesn’t. He seems to imply that it does though.
GetTickCount also works just fine on 16-bit and 64-bit CPUs, though the author implies that it doesn’t.
The article brings up cases where counters are non-monotonic, where the counter jumps backwards by a few ticks. If that happens, it means the timer is not implemented correctly by the OEM. Most likely something in OEMIdle is not right. We do provide documentation of how to implement a timer (here’s some), standard implementations for different CPUs, and tests to verify that the timer is implemented correctly. But I’m sure some people would argue that we don’t do enough to help OEMs get this right, and maybe they’re right. Let’s discuss it. What else would you like to see? What trouble have you had?
All the other examples of badness that the author uses come from desktop Windows as far as I can tell, and they are problems with applications that use GetTickCount improperly, not with Windows itself.
You could argue that this is too complicated, that GetTickCount should just read the timer hardware straight, and scale to milliseconds. I think the main reason it was done this way was to standardize between hardware with a count-compare type timer and hardware with a count-down timer. It also avoids doing division to convert to milliseconds, especially 64-bit division since 64-bit timers are common.
The author actually suggests using a stopwatch to measure elapsed time during a performance test. Even if that was accurate enough, it’s a pain and it’s not automatable. How could you run tests regularly to make sure that nothing got worse? I fully support the author’s idea that you should loop many times, so that the performance test runs long enough that you could time it with a stopwatch. But that’s for reasons of repeatability, to get rid of variance. I don’t believe a stopwatch is the right answer. For timing code run-times, GetTickCount and QueryPerformanceCounter have satisfied every need I’ve seen so far.

I guess the thing I disliked the most was that the author took a list of implementation complications, application bugs from desktop Windows, incorrect information and “what if’s,” and turned it into a negative portrayal of Windows CE. Maybe I am just too sensitive about the product I pour so much energy into. Too many people assume that Microsoft developers don’t care, when the truth is very much the opposite.

Oh man has this discussion gotten long. Have I ever got just a few words to say about something? Oh well. Write back if you have opinions to add or if you think I got any technical details wrong.

Sue

Resolving Symbols Manually on Windows CE (ADDRESS --> SYMBOL)

sloh — Mon, 28 Feb 2005 19:37:00 GMT

You know you've been there. You are looking at a callstack, and the one function you want to know the identity of, failed to resolve symbols or has bad symbols. Dang! If only you knew what function 0x12345678 refers to, you'd find your bug.

First thing's first: you have to understand the Windows CE virtual memory layout. You have to understand what slots are, and recognize slot 0 & 1 addresses versus addresses from process slots. That’s why I blogged about that first. If you internalized all of those details already, then you’d know that 0x12345678 was inside the process that’s running in the slot starting at 0x12000000. In fact, it’s at offset 0x00345678 from the start of that .exe. Heck, you’re almost to the symbol already!

But this shows that the 2 main steps to resolving a symbol are: figuring out what module the address lies within, and figuring out what symbol lies at that offset within the module. If you need to get all the way down to the line of source code involved, you can figure that out too, by figuring out what line of source code is at that offset within the function.

Figuring out what module it is

To figure out what function or variable is at a particular address, first you need to know what module (DLL or EXE) is loaded into that chunk of RAM. So first you eyeball the address and decide whether it’s a DLL or an EXE. Since EXE code & data starts at the bottom of the slot, and DLL code & data starts at the top, it’s pretty easy to figure out. 0x12345678 is an EXE, 0x13987654 is a DLL.

Then you look for the DLL or EXE that is the closest match. There are a bunch of ways to do this. You can open up the Modules and Symbols Window, sort by address, and find the module that matches. If you are looking up the symbol for a function, sort by “Image Address Range.”

Modules and Symbols Window
Module         Image Address Range    Relocated Data Address Range
lmemdebug.dll 0x01AC0000-0x01AC4FFF
kbdmouse.dll   0x01BA0000-0x01BC1FFF
devmgr.dll     0x03EE0000-0x03EE9FFF 0x01FE9000-0x01FE92A4
fsdmgr.dll     0x03F50000-0x03F60FFF 0x01FF7000-0x01FF73F0
coredll.dll    0x03F80000-0x03FD8FFF 0x01FFB000-0x01FFBD64
filesys.exe    0x04010000-0x04041FFF
shell.exe      0x06010000-0x06022FFF
device.exe     0x08010000-0x08012FFF
gwes.exe       0x0C010000-0x0C0CBFFF
services.exe   0x12010000-0x12017FFF
nk.exe         0x81200000-0x8123DFFF 0x81FA0000-0x81FFD7BF
hd.dll         0x8123E000-0x8123FFFF 0x82002000-0x82002954
osaxst0.dll    0x81240000-0x81246FFF 0x82003000-0x82006328
giisr.dll      0x81249000-0x8124AFFF 0x82008000-0x82008504
osaxst1.dll    0x82042000-0x82046FFF
kd.dll         0x82047000-0x82060FFF

If you’re looking up the symbol for a variable, then for a DLL or EXE with a “Relocated Data Address Range” the variable would fall inside that range; for a DLL or EXE that does not have relocated data, the variable would fall inside the image address range. So you may need to sort on both in order to find the right address range.

Another option you have is that you can type “gi proc” or “gi mod” in the Target Control Window to get the location where the code is loaded. The exception for this is the kernel, nk.exe, which reports a different value under “gi proc” than where its code is loaded. (These lists don’t show relocated data addresses, so they’re only useful for looking up function symbols.)

Target Control Window

Windows CE>gi proc
PROC: Name            hProcess: CurAKY :dwVMBase:CurZone
P00: NK.EXE          03fb4002 00000001 c2000000 00000000
P01: filesys.exe     03f4b69e 00000002 04000000 00000020
P02: shell.exe       03dcae86 00000004 06000000 00000000
P03: device.exe      e3d60f6e 00000008 08000000 00000000
P05: gwes.exe        4389f8aa 00000020 0c000000 00000000
P08: services.exe    03497182 00000100 12000000 00000000

Windows CE>gi mod
MOD: Name            pModule :dwInUSE :dwVMBase:CurZone
M35: coredll.dll     83fb4794 7ffbffff 03f80000 00000000
M45: devmgr.dll      83d36250 00000008 03ee0000 00000000
M58: fsdmgr.dll      83fb1ea0 00000002 03f50000 00000000
M60: giisr.dll       8390c5dc 00000009 81249000 00000000
M65: hd.dll          83fb425c 00000001 8123e000 00000000
M75: kbdmouse.dll    8383be54 00000020 01ba0000 00000000
M76: kd.dll          82ce7e40 00000001 82047000 00000000
M80: lmemdebug.dll   83d60d30 0000003c 01ac0000 00000000
M98: osaxst0.dll     83fb44c0 00000001 81240000 00000000
M99: osaxst1.dll     83103d6c 00000001 82042000 00000000

Other possible options – if you have data from a Remote Kernel Tracker / CeLog output file (.clg), that file usually contains a “re-sync” (the output of the CeLogReSync API) where CeLog dumps a list of all the existing threads, processes and modules. (This list doesn’t contain relocated data addresses either.) Here is some sample output from the “readlog” command-line tool which parses CeLog output files, edited a little to fit this web format.

Readlog.exe Output File
--:--:--.---.--- : Marker, counter frequency=1193180 Hz, default thread quantum=100
0:18:33.578.716 : ProcessCreate, dwVMBase=0x81200000, NK.EXE
0:18:33.578.779 : ProcessCreate, dwVMBase=0x04000000, filesys.exe
0:18:33.578.870 : ProcessCreate, dwVMBase=0x06000000, shell.exe
0:18:33.578.901 : ProcessCreate, dwVMBase=0x08000000, device.exe
0:18:33.579.666 : ProcessCreate, dwVMBase=0x0C000000, gwes.exe
0:18:33.579.979 : ProcessCreate, dwVMBase=0x12000000, services.exe
0:18:33.581.421 : ModuleLoad, dwBase=0x82047000, kd.dll
0:18:33.581.450 : ModuleLoad, dwBase=0x82042000, osaxst1.dll
0:18:33.582.422 : ModuleLoad, dwBase=0x01BA0000, kbdmouse.dll
0:18:33.582.559 : ModuleLoad, dwBase=0x81249000, giisr.dll
0:18:33.583.481 : ModuleLoad, dwBase=0x03EE0000, devmgr.dll
0:18:33.583.494 : ModuleLoad, dwBase=0x01AC0000, lmemdebug.dll
0:18:33.583.673 : ModuleLoad, dwBase=0x03F50000, fsdmgr.dll
0:18:33.583.699 : ModuleLoad, dwBase=0x03F80000, coredll.dll
0:18:33.583.714 : ModuleLoad, dwBase=0x81240000, osaxst0.dll
0:18:33.583.727 : ModuleLoad, dwBase=0x8123E000, hd.dll
--:--:--.---.--- : Sync End Marker

Another option you have is to use the ToolHelp APIs (CreateToolhelp32Snapshot, etc) to programmatically gather this data.

It’s relatively straightforward to figure out, given an address, which module that address falls inside. Use the “Price Is Right” matching: whichever module is closest to that address, without going over, is the matching module. [“The Price is Right” is an old US TV game show.] Address 0x8123FFFF is inside hd.dll; 0x81240000 is inside osaxst0.dll.

The main thing you have to be careful about is recognizing addresses that are inside process slots, and converting them to slot 0 addresses. Address 0x09EE5678 is inside devmgr.dll, not device.exe. (ERRATA: See comments below for the error in this statement. I've left the text here unchanged so that you know what I was talking about. For the sake of the remainder of this write-up, just pretend devmgr.dll loads at address range 0x01EE0000-0x01EE9FFF instead of the range I listed in all the examples above.)

Figuring out what function or global variable it is

Once you know what DLL or EXE the address is within, you can simply subtract the DLL or EXE base address from the address in question, to get the offset of the address from the start of the DLL or EXE. So address 0x09EE5678 is at offset 0x00005678 from the start of devmgr.dll.

Then you look up the offset from the .map file for the DLL or EXE. You should already have the .map files from most modules you work with, but if you are missing one, it may be because the WINCEMAP environment variable must be set to 1 when the module is linked, in order to produce the .map file.

Here’s an excerpt from devmgr.map. The important things in the .map file are the function names and the addresses on the right side (highlighted):

Preferred load address is 10000000

0001:000044b0 _ConvertStringToGuid       100054b0 f devcore:devpnp.obj
0001:00004546 _FindDeviceInterface       10005546 f devcore:devpnp.obj
0001:0000456c _I_AdvertiseDeviceInterface 1000556c f devcore:devpnp.obj
0001:00004774 _PnpAdvertiseInterfaces    10005774 f devcore:devpnp.obj
0001:00004a9b _PnpDeadvertiseInterfaces 10005a9b f devcore:devpnp.obj
0001:00004ae5 _InitializePnPNotifications 10005ae5 f devcore:devpnp.obj

The address on the right is saying where the function would live in RAM if the DLL was loaded at its PreferredLoadAddress – which is stored at the top of the .map file, but is almost always 0x10000000 for a DLL. Which makes it easy, because you just chop off the top part of the address to find out where the function would live if the DLL was loaded starting at 0. So offset 0x00005678 from the beginning of devmgr.dll is the symbol I_AdvertiseDeviceInterface. Any offset between 0x0000556C and 0x00005773 inside devmgr.dll falls inside I_AdvertiseDeviceInterface. And, since devmgr.dll is loaded at address 0x03EE0000, any address between 0x03EE556C and 0x03EE5773 corresponds to I_AdvertiseDeviceInterface. So again you are basically using “Price is Right” matching.

EXEs are a little different, but even easier. Most EXEs will be linked with a PreferredLoadAddress of 0x00010000, but that is the spot the Windows CE kernel always loads an EXE at anyway. So there’s no subtraction involved – just use the address to the right of the symbol.

Once you get used to doing this stuff, it’s not that hard, just a little manual labor. In general the formula for going from ADDRESS à SYMBOL is:

(.map file offset) = (address) – (module base addr) + (module preferred load addr)

And remember to use “Price is Right” matching for the module and the symbol: whichever has the closest address without going over is the closest match.

Figuring out what source line it is

You can even get down to the source line if you have the COD files to look at. I have rarely had to do this kind of lookup, but it is occasionally useful. The two main uses I’ve had for it are: figuring out exactly which line of source/disassembly a crash or debug break occurred on, and figuring out which instance of a function call is involved in a callstack. For example if function Foo calls function Bar twice, and I’m looking at a callstack which involves Foo calling Bar, I may want to know which invocation of Bar it was inside Foo. The address given for Foo in the stack is the address where the jump to Bar occurred, so I can use the address given for Foo in the stack to figure out what line of code the jump occurred on.

COD files come from compiling the code with the WINCECOD environment variable set to 1. By default WINCECOD is not set, so you will not have COD files by default. And COD files are not shipped with the Platform Builder CDs or with the Windows CE source. So unless you have buildable source for the OS, this work is only possible for your own code.

You may have noticed that the MAP file lists the OBJ file the function came from. For example I_AdvertiseDeviceInterface came from the devpnp.obj file which was linked into devcore.lib. This means the code was in “devpnp.c” or “devpnp.cpp.” If you know where this source file is, you can go to the right directory, set WINCECOD=1, build –c, and then open the COD file from obj\%_TGTCPU%\%WINCEDEBUG%. It’ll be named after the C or CPP file – in this case devpnp.cod. The function name will appear several times in the COD file, but in only one case will the function name be followed by the word “PROC,” and this is the beginning of the assembly code for that function. Here’s a sample that’s munged a little from reality.

_I_AdvertiseDeviceInterface PROC NEAR ; COMDAT

; 348 : {

001e8     55          push ebp
001e9     8b ec       mov ebp, esp
001eb     81 ec 14 02 00
      00          sub esp, 532         ; 00000214H
001f1     a1 00 00 00 00    mov eax, DWORD PTR ___cke
001f6     53          push ebx
001f7     56          push esi

; 349 : ULONG result = ERROR_SUCCESS;

001f8 33 f6 xor esi, esi

As you can see, the source code and line numbers are included in the COD file next to the assembly code that implements the source. Each assembly instruction is labeled with an address. So the COD file provides the mapping from function offset to source line. Sometimes the assembly for the function is labeled starting with 0, and sometimes not. (I believe it may depend on which compiler is being used – meaning it varies by CPU type.) If not, you have to take the function start label into account. So in this example the function starts at label 0x001E8.

So the formula for the line of assembly to look for is:

(assembly label) = (.map file offset) – (function start addr from .map) + (function start label from .cod)

And to complete my example, subtracting the start offset of I_AdvertiseDeviceInterface 0x0000556C from the address 0x00005678, I find that the address I'm interested in is at offset 0x0000010C from the beginning of the function. Add that to the start label 0x001E8 of I_AdvertiseDeviceInterface from the COD file, and I am looking for the assembly line labeled 0x002F4. This line will tell me exactly what assembly code is at that address, and what line of source code that corresponds to.

Looking for a good tool for doing hex math

sloh — Fri, 25 Feb 2005 23:34:00 GMT

While I'm writing about all of these address tricks -- I figured I'd ask around if there's a good tool for manipulating hex values. I just use "calc" which comes with Windows; if you put it into scientific mode it will handle hex values for you. But it doesn't have very good shifting or bitwise functionality, and it always uses 64-bit values. Most of the time I want to subtract and get the 32-bit value, not FFFFFFFF FFFFFFFF. I like how easy it is to flip between hex and decimal in calc, but I wish I could do it without using the mouse.

So I figured I'd ask and see if anyone else knows of a better tool.

Windows CE Virtual Memory Layout for Debugging

sloh — Fri, 25 Feb 2005 20:15:00 GMT

I want to blog about how to resolve symbols manually, and realized I would have to assume that the reader would understand the CE VM layout. So I figure I’d better explain that first. You don’t need to know everything about the VM layout in order to work with Windows CE, but it can be handy at times to at least understand the basics. You need to understand some of it in order to be effective at debugging.

There are a few spots in this discussion that I could go into more “whys” but I avoided them because the answers aren’t necessary in order to debug. We can discuss them separately if they bug you.

Anybody who has worked with CE for very long knows that there are a maximum of 32 processes, and each process gets a maximum of 32MB of virtual address space to work inside. Part of the reason for that is because Windows CE keeps all processes’ address spaces available at all times, even when those processes are not running. What it means is that the lower part of the address space is split into 32MB “slots.” The important details to internalize are:

32MB in hex is 0x02000000
Slot 0 is 0x00000000 through 0x01FFFFFF
Slot 1 is 0x02000000 through 0x03FFFFFF
The last process ends at 0x41FFFFFF
Memory from 0x42000000 to 0x7FFFFFFF is mostly the “shared area” used for VirtualAllocs and memory-mapped files. In other words there will never be any symbols there.
Memory above 0x80000000 is “kernel stuff” – the kernel does run there, as well as DLLs that load into the kernel, like installable interrupt service routine (ISR) DLLs.

If you do the math, that adds up to 33 slots. Slots 0 & 1 are special slots that aren’t used by processes, and I’ll explain that in a bit. That leaves 31 slots for process to run inside – and the 32nd process is the kernel, which lives elsewhere in RAM somewhere above the 0x80000000 mark. [OK detail hounds, in CE 3.0 and earlier, the kernel ran in slot 1 instead of in the upper 2GB.]

Slot 0 is a special slot whose contents change depending on what process is running. The current process is always mapped into slot 0 in addition to having its own unique slot. Most of the addresses a processes is given to work with, are in slot 0. Get the address of a variable or function from an EXE in the debugger, and you’ll see something like 0x00012345. But if that variable is from a process that’s running in the slot from 0x12000000 to 0x13FFFFFF, that same variable will be present at address 0x12012345.

Try it out. Break into the debugger while your EXE is running. Use the Modules & Symbols Window to figure out what slot your process is in, then change a slot 0 address into one from the process slot and see if you get the same data.

Understanding slot 0 is actually very important for debugging. If you hit the “break” button at a random time, your process might not be running at that time. If you use the Watch Window to try to look at a global variable inside your process – the debugger might know what address the variable is at, without knowing the value of the variable. For example, John Eldridge already blogged a bit about how to use the Watch Window to look at a variable inside a specific module. Suppose you did this:

      Watch Window
      {,,helloworld.exe} &MyGlobalVar     0x00012345
      {,,helloworld.exe} MyGlobalVar      ???

If helloworld.exe isn’t the process that’s running at the moment you break into the debugger, the debugger knows what the address of the global variable is, but not the value. But you can manually map the slot 0 address to the right slot, if you know what slot the process is running in. If helloworld.exe is running in the slot starting at 0x12000000, then you can add the slot start address to the slot 0 address, like this:

      Watch Window
      {,,helloworld.exe} &MyGlobalVar     0x00012345
      {,,helloworld.exe} MyGlobalVar      ???
      *((DWORD*)0x12012345)               72

(yes, that’s not a DWORD-aligned address, please close your eyes if it bugs you)

Also, if you switch the debugger to helloworld.exe, it can figure things out. I usually use the Callstack Window and select a thread from helloworld.exe to switch to. Then the contents of the Watch Window might change.

      Watch Window
      {,,helloworld.exe} &MyGlobalVar     0x00012345
      {,,helloworld.exe} MyGlobalVar      72

The difference is that now the slot 0 address refers to the same slot the debugger is currently pointed at.

Slot 1 is also a special slot. All DLLs that are stored in the “MODULES” section of ROM load in slot 1. The code – not the data – for as many DLLs as possible loads in slot 1. Only ROM DLLs load in slot 1. If you drop a new copy of a DLL onto a device, to replace a DLL that was in ROM, that new DLL will not load in slot 1.

Check it out. Open the Modules & Symbols Window and view the addresses that various DLLs are loaded at. Some will be in slot 0, some will be in slot 1. A few will be in the kernel range.
So now look back at the addresses John Eldridge posted on his blog entry, and you can see that the DLL addresses he posted were all from slot 1.

Slot 1 stays the same no matter what the current process is. That’s why only code can load there; if two processes load the same slot 1 DLL, they will share the same code, but the two instances of the DLL will have different globals. The globals are stored in the process slot – so they will most often be referenced in slot 0 just like everything else the process owns.

The main point to all of this description of slot 0 & 1 is that you need to understand when the memory you need to look at is process-specific, and how to view process-specific memory. If a DLL loads into multiple processes, and you want to look at a global variable inside the DLL, the value could be different in different processes. You need to know which process you’re looking at, and how to change it if necessary.

One other detail that’s handy to know, is that EXE code loads at the very bottom of the slot (closest to 0x00000000), while DLL code & data load from the top of the slot down (closest to 0x1FFFFFFF). In between are heap allocations, thread stacks, small VirtualAllocs, and other “goodies” that don’t have symbols.

With practice you will develop one of the ninja debugging skillz: recognizing the location of an address just by looking at it. Here’s some practice:

0x00012345 is probably code or global variables from an EXE. 0x0056789A is probably not. Why? Too big, that’s a gigantic EXE. It’s more likely to be stack or heap, some other dynamically allocated memory, not EXE or DLL contents. It’s easy to ballpark if you can look at the actual size of the EXE.
0x03123456 is a slot 1 DLL. So is 0x02123456.
Address 0x3456789A is inside the process that’s running in the slot that starts at 0x34000000; you can use the Modules and Symbols window to figure out which process that is. But it’s most likely not code or global variables – for the exact same reason as 0x0056789A in my first bullet. This is just a “slot-mapped” version of the same address.
Address 0x019ABCDE is probably from a DLL. It's near the top of the slot so it's more likely to be a DLL than heap or stack. The same with 0x3519ABCDE. Get used to noticing whether the 2nd nibble of the address is odd or even.
Address 0x6789ABCD is not from a module. Maybe something from VirtualAlloc or a memory-mapped file, but you’ll never find a symbol that matches that address.
While we’re at it, okay, I am using bad sample addresses. 0x00012345 is not DWORD-aligned. Things are much more likely to be aligned on a 4-byte boundary: ending in 0, 4, 8 or C. Using consecutive numbers just seems more “friendly” to me, so I’m sorry if that bugs you. If you noticed, then you already have some ninja debugging skillz. It’s a good thing to be attuned to, since non-x86 processors can’t cope with accessing 4-byte values that are not 4-byte aligned.

Doug Boling, one of our MVPs, has written a paper that explains the virtual address space, and its implications, in much more detail. You should review that if you have more questions. I was just trying to cover the aspects that are important for debugging.

Next I’ll write about how to resolve symbols for addresses (how to go from address --> symbol), and after that I’ll write about the other direction, how to figure out an address for a particular function or variable (symbol --> address).

[Note: This applies to all Windows CE versions up through 5.0; with the exception I already noted about the kernel running in slot 1 in CE 3.0 and earlier.]

Debugging tricks

sloh — Fri, 25 Feb 2005 02:27:00 GMT

John Eldridge has posted some debugging tricks on his blog, and that has inspired me to write up some of my own. I'll start working on that... In the meantime I wanted to share a link to his tips: http://blogs.msdn.com/kitlfirst/

John has way more ninja debugging skillz than me, but I've picked up a few in my time at MSFT. I'll talk about how to resolve symbols manually using .map and .cod files, debugger extensions, and some ways to debug on a standalone device when a debugger's not an option.

Sue

It has been a while

sloh — Thu, 17 Feb 2005 20:21:00 GMT

I would use being busy as an excuse, except I'm always busy. ;-) Call it, lack of creativity about coming up with blog entries. I don't know how Mike Hall does it.

Anyway I've gotten a couple of questions about allocating large amounts of memory, that really come down to understanding the Windows CE virtual memory architecture, and especially to understanding the 32MB per process limit. So I thought I'd post a link to Doug Boling's useful paper, http://msdn.microsoft.com/library/en-us/dncenet/html/advmemmgmt.asp

CE memory division between "storage" & "program" memory

sloh — Fri, 10 Sep 2004 00:29:00 GMT

Somehow I've been involved in a bunch of discussions on this subject lately, so I figured I might as well lay out some of the issues.

Q: What are the ways to set the way memory splits between storage & program memory? Or in other words, how do I set the size of the object store?

A: You have 3 options: Set the size in OAL code by implementing the pOEMCalcFSPages function. Set the size in OAL settings by changing the value of FSRAMPERCENT in config.bib. Set the size at run-time by calling the SetSystemMemoryDivision function. The CE control panel "system" applet has a slider that users can drag to adjust the size; that just calls SetSystemMemoryDivision.

Q: What is the minimum size for the object store? If all of my device settings are stored on a persistent file system, so that I don't use the object store at all, can I set the object store to 0?

A: There is no way to completely get rid of the object store, even in the most recent OS version (5.0). The absolute minimum value you can use is 28KB. If you're using FSRAMPERCENT, use FSRAMPERCENT=0x00000004 which gives you 32KB. If you are actually using the object store for some data, I can't really give you a minimum. (eg. I can't say what the min. size required to boot successfully is.) I suggest you find it by booting and seeing how much of the object store is consumed, using GetStoreInformation or GetDiskFreeSpace.

Q: What is the maximum size for the object store?

A: 256MB. But since FSRAMPERCENT can't go that high, you'll have to use pOEMCalcFSPages for that.

Q: What is the minimum size for the program memory? How do I guarantee that my device can boot?

A: Again, there isn't any one minimum size I can give you. It varies based on what components you have included in your image, and based on your own code. What I suggest is that you find out what the minimum number of free pages was, after booting a system that has enough RAM. Use the "mi" command from the target control window, and look for the "minfree pages" value. That is a "low water mark" for the lowest number of free pages that were available at any one time. Subtract that from the total pages to find out the maximum number of pages that were used at any one time. I think the object store pages count in that total, so you should subtract the size of the object store, to get the maximum number of pages that were used for program memory. Then add some space for safety -- say 24 pages. (I just made that number up.) Then use that as your minimum size for program memory. So, that means use this formula:

(minimum RAM size) = (total pages) - (minfree pages) - (number of object store pages) + (safety margin)

Q: What is the maximum size for the program memory?

A: 512MB, minus the 28KB minimum for the object store. There is a way to extend past 512MB but that is a story for another day.

Q: How does FSRAMPERCENT work? It is confusing.

A: Yes, it is confusing. To help you understand where it is coming from: back in the days when memory was much smaller, OEMs wanted ways to give stair-step functions of memory, so that they could create multiple devices that vary only by the amount of RAM inside them. FSRAMPERCENT says, for each of the first four 2MB chunks of RAM, how many 4KB pages per 1MB should be given to the object store. So you could say, if my device has 2MB of RAM use this much; if it has 4MB use this other amount, etc. pOEMCalcFSPages is a later way to solve the same problem, and is much more flexible. Anyway here are some tips for reading and setting FSRAMPERCENT:

To interpret a value like 0x40302010, take each byte separately. Each byte represents the number of 4KB blocks per MB for 2MB. So 0x40 x 4KB = 256KB per MB for 2MB; so the top byte gives 512KB of RAM to the object store, and the remaining 1.5MB to program memory. Likewise 0x30 --> 384KB storage, 1.625MB program; 0x20 --> 256KB storage, 1.75MB program; 0x10 --> 128KB storage, 1.875MB program. Add them together for the total: 1.25MB storage and 6.75MB program. Check that they add to 8MB. Beyond the first 8MB, all the memory is used as program memory (I think). Don't take my word for it. :-)
To specify that N bytes of RAM (where 28KB <= N < 2MB) should be given to the object store, round to the nearest 8KB boundary [N], then set the bottom byte of FSRAMPERCENT to ([N] / (4096 * 2)). So for 28K, round up to 32K, divide by 8KB to get FSRAMPERCENT=0x04. To specify amounts between 2MB and 8MB set the bottom bytes to 0xFF, subtracting 2MB for each.
To specify that P% of RAM (where 0 <= P <= 100) should be given to the object store, set each byte of FSRAMPERCENT to ((P / 100) * (1MB / 4KB)), or in simplified form (P * 2.56). So for example to use 12.5% of RAM for the object store, use (12.5 * 2.56) = 32 = 0x20. So the 4-byte value of FSRAMPERCENT would be 0x20202020. To use 33.3% of RAM, set FSRAMPERCENT to 0x55555555.

Q: When I change the division dynamically using SetSystemMemoryDivision or by moving the control panel slider, does that value persist?

A: If your device always warm-boots, the object store will always be there so its size will not change on reboot. If your device always cold-boots, then the object store is recreated on every boot. The division is NOT saved anywhere so it will not persist across cold boots. Every time you cold boot you get whatever size is specified by FSRAMPERCENT / pOEMCalcFSPages. (The other variant of this question is: Why doesn't the registry value that stores the memory division seem to persist when the rest of my registry does? The answer is that there is no such registry value.)

Q: Does all of this apply to PocketPC / MS Smartphone?

A: Well, yes, except that PocketPC (and maybe SP, I'm not sure) has a background thread that dynamically moves the memory division in the background, so that you always have some storage memory and always have some program memory. So any changes you make on PPC/SP will quickly go away when that thread kicks in.