The Windows Mobile RSS (Reed and Steve Stuff) Feed : virtual memory

Mobile Buzz at Tech-Ed

ReedR — Wed, 13 May 2009 17:48:00 GMT

There is a lot of good stuff going on a Tech-Ed this week in LA. In you are here, come by and see us at the Windows Mobile booths. This is one of the first big events to really start sharing details and demonstrating the technology Windows Mobile 6.5.

Some highlights thus far…

· Take your pick from several sessions that overview WM 6.5 and show off new features. You can finally see the UI improvements in action and get a feel for the rich users experience behind this release.

· The Windows Mobile Marketplace team is here! Come by the Marketplace booth or attend one of the breakouts to get your Marketplace questions answered. Feedback Welcomed!

· Nickolas Landry spotlighted “The Windows Mobile Unified Sensor API” in his session earlier this week. This is a handy library up on Codeplex that you can use to take advantage cool hardware like accelerometers and light sensors on popular WM devices.

· Curious about this thing called a Widget? There is a good Hands-On-Lab (HOL) that will have you up and writing your own widgets in 15min. They are easy, they are connected, they are … well, Widgets and they are pretty cool.

· Rob Tiffany published a post recently that covered some techniques to maximize available memory space with managed applications. He followed it up this week with a session that covered this technique in depth. This is good stuff, but we also haven’t talked enough about some major changes that occurred in the platform under 6.1 to automatically help with the old Virtual Memory Monster. Doug talked about it here. Net, net… “While execute-in-place(XIP) DLLs are still loaded from the top of Slot 1, any DLL that is larger than 64 KB and isn’t XIP is loaded in slot 60. If the 32 MB of slot 60 fills up with DLLs, the OS uses slot 61 for the DLLs. This additional 64 MB of DLLs space should go a long way in reducing the problem of too many DLLs in Windows Mobile systems.” This is… HUGE.

· Maarten Struys gave a smoking session on Creating Location-Aware Applications on Windows Mobile. Maarten rocks…what else can I say.

…and that’s just the first two days!

Cheers,

Reed

Developer Patterns for Saving Memory on Windows Mobile

hegenderfer — Fri, 10 Apr 2009 20:44:33 GMT

Check out this new post over on Rob's blog around saving memory on Windows Mobile...very cool stuff :)

You can see him present this live at TechEd as well...

Slaying the Virtual Memory Monster - Part II

ReedR — Mon, 01 Oct 2007 23:46:00 GMT

Someday I’ll learn to write a simple blog post—a couple of paragraphs about something cool and edgy. Not today. There is no way to cover Virtual Memory in brevity and still get anything out of it so... we’re going to trudge through this. In my last post, we covered some important VM fundamentals. In this post, we’re going to get a little more “hands on” and actionable.

Let’s talk about ways to diagnose and deal with VM issues. Because the failure characteristics are somewhat “random” in nature and there are several ways to tackle them, the first step is to profile the VM space on the device and try to figure out what you are up against. This is a good thing to do for a number of reasons:
   1) You need know which processes are the biggest consumers of VM space reserved for loading DLLs
   2) You need to know how much VM space is still available inside YOUR process
   3) How you attack a VM problem depends on #1 and #2 above

Profiling the Virtual Memory Situation

When I first started debugging VM issues on Windows Mobile, I was surprised to find that we didn’t release any decent low level memory tools in the public WM SDK. There are a couple of OEM tools we provide to device makers, but since they are not included in the public SDK, they won’t help us much and I’m not going to use or discuss them here. The good news is that the basic APIs you need to understand VM use are publically exposed within the ToolHelp API set. These APIs work across all versions of the WM platforms and for the most part – this technique gets the job done. The best way to get started with this approach is via and old support tool we released way back in 2002 which remains available in our downloads -- called DumpMem. DumpMem simply uses the ToolHelp APIs to dump information about all the running processes into a text file and includes some basic heap data. The files are not particularly fun to work with (it’s a big messy text file), but I’ve seen several versions of home grown tools built around the underlying DumpMem code and parsing tools for the logs. I’ve written several myself. With a little practice, you can learn to make sense of these files and gain a lot of insight from them.

One concept I didn’t cover in the last post was “slots”. A slot is a basic unit for maintaining virtual memory within the Windows Embedded CE kernel. Without going into great detail, I’ll just say that each slot in the user address space is 32 MB and there are 32 slots set aside for the maximum number of running process. Two slots are special. Slot 0 is used for the current running process and Slot 1 is reserved exclusively for in-ROM components that have been included as part of the device image. Slot 0 is represented by the area below 0x02000000 and since it’s where the active process runs, that’s what we’re most interested in. For all practical purposes, that’s our 32MB virtual address space area we have to make it all work.

Virtual Memory in a “perfect world”

Our emulators are a good place to start. They don’t have the OEM extras -- code, drivers, startup software, etc., and thus, generally have more free VM than you’ll find on commercially available devices.

What are we looking for?

1) First we want to know where the lowest DLL is loaded in memory (regardless of process).
Since we established that every DLL has to load lower than previous DLLs, we’re basically looking to see how far that load position has gotten pushed down. All running processes that load DLLs can contribute to this position. This is often the biggest contributor in VM failure scenarios. When the DLL load position get pushed really low, it’s more likely to bump into a process heap (which is growing upward). When that happens, LoadLibrary fails and you won’t be able to load any more components in your process. This is also a very important metric to consider for all processes since the same low DLL load position has implications for all apps.

2) Next we want to identify the biggest free area between our heap (growing upward) and the lowest DLL load position in our process.
We want to know this because it tells us how much room is left to load DLLs in our process and is often the largest single piece of contiguous memory left for the heap (important if you need to allocate a sizable chunk of memory without going to the large memory area like an image, etc).

Step by Step – Create a Memory Dump using DumpMem

Let’s start with DumpMem and a “Hello World” application. I’m just going to let Visual Studio build a simple “Hello World” application and launch it on the WM6 Professional emulator. Then I’ll use DumpMem to dump the process and we’ll get messy.

I simply select the process I want to dump (HelloWorld_1), choose “Create File” and dumpmem writes everything into a file called “dumpmem.txt” in the root directory of the device. You can choose any process in the list, but DumpMem only grabs extra heap information for the specific you select. We’re not going to use heap data for this exercise, but sometimes, it’s good to see what kind of allocations are in there.

*DumpMem uses a fairly memory intensive API called CreateToolhelp32Snapshot to grab heap info about the selected process. If VM is already very tight on the device you are profiling, this API can cause DumpMem to fail. If this happens, you can change the call in the DumpMem source to pass in TH32CS_SNAPNOHEAPS flag. This won’t give you a dump of the actual heap data, but it will allow you to get the process info with all the loaded modules which is usually all you need.

The DumpMem log is broken into several sections:

· Platform Information

o This is simply the platform, CPU type, and version number

· Virtual Memory Dump (of the selected process)

o This is a map of the 32MB process space for the selected application. The format is address, size, state, type, use.

· Heap Dump of the selected process

o The heap(s) break down every chunk of raw heap memory it contains and shows you the address, size of the chunk, and if it’s in use (fixed or free)

· Processes and their modules

o An enumeration of all the running processes, all the components each has loaded, and then for each component – the name, load address, size, and module attributes

We’ll go into these sections in more detail later on, but first let’s just focus on our first goal – identifying the lowest DLL load position. To do this, skip over all the sections in the log file you just created to the last one – “Processes and their modules”.

What you find here is a list of all the running processes and more importantly – all the components (DLLs, MUIs, etc.) they have loaded along with their load positions. We’re mostly interested in components that are loaded into Slot 0 (below 0x02000000). One quick way to identify these components is by their attribute in the list. If the attribute does not include “XIP” we want to pay special attention because they will always be in this space and usually one of the lowest DLLs in memory (see bold items below). Another way might be to look at the load position of each component in something like Remote Process Viewer and see if it’s using loading below 0x2000000. If the component has an XIP attribute, then it’s a ROM based module and usually loaded into a special area in Slot 1 (outside the area we’re worried about). Sometimes, when the XIP region gets full, these components will start getting pushed down into Slot 0. When that happens, they affect our VM space the same way as non-XIP components. Anything below 0x02000000 that is not labeled XIP is what we’re primarily concerned about. If I look at the dumpmem.txt file I just created, that section looks something like the list below (I’ve trimmed off each module list after the first 3-4 items).

…

Processes and their Modules

Module File Attributes Legend

RO - Read Only

H - Hidden

S - System

A - Archive

C - Compressed

NK.EXE base address: C2000000

=========================

coredll.dll 03F4E000 96000 RO, H, S, XIP

coredll.dll.0409.mui 7FFE0000 15000 RO, S, XIP

hd.dll 88079000 4000 RO, H, S, XIP

…

filesys.exe base address: 04000000

=========================

rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM

vcefsd.dll 01990000 1C000 RO, H, S, XIP

ramfmd.dll 01A80000 9000 RO, H, S, XIP

…

device.exe base address: 06000000

=========================

rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM

serial_smdk2410.dll 019E0000 A000 RO, H, S, XIP

serdma.dll 01A10000 7000 RO, S, XIP

...

services.exe base address: 08000000

=========================

lap_pw.dll 01910000 18000 RO, S, C, RAM from ROM

rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM

obexsrvr.dll 02416000 A000 RO, H, S, XIP

…

gwes.exe base address: 0A000000

=========================

kbdmouse.dll 01960000 9000 RO, H, S, XIP

touch.dll 019B0000 A000 RO, H, S, XIP

deviceemulator_lcd.dll 01B30000 2D000 RO, H, S, XIP

…

shell32.exe base address: 0C000000

=========================

wlmhssearchbarcode.dll 01890000 8000 RO, S, C, RAM from ROM

wlmshared.dll 018A0000 3C000 RO, S, C, RAM from ROM

wlmuiframework.dll 018E0000 19000 RO, S, C, RAM from ROM

wlmtodayscreen.dll 01900000 6000 RO, S, C, RAM from ROM

siminit.dll 02090000 6000 RO, S, XIP

…

EmulatorStub.exe base address: 0E000000

=========================

coredll.dll 03F4E000 96000 RO, H, S, XIP

coredll.dll.0409.mui 7FFE0000 15000 RO, S, XIP

poutlook.exe base address: 10000000

=========================

ossvcs.dll 0300E000 42000 RO, S, XIP

oleaut32.dll 03050000 2F000 RO, H, S, XIP

ole32.dll 0307F000 2C000 RO, H, S, XIP

…

connmgr.exe base address: 12000000

=========================

rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM

ril.dll 02096000 11000 RO, S, XIP

cspvoice.dll 02125000 10000 RO, S, XIP

…

ConManClient2.exe base address: 14000000

=========================

devicedma.dll 01860000 B000 A

edbgtl.dll 01870000 14000 A

coredll.dll 03F4E000 96000 RO, H, S, XIP

…

HelloWorld_1.exe base address: 16000000

=========================

mscoree2_0.dll 01C70000 C4000 RO, H, S, XIP

netcfagl2_0.dll 0218C000 3B000 RO, H, S, XIP

mscoree.dll 021C7000 10000 RO, H, S, XIP

…

fexplore.exe base address: 18000000

=========================

ossvcs.dll 0300E000 42000 RO, S, XIP

oleaut32.dll 03050000 2F000 RO, H, S, XIP

ole32.dll 0307F000 2C000 RO, H, S, XIP

…

dumpmem.exe base address: 1A000000

=========================

toolhelp.dll 02E5F000 4000 RO, H, S, XIP

ossvcs.dll 0300E000 42000 RO, S, XIP

oleaut32.dll 03050000 2F000 RO, H, S, XIP

…

Step by Step – Build a list of the DLLs loaded below 0x02000000

If you make a list of all the DLLs in every process that have been loaded under 0x02000000 and order them by load position in memory, you get a pretty good idea of what’s going on with the low DLL load position and which processes have contributed to pushing it down to its current position. Our current list of all DLLs loaded under 0x02000000 looks like this:

Component –––––––––––––––– load position – size -- attributes

devicedma.dll 01860000 B000 A (lowest DLL in memory)

edbgtl.dll 01870000 14000 A

wlmhssearchbarcode.dll 01890000 8000 RO, S, C, RAM from ROM

wlmshared.dll 018A0000 3C000 RO, S, C, RAM from ROM

wlmuiframework.dll 018E0000 19000 RO, S, C, RAM from ROM

wlmtodayscreen.dll 01900000 6000 RO, S, C, RAM from ROM

lap_pw.dll 01910000 18000 RO, S, C, RAM from ROM

rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM

…

kbdmouse.dll 01960000 9000 RO, H, S, XIP

touch.dll 019B0000 A000 RO, H, S, XIP

serial_smdk2410.dll 019E0000 A000 RO, H, S, XIP

serdma.dll 01A10000 7000 RO, S, XIP

s3c2410x_wavedev.dll 01A20000 9000 RO, H, S, XIP

pwrbtn2410.dll 01A90000 4000 RO, H, S, XIP

pcmcia.dll 01AA0000 A000 RO, H, S, XIP

pcc_smdk2410.dll 01AB0000 9000 RO, H, S, XIP

pcc_serv.dll 01AC0000 F000 RO, H, S, XIP

nullcam.dll 01AD0000 C000 RO, H, S, XIP

nleddrvr.dll 01AE0000 4000 RO, H, S, XIP

irsir.dll 01B00000 6000 RO, H, S, XIP

emulserv.dll 01B10000 5000 RO, H, S, XIP

dmatrans.dll 01B20000 5000 RO, H, S, XIP

battdrvr.dll 01BF0000 5000 RO, H, S, XIP

backlight.dll 01C00000 4000 RO, H, S, XIP

asyncmac.dll 01C20000 6000 RO, H, S, XIP

scard.dll 01C40000 B000 RO, H, S, XIP

mscoree2_0.dll 01C70000 C4000 RO, H, S, XIP

Step by Step – Analyze the memory data as it applies to your process

Low DLL load position

This is a pretty small list of DLLs using a simple environment (it’s the WM 6.0 SDK refresh Pro emulator). You can see the lowest DLL (devicedma.dll) loaded at 0x01860000 (~24MB). You may also notice that if you add the DLL size to a load position, there may be a small gap before the next DLL loads in memory. In WM5, we had to load components on a 64K boundary. This means even if you loaded two DLLs that were 10K in size, they would eat up 128K of VM. In WM6, we use a 4K boundary which makes more efficient use of memory.

From our data, it appears that the next DLL loaded into memory (from any process) is going to find a home right under 0x01860000. That’s where the lowest module is loaded right now and the next one has to load underneath it. This answers our first question about where the low DLL load position is. As you can see, we’re already a long way down from 32MB since this is ~24 MB in the VM map. The low DLL load mark has already been reduced by 8MB. What pushed it down that far?

You could go back through the list of component under 0x02000000 and find the processes that loaded them. That would tell you which processes are the biggest culprits. If you did this, you would see that shell32.exe is responsible for loading several Windows Mobile Live components, but even so – the largest one is only about 240K. The real culprits are all the system components and drivers that have been pushed down into Slot 0 because slot 1 was full. This is very typical of commercial devices. That’s what pushed down the low DLL load position by 8MB. This number is highly variable for retail devices and something you typically don’t have a lot of control over. It’s also why if you have a big application that requires a lot of VM space to load DLLs, you need to be especially careful to profile your target devices.

What about our heap space?

To get an understanding of available heap space left, I usually look at the first section of the dumpmem.txt file called “Virtual Memory Dump” (left column below). This is a simple outline of your memory space and how it’s being used. If I identify the heap area between where my EXE is loaded and before the first DLLs appear in this VM Map, I can find the largest available free memory area left in my process.

Before we look at the virtual memory dump, let’s clarify one point. You will notice that all the memory positions seem to be offset from 0x16000000. You may recall that the base address for HelloWorld_1 was 0x16000000 (where it lives when it’s not the active process in slot 0). Just subtract 16000000 from all the memory positions if it’s easier to think about it in the context of Slot 0. Here we go…

Virtual Memory Dump of HelloWorld_1.exe

16000000 10000 F NA

*The first little block of memory is 10000 in size. This is the reserved position at the bottom of each process and will always be here for any process.

Next comes our EXE…

16010000 2000 R NA Image

16012000 1000 C RW Image

16013000 1000 R NA Image

16014000 1000 C RO Image

16015000 1000 R NA Image

16016000 1000 C RO Image

16017000 1000 R NA Image

*The memory from 16010000-16017000 is labeled as “image”. This is the bottom of the VM position and where our EXE is loaded. The lowest section labeled “Image” is always your EXE.

Next comes our Heap and Stack memory…

16018000 8000 F NA

16020000 1000 C RW Private

16021000 F000 F NA

16030000 D000 R NA Private

1603D000 3000 C RW Private

16040000 5000 C RW Private

16045000 2A000 R NA Private

1606F000 1000 F NA

16070000 5000 C RW Private

16075000 2A000 R NA Private

1609F000 1000 F NA

160A0000 D000 C RW Private

160AD000 22000 R NA Private

160CF000 1000 F NA

160D0000 20000 C RW Private

160F0000 F000 R NA Private

160FF000 1000 F NA

16100000 1000 C RW Private

16101000 2E000 R NA Private

1612F000 1000 F NA

16130000 F000 R NA Private

1613F000 1000 C RW Private

16140000 10000 C RW Private

16150000 F000 R NA Private

1615F000 1000 C RW Private

*The area from 1601800-1615F000 is dynamic memory being used by the process – heap, stack, etc. Notice there are a few free areas marked “F” but most of it is “Private” memory which has been committed or reserved. This is where the majority of your dynamic memory allocations occur and your heap expands upward.

16160000 1B10000 F NA

*At the top of your heap area you will usually find a big free chunk of memory. This is largest contiguous free chunk of memory available inside your VM space (Slot 0). In this case, it’s 1B10000 or ~27MB.

Next (at the top of the VM area working its way down) you will find any DLLs loaded in our process. The first one shows up at 0x17C70000…

17C70000 1000 R NA Image

17C71000 7000 C RO Image

17C78000 1000 R NA Image

17C79000 3000 C RO Image

17C7C000 18000 R NA Image

17C94000 1E000 C RO Image

17CB2000 1000 R NA Image

17CB3000 14000 C RO Image

17CC7000 1000 R NA Image

17CC8000 12000 C RO Image

17CDA000 5000 R NA Image

17CDF000 3000 C RO Image

17CE2000 4000 R NA Image

17CE6000 1000 C RO Image

17CE7000 5000 R NA Image

17CEC000 6000 C RO Image

17CF2000 4000 R NA Image

17CF6000 1000 C RO Image

17CF7000 9000 R NA Image

17D00000 4000 C RO Image

17D04000 4000 R NA Image

17D08000 2000 C RO Image

17D0A000 2000 R NA Image

17D0C000 B000 C RO Image

17D17000 1000 R NA Image

17D18000 2000 C RO Image

17D1A000 1000 R NA Image

17D1B000 1000 C RO Image

17D1C000 1000 R NA Image

17D1D000 1000 C RO Image

17D1E000 3000 R NA Image

17D21000 1000 C RO Image

17D22000 1000 R NA Image

17D23000 2000 C RO Image

17D25000 2000 R NA Image

17D27000 5000 C RW Image

17D2C000 8000 R NA Image

17D34000 EC000 F NA

17E20000 14000 R NA Image

17E34000 2000 C RW Image

17E36000 1A000 R NA Image

17E50000 1000 C RW Image

17E51000 53000 R NA Image

17EA4000 1000 C RW Image

17EA5000 58000 R NA Image

17EFD000 1000 C RW Image

17EFE000 8000 R NA Image

17F06000 5000 C RW Image

17F0B000 10000 R NA Image

17F1B000 1000 C RW Image

17F1C000 1A000 R NA Image

17F36000 1000 C RW Image

17F37000 E000 R NA Image

17F45000 1000 C RW Image

17F46000 1C000 R NA Image

17F62000 1000 C RW Image

17F63000 5000 R NA Image

17F68000 1000 C RW Image

17F69000 3000 R NA Image

17F6C000 2000 C RW Image

17F6E000 32000 R NA Image

17FA0000 40000 F NA

17FE0000 1C000 R NA Image

17FFC000 1000 C RW Image

17FFD000 3000 R NA Image

*Memory from 17C70000 up is labeled as “image”. This is memory used by DLLs – remember they load from the top of the VM space and work their way down. All I care about at this point is the lowest position used in my process. If I map address lowest DLL loaded at 17C70000 back into Slot 0 (subtracting the base address), I get 1C70000 which can be easily identified as mscoree2_0.dll in the module section of HelloWorld.exe.

From this information, we achieve our second goal. We know that there is a large free chunk of VM space (0x01B10000 bytes –or- ~27MB), between our EXE and the lowest DLL in our process. The lowest DLL loaded into our process is mscoree2_0.dll at 0x01C70000.

Making sense of it all

Let’s now regroup on our objectives and put it all in context. Our goals were:

1) Understanding the lowest DLL load position in all processes:
>>> devicedma.dll at position 0x01860000 (~24MB)

2) Identifying the largest chunk of available VM and lowest DLL in OUR process
>>> we found 0x1b10000 bytes FREE right above our heap and we located mscoree2_0.dll at 1C70000 giving us about 27MB of free space to work with.

This was simply example and done in a very clean environment so we’re still in a pretty healthy situation. Goal #2 tells us we have plenty of space to grow (heap) and plenty of room to load more DLLs.

Goal #1: The lowest DLL load position in OUR process is 0x1C70000 so you might expect the next DLL to load right under it. Considering #1 though, the next DLL would actually load significantly lower – just below 0x01860000. Remember that when our process loads another component, it has to find a position under every other component that has already been loaded by any process. This, in effect, restricts our heap range a bit. Where we had a giant contiguous chunk of almost 27MB worth of VM space—loading a component now would split up that area by positioning itself around 23-24MB in the map. Our heap can still grow upward and claim most of this available space, but it’s no longer single contiguous chunk. This becomes important when you have an app that might need to make a big allocation (e.g. – manipulating an image, media file, etc.). I say "most" of this space because if you are not running with full privileges (e.g. - untrusted on a 2-tier phone), then you are limited to available VM space under the low DLL load position -- even if there is plenty of free space up there!

This is a really simple exercise, but you can imagine how complex this can get with a lot of applications running – especially if they load a bunch of other components. While there may only be 32 processes running, don’t forget that services.exe and driver.exe can become quite full with dlls that load on boot. This can contribute to pushing down that low DLL load position. Once the DLL position gets pushed WAY down, the available area left to grow your heaps can get quite constrained. If you start running into problems with VM issues or need to get an idea of what is going on, you can use the above techniques to get an idea of what you have to work with (and why it might be failing).

Where’s the smoking gun?

We already noted that when applications experience VM related failures, there’s not always a smoking gun. The failures can be appear quite random. Let’s talk about some of the most common failures and contributors.

Components that fail to load - The most common failure occurs when an application attempts to load a dependant component – either on startup or through a call to LoadLibrary. There has to be enough space between your heap and the low DLL load position to succeed, so when the DLL load position gets pushed too far down, LoadLibrary will fail. At this point, an application can fail to start or a manual call to LoadLibrary will return an error. The actual affect on an application can vary depending on how it handles this failure.

Memory allocations that fail - When a heap can no longer expand, memory becomes too fragmented, or there is insufficient space for a large contiguous allocation-- memory allocations can begin fail. Few developers write to code to adequately check for memory failures so this very often results in unhandled exceptions for the application (e.g. – crash).

Dynamic failures - It’s pretty easy to take responsibility for problems in your own application—failure to start, exceptions, etc., but it’s even more complex that. VM problems commonly extend into other processes because of the DLL loading and memory rules. How does this happen?

Because every process can contribute to the low DLL load position, this means that the order in which they start can make a big difference. Suppose the low DLL load position has been pushed down to the point where only 10MB of space is left to load more DLLs. What happens then if you have three application—“A”, “B”, and “C” that all load 5MB worth of unique components. If you load “A” and “B”, then “C” fails. If you load “B” and “C”, then “A” would fail.

Part of the challenge today is that VM health is very much a little ecosystem with many contributors. The OEM shares responsible for delivering a device that doesn’t eat up all the VM space before you install the first app. The developer shares responsible for not taking more than their share of resources. The end user shares responsible for all the apps they install. Microsoft shares responsibility for mitigating this problem at a platform level so nobody has think about it. In the not so distant future, we will likely adopt changes in the platform (like we did with CE6) that expand the VM range so that this isn’t an issue anymore. Right now, we have to work with what we have.

Tactics developers use to address VM problems

· Profile the VM space first. You cannot solve this problem unless you know what is going on with your memory.

· Static linking is your friend. When you statically link code, it becomes part of thee EXE which means it loads in the bottom of the VM space (the best possible area). You don’t have to worry about DLL load rules and you don’t impact the VM of other processes. You may end up using a little more RAM if multiple apps use the same code, but the trade-off is probably worth the headaches you can avoid. Plus, you avoid extra signing costs by eliminating DLLs.

· Start order matters. While more of a band-aid than a solution, controlling the load order of processes on a device can help you work around VM issues. For this to work, you really need a good understanding of the memory profile of the device as well as which processes are contributing to the problem. The first app gets the most DLL space, the second gets the slightly less, etc. By controlling which apps start first and/or disabling services or drivers you don’t need – you can sometimes avoid VM problems.

· Merge small DLLs. Windows Mobile 5.0 had to load components on a 64K boundary which meant that every a small DLL used a minimum of 64K. If you have a bunch of small DLLs, it may make sense to merge them to save space. Windows Mobile 6 loads on a 4K boundary which makes much better use of space.

· Move resources to resource-only DLLs (dll’s with no entry point). Resource only DLLs are loaded up in slot 63 and don’t use up your valuable VM space in slot 0.

· For large memory allocations, consider using VirtualAlloc and the Large Memory Area (aka – High Memory Area). Allocations of 2MB or greater use space well outside of slot 0 and leave your process with more room to work.

· Trim your code. Make it smaller and lighter and save on VM space

In Summary

I’ve rambled long enough about the Virtual Memory Monster. Somewhere this is a blog post turned into whitepaper. Hopefully those of you who read this now have a better understanding of the creature and some skills you can use to battle it. Like most nasty, bad-guy type, it’s more misunderstood than evil. Hopefully you never run into it, but if you do – you have some tools under your belt.

If you guys want to dig into this in more detail, let me know. I could blog through a real world problem or two to help you get the hang of it.

Some of you will undoubtedly want to know more about how NETCF optimizes memory inside of this limited area. You will notice that if you load a NETCF assembly, it doesn’t use up DLL space like native DLLs. NETCF goes to great lengths to make good use of memory transparently. Since there are already some great posts on this topic, I’ll simply reference them here:

http://blogs.msdn.com/stevenpr/archive/2005/12/12/502908.aspx

http://blogs.msdn.com/mikezintel/archive/2004/12/08/278153.aspx

Next up…. Power Management or maybe IE Mobile? Both of these areas seem to be generating a lot of interesting support activity right now.

Cheers,
Reed

Slaying the Virtual Memory Monster

ReedR — Fri, 31 Aug 2007 07:51:00 GMT

If you have ever heard one of my talks on Windows Mobile development, you may remember me ranting a bit about Virtual Memory issues. It’s one of those topics that many developers never need to worry about, however, if you are one of the devs that have fought this battle – you can appreciate why I think it deserves some attention. Especially now because…

It’s easy to be a Windows Mobile Developer

When developers first start working with Windows Mobile, they are often surprised at just how much they can do and how much knowledge they can reuse. Built on top of Windows CE, the platform inherits all the flexibility and power of a proven operating system used for years on small footprint and special purpose devices. If you have done any desktop development, you will find most of the same APIs and libraries you have come to depend on. Windows Mobile may have fewer or smaller versions of the APIs, but many are nearly identical to the desktop. In the same respect, the Compact Framework provides a comprehensive subset of the full .NET Framework functionality. All in all, it’s a natural extension of what you already know how to do. Visual Studio enables mobile development beside all your other projects so -- everyone can be a Windows Mobile developer. This is a good thing.

It’s the same but it’s different

As similar as Windows Mobile and CE development is to the desktop world, there are some fundamental differences that new mobile developer should take time to understand. Remember, the WM 5 and 6 platforms are built on CE 5.x— designed with a very small memory footprint in mind. Most notably, the virtual memory space is considerably smaller than our desktop platforms and DLLs have special requirements for loading into the same memory area across applications. Did I mention that the platform may automatically shutdown applications when memory gets low and that only 32 processes can run at the same time? Many of these differences are easy enough to accommodate, but if you don’t take the time to learn some of these unique rules of mobile development, you may find yourself face to face with the nastiest of these -- the Virtual Memory Monster.

Did someone say monster? Yeah, well… we could just say “a rock and a hard place” but you wouldn’t remember that post and it isn’t very exciting is it? If we’re going to tackle a problem, let’s give it a name. The “Virtual Memory Monster” is actually a condition when your process runs out of VM space long before your device runs out of RAM. As a result, you can see a number of “memory” related failures that never really point to a specific piece of code. Sometimes they don’t even point to a specific application. It’s a difficult condition to address even when you know what is going on. To the novice WM developer, it can be just plain baffling. It doesn’t happen a lot, but it happens. Let’s talk about it…

First, let’s agree on the terms

This is not a new topic. It’s been around for years with previous versions of CE and Windows Mobile, but it still may be the most misunderstood blocker in the Windows Mobile handbook of “gotchas”. It can be tricky to talk about, so we need to lay down some fundamentals. I’m going to take a few liberties in my explanation for the sake of keeping it simple, but I encourage you to read the technical papers linked to the end of this post for deeper insight.

· Virtual Memory – When I talk about Virtual Memory, I’m really talking about addressable memory space. Think of this as an abstracted “work area” for a process. On the desktop, we had a 4GB address space and each application would get a 2GB area by default. On Windows Mobile, each application has to work with a 32MB slot. That means, in a perfect world, you have a 32MB VM area to leverage in your process space.

· RAM – RAM is the physical resource each process consumes to fill memory requests. Your process may have a 32MB virtual address space, but you don’t see 32MB of RAM disappear from the system when you start a process right? When the process loads resources and commits memory, RAM is then “consumed” to fill this need. For example, each process starts with a default heap that is created within your virtual memory area. It grows as your application allocates objects and as a result, consumes memory out of the available system RAM and begins to use up your 32MB address space.

· RAM vs. VM - RAM and VM are two completely different categories of memory that are sometimes (and mistakenly) used interchangeably. They have different failure characteristics when they run out. When you run out of RAM, you are out of physical memory. When you run out of VM, you run out of space to use the memory. Having tons of RAM won’t help your process if the VM space is full. Similarly, you can run out of RAM and still have plenty of VM space left in a process.

Yet another way to think about it…

Imagine a conference room with a limited number of seats. Let’s make a rule that if you want to get any work done, it has to happen in the conference room. A WM process is like the conference room. It’s where the “work” happens and the virtual address space is much like the seating arrangement. You may have hundreds of employees, but if they don’t fit inside the room and have a seat, you can’t use them – just like RAM won’t do you any good if your process VM space is full.

The devil is in the details…

A large, resource hungry application could simply run out of VM space by using a lot of RAM and/or loading a lot of components. Those are pretty easy to spot and deal with because they stand out like a sore thumb. The real problem comes from a combination of contributing factors that are quite harmless by themselves but problematic when mixed together.

1) As memory is getting bigger and cheaper – devices are shipping with more of it. When you can actually allocate 32MB worth of RAM and still have some to spare, then it’s easier to exhaust your VM space.

2) Developers are building desktop sized apps for Windows Mobile without thinking about (or worrying enough) about memory implications on each other

3) Windows Mobile inherits CE’s age-old technique for loading DLLs

Technical Explanation

All the contributing factors share responsibility for VM related failures, but the little “optimization” technique that CE used to be efficient sort of becomes the bad guy. Without going into great detail about WHY it works the way it does, let’s just jump to the HOW. I’ll first say that some DLLs loaded into ROM are not really part of this problem. For the most part, most live in a special place in memory between 0x02000000 – 0x03ffffff and specifically set aside for ROM components outside the VM area we’re worried about. Managed DLLs are also special cases (we’ll get to that later). All other DLLs and components (which are not part of the device ROM) that are loaded into your process space deserve the special attention. When they are loaded, they must be mapped into exactly the same VM position in any process that uses them. To do this they all share some restrictions about where and how components get loaded. A DLL cannot be loaded into a position being used by any other DLL. (pause and ponder) Let’s walk through it…

A Windows Mobile process space starts with 32MB of virtual memory. When you start an app, the EXE is loaded at the bottom of this space (above a little reserved spot) starting at 0x10000 (64k) and grows upwards (static data, heap, stack, etc.). The heap will grow upward into the process space as you start allocating objects in your app. Everything grows up – from the bottom of the space to the top except DLLs. DLLs are special -- they are loaded at the very top of the address space under any other previously loaded DLLs and work their way down – always considering the load position of other DLLs and loading at an address underneath them. In a perfect world, this all starts up around position 0x2000000 (32MB). I say “in a perfect world” because there is almost always other components loaded ahead of you by the OEM, operator, and other applications. Realistically, the starting DLL load position is often somewhere between the 14-18MB mark on a freshly reset, commercial device and can vary greatly depending on what “ships” on the device. Net, net -- your free VM (“work area”) can vary and you have two sides of the VM space – one growing up and one growing down. When they meet – it’s often “game over” for loading any more components into that process and the failures can start. The platform tries really hard to make it all work automatically and transparently. By loading DLLs at the top of the address space and at a location under any other system wide DLLs, it usually avoids collisions and conflicts across the VM space – but not always.

For example – when process 1 loads a “A” DLL, it loads in the top of its VM space. The next “B” DLL has to load below the first one. If process 2 now loads a “C” DLL, it loads in the new process but it does so in a VM position below the DLLs from process 1 – keeping space open in case process 2 needs to load “A” or “B” at some point. It also means that Now if process 2 loads a C.DLL into memory at 0x00120000 – and process 1 wants to load the same DLL, then VM at 0x00120000 needs to be free in process 1 as well. If the heap in process 1 grows beyond 0x00120000 and then process 1 attempts to load a DLL it fails. Why? The VM area where the DLL must load has been used. You can imagine how complicated this can get with 32 processes – all pulling in DLLs that have to consider each other’s load positions.

What makes this a real challenging problem is that no single application is responsible for the issue. Since it’s perfectly legal for any application load a DLL and every other app has to load components in a position lower than its predecessor, it becomes a little dependant ecosystem. The problem is typically a result of multiple apps all contributing “a little” to a larger issue. If you are the last guy to load your components, you may have to load them very low in the address spaced. Now if your application is “big” – your process may already consume much of the VM space by simply loading the EXE and setting up the heap (remember, it grows up). Your heap may have already grown into the addressable area you need to load that DLL. This is the most common failure scenario.

DLLs are treated like VIPs

Back to the conference room analogy… think of a DLL like a VIP you invited to sit in on your meeting. You give him the best seat in the room (top of the memory space) and let everyone else in the building know he is here. Word gets out and so others teams leave an empty seat in their conference rooms so that if our “special guest” decides to attend another meeting, there’s an open seat right where they expect it. Maybe other VIPs show up and pretty soon, you have a bunch of seats reserved. Then you have to use the reserved seats because space gets tight… and the VIP shows up! Sparks fly! This is similar to how Windows Mobile treats DLLs in when they are loaded into a process space.

Why don’t desktop developers struggle with this?

Running out of VM space is not a new problem nor a mobile specific problem, but there are reasons desktop developers rarely deal with it. First of all, this problem is hard to create when your address space is significantly larger than your available RAM. In the desktop world, Windows used a 4GB model with 2GB of that space dedicated for each process. It’s fairly rare for a single application to use 2GB of RAM. Many systems didn’t even have this kind of RAM available. Most desktop systems would run out of RAM or start paging to disk (setting off a big red flag) before they ever filled up the VM space. When you load a DLL in the desktop world, you don’t have to worry about DLL load positions.

What makes Windows Mobile particular different from the desktop is that it has a small 32MB VM space and special rules about how DLLs load within it. A 2MB DLL loaded in a desktop process is a drop in the VM bucket and has little impact on anything. A 2MB component loaded on Windows Mobile is a sizable chunk of your VM and can place pressure on other apps because of DLL loading rules. Devices are starting to ship with more RAM that makes it easier to fill up a VM space. Combine plentiful RAM, resource intensive applications, and DLL load restrictions and you have the makings for some interesting VM problems. By the way – CE 6 changes all this, but Windows Mobile (both versions 5.0 and 6) is still built on CE v5.x.

Identifying the failure…

When you run out of RAM, it is easy to understand and identify. The system tells you that memory is low and you can observe the memory status easily in system information. Running out of VM space is… well, nearly invisible. RAM looks fine and there is no way to visibly determine or verify VM issues within the UI. Components start failing to load so apps may crash or fail to start. Drivers may fail to work as expected. Ever have to disable Bluetooth to start an application? Ever had to start applications in a certain order to make it all work? While not the only cause, VM issues are often the cause of these failures and they can leave you scratching you head and going in circles with a debugger.

So who needs to be concerned and how do you avoid/debug this problem? … I mean, slay the monster?

Every developer should at least understand the concepts—VM size, DLL load rules, and the size of their application footprint. If you are building or loading native components from your application it’s especially important. If you have a memory intensive application it’s important. If you are building out an enterprise device configuration, you should do some investigation to make sure all the apps you load are good “memory” citizens and don’t create problems for each other.

It’s too much for one post so I wanted to touch on some fundamentals first. In the next part of this post, I’m going to cover examples, techniques, and tools you can use to identify, profile, and work around common VM issues. In the meantime, I suggest a couple of additional articles on the CE and WM memory architecture.

Windows CE .NET Advanced Memory Management

http://msdn2.microsoft.com/en-us/library/ms836325.aspx

Effective Memory, Storage, and Power Management in Windows Mobile 5.0

http://msdn2.microsoft.com/EN-US/library/aa454885.aspx

Cheers,

Reed