HoppeRx - the cure for your ailing device

A community site dedicated to the support of device problems found by Hopper

Saving VM by using OEMDRIVERSHIGH package

Saving VM by using OEMDRIVERSHIGH package

  • Comments 15

The release of Windows Mobile 6 (WM6) gave us a few more tools for dealing with Virtual Memory (VM) issues. Compaction of slot 1 is new for WM6, in previous releases all slot 1 modules were aligned on 64k boundaries, causing VM address space to be wasted when modules sizes aren’t perfectly divisible by the alignment size of 64KB were mapped into slot 1. The new alignment size of 4k allows more modules to be packed closer together, thus better compaction.

To illustrate this point, Figure 1 below shows 4 files totaling 142k in size taking up 320k using 64k alignment. Figure 2 shows the same 4 files only taking up 144k using 4k alignment. The white space is unused memory.

Along with compaction, the WM6 release also provides a way to map OEM driver’s modules into slot 1 with the package OEMDRIVERSHIGH. This is important because all the modules that do not fit into slot 1 rollover into slot 0, and thus reducing available VM for all slots. Further exacerbating the situation of modules rolling over into slot 0 is the fact slot 0 aligns modules on 64k boundaries. The OEMDRIVERSHIGH package enables OEMs, SVs, and system integrators to change the packaging priority of OEM drivers, which is typically near the end in the list of packages processed by makeimg.

When it comes to reconfiguring modules, OEM’s don’t have the ability to rearrange packages or to change the modules that go into a particular package. All that is available at this stage of building an image is reconfiguring the modules that go into the OEMDRIVERS package. Now with the addition of OEMDRIVERSHIGH package an OEM can target selected DLL’s to be processed, which will push larger modules out of slot 1 and into slot 0.

 

Optimizing OEMDRIVERSHIGH usage

To save VM using OEMDRIVERSHIGH we are going to identify the modules at the bottom of slot 1 on the verge of rolling over into slot 0. Through careful comparison of these modules and those currently in the OEMDRIVERS package we will reassign OEMDRIVERS modules to OEMDRIVERHIGH. By doing so the modules in OEMDRIVERSHIGH will be processed before the modules at the bottom of slot 1, and through this careful calculation the OEMDRIVERSHIGH modules targeted for slot 1 will rollover files on the slot 1 / 0 boundary. The VM savings will come from rolling over larger modules that are 100’s of KB into slot 0 which is 64KB align and filling the freed slot 1 space with smaller modules. It is the relocation, for example, of three 16KB modules to slot 1 that will free 192KB in slot 0 and only consume 48KB in slot 1.

To achieve this, the first place to start is analyzing the output of makeimg from a baseline build. The output isn’t always captured so you many need to run it once capturing the output (makeimg > makeimg.out 2>&1). Search through makeimg.out and look for “fixing up”. These are modules rolling over from slot 1 into slot 0. Below is an example of modules that just made it into slot 1 and modules that started rolling over into slot 0. This list of modules will be different for every project based on SYSGEN, BSP flags, IMG flags and the type of image.

Using the data from makeimg.out we will build a table to calculate the size of each module sitting at the bottom of slot1. Later we will use this data to determine how many modules to move into OEMDRIVERSHIGH, which will cause these modules to rollover.

 

                                                        FIGURE 3

 

Module cespell.dll          at offset 01d8a000 data, 0210b000 code

Module MsgStore.dll         at offset 01d89000 data, 02106000 code

Module msim.dll             at offset 01d87000 data, 020e4000 code

Module mscoree.dll          at offset 01d86000 data, 020d4000 code

Module netcfagl2_0.dll      at offset 01d85000 data, 02099000 code

Module netcfd3dm2_0.dll     at offset 01d84000 data, 02072000 code

Module ccoredrv.dll         at offset 01d83000 data, 02051000 code

Module ccoreutl.dll         at offset 01d82000 data, 02042000 code

Module cspvoice.dll         at offset 01d81000 data, 02032000 code

Module simsec.dll           at offset 01d80000 data, 02026000 code

Module sms.dll              at offset 01d7f000 data, 02020000 code

Code space full, fixing up supsvcs.dll to ram space

Code space full, fixing up btagsvc.dll to ram space

Code space full, fixing up BthAGPhonebook.dll to ram space

Code space full, fixing up celltsp.dll to ram space

Code space full, fixing up cplphone.dll to ram space

Code space full, fixing up phone.dll to ram space

Code space full, fixing up ril.dll to ram space

 

Taking the modules code offset from Figure 3, and subtracting it from the code offset of the previously listed module will determine the slot 1 space that can be recouped. (The data segment is need regardless of what we do and will remain the same.) So for msim.dll the code size is 0x02106000 – 0x020E4000, which is 136KB. The rollover order will be the same for every run of makeimg, the only exception is when slot 1 is low on space, it will pack in modules small enough to fit into remaining space. This is good news because one less module in slot 0 saves an additional 64KB. The consistent rollover order enables us to predict which files will rollover and in what order, this is illustrated in Table 1. The last two columns of this table illustrate the total space saved in slot 1 and total space used by these modules as they rollover. Notice how sms.dll consumes 24K of space in slot 1 and will consume 64K in slot 0.

 

Table 1

 

Rollover Order

Name

Space used

in slot 1 (KB)*

Space used

in slot 0 (KB)

 if rolled over*

Incremental space freed in slot 1 (KB)

Incremental space used in slot 0 (KB)

1st

sms.dll

24

64

24

64

2nd

simsec.dll

48

64

72

128

3rd

cspvoice.dll

64

64

136

192

4th

ccoreutl.dll

60

64

196

256

5th

ccordrv.dll

132

192

328

448

6th

netcfd3dm2_0.dll

156

192

484

640

7th

netcfag12_0.dll

236

256

720

896

8th

mscoree.dll

64

64

784

960

9th

msim.dll

136

192

920

1152

10th

MsgStore.dll

20

64

940

1216

11th

cespell.dll

160

192

1100

1408

*Calculated with equation roundup(size/slot alignment)*slot alignment; sms.dll size in slot 0 is roundup((0x02026000 – 0x0202000)/1024/64)*64.

 

Now we need to generate a list of OEMDRIVERS modules that can be relocated into the OEMDRIVERSHIGH package. A dir of %_FLATRELEASEDIR%/OEMDrivers_PACKAGE_FILES/ will result in a rough list. Further refine the list by making sure the modules are not being filtered out of the image, so search makeimg.out to verify their presence.

The next step is to determine the module size that will be used in slot 1. For DLLs and EXEs this can be done with the command dumpbin /headers <DLL name> and subtracting the size of .reloc from the total size of the image. For the example below we get 0x5000KB (20KB).

 

D:\_FLATRELEASEDIR>dumpbin /headers gpio.dll

  ...

             6000 size of image

  ...

        1000 .reloc

  ...

 

Table 2 is a list of modules going into OEMDRIVERS package that can be moved into the OEMDRIVERSHIGH package. The table is sorted by a ratio of wasted VM / module size, size is taken from the dumpbin output. The columns “Total space freed in slot 0” and “Total space used in slot 1”list the sum total of space used by that module and the others that precede it in the table based on the compaction of each slot. The column “Wasted VM in slot 0” illustrates the VM not utilized by 64KB alignment. To populate “Modules from Table 1 rolled over into slot 0” we compare the “Incremental space freed in slot 1” column of Table 1 and the “Total space used in slot 1” of Table 2. This comparison determines how many modules we can put into OEMDRIVERSHIGH thus causing modules from Table 1 to rollover into slot 0. The intersections of the two columns are marked with an asterisk to indicate the optimal amount of savings to get an additional module listed in Table 1 to rollover. As illustrated in this table, it would only take one module (gxdma.dll) to get one module from Table 1 to rollover. However if we wanted to rollover two modules we could relocate the first 2-4 modules, four being to optimal number.

 

Table 2

 

Name

Module size (KB) *

Total space freed in slot 0 (KB)

Total space used in slot 1 (KB)

Waist in 64k Slot (KB)

Ratio
Waste/Size

Files rolled over into slot 0

gxdma.dll

16

64

16

48

3.00

1*

usbOtg.dll

16

128

32

48

3.00

2

nleddrvr.dll

16

192

48

48

3.00

2

keypad.dll

20

256

68

44

2.20

2*

backlight.dll

20

320

88

44

2.20

3

battdrvr.dll

20

384

108

44

2.20

3

gpio.dll

20

448

128

44

2.20

3*

sdioclient.dll

24

512

152

40

1.67

4

kbdmouse.dll

36

576

188

28

0.78

4*

drvr005.dll

76

704

264

52

0.68

5

drvr001.dll

40

768

304

24

0.60

5*

uart.dll

40

832

344

24

0.60

6

touch.dll

44

896

388

20

0.45

6

drvr002.dll

48

960

436

16

0.33

6

drvr003.dll

48

1,024

484

16

0.33

6*

ddi.dll

536

1,600

1,020

40

0.07

11*

rilgsm.dll

596

2,240

1,616

44

0.07

 

usbfn.dll

60

2,304

1,676

4

0.07

 

drvr006.dll

368

2,688

2,044

16

0.04

 

drvr004.dll

64

2,752

2,108

0

0.00

 

* To achieve optimal savings when rolling over this number of files from slot 0 push this file and all pervious files into OEMDRIVERSHIGH

 

We can now use the optimal number of modules to relocate to estimate VM savings. We can assume VM usage in slot 1 stays the same, though it isn’t exactly the same as the relocated modules aren’t exactly the same size as the ones being rolled over. Finally we compare the differences in slot 0 usage. Table 3 brings together data from Table 1 and optimal data from Table 2 to calculate VM savings. Looking that the data under the column “Compaction savings in slot 0 (KB)” we can see a clear point where causing 6 modules to rollover will save 384KB in VM. We will relocate the first 15 modules of table 2 so that six modules of Table 1 rollover. We should see about 384KB of VM savings. It is fathomable, because VM is conserved in 64KB chucks that there are multiple points of optimal savings, if this were to happen in your optimization you should implement each scenario to see which one results in the better optimization.

 

Table 3

 

Space freed in slot 0 by relocation (KB) (From Table 2)

Space used in slot 0 by rollover (KB)

(From Table 1)

Compaction

savings in slot 0 (KB)

Modules rolled over

Modules

relocated to OEMDRIVERSHIGH

64

64

0

1

1

256

128

128

2

4

448

192

256

3

7

576

256

320

4

9

768

448

320

5

11

1024

640

384

6

15

-*

896

-

7

-

-*

960

-

8

-

-*

1152

-

9

-

-*

1216

-

10

-

1600

1408

192

11

16

* Based on the sort order of Table 2 there is a large jump between the 15th and 16th module; because of the ddi.dlls (16th) size. Relocated the ddi.dll will cause the 7th, 8th, 9th, 10th, and 11th modules of table 1 to rollover.

 

Enable OEMDRIVERSHIGH

·         Set environment variable IMGOEMDRIVERSHIGH=1

·         In platform\<platform>\files\wpc\oem.cpm.csv or platform\<platform>\files\smartfon\oem.cpm.csv add the line ‘PACKAGE_OEMDRIVERSHIGH,OEMDriversHigh’

·         In platform\<platform>\files\platform.bib change @XIPREGION [END]IF PACKAGE_OEMDRIVERS to @XIPREGION [END]IF PACKAGE_OEMDRIVERSHIGH for targeted modules.

·         Do the same for platform.reg and the corresponding settings for modules moved in previous step.

·         In platform\<platform>\files\wpc or platform\<platform>\files\smartfon make a copy of oemdrivers.pkd.xml and name it oemdrivershigh.pkd.xml

·         Use guidgen tool to create a new packageguid for oemdrivershigh.pkd.xml.

·         In oemdrivershigh.pkd.xml  change the name entry in the XML file from OEMDrivers to OEMDriversHigh.

 

Example:

This example is for the GPIO driver which is typically pretty small.

; @CESYSGEN IF COREDLL_CORESIOA

; @XIPREGION IF PACKAGE_OEMDRIVERS

IF BSP_NOGPIO !

IF IMG_NOGPIO !

    gpio.dll        $(_FLATRELEASEDIR)\gpio.dll                NK SH

ENDIF IMG_NOGPIO !

ENDIF BSP_NOGPIO !

; @XIPREGION ENDIF PACKAGE_OEMDRIVERS

; @CESYSGEN ENDIF COREDLL_CORESIOA

 

This is what the same section would look like after moving the module to the OEMDRIVERSHIGH package.

; @CESYSGEN IF COREDLL_CORESIOA

; @XIPREGION IF PACKAGE_OEMDRIVERSHIGH

IF BSP_NOGPIO !

IF IMG_NOGPIO !

    gpio.dll        $(_FLATRELEASEDIR)\gpio.dll                NK SH

ENDIF IMG_NOGPIO !

ENDIF BSP_NOGPIO !

; @XIPREGION ENDIF PACKAGE_OEMDRIVERSHIGH

; @CESYSGEN ENDIF COREDLL_CORESIOA

 

Results

For this example I modified platform.bib to move the first 15 modules of Table 2 into OEMDRIVERSHIGH, so that the last 6 modules of Table 1 would rollover into slot 0 and a goal of saving 384KB. Here is snippet from the optimized makeimg.out showing the target modules of slot 1 are now rolled over into slot 0. Occasionally, though not depicted here, the relocated VM space used in slot 1 will be smaller than the VM space being rolled over. When that happens you will see additional files this area also being assigned data and code offsets; each extra module that makes it in is saving at least an additional 64KB. We don’t see this because we rolled over 484KB by replacing it with exactly the same size, which I suspect won’t happen all that often.

 

...

Module MsgStore.dll         at offset 01d76000 data, 0208e000 code

Module msim.dll             at offset 01d74000 data, 0206c000 code

Module mscoree.dll          at offset 01d73000 data, 0205c000 code

Module netcfagl2_0.dll      at offset 01d72000 data, 02021000 code

Code space full, fixing up netcfd3dm2_0.dll to ram space

Code space full, fixing up ccoredrv.dll to ram space

Code space full, fixing up ccoreutl.dll to ram space

Code space full, fixing up cspvoice.dll to ram space

Code space full, fixing up simsec.dll to ram space

Code space full, fixing up sms.dll to ram space

Code space full, fixing up supsvcs.dll to ram space

Code space full, fixing up btagsvc.dll to ram space

...

 

Using makeimg.out from a base line image and the optimized image I got these results.

 

Results from baseline makeimg:

...

Module wendyser.dll         at offset 01220000 data

Module drvr007.dll          at offset 010e0000 data

Module drvr002.dll          at offset 010d0000 data

Module drvr005.dll          at offset 010b0000 data

Module drvr001.dll          at offset 010a0000 data

Module shellcelog.dll       at offset 01090000 data

***** BuildPkg Warnings *****

...

 

Results from optimized OEMDRIVERSHIGH package:

...

Module drvr006.dll          at offset 01370000 data

Module wavedev.dll          at offset 01260000 data

Module wcestreambt.dll      at offset 01250000 data

Module wendyser.dll         at offset 01240000 data

Module drvr007.dll          at offset 01100000 data

Module shellcelog.dll       at offset 010f0000 data

***** BuildPkg Warnings *****

...

 

The last address assigned by makeimg is for shellcelog.dll; for the baseline is 0x1090000, and 0x10F0000 when optimized for OEMDRIVERSHIGH. This is a savings of 384KB.

 

Summary

The potential amount of VM savings will change from platform to platform. Potential savings can be affected by modules sizes sitting on the slot 1 / 0 threshold and the libraries configured to be in the devices modules section. The potential saving is also dependent on how many small to medium size OEM drivers are available to be relocated.  The more you have the greater potential there is for savings.

 

This is not the only way to save VM, nor is this the first step that should be taken to preserve VM. To trouble shoot VM issues a clear understanding of the CE memory model is also needed.

 

Comments
  • Hi wesbarc :

    Thansk for this article! We do enconter VM issue by using WM6 to develop. With your way to investigate, the problem we encounter now is that after choosing some optional components, the free space of slot 1 is quite few for OEM drivers, the following is the part of our makeimg.out:

    ===========================

    Module ril.dll              at offset 01e14000 data, 020be000 code

    Module siminit.dll          at offset 01e13000 data, 020b8000 code

    Module ipsec.dll            at offset 01e12000 data, 020a6000 code

    Module ipsecsvc.dll         at offset 01e0f000 data, 02062000 code

    Module l2tp.dll             at offset 01e0e000 data, 02052000 code

    Module scard.dll            at offset 01e0d000 data, 02047000 code

    Module winscard.dll         at offset 01e0c000 data, 02040000 code

    Module OEM_SMS_Providers.dll at offset 01e0b000 data, 02039000 code

    Module OEM_BTPower.dll      at offset 01e0a000 data, 02034000 code

    Module OEM_Daemon.dll       at offset 01e09000 data, 02030000 code

    Module asyncmac.dll         at offset 01e08000 data, 0202a000 code

    Module atadisk.dll          at offset 01e07000 data, 02023000 code

    Code space full, fixing up battdrvr.dll to ram space

    Code space full, fixing up bthuart.dll to ram space

    Code space full, fixing up bthusb.dll to ram space

    Code space full, fixing up PwrButton.Dll to ram space

    ...

    ===========================

    We like to keep those optional components because that is the new and attracting features of WM6. So any good idea for us to solve the problem?

    Thanks in advance!

    --

    Motanis

  • From the looks of your makeimg.out sample these are modules targeted to OEMDRIVERS and are at the threshold for rolling over into slot 0. If most of your modules were already in Slot 1 then using OEMDRIVERSHIGH probably won’t save you any VM because they were already in slot 1.

    The modules listed in the sample makeimg.out will definitely be part of your image whether they end up in slot 1 for slot 0.

  • Hi Wesbarc:

    It's interesting and useful article, thanks!

    From the reply of Motanis quesiton, you said 'If most of your modules were already in Slot 1 then using OEMDRIVERSHIGH probably won’t save you any VM because they were already in slot 1.' What's the most means, it's means just a few files with fixing up or have other flag to determine?

    Thanks in advance.

  • It seems that taking adv. of OEMDRIVERSHIGH package just re-arrange the order of entering slot1 within "OEM modules" during makeimg, right? However, the slot 1 is almost full and occupied by Microsoft's modules while extracting OEMDRIVERS packages in makeimg. Therefore, most of OEMDRIVERS modules are forced to be put in slot 0 due to "Code space full".

    The skill you provide still has benefits for Virtual Memory usage in this situation?

  • This is a very difficult subject to explain so thank you for asking these questions to help clarify the subject.

    Your understanding of the process is correct in that we change the order of modules entering slot 1. It sounds to me like the difficulty you are encountering is that before enabling OEMDRIVERSHIGH it is the modules targeted to OEMDRIVERS that are rolling over from slot 1 to slot 0. In this situation you have to modify the documented technique because the modules rolling over and the module being reassigned to OEMDRIVERSHIGH are the same. Some VM savings could come from your effort, but I suspect that if it is OEMDRIVERS modules that are rolling over that you aren’t losing a lot of VM due to slot 0 rollover. In this example slot 0 rollover is consuming 15.4 MB of slot 0 (0x02000000-0x01090000). I suspect yours is much less.

  • luckytigerwood:

    You are correct. In this case “most” meant a majority of the OEMDRIVERS modules. This stems from the fact that if, for example, 50% of the OEMDRIVERS modules rollover into slot 0, then relocating modules to OEMDRIVERSHIGH would only serve to push the remaining OEMDRIVERS modules that were in slot 1 into slot 0. By taking into account compaction some VM may be saved.

  • Hi Wesbarc:

    Sorry for disturb you again with a question which isn't related with this article, because I really don't know who  

    can help me to understand this problem, thanks in advance if you can help me on this or recommend me to consult other expert.

    I do the hopper test with my devices,and I get such result which I never seen before.

    Build = 17740 (OS 1235)(Hop 2.0.16,6225).

    Random Seed = 132143.

    Previous runtime = 1007 mins (16 hrs 47 mins).

    Ended by: UI Unresponsiveness caused by CPU starvation!

    Boot count (prev): 1 (0)

    ACTIONS/min = 107

    Total States = 218

    ......

    ......

    ......

    STATS: System: RUNTIME = 16 hrs 35 mins; ACTIONS = 106966; ACTIONS/min = 107; Total States = 218

    FREE MEMORY: 2400 KB

    FREE DISK:   54219 KB

    EndType = UI Unresponsiveness caused by CPU starvation

    WatchDog: Highest starved priority 251 should ping 618045 ticks ago.

    Would you please help me on it, thanks!

  • Have you seen this Wiki page? I like using CEDebugX for system hangs.

    http://channel9.msdn.com/wiki/default.aspx/CeDeveloper.BSPHopperInvestigation

  • It's really a good web site, thanks!

  • Hello:

    Can I ask you if we can use IMGVMCOMPACT=1 safely.

    I have noticed this definition in \public\wpc\oak\files\wpc.sku.xml.

    It seems it can put many dlls from MODULES to FILES thus give us more Virtual Memory.

    But I found nothing about its in help.

    So I wandering if we can set IMGVMCOMPACT=1 safely to reserve more VM?

    Thanks

    Jun

  • I was able to dig up a little information about this flag and I posted it here. http://channel9.msdn.com/wiki/default.aspx/CeDeveloper.BSPVMOptimization. I have also filed a doc bug to get MSDN/OEM docs updated appropriately.

  • Thanks Wesbarc,

    I used this information to gain 0x50000 bytes by moving in/out the dlls between OEMDrivers/OEMDriversHigh packages. -Jack

  • Wesbarc,

    We are eliminating the XIP DLL VM here, is it possible to reduce the heap usage by reducing the Thread Stack by either linker option or CreateThread? It's now set to 64K. 10 threads takes up to 640KB. although it's only 40~100K used. Thanks,

    -Jack

  • Have you seen this link?

    http://channel9.msdn.com/wiki/default.aspx/CeDeveloper.BSPVMOptimization

    This is an excerpt:

    “You can also create a heap to allocate from shared memory. Instead of using LocalAlloc for heap allocations from the default process heap, create a heap to allocate from shared memory to preserve process VM. Use CeHeapCreate to create a new heap and define custom allocator/deallocator functions to allocate/deallocate blocks from shared memory. The custom allocator/deallocator functions will be called when the heap size needs to change. Use HeapAlloc to allocate from your heap.”

    Although this is not a complete answer, hopefully this is enough information to get you going.

  • Hi wesbarc:

    I encounter this problem,when compile WM611.

    [makecif Error:] (-1): Unable to find file D:\WM611\release\WPC_CITRINE_Ship\WPC\Packages\OEMDRIVERSHIGH.cab.pkg. Exiting.

    please help me!

    Thank you

    Bob

Page 1 of 1 (15 items)
Leave a Comment
  • Please add 2 and 4 and type the answer here:
  • Post