Pei Zheng's Mobile Computing Blog : Windows Mobile

ConnectivityManager in Android SDK: Compared with Connection Manager in Windows Mobile

zhengpei — Wed, 23 Sep 2009 08:03:00 GMT

Anyone who has ever used Windows Mobile (actually WinCE) Connection Manager will probably share mix feelings: on one side, the ConnMgr APIs provide a nice abstraction for applications to request and use a connection without knowing and controlling the physical connectivity; on the other side, it has been a pain in the neck to override or bypass Connection Manager's connection selection logic if the application should use a specific physical connection (for example, GPRS rather than WiFi). And the concept of metanetworks, Internet, Work, WAP network, Secured WAP, is indeed very confusing and difficult to manipulate.

So how does Android manage connections?

In Android SDK 1.5 (code named cupcake), the android.net.ConnectivityManager class (this is Java, not C++) provides connection state management and notification functionality. Specifically, it can be used to:

Obtain broadcast intents when network connectivity changes by checking ConnectivityManager.CONNECTIVITY_ACTION and ConnectivityManager.EXTRA_NETWORK_INFO in a class extended from BroadcastReceiver. (For a brief introduction to Android application fundamentals, read this document. An example of network connectivity listener can be found here.
Query the coarse-grained or fine-grained state of the available networks using ConnectivityManager.GetActiveNetworkInfo() or ConnectivityManager.GetNetworkInfo(int networkType).

This is same-old same-old as in Windows Mobile, which provides similar APIs like ConnMgrRegisterStatusNotification(), ConnMgrConnectionStatus, and ConnMgrQueryDetailedStatus() (examples of these APIs can be found in WM SDK cmhelper).

However, the ConnectivityManager class does offer an interesting method called requestRouteToHost():

public boolean requestRouteToHost (int networkType, int hostAddress)

Since: API Level 1

Ensure that a network route exists to deliver traffic to the specified host via the specified network interface. An attempt to add a route that already exists is ignored, but treated as successful.

Parameters

networkType the type of the network over which traffic to the specified host is to be routed

hostAddress the IP address of the host to which the route is desired

Returns

true on success, false on failure

networkType	the type of the network over which traffic to the specified host is to be routed
hostAddress	the IP address of the host to which the route is desired

This method is very useful--it allows your application to ensure that a desired physical connectivity (represented by the networkType) is available to reach a specific destination. If a route to the destination IP address from the desired network type is in place (in the routing table), then it simply returns success. Otherwise, it will try to make that happen--this is not mentioned in the above description but can be traced to ConnectivityService class:

In ConnectivityManager class where mService is a ConnectivityService object:

/**
     * Ensure that a network route exists to deliver traffic to the specified
     * host via the specified network interface.
     * @param networkType the type of the network over which traffic to the specified
     * host is to be routed
     * @param hostAddress the IP address of the host to which the route is desired
     * @return {@code true} on success, {@code false} on failure
     */
    public boolean requestRouteToHost(int networkType, int hostAddress) {
        enforceChangePermission();
        if (!ConnectivityManager.isNetworkTypeValid(networkType)) {
            return false;
        }
        NetworkStateTracker tracker = mNetTrackers[networkType];
        /*
         * If there's only one connected network, and it's the one requested,
         * then we don't have to do anything - the requested route already
         * exists. If it's not the requested network, then it's not possible
         * to establish the requested route. Finally, if there is more than
         * one connected network, then we must insert an entry in the routing
         * table.
         */
        if (getNumConnectedNetworks() > 1) {
            return tracker.requestRouteToHost(hostAddress);
        } else {
            return tracker.getNetworkInfo().getType() == networkType;
        }
    }

The mNetTrackers array contains NetworkStateTracker objects for different network types (mobile data and WiFi).

In MobileDataStateTracker class, which extends from NetworkStateTracker:

/**
     * Ensure that a network route exists to deliver traffic to the specified
     * host via the mobile data network.
     * @param hostAddress the IP address of the host to which the route is desired,
     * in network byte order.
     * @return {@code true} on success, {@code false} on failure
     */
    @Override
    public boolean requestRouteToHost(int hostAddress) {
        if (mInterfaceName != null && hostAddress != -1) {
            if (DBG) {
                Log.d(TAG, "Requested host route to " + Integer.toHexString(hostAddress));
            }
            return NetworkUtils.addHostRoute(mInterfaceName, hostAddress) == 0;
        } else {
            return false;
        }
    }

Notice the NetworkUtils.addHostRoute() call. This will add a host route into the system's routing table for the specified destination address via the mobile WWAN interface. This enables the scenario where you want your MMS go through a specific APN for MMS while other traffic on another APN. As of Android SDK 1.5, only MobileDataStateTracker class will actually does this--the WiFiStateTracker class does not do anything yet at this point.

3rd Party Software Integration: What Quality Gates are Needed?

zhengpei — Tue, 18 Aug 2009 20:25:00 GMT

One of the key phases for a mobile device project is 3rd party software integration, which is usually done after initial software baseline has been established (BSP migration complete, driver stabilized, hopper & battery life pass). According to device software product definition, a number of 3rd party applications and modules (for example, some DShow filters) have been identified and made available for integration at this phase. Some of these software may not be the final release as the vendors may also need to conduct technical evaluation and testing with a sample device on their side before releasing a final release. Vendors will provide in-ROM integration instructions, and the ROM build team just needs to follow the instructions to generate test images.

Nonetheless, a well-defined strategy and process for 3rd party software quality gating turn out to be highly critical for device software stability. At minimum, for each 3rd party application, the following quality gates must be used:

Detecting leaks using Application Verifier: Make sure the application does no have significant memory leaks, handle leaks, GDI leaks. For modules/drivers, develop test tools to run within Application Verifier. This can be done within 3 days for a single application or module.

MTTF using hopper: Any application should not take down hopper number by more than 10%. Some applications were not designed with hopper testing in mind (for example, an application runs full-screen but does not enable "Start" button). These will bring down hopper numbers so they must be fixed. Having a focused hopper tool handy such that you can stress a specific application. Also useful is to enable debug messages in those 3rd party applications such that you can always check CElogs for a failed hopper run. This will usually take 3-6 days depending on the degradation of hopper numbers caused by the application.

Battery life/DoU: Windows Mobile application design guideline suggests that an application should follow device power management (by registering for power notification events) and react to system power state changes accordingly. No polling or animations when the application is in background. Use power monitor to check power consumption when the application is running and the device is in various power states. DoU test can be performed to ensure an application does not bring down more than 10% of the DoU number. This will take one or two weeks if the application needs to be modified and re-tested.

No device crashes/hang: this is quite obvious. A 3rd party application often comes with an acceptance test plan, which identifies test cases in different priorities. Perform all the tests and make sure there is no crash (we're sorry messages) or device hang. If there are crashes, the root cause must be found and fixed (if it is in the application) by looking into the dump files (kdmp) with symbols. A consistent repro of a crash is ship-blocking and must be fixed. Depending on the complexity of the test plan, this step may take 2-3 weeks and even longer if the 3rd party software does not come with good quality.

In summary, for a single 3rd party application, it may take several weeks to perform the integration and quality gating, during which effective communication with the vendors is important and both sides understand the device will not ship unless these quality gates are passed (according to the license agreement with the vendors). Issues uncovered should be properly prioritized such that key blocking/critical issues are resolved in the new builds from the vendor, whereas low-user-impact issues can be punted to maintenance releases later.

To ensure high quality of the 3rd party software and overall device stability, and importantly device software will be delivered as planned, an earlier planning of integration and quality gating for each 3rd party software is a must-do for overall device software engineering management, along with required QA/Dev resources pre-allocated.

Windows CE Storage Manager Breakdown

zhengpei — Sat, 15 Aug 2009 00:45:00 GMT

The best architectural pictures for Windows CE / Windows Mobile Storage Manager are the followings (from msdn site):

Now, if you look at the following call stack on a WM6.5 device emulator (CE5), the above picture is much easier to understand:

Call Stack: filesys.exe: 0x1FFE9356

0x0406eb18 AMDNORD!CNorFmd::ReadSector(unsigned long) fmd.cpp line 256
0x0406eb18 AMDNORD!FMD_ReadSector() fmd.cpp line 1168
0x0406eb24 AMDNORD!Fal::ReadFromMedia() fal.cpp line 160 + 36 bytes
0x0406eb50 AMDNORD!ReadFromMedia() falmain.cpp line 1064 + 20 bytes
0x0406eb7c AMDNORD!DSK_IOControl(unsigned char *, unsigned long, unsigned long *) falmain.cpp line 695 + 8 bytes
0x0406ebb4 FSDMGR!CBlockDevice::Control(void *, unsigned long, unsigned long *, _OVERLAPPED *) blockdev.h line 90 + 40 bytes
0x0406ebe0 FSDMGR!FS_DevDeviceIoControl(void *, unsigned long, unsigned long *, _OVERLAPPED *) blockdev.cpp line 24
0x0406ec00 NK!SC_DeviceIoControl(void *, unsigned long, unsigned long *, _OVERLAPPED *) kmisc.c line 2871 + 52 bytes
0x0406ec74 COREDLL!xxx_DeviceIoControl(void *, unsigned long, unsigned long *, _OVERLAPPED *) twinbase.cpp line 49 + 52 bytes
0x0406ecac FSDMGR!CStore::InternalStoreIoControl(void *, unsigned long, unsigned long *, _OVERLAPPED *) store.cpp line 887
0x0406eccc FSDMGR!FSDMGR_DiskIoControl(void *, unsigned long, unsigned long *, _OVERLAPPED *) fsdserv.cpp line 567 + 60 bytes
0x0406ed10 MSPART!ReadPartition(unsigned long, unsigned long *) part.cpp line 600 + 60 bytes
0x0406ed40 MSPART!PD_DeviceIoControl(unsigned char *, unsigned long, unsigned long *) part.cpp line 754 + 36 bytes
0x0406ed74 FSDMGR!CPartDriver::DeviceIoControl(unsigned long, void *, unsigned long, unsigned long *) partition.h line 273 + 52 bytes
0x0406edac FSDMGR!PartitionIoControl(void *, unsigned long, unsigned long *, _OVERLAPPED *) partition.cpp line 57 + 40 bytes
0x0406edd8 FSDMGR!FSDMGR_DiskIoControl(void *, unsigned long, unsigned long *, _OVERLAPPED *) fsdserv.cpp line 567 + 60 bytes
0x0406ee1c FSDMGR!ReadWriteDiskEx() fsdserv.cpp line 215 + 48 bytes
0x0406ee4c FSDMGR!FSDMGR_ReadDiskEx() fsdserv.cpp line 317
0x0406ee50 IMGFS!CVolume::ReadDisk() imgfsvol.cpp line 1715 + 32 bytes
0x0406eeac IMGFS!CStream::LoadBlockAllocationTable() imgfsfile.cpp line 326 + 16 bytes
0x0406eedc IMGFS!CStream::CStream(unsigned long) imgfsfile.cpp line 215
0x0406eee4 IMGFS!CFile::GetStream() imgfsfile.cpp line 172 + 36 bytes
0x0406ef00 IMGFS!CVolume::FS_CreateFileW_read(unsigned long, unsigned long, unsigned long) imgfsvol.cpp line 944 + 12 bytes
0x0406f138 IMGFS!FSD_CreateFileW(unsigned long, _SECURITY_ATTRIBUTES *, unsigned long, unsigned long, void *) imgfsapi.cpp line 95
0x0406f150 FSDMGR!FSDMGR_CreateFileW(unsigned long, _SECURITY_ATTRIBUTES *, unsigned long, unsigned long, void *) fsdapis.cpp line 1020 + 76 bytes
0x0406f1b0 COREDLL!xxx_AFS_CreateFileW(unsigned long, _SECURITY_ATTRIBUTES *, unsigned long, unsigned long, void *) textfile.c line 95
0x0406f1dc FILESYS!FS_CreateFileW(unsigned long, unsigned long, void *) fsmain.c line 2246 + 60 bytes
0x0406f6fc NK!SC_CreateFileW(unsigned long, unsigned long, void *) kmisc.c line 2600 + 56 bytes

From the top to the bottom of the stack:

AMDNORD.DLL: flash driver, which is a block driver FMD (Flash Media Driver) linked with FAL (Flash Abstraction Layer) lib. Source code for this FMD used in the device emulator is in \PLATFORM\DeviceEmulator\SRC\COMMON\AMD\FMD\. Real world examples of MSFLASHFMD are avaliable here PUBLIC\COMMON\OAK\DRIVERS\BLOCK\MSFLASHFMD.

FMD exposes standard stream interface to device manager. The associated storage file is specified in the Drivers registry key.

[HKEY_LOCAL_MACHINE\Drivers\BuiltIn\AmdNorFMD]
   "Dll"="amdnord.dll"
   "Order"=dword:2
   "Prefix"="DSK"
   "Ioctl"=dword:4
   "Profile"="MSFlash"
   "IClass"="{A4E7EDDA-E575-4252-9D6B-4195D48BB865}"
   "MemBase"=dword:A8020000
   "MemLen"=dword:05FE0000
   "Flags"=dword:00010000

FSDMGR.DLL: file system driver manager, which looks into the storage profile in the registry to find partition driver (if any) and file system drivers (FSDs) for those partitions (and possiblely file system filter drivers as well). If no partition driver is specified, then entire device is exposed as a single partition. In this example, the file systems to be loaded are Imgfs and FATFS (FMD for RAM file system (root) plus hive storage is ramfmd.dll in PUBLIC\COMMON\OAK\DRIVERS\BLOCK\MSFLASHFMD\RAM). Note that the MSFLASH flash block driver has been specified as auto-load block driver, meaning that it will be loaded by storage manager instead of device manager, and it cannot be unloaded.

[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\MSFlash]
    "PartitionDriver"="mspart.dll"
    "MountAsROM"=dword:1
    "MountHidden"=dword:1
    "Folder"="Flash Disk"
    "Name"="FLASH Disk Block Device"

; Keep FATFS from trying to shadow \Windows
[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\MSFlash\FATFS]
    "MountAsROM"=dword:0
    "MountHidden"=dword:0

; Support XIP in IMGFS
[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\MSFlash\IMGFS]
    "XIP"=dword:1

; Default settings for ImgFS. These can be overriden per profile.
[HKEY_LOCAL_MACHINE\System\StorageManager\IMGFS]
    "FriendlyName"="Image-Update Filesystem"
    "Dll"="imgfs.dll"
    "Paging"=dword:1
; @CESYSGEN IF IMGFS_IMGFS_NOWRITE
    ; By default, a read-only imgfs partition will shadow ROM.
    "ShadowROM"=dword:1
; @CESYSGEN ENDIF

; Override names in default profile
[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\MSFlash]
    "Name"="MSFLASH for AMD Nor"
    "Folder"="NOR Flash"

[HKEY_LOCAL_MACHINE\System\StorageManager\AutoLoad\MSFlash]
    "DriverPath"="Drivers\\BuiltIn\\AmdNorFMD"
    ; LoadFlags 0x01 == load synchronously
    "LoadFlags"=dword:1
    "Order"=dword:0
    "BootPhase"=dword:0

MSPART.DLL: partition driver to be used for the MSFLASH profile bound with the flash block driver.

IMGFS.DLL: file system driver for Imgfs which is mounted at \Windows.

So from the bottom to the top, when the file system (filesys.exe) is about to create a file, file system manager (fsdmgr.dll) finds the file system driver (imgfs.dll) which in turn asks for ReadDisk from file system manager. Then the file system driver issues FSDMGR_DiskIoControl to partition driver (mspart.dll), which eventually goes to the flash block driver (amdnord.dll). If there is a file system filter driver in place, then it will reside as an extra lightweight file system driver on top of the standard one. I hope this helps clarify the architecture of CE/Mobile storage management.

Using Power Monitor to Tackle Battery Life Problems: Examples

zhengpei — Tue, 11 Aug 2009 20:37:00 GMT

Power monitor (made by Monsoon solutions) is a very useful to improve device battery life. I think every OEM/ODM engineering team should have on handy to watch and troubleshoot battery life issues. Even better, bring the power monitor with your devices into the field, and perform filed tests or comparison tests. For example, how's power consumption of the GPS chip on your device? Just take a drive with your device and connected power monitor with GPS on and capture the logs. Then power monitor will parse the logs and give you a power consumption graph for the test.

The following are some examples of using power monitor to uncover mysterious battery life issues.

1. Standby Current - 30mA !

This graph shows power consumption data of several scenarios on a device: 2G MT call, 2G MO call, Switching to 3G (by changing phone settings), 3G MT call, 3G MO call, and then device gets back to standby with screen off. Device backlight, which burns about 50mA of current, will be turned off after 30 seconds of idle time. When that happens the standby current is around 2-8mA, as can seen after the 2G MT, 2G MO, and 3G MT calls.

The problem is that after the 3G MO call, and device backlight off and suspends, the standby current stays at around 30mA forever. Back-of-the-envelop calculation shows for a whole day 30mA * 24h = 720mAH, about half of the battery capacity on the device.

As this can be reproduced pretty easily, modem log and CE log can be collected and investigated. Eventually this turned out to be the audio path not being properly closed when the 3G MO out ended. After fixing it in RIL driver, the problem was solved.

2. Mysterious thread

This is a classic example of power-unaware programming. At some point the device starts to run a mysterious thread exactly every minute (see the 120mA spikes, even if in suspend mode. Essentially this leads to an average current of about 20-30mA, which is definitely unacceptable. More specifically, when the thread runs, it drains an average of 50mA for 16 seconds, and that will be 13.3mAh for an hour, or 320mAh a day!

How to find the culprit thread? Use Kernel Tracker. Try to look for WaitForSingleObject() calls that has a timeout of 1 minute. If the kernel tracker GUI does not work for you, get the CE log and look for mysterious threads that run every 60 seconds:

0:10:35.890.640 , Switch Thread , hThread=0x07A9FFAE
0:10:35.890.645 , Migrate Thread , hProcess=0x07AB230E
0:10:35.890.741 , Wait for Multiple Objects , Timeout=60000ms , WaitAll=0 , NumObjects=8 , Object 0: 0x07AA8C2A , Object 1: 0x07A9A2C6 , Object 2: 0xC7A9A5D2 , Object 3: 0x07A9A392 , Object 4: 0xC7A9A506 , Object 5: 0x07A9FAAA , Object 6: 0x47A9A986 , Object 7: 0x07AC15E2

0:11:15.961.408 , Switch Thread , hThread=0x07A9FFAE
0:11:15.961.412 , Migrate Thread , hProcess=0x07AB230E
0:11:15.961.508 , Wait for Multiple Objects , Timeout=60000ms , WaitAll=0 , NumObjects=8 , Object 0: 0x07AA8C2A , Object 1: 0x07A9A2C6 , Object 2: 0xC7A9A5D2 , Object 3: 0x07A9A392 , Object 4: 0xC7A9A506 , Object 5: 0x07A9FAAA , Object 6: 0x47A9A986 , Object 7: 0x07AC15E2

This can be further mapped to a thread name as long as you have flat release directory. For the issue, it turned out a battery monitor thread in devices.exe that has a bug which will keep the thread running forever even if the device has been suspended.

Summary:

Power monitor is a great weapon to attach battery life problems for mobile devices. Use it along with other tools such as kernel tracker, CE Log, and KITL to pinpoint what has been wrong in your system. Always has a reference device handy so you know what are normal and what are not. When you perform a scenario test, have CE Log running will be a good idea, but you have to be aware that CE Log flush will drain some battery and create some spikes in your power graph.

Windows Mobile Device DRM Integration

zhengpei — Mon, 10 Aug 2009 01:26:00 GMT

Many devices today provide an OMA DRM 1.x/2.0 solution from a 3rd party (for example, from irdeto). So how to verify if the integration has been properly done? Well, aside from the tests that your vendor performed, you can also use the following site to verify:

http://www.drmtest.com

Common use of this site is to test those Forward Lock and Combined Delivery content such as images (JPG, GIF) and media formats (WAV, mp3, etc).

To make a Windows Mobile phone DRM-ready, i.e., be able to recognize DRM content received via MMS, email, or from the web, a specific file system filter driver has to be loaded into the system. This is often done by adding the the filter into the MODULES section of the memory bib file such that it will be loaded into Slot 1 (unless Slot 1 is so crowded that the module overflows into Slot 0), and put required registry values (for registering the filter driver with file system) to BOOT HIVE.

; HIVE BOOT SECTION
[HKEY_LOCAL_MACHINE\System\StorageManager\Filters]
[HKEY_LOCAL_MACHINE\System\StorageManager\Filters\DrmFSFilter]
"Order"=dword:00000001
"Dll"="DrmFSFilter.dll"
; END HIVE BOOT SECTION

MA DRM specifications can be downloaded from http://www.openmobilealliance.org/technical/release_program/drm_archive.aspx

DirectShow Tool GraphEdt on Windows Mobile

zhengpei — Sat, 08 Aug 2009 02:49:00 GMT

On Windows and Windows CE, the GraphEdt tool can be used to investigate (visually) what DShow filters are used for a given media file, and it is also very useful to verify if your filters work as expected. For desktop Windows, this tool is included in the platform SDK or so called Windows SDK. On Windows CE, the following steps have been suggested to do this:

To use graphedt on CE 5.0, you need to do the following:

1. set WINCEREL=1

2. graphedt uses MFC, so set SYSGEN_MFC=1. Then do "sysgen -p dcom mfc"

3. sysgen -p directx graphedt

Then you should see graphedt.exe in your _flatreleasedir.

On a Windows Mobile AKU build environment, if you follow the steps above, you will hit errors like there at Step 3:

\public\common\sdk\lib\ARMV4I\retail\corelibc.lib
graphedt.lib(filtervw.obj) : error LNK2001: unresolved external symbol IID_IDirectDrawSurfaceKernel
graphedt.lib(filtervw.obj) : error LNK2001: unresolved external symbol IID_IDirectDrawKernel
graphedt.lib(filtervw.obj) : error LNK2001: unresolved external symbol IID_IDDVideoPortContainer

Meantime I am looking into this and see if this can be fixed. Or it can't be fixed because it is by-design because officially Microsoft does not support GraphEdt on Windows Mobile?

Best Client Apps of Twitter, Facebook, LinkedIn, Youtube, Picassa for Windows Mobile

zhengpei — Sat, 27 Jun 2009 20:34:00 GMT

Here is the outcome of my little research...

Best Twitter Client for Windows Mobile

PocketTwit

---------------------------------------------------------------------------------------------------------------------------------------

Best Facebook Client for Windows Mobile

Offical Facebook Client by Microsoft

---------------------------------------------------------------------------------------------------------------------------------------

Best LinkedIn Client for Windows Mobile

There is no LinkedIn client for Windows Mobile yet at this moment.

---------------------------------------------------------------------------------------------------------------------------------------

Best Youtube Client for Windows Mobile

The official Youtube client

---------------------------------------------------------------------------------------------------------------------------------------

Best Picasa Client for Windows Mobile

No standalone app available yet. From a web browser you can visit m.google.com/photos to acccess the mobile picasa site.

Remote-Desktop into MacOS from Windows Mobile

zhengpei — Sat, 07 Mar 2009 01:23:00 GMT

Windows Mobile 6 has the built-in remote desktop client. So you can use it to remote desktop into any Windows machine running terminal services (i.e., running the RDP service that listens at Port 3389). For example, once you enable remote desktop on your XP box, you can remote into it using the client on Windows Mobile. For any other platforms, as long as you have the open source xrdp package installed (which is a RDP server), you can remote into it.

RDP is only one flavor of the remote-desktop world. VNC (Virtual Networking Computing) has been there way before RDP becomes pupular on Windows.

For MacOS, there is an easier way to remote-desktop into it:

Enable Apple Remote Desktop (an implementation of VNC service): Preferences > Sharing > select Apple Remote Desktop
Install a VNC client on your Windows Mobile device. I tried several, but only ZoomVNC works on my WM6.1 device.

Done. Isn't that simple?

Mobile OS Virtualization

zhengpei — Tue, 25 Nov 2008 23:29:00 GMT

Even before VMWare acquires Trango, the French company specialized in mobile OS virtualization, I've heard some OEMs in talks with Trango for future Windows Mobile devices.

A picture is better than 1000 words. Here is a picture of Trango's virtual processor:

The hypervisor is the Virtual Machine Monitor. From Trango's website:

The TRANGO hypervisor is a virtualization layer that allows multiple Virtual Machines to run on a single Hardware platform. It is based on paravirtualization technology which requires that the guest OS be slightly modified. "Privileged" or kernel mode instructions in the OS that access hardware directly are modified to API calls to the hypervisor, or "hypercalls".
Para virtualization is the most appropriate virtualization technique for embedded applications as it allows the best real-time performance compared to emulation or full virtualization which impose considerable resource burdens on the system both from the standpoint of memory and CPU overhead.

This is like Microsoft's Hypervisor for desktop systems, as opposed to VMWare's. This solution can be used for enabling something that Windows Mobile does not support, for example, iMode for Japanese market, or maybe running Windows Mobile apps on iPhone, or the other way around?

Cellular Based Video Telephony on Windows Mobile

zhengpei — Fri, 14 Nov 2008 11:47:00 GMT

How do you enable Video Telephony on Windows Mobile? What are the things you have to be careful of?

First of all, what is the underlying bearer? WiFi, WiMax, or 3G? For WiFi and WiMax, VT will be just a Video over IP application. For 3G, however, most likely it will be a circuit-switching based application. This article is focused on the 3G side of video telephony on Windows Mobile. The standard 3G-324M is based on ITU H.324 standard that specified a data rate of 64 kilobits per second over a circuit-switched UMTS or TD-SCDMA connection.

The following picture shows the 3G-324M architecture (copied from an article published at www.smartphonemag.com):

Overall, to enable Video Telephony on Windows Mobile, there are four major work items to be done:

Develop a VT application must enable circuit switching for video call with the baseband processor using TAPI;
Must have 3G-324M protocol stack, such as the one from Radvision or Dilithium;
Must have Video codecs for H.264 (a.k.a MPEG-4 AVC), H.263, 3gp, etc.
VT application should be tightly integrated with Windows Mobile phone app;

The following picture from Radvision shows the architecture of a VT app on Windows Mobile:

Note that the VT application goes directly to TAPI and then RIL. Contrast to handling cellular voice calls, connection manager is not involved for VT calls. However, connection manager will be able to detect radio resources used by VT calls and manage them as "alien calls" which has a higher priority of data connection but lower priority of cellular voice call.

Phone canvas integration with the VT application is easy - Windows Mobile provides many ways to customize controls on the phone canvas. The AKU has numerous examples of doing that. Outlook and contacts integration can also be done without too much difficulty. Codec-wise, the programming interface is standard Direct Show based.

The major work is the 3G-324M stack, such as implementing call establishment, control control and quality of service support. Doing that from scratch does not seem to worth the effort. Adopting a high-performance stack is a common practice among device manufacturers.

Debugging Stack Fault of Hopper Failure on Windows Mobile devices

zhengpei — Fri, 12 Sep 2008 09:35:00 GMT

1. Introduction

Stack fault happens whenever a thread’s stack is almost used up, and the Windows CE kernel will generate a message of the problem like this:

9381584 PID:47c40bda TID:24ccc7ce [Stack fault]: Thread=85fa3a40 Proc=8060e640 'device.exe'

9381584 PID:47c40bda TID:24ccc7ce AKY=00020013 PC=01a23be0(dummy.dll+0x00003be0) RA=03ed33a0(devmgr.dll+0x000033a0) BVA=24081d24 FSR=00000007

The data in the message is important to understand the failure, which can be a data abort, stack fault, page fault, etc.

AKY "Access Key": Process slot bitmask corresponding to the processes the excepting thread has access to. Platform Builder can show Access Keys of each process in the “Processes” window.
PC "Program Counter": Represents the current line of instruction. On ARM platforms, this is the current value of the PC register and EIP (Instruction Pointer) on x86 platforms. If symbols are available, the exception handler will attempt to provide an offset line into the DLL that caused the exception.
RA "Return Address": Pointer to the instruction address of the function that called the current function. Had the current function NOT caused an exception, this is where we would return to. For a DLL on Windows CE, simply use the last two bytes of the address plus the preferred load address of the module (0x10000000) to search Rva+Base column in the corresponding map file and you will find the function the address falls into. For an EXE, use the last two bytes of the address plus the preferred load address (0x00010000) to search the Rva+Base column. The same calculation applies to BVA below.
BVA "Base Virtual Address": The contents of BVA depend on the type of exception found. If the exception is a Prefetch Abort, the value points directly to the PC register (execution point). If the exception is a Data Abort, then this value points to why the exception was caused. It is a combination of the Virtual Memory base of the module found plus the value that caused the exception.
FSR "Fault Status Register": The FSR represents several flags that will help you understand the nature of your exception. For ARM devices the following flags can be set:

#define FSR_ALIGNMENT 0x01

#define FSR_PAGE_ERROR 0x02

#define FSR_TRANSLATION 0x05

#define FSR_DOMAIN_ERROR 0x09

#define FSR_PERMISSION 0x0D

To confirm a stack fault failure, you should at least have the system dump when the problem occurred. Or better yet, you have a KITL enabled device and you can reproduce the problem.

2. A Real Example

The following is a real world example of stack fault in while running Hopper with focus on tmail.exe.

9381584 PID:47c40bda TID:24ccc7ce [Stack fault]: Thread=85fa3a40 Proc=8060e640 'device.exe'

9381584 PID:47c40bda TID:24ccc7ce AKY=00020013 PC=01a23be0(dummy.dll+0x00003be0) RA=03ed33a0(devmgr.dll+0x000033a0) BVA=24081d24 FSR=00000007

Failed to initialize bug tagger!

9383377 PID:47c40bda TID:24ccc7ce RaiseException: Thread=85fa3a40 Proc=8060e640 'device.exe'

9383463 PID:47c40bda TID:24ccc7ce AKY=00020013 PC=03f6c3c4(coredll.dll+0x0001e3c4) RA=8030a514(NK.EXE+0x0000a514) BVA=00000001 FSR=00000001

9384183 PID:a79698b2 TID:a5f6ac92 OEMIoControl: Unsupported Code 0x1010058 - device 0x0101 func 22

Assembly Dump:

DUM_IOControl:

01A23BA4 E1A0C00D mov r12, sp

01A23BA8 E92D5FF0 stmdb sp!, {r4 - r12, lr}

01A23BAC E28DB028 add r11, sp, #0x28 <<r11 points back to last sp (0x28 accounts for 40 bytes for those saved registers)

01A23BB0 E59FCD48 ldr r12, [pc, #0xD48] << Load value into r12 from address 0x01A24900. Note that the value is negative. (FFFF7EFC or -33028, the size of the local variables)

01A23BB4 E08DD00C add sp, sp, r12 << Sp = 24081D44 and only 7K (1D44) left on the stack after this.

$L36859:

01A23BB8 E1A06002 mov r6, r2

01A23BBC E1A04001 mov r4, r1

01A23BC0 E59F3D34 ldr r3, [pc, #0xD34]

01A23BC4 E5933000 ldr r3, dwLenIni

01A23BC8 E50B302C str r3, [r11, #-0x2C]

01A23BCC E59F0D24 ldr r0, [pc, #0xD24]

01A23BD0 E3A08001 mov r8, #1

01A23BD4 E3A03000 mov r3, #0

01A23BD8 E3A07057 mov r7, #0x57

01A23BDC E24BC902 sub r12, r11, #2, 18 << r12 = r11 -2>>18 = 0x24081e70

01A23BE0 E50C7114 str r7, [r12, #-0x114] << Crash here since a page in request failed due to stack overflow

01A23BE4 E24BC902 sub r12, r11, #2, 18

…

01A24900 FFFF7EFC ???

Registers Dump:

R0 = 01A23138 R1 = 0001000D R2 = 24089F80 R3 = 00000000

R4 = 0001000D R5 = 0005E180 R6 = 24089F80 R7 = 00000057

R8 = 00000001 R9 = 24089F80 R10 = 00000000 R11 = 24089E70

R12 = 24081E70 Sp = 24081D44 Lr = 03ED33A0 Pc = 01A23BE0

Cpsr = 4000001F

Negative=0 Zero=1 Carry=0 Overflow=0 Q=0

Call Stack Dump:

Call Stack: tmail.exe: 0x24CCC7CE 11:36:54 12/07/2007 Taipei Standard Time

0x24081d44 DUMMY!DUM_IOControl(void * 0x00000000, unsigned long 0x00000000, unsigned long * 0x00000000) dummy.cpp line 491 + 20 bytes

0x24089e70 DEVMGR!DM_DevDeviceIoControl(void * 0x00000000, unsigned long 0x00000000, unsigned long * 0x00000000, _OVERLAPPED * 0x00000000 {Internal=??? InternalHigh=??? Offset=??? ...}) devfile.c line 464 + 44 bytes

0x24089eb4 NK!SC_DeviceIoControl(void * 0x00000000, unsigned long 0x0001000d, unsigned long * 0x80321034, _OVERLAPPED * 0x0ba6169c {Internal=0x00550044 InternalHigh=0x0031004d Offset=0x0000003a ...}) kmisc.c line 2860 + 52 bytes

0x24089f28 COREDLL!xxx_DeviceIoControl(void * 0x00000000, unsigned long 0x00000000, unsigned long * 0x00000000, _OVERLAPPED * 0x00000000 {Internal=??? InternalHigh=??? Offset=??? ...}) twinbase.cpp line 49 + 52 bytes

0x24089f60 CAMERA_MAINSTONEII!SetCameraPresentFlag() cameradriver.cpp line 419

0x24089f90 CAMERA_MAINSTONEII!CAM_Open() cameradriver.cpp line 461

0x24089fb4 DEVMGR!I_CreateDeviceHandle(void * 0x0005e020) devfile.c line 98 + 24 bytes

0x2408a004 DEVMGR!DM_CreateDeviceHandle() devfile.c line 182 + 24 bytes

0x2408a084 COREDLL!xxx_CreateDeviceHandle() tdevice.c line 122

0x2408a08c FILESYS!FS_CreateFileW(unsigned long 0x00075c08, unsigned long 0xc0000000, void * 0x00000001) fsmain.c line 2275 + 28 bytes

0x2408a5ac COREDLL!xxx_CreateFileW(unsigned long 0x00000003, unsigned long 0x00000080, void * 0x00000000) twinbase.cpp line 100 + 52 bytes

0x2408a5e0 QUARTZ!CCaptureAdapter::Load() adapter.cpp line 69 + 44 bytes

End Call Stack: tmail.exe: 0x24CCC7CE 11:36:54 12/07/2007 Taipei Standard Time

Combining the call stack and the registers, we can draw the following figure of the thread stack:

The local variable size in the function DUM_IOControl() is 33028 (Note that -33298= FFFF7EFC). However, the thread stack is almost used up, with only 7492(0x1D44) bytes left. The problem should not occur if OEM checks every warning message when they compile the code.

The following instruction results in a device hang:

01A23BE0 E50C7114 str r7, [r12, #-0x114] << Crash here since a page in request failed due to stack overflow

At this point r12 is 0x24081E70. Thus r7 will be stored in 0x2E081D5C. Note that there are two 4K guard pages for a 64Kb thread stack (in this case the thread stack is 0x24080000 ~ 0x2408FFFF). Note that when you build applications within Platform Builder or the Windows Mobile Adaptation Kit, the default thread stack is 64Kb of reserved virtual memory (the physical RAM is committed one page at a time). This is a pretty reasonable limit for most things on an embedded device. However, when you build your applications in Visual Studio, the default thread stack is 1MB of reserved virtual memory, which is often much more than needed. That is why experienced Windows Mobile application developers will set the default thread stack to 64Kb in Visual Studio (in Project properties à Linker à System à Stack Reserve Size) and then allocate larger stacks only when necessary.

Back to the stack fault above, the two guard pages are: Page A: 0x24082000-0x24083FFFF and Page B: 0x24080000-0x24081FFFF. When Page B is being hit, the kernel will throw an exception for the application to handle. If the application hits Page A, the system will terminate the thread. The above instruction is hitting Page B and the thread will be terminated immediately. For more on thread stack guard pages, see http://blogs.msdn.com/hopperx/archive/2006/02/03/524170.aspx

3. A Faulty Sample Driver

Now let’s take a look at the sample driver that demos the sample problem. Let’s first see the failure logs and call stacks.

Debug messages:

920555 PID:a377f6ca TID:624727c2 [Stack fault]: Thread=83451bb4 Proc=80314240 'device.exe'

920557 PID:a377f6ca TID:624727c2 AKY=00000411 PC=019b14dc(baddrv.dll+0x000014dc) RA=019b151c(baddrv.dll+0x0000151c) BVA=161b19a4 FSR=00000005

Call stack:

0x161b19cc BADDRV!StackOverflow(unsigned long 0x00000010) line 25

0x161b59d8 BADDRV!StackOverflow(unsigned long 0x00000010) line 34

0x161b99e4 BADDRV!StackOverflow(unsigned long 0x00000011) line 34

0x161bd9f0 BADDRV!Stack_Fault(unsigned char * 0x00000000, unsigned long 0x00000000, unsigned char * 0x00000000, unsigned long 0x00000000, unsigned long * 0x161bdb6c) line 20

0x161bda0c BADDRV!Launch_Test_case(unsigned long 0x00000300, unsigned char * 0x00000000, unsigned long 0x00000000, unsigned char * 0x00000000, unsigned long 0x00000000, unsigned long * 0x161bdb6c) line 40

0x161bda38 BADDRV!BAD_IOControl(unsigned long 0x00000066, unsigned long 0x00000300, unsigned char * 0x00000000, unsigned long 0x00000000, unsigned char * 0x00000000, unsigned long 0x00000000, unsigned long * 0x161bdb6c) line 78 + 36 bytes

0x161bda68 DEVMGR!DM_DevDeviceIoControl(void * 0x00000000, unsigned long 0x161bdb6c, unsigned long * 0x00000000, _OVERLAPPED * 0x161c5970) line 464 + 44 bytes

NK!80036520()

Registers:

R0 = 00000010 R1 = 00000000

R2 = 00000000 R3 = 00000001

R4 = 019B14A0 R5 = 00000003

R6 = 00000300 R7 = 00000000

R8 = 00000000 R9 = 00000000

R10 = 00000000 R11 = 161BDAA8

R12 = 161B59D8 Sp = 161B19CC

Lr = 019B151C Pc = 019B14DC

Cpsr = 2000001F

Negative=0 Zero=0 Carry=1 Overflow=0

Q=0

IRQ=0 FIQ=0 Thumb=0

M4=1 M3=1 M2=1 M1=1 M0=1

Disassembly:

StackOverflow:

019B14CC mov r12, sp

019B14D0 stmdb sp!, {r0}

019B14D4 stmdb sp!, {r12, lr}

019B14D8 sub sp, sp, #1, 18

$M26864:

019B14DC add r3, sp, #1, 18

019B14E0 ldr r3, [r3, #8]

019B14E4 cmp r3, #0

019B14E8 bhi |$M26864+14h (019b14f0)|

019B14EC b |$M26864+40h (019b151c)|

019B14F0 add r3, sp, #1, 18

019B14F4 ldr r3, [r3, #8]

019B14F8 sub r3, r3, #1

019B14FC add r12, sp, #1, 18

019B1500 str r3, [r12, #8]

019B1504 mov r3, #1

019B1508 add r12, sp, #3, 20

019B150C str r3, [r12, #0xFFC]

019B1510 add r0, sp, #1, 18

019B1514 ldr r0, [r0, #8]

019B1518 bl |StackOverflow (019b14cc)|

019B151C add sp, sp, #1, 18

019B1520 ldmia sp, {sp, lr}

019B1524 bx lr

The faulting instruction is PC=019B14DC. At that point SP is 161B19CC. We know that for each thread’s stack ranges from xxxx0000 ~ xxxxffff, and 161B19CC falls into the topmost guard page (Please refer to the stack figure in the above example). Therefore when the kernel sees this as one of the operands in this instruction, it will generate the stack fault.

The code that creates big arrays on the thread stack is shown below:

void Stack_Fault( PBYTE pBufIn,

DWORD dwLenIn,

PBYTE pBufOut,

DWORD dwLenOut,

PDWORD pdwActualOut )

{

DWORD dwStackDepth = 18;

StackOverflow(dwStackDepth);

}

void StackOverflow(DWORD dwDepthCount)

{

if(0 >= dwDepthCount )

return;

dwDepthCount --;

// Create an array on the stack

DWORD buf[4096];

buf[4095] = 1;

StackOverflow(dwDepthCount);

}

Stack_Fault() calls StackOverflow() recursively. Each time 4Kb of space is allocated on the thread stack. This explains why the thread stack is all used up after three calls of StackOverflow(), as shown in the call stack dump above.

4. Conclusion

This article discusses stack fault, a common problem with a device driver on Windows Mobile platform. The problem occurs when the guard pages of the 64Kb thread stack is being hit. When you see this problem while running Hopper or other tests, make sure you have the call stack and registers, as well as the assembly code such that you can verify the cause.

Debugging Attached Process on Devices

zhengpei — Sat, 06 Sep 2008 00:24:00 GMT

Very often I need to build a quick test EXE or DLL and test it using device emulator. I can of course do this by using Platform Builder's "load from release directory" feature with a KITL enabled image. But this will require a full booting up of the image.

I can also use the "Attach to Process" feature of Visual Studio Smart Device component. First run the test application on a device emulator, then attach the VS debugger to the process. Then when you break the process, you probably see some disassembly code with unresolved addresses. So the question is, how to make better use of this feature for source code debugging?

First, you need to check if the symbol of the test application is loaded. Open the 'Modules' windows and see if the symbol file (PDB) of the test application is showing up there. You should see 'symbol loaded' if you have the PDB file and the EXE/DLL in the same directory where you run the test application.

Then, think about what code is running when you break the process. If you see symbol of the test application is loaded but you did not see any symbol in the disassembly window when you break the process, chances are the process is running code that you don't have the symbols. For example, for a simple Win32 window application created using the wizard, if you break it, you will end up in the message loop somewhere in coredll.dll. If you don't have symbol for it, you will not see any meaningful information in the disassembly window and the call stack window.

A good way to break the test application is to create a breakpoint that you know it will be hit. After attaching the debugger to the process, add a new breakpoint such as "func" (whereas func is a function in the test application).

NOTE: DO NOT USE THE PB METHOD OF ADDING A BREAKPOINT HERE! That is, just enter the function name, not the PB way of "{,,yourapp.exe} func".

Then you should see the breakpoint is resolved and a solid red circle shows up next to it in the breakpoint window. Then when you do something with the application to trigger this breakpoint, you will see the application breaks and its source code shows up! Well, that is based on the assumption that the source code path indicated in the PDB file exists on this computer. If the debugger has trouble finding the source files using that path, it will pop up a nice dialog box for you to specify a path. In that case, just browse to the place where you have the source files.

Now, you can do source code debugging of the process you just attached.

Cloud Computing: "Save To My SkyDrive"

zhengpei — Sat, 02 Aug 2008 03:43:00 GMT

Disclaimer: this post does not imply anything related to Microsoft Cloud Computing strategy.

What does Cloud Computing (CC) mean to me? Two services for now:

Microsoft SkyDrive is a service to store and share files on the web, and Google Docs enables online editing and sharing of documents and calendars. (Google Gear allows offline editing).

One feature missing from both services is what I call the "Save To My SkyDrive" on the "File" menu, with which you can edit your stuff in your client application and save it directly to your SkyDrive. Even better, you should be able to create a network drive that maps your SkyDrive to a drive letter, and drag and drop your stuff directly into the drive.

Looking ahead, I expect a lot (probably not all) of your communication materials on your hard drive will be actually just local copies of the same stuff in the cloud. We already have emails stored somewhere on the Internet so we can access anywhere, anytime, from any computer. We will see a lot more going this direction: your Office docs, your calendar, your personal documents, your health care record, your family photos, etc. In other words, the web, or the cloud, is your big "My Documents" folder, and you are either working offline with some client applications such as Microsoft Office and sync with the cloud later, or working online using web browser based applications to edit your files directly.

The files are not even stored and managed in one single place - for example, your family video may not be saved as a single file on Cloud Server # 314 at Mountain View, CA. Instead, pieces of this file may be stored in various places in New York City, Phoenix, Chicago, and Seattle, and some networking coding scheme is employed to achieve excellent performance and reliability (P2P users should know what I mean here).

And your mobile devices should be doing the same thing - offline editing and online syncing to your corner in the cloud.

One word: web-sourcing. We will see our local storage sourced to the cloud, and a lot of our communications (file editing and review, meeting request, trip planning and expense tracking, etc) can be sourced to the cloud as well.

Why are we heading this way? Two reasons:

Files in your local storage can not be shared with others easily; How often do you have to send attachments of file copies back and forth to your colleagues and family, and to yourself?
How much space do you really need to store your docs/videos/mp3/books? I always need more space ...

Is this trend killing box software manufacturers like Microsoft? Not at all. Client applications are still much more powerful and user-friendly than those browser-based solutions. People still need them. The client software just need to be more CC-ready, creating an enriched user experience that is closely, naturally integrated into the web.

This requires a company-wide initiative to be successful, something like Cloud-Ready Computing (CRC (r) all rights reserved) , just like what Microsoft did with security (Trustworthy Computing). Small features such as "Save To My SkyDrive" should be identified by every participating product group, and the Windows Live services should be extended to provide could hosting, and the Windows Mobile platform should be Cloud-Ready together with other Microsoft products as well instead of always playing the porting/catching-up game.

Update: Microsoft Office Live Workspace (beta) offers similar service as Google Docs.

How to Display all IE Mobile Favorites?

zhengpei — Fri, 11 Jul 2008 09:55:00 GMT

Let's say we want to write an application that displays all the favorites of IE Mobile and allows user to click them. Then IE Mobile should go to that URL.

We can certainly write some code to retrieve all the favorites items from \Windows\Favorites by calling FindFirstFile(). Note that some default favorites are saved in a registry key, so we might want to retrieve those as well. Then we can create a UI to show them, and then launch IE Mobile with the selected URL user clicked.

Or, we can utilize the default home page of IE Mobile, \windows\default_0409.htm. This page has some javascript that will call a window.external.favorites object which will retrieve all favorites into a string array. The javascript code looks like this:

    // Loads the current favorites
    function LoadFavorites()
    {
        // favorites
        var favorites = window.external.favorites;

        for(var i = 0; i < favorites.length; ++i)
        {
             strFavHTML += "  <a class=\"collection\" href='' onclick=\"window.external.favorites(" +
                            i + ").goTo();return false\">" + favorites(i).name +
                            "</a><br>";
        }

    }

We can create a new page to do the same. However, when I copied the page onto the device and loaded it into IE Mobile, I did not see any favorites. What is the problem?

Security. The default home page default_0409.htm has a 'system' attribute! Okay, I changed my html file to be 'system' as well using Pocket Controller, then it worked!

Remaining work is quite simple. Write an application to call CreateProcess() and pass my own htm file name as parameter to IExplore.exe:

    SHELLEXECUTEINFO info;
    memset(&info, 0, sizeof(SHELLEXECUTEINFO));

    info.cbSize = sizeof(SHELLEXECUTEINFO);
    info.fMask = SEE_MASK_FLAG_NO_UI;
    info.lpVerb = _T ("open");
    info.lpFile = L"\\Windows\\IExplore.exe";
    info.lpParameters = L"file://\\windows\\fav.htm";
    info.nShow = SW_SHOW;
    ShellExecuteEx(&info);

My simple page fav.htm looks like this:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<meta name="MobileOptimized" content="240">
<title>IE Mobile Favorites</title>

    <script>
    var strFavHTML   = "";
    var strHistHTML = "";

    // Loads the current favorites
    function LoadFavorites()
    {
        // favorites
        var favorites = window.external.favorites;

        for(var i = 0; i < favorites.length; ++i)
        {
             strFavHTML += "  <a class=\"collection\" href='' onclick=\"window.external.favorites(" +
                            i + ").goTo();return false\">" + favorites(i).name +
                            "</a><br>";
        }

    }


</script>
</head>

<body align=center>
<p>My Favorites</p>

<script>
LoadFavorites()
document.write(strFavHTML);

</script>
</body>
</html>

Device Hangs: Anti-Virus Software and ActiveSync Deadlock

zhengpei — Fri, 27 Jun 2008 02:52:00 GMT

I've been working on an interesting problem reported by a Windows Mobile partner. They have a anti-virus software installed on the WM6 device. And whenever they copy an infected file from desktop to the device, that anti-virus software will detect that infected file and delete it. After the user sees 'cannot copy file blah blah' in ActiveSync on the desktop side, the device hangs.

So the question is: what causes the device to hang?

Apparently the AV software has something to do with it. It is constantly monitoring any file system changes on the device, trying to scan every new file copied from the desktop side.

Okay, I've got a full kdmp file, with a corresponding flat release directory with all the symbols (including the AV software symbol files). Attach it to PB, run CEDebugX, I see this (Note I slightly modified the stuff to hide some sensitive information):

Thread A:
Thread: 0x89395640 (rapiclnt) hThread=0x6939595a
===========================================================================
NK!SC_TakeCritSec() line 5233
NK!EnterCriticalSection() line 104
NK!SC_LoadLibraryExW() line 4699
COREDLL!int_LoadLibraryExW() line 598 + 60 bytes
COREDLL!stub_LoadLibraryExW() line 352
COREDLL!xxx_LoadLibraryW() line 649
FAVFFILT!DLL_F_CEL_LogMsg(const wchar_t * 0x03ec4134, unsigned long 0x00000002) + 40 bytes
FAVFFILT!FILTER_MoveFileW() + 260 bytes
CACHEFILT!FilterHook_t::MoveFileW() line 379 + 16 bytes
CACHEFILT!CachedVolume_t::MoveHelper() line 903
CACHEFILT!FCFILT_MoveFileW() line 281 + 8 bytes
FSREPLXFILT!FILTER_MoveFileW() line 410 + 28 bytes
FSDMGR!FSDMGR_MoveFileW() line 634 + 32 bytes
COREDLL!xxx_AFS_MoveFileW() line 118
FILESYS!FS_MoveFileW() line 2449 + 16 bytes
COREDLL!xxx_MoveFileW() line 62
RAPICLNT!xxxMoveFile() line 117

Thread B:

Thread: 0x89397000 (av.exe) hThread= 0xea63f872
===========================================================================
NK!SC_TakeCritSec() line 5233
NK!UB_TakeCritSec() line 5274
COREDLL!xxx_TakeCritSec() line 1857
COREDLL!EnterCriticalSection() line 104
CACHEFILT!CachedVolume_t::CreateFileW(unsigned long 0x00000001, _SECURITY_ATTRIBUTES * 0x00000000, unsigned long 0x00000003, unsigned long 0x00000000, void * 0x00000000) line 665
CACHEFILT!FCFILT_CreateFileW(unsigned long 0x00000001, _SECURITY_ATTRIBUTES * 0x00000000, unsigned long 0x00000003, unsigned long 0x00000000, void * 0x00000000) line 267
FSREPLXFILT!FILTER_CreateFileW(unsigned long 0x00000001, _SECURITY_ATTRIBUTES * 0x00000000, unsigned long 0x00000003, unsigned long 0x00000000, void * 0x00000000) line 76 + 64 bytes
FSDMGR!FSDMGR_CreateFileW(unsigned long 0x00000000, _SECURITY_ATTRIBUTES * 0x00000003, unsigned long 0x00000000, unsigned long 0x00000000, void * 0x0b392002) line 1013 + 76 bytes
COREDLL!xxx_AFS_CreateFileW(unsigned long 0x00000001, _SECURITY_ATTRIBUTES * 0x00000000, unsigned long 0x00000003, unsigned long 0x00000000, void * 0x00000000) line 95
FILESYS!FS_CreateFileW(unsigned long 0x00000000, unsigned long 0x80000000, void * 0x00000000) line 2226 + 48 bytes
NK!SC_CreateFileW(unsigned long 0x00000003, unsigned long 0x00000000, void * 0x00000000) line 2589 + 56 bytes
NK!OpenFileFromFilesys() line 571 + 48 bytes
NK!TryDir(int 0x0000000e, int 0x00000000) line 690 + 28 bytes
NK!TrySameDir() line 728
NK!OpenExecutable() line 806 + 12 bytes
NK!OpenADll() line 3441 + 20 bytes
NK!PerformCallBack4Int() line 2386 + 68 bytes
NK!InitModule(unsigned short 0x0000) line 4105 + 44 bytes
NK!LoadOneLibraryW() line 4442 + 24 bytes
NK!SC_LoadLibraryExW() line 4699 + 20 bytes
COREDLL!int_LoadLibraryExW() line 598 + 60 bytes
COREDLL!stub_LoadLibraryExW() line 352
COREDLL!xxx_LoadLibraryW() line 649
SHUTIL!CePerfOpenSession() line 489 + 24 bytes

[Diagnose] Invoking deadlock diagnose tool
Creating thread list ...
Creating the proxy list...
920 thread proxies.
DEADLOCK DETECTED!
===========================================================================
|
| Deadlock between rapiclnt av.exe
|
===========================================================================
--> Critical Section (lpcs = 0x042902cc)
      owned by ...
rapiclnt thread 0x89395640
      blocked on ...
Critical Section (lpcs = 0x842f0ba0) LLcs
      owned by ...
av.exe thread 0x89397000
       blocked on ...
--> Critical Section (lpcs = 0x042902cc)

rapiclnt.exe is the RAPI client that handles file operations of ActiveSync. av.exe is the AV software. As you can see from the CedebugX output, the two threads deadlocked because Thread B is holding the LLcs, the OS loader lock while trying to get into a critical section. Thread A is doing MoveFile and somehow it is trying to get the loader lock after getting into that critical section that Thread B is waiting to get in. So deadlock occurred. I looked into the code path of Thread A, and figured out the critical section it holds is the file system volume access critical section. When Thread B calls CACHEFILT!CachedVolume_t::CreateFileW(), it tries to enter that file system volume critical section, which is taken by Thread A.

Then the question is, why Thread A, which is supposedly performing a MoveFile operation, tries to get the OS loader lock? Take a look at Thread A's stack, you will see a file system filter FAVFFILT.dll that performs a LoadLibrary call. Yes, that is why the loader lock is needed. I was told that the file system filter is a third-party software installed on the device. Without knowing the source of the filter, I cannot dig deeper into this issue any more.

Anyway, things have been clear now. The file system filter driver should be fixed: do not call LoadLibrary() because that will require a OS loader lock while the underlying thread is probably holding a file system volume lock. This may lead to a deadlock with other threads calling LoadLibrary().