HoppeRx - the cure for your ailing device : KITL

MemoRx incorrectly displaying VM overlap in pre-release Windows Mobile versions

shende — Fri, 11 Jan 2008 03:07:00 GMT

Many OEM’s have already noticed that Memory Doctor doesn’t correctly represent VM overlap in some pre-release versions of Windows Mobile. There have been some changes in WM VM architecture that contradict an assumption made by MemoRx which results in...(read more)

Passive KITL to the rescue

deanmel — Fri, 02 Nov 2007 18:59:00 GMT

I'm sure many of you have been in a situation where your device hangs during field testing. Or sometimes you are trying to track down a problem which only repros at a certain location. The best thing you can have in these situations is, of course, a live...(read more)

Debugging Suspend/Resume Issues

JeCahill — Mon, 11 Jun 2007 03:04:00 GMT

The blog "Snow White's Poison Apple" describes how to test the most common suspend/resume hang. Other suspend/resume issues can be debugged over a "Passive KITL" connection. A standard KITL connection (a.k.a. "Active KITL") gets dropped when a device suspends, however, Passive KITL doesn't suffer from this problem.

To enable passive KITL on a device, build a KITL image but change OEMInit() so it calls KitlInit(FALSE). The FALSE parameter will tell the kernel to initialize the KITL structures without initializing the hardware. Since the hardware isn't initialized, KITL won't be affected when the suspend path powers off the hardware. Check out the CE6.0 documentation for more information about implementing KITL and enabling active or passive KITL.

Once passive KITL is enabled, a trigger is needed to activate KITL and break into the debugger when an issue occurs. The best way to do this is to modify the keyboard driver so that it calls DebugBreak() when a specific key is pressed. With the device configured and the trigger in place, a tester can suspend/resume the device repeatedly until the device hangs, then press the specific button and connect to the device in Platform Builder. From this point on, the hang can be debugged using Platform Builder’s debugging tool’s just like any other system hang.

Map File Breakpoints

shende — Wed, 14 Feb 2007 17:57:00 GMT

While debugging, I am often faced with trying to set breakpoints in module code that I do not control. Said modules rarely have current symbols and access to source files either not possible or practical. I usually face these issues armed only with a call stack and a [possibly ancient] map file in which to determine the root cause of the error. I like to convince myself that single-stepping through code would be too easy and debugging ASM to find the problem makes things more interesting. Sometimes it works.

To get started, you need to guess where it is you would like to go – if you have a call stack, this is the function name at the top of the list. First try to set this breakpoint in the breakpoint window using the following format:

{,,ModuleName.dll} FunctionName

Please note that the FunctionName is case sensitive, but the ModuleName is not. If you run through your repro and the debugger did not break, we need to dig a little deeper and make sure we are actually using the correct module name. First, load the Platform Builder “Modules” window (View->Debug->Modules) and look for the module name in the list – remember that during makeimg, modules will often be renamed as they are added to the ROM. If you are certain of the module name, lets find the “base” of this module by breaking at DllMain (or WinMain for EXE’s).

{,,ModuleName.DLL} DllMain

{,,ModuleName.EXE} WinMain

DllMain is often a convenient breakpoint since it is called often, not just on load/unload. Once we are stopped at DllMain, the ease of finding other breakpoints within this module will depend on how old your map file is. If the module has undergone significant changes, the below may not work, but in my experience map files are usually “close enough” to get what you need. Open the map file and find the function you are trying to break at:

...

0001:0002fab4       ?ParseGetCallerIdSettings@@YAJPBDAAPAXAAI@Z 10030ab4 f   supsvc.obj
0001:0002fdd4       ?ParseGetHideIdSettings@@YAJPBDAAPAXAAI@Z 10030dd4 f   supsvc.obj
0001:00030138       ?ParseGetDialedIdSettings@@YAJPBDAAPAXAAI@Z 10031138 f   supsvc.obj
0001:000303f8       ?ParseGetClosedGroupSettings@@YAJPBDAAPAXAAI@Z 100313f8 f   supsvc.obj
0001:00030810       ?ParseGetCallForwardingSettings@@YAJPBDAAPAXAAI@Z 10031820 << HERE HERE HERE
0001:00031338       ?ParseGetCallWaitingSettings@@YAJPBDAAPAXAAI@Z 10032338 f   supsvc.obj
0001:00031920       ?BSCompareChars@@YAHPBX0@Z 10032920 f   util.obj
0001:00033018       ?ParseGetAudioGain@@YAJPBDAAPAXAAI@Z 10034018 f   voice.obj

...

The function I want to stop at is the RIL component ParseGetCallForwardingSettings() and is located at: 10031820. This number is not an address, but rather an offset that we can use to calculate how much code lies between this function and DllMain. Looking at the same map file, locate DllMain:

…
0001:00029914       ?StartReadyStateQuery@CRilHandle@@QAAXXZ 1002a914 f   rilhand.obj
0001:00029960       ?Register@CRilEventLog@@QAAJXZ 1002a960 f i rilmain.obj
0001:000299c0       DllMain                    1002a9c0 f   rilmain.obj
0001:00029a10       ?RIL_Deinit@@YAHK@Z        1002aa10 f   rilmain.obj
0001:00029aec       ?RIL_Read@@YAKKPAXK@Z      1002aaec f   rilmain.obj
0001:00029af4       ?RIL_Write@@YAKKPBXK@Z     1002aaf4 f   rilmain.obj
…

Using calculator, find the amount of code that exists between these two functions. In this case, my function is later in the map file than DllMain so I Subtract:

10031820 - 1002a9c0 = 0x6E60

To get the code offset between these two functions. Going back to the debugger, locate the instruction address of DllMain and drag this into a watch window.

	… 0102A9B8 bx lr 0102A9BC andhi r4, r0, r5 DllMain: 0102A9C0 stmdb sp!, {r4, lr} <<< HERE $M46300: 0102A9C4 cmp r1, #1 0102A9C8 bne \|$M46300+18h (0102a9dc)\| 0102A9CC ldr r0, [pc, #0x38] 0102A9D0 bl \|CRilEventLog::Register (0102a960)\| …

Next, use the watch window to help your function by adding your offset value to the instruction address of DllMain. Note: if your function was less than DllMain, you would actually subtract:

Click and drag this resulting value back to the ASM window to find code close to your function. Looking at the ASM, you will see that we are very close to the top of a function and should set a breakpoint here.

Hint: Look for stmdb instructions and SP register manipulation to show you the beginning and end of functions.

...

010317EC    ldmia       sp!, {r4 - r8, lr}
010317F0    bx          lr
010317F4    andhi       r4, r0, r5
010317F8    ???
010317FC    smlabbeq    r0, r12, r3, r2
01031800    ldreqd      r8, [r0, -r12]
01031804    smlatteq    r0, r12, r2, r8
01031808    andhi       r0, r7, r7, asr r0
0103180C    ???
01031810    mov         r12, sp << Func begins here
01031814    stmdb       sp!, {r0 - r3}
01031818    stmdb       sp!, {r4 - r12, lr}
0103181C    sub         sp, sp, #0x86, 30
01031820    mov         r7, r2   << HERE
01031824    mov         r8, r1
01031828    str         r0, [sp, #0x240]
0103182C    ldr         r3, [pc, #0x3C4]
01031830    ldr         r3, [r3]
01031834    str         r3, [sp, #0x214]
01031838    mov         r1, #0
0103183C    str         r1, [r8]
01031840    str         r1, [r7]

...

The specific case above puts us very close to the top of a function – but is it the function we are looking for? It is hard to say without setting breakpoints and continuing to see if we stop. Don’t be shy about scrolling through the ASM looking for other functions nearby and setting multiply breakpoints (they are free!) on those as well. Function prologs have a certain “look” about them and your eye will train to identify them quickly.

The above steps will not work under all conditions and significant module changes can render older map files useless. But, as the device matures and code churn is reduced – I have found this method very useful. Good Luck!

KITL timeouts with USB Serial Part 2

deanmel — Sun, 14 Jan 2007 08:18:00 GMT

In the first part we talked briefly about KITL timeouts. In this article we are going to give you a little bit more details and try to explain why it behaves the way it does.

How did we run into this issue?

It all started with Hopper (apparently Hopper is also good at finding KITL problems as well as all the other problems it usually finds :-). When we ran Hopper on one of the devices, it would behave in a very strange way. Sometimes it would start spewing a lot of data into the debug output and stop completely, the other times it would keep going, sometimes it would stop for 20 minutes and then continue like nothing happened. And when debug output stopped we could still type in Shell window and get the response back from the Shell commands. Most of the time, however, we would see a small hick up (sometimes less then a second) and debugging would continue after that. As you see it was completely random. Obviously if the only thing we saw were hick ups we would have never even noticed the problem.

Everyone knows how frustrating these ambiguous bugs could be when you don't have a consistent repro. We got lucky because we saw that it would happen when Hopper tried to print out a 27 character string.

Why 27 characters? As it was mentioned in the previous post the way USB Serial KITL indicates the end of the transmission is by sending a partial packet (less then max packet size). However, if the data sent over the wire happens to be a multiple of max packet size then a zero length packet needs to be sent to indicate the end of the transmission. The default packet size that most manufactures use right now is 64 bytes. When you send 27 character text you will get exactly 64 bytes. Let's look at the debug message packet to see why it is 27 character string. Here is how the Debug Message packet looks when sent over the USB Serial:

If you look at the header file you'll see the following sizes:

sizeof(OAL_KITL_SERIAL_HEADER)

sizeof(KITL_HDR)

sizeof(KITL_DBGMSG_INFO)

That's 34 bytes plus 27 byte string. Which is 61 bytes. Where is the other 3? Let's look at the packet that was sniffed from the line:

There are 3 extra bytes at the end: 0x0D, 0x0A and 0x00. They are \n, \r and \0.

So that's how the mystery of 27 byte string was solved.

The Explanation

We know that this hick up or hang happens when the data sent by the client is aligned to exactly max packet size. But why in some cases do we see a hick up and in the others a hang? And why would we still be able to type shell commands? That didn't make any sense.

Why would we still be able to type shell commands? The answer lies in the KITL Architecture. KITL has a full duplex architecture meaning it has separate channels for sending and receiving data. It also has 16 separate clients (see the picture blow). Debug Message is client number 0 and Shell command window is client number 10 then you can bring down client number 0 but client number 10 would still work. The problem we were having was on the Transport level which is below the Client level but since the Transport was still Polling the data underneath it was able to respond to our Shell requests and only the Debug Message Client was stuck.

But why would it sometimes stop just for a split second and other times it would hang? The answer is simply luck! Let's say the device was trying to send 4 packets which were exactly 64 bytes each. The host would receive all 4 packets however on the last packet it would just stop because it is waiting for a transmission end packet which never comes because there is no more data in the queue on the device. That is the case when the Debug Message output would get stuck. If the device were to send something else at that moment then the data flow would resume and packet number 4 would be flashed out.

There are other ways that would allow the data flow to be resumed. When the host waits for the packet to come but it never does for any reason then the host would send a packet back asking the device to retransmit the data. However if the only thing the device does is retransmit the last packet it will still be aligned at 64 bytes and the host would still think that there is more data. In most cases though the device send something else with the retransmit (echo, acknowledgment etc.) which will generate a partial packet in the end and the data flow will be resumed. I hope now you see why I said "luck" in the beginning.

The Conclusion

In the end this was a very challenging issue and I'm glad that we had USB sniffer that allowed us to look at the raw packets. Some of you might not have the USB sniffers that's why we thought it is important for us to share our finding with you.

KITL timeouts with USB Serial

deanmel — Sun, 24 Dec 2006 01:49:00 GMT

Recently the JDP team investigated a group of KITL timeout failures. Each timeout occurred under similar conditions and seemed to be associated with a single bug. At first we were puzzled as to why we were seeing these failures on different devices with different OS versions. As the investigation progressed it became clear that the bug was located in a section of BSP code that handles KITL packets sent over the USB Serial transport. To indicate a transmission is complete the USB client sends a partial packet (i.e. a packet smaller then the max packet size) to USB host. This means that if the data being sent is equal to the exact size of the USB max packet size, the USB client must send a zero length packet at the end of the transmission. On the BSPs we were using, this zero length packet was never sent, so in some cases KITL would think it was hung waiting for a transmission to complete; in other cases, the debug message flow would simply stop for a split second and then continue. To test for this bug you can write a very simple application that will send 64 byte packets to the debug output. Keep in mind the standard KITL header is 34 bytes (do not worry about these details if they do not make sense right now, I will explain KITL header and transmissions in more detail in a follow-up blog). Also, do not forget that debug messages are sent in ASCII therefore each character will take 1 byte. The following code will detect the KITL packet bug if the bug is present, a hang or a pause will be shown in the debug messages when this code is executed:

char strng[101];

for(int a = 0; a < 100; a++)

{

memset(strng, 'k', a);

strng[a] = '\0';

printf("%s\n", strng);

//!!If the bug exists, the second printf should cause a hickup or hang!!

printf("%s\n", strng); //Remember '\n' is compiled into two characters (CR and LF).

}

The Fix
This bug has a simple fix. Find the KITLSend function in your KITL implementation, (it's called USBKitlTxIntHandler in some BSPs), then find the spot right before a packet is sent to USB and insert these 3 lines:

// here we split the last trunk of data into 2 short packets(EP1Len - 4) and 4 bytes

if (len == EP1Len)

{

// if the requested send size happens to be multiple of EP1Len,

// we should send a zero length packet to indicate end of data,

// here we split the last trunk of data into 2 short packets(EP1Len - 4) and 4 bytes

//, so we never need to send a zero length packet

lastPacketLen -= 4;

}