HoppeRx - the cure for your ailing device : DeanMel

Passive KITL to the rescue

deanmel — Fri, 02 Nov 2007 18:59:00 GMT

I'm sure many of you have been in a situation where your device hangs during field testing. Or sometimes you are trying to track down a problem which only repros at a certain location. The best thing you can have in these situations is, of course, a live...(read more)

Running Platform Builder 6 on Vista

deanmel — Thu, 25 Oct 2007 20:39:00 GMT

There are a lot of people that afraid to switch of Vista because they are afraid that their stuff will not work. Well truth be told, I've been running Vista+VS2005_SP1+PB6 since March of this year and haven't had any major problems. The only two problems...(read more)

Where did Callstacks go from the Hopper logs?

deanmel — Thu, 04 Oct 2007 23:03:00 GMT

If you upgraded from an older version of Hopper to a more recent one, you probably noticed that the callstacks are gone from the Hopper logs. We made them optional for two reasons: 1. Printing out callstacks slowed down the run significantly 2. They did...(read more)

Why my private binaries do not show up in the image?

deanmel — Fri, 28 Sep 2007 21:34:00 GMT

Why my private binaries do not show up in the image? I've been asked this question too many times by now. Many partners when testing their private changes have to figure it out the hard way. The reason why your updated binary doesn't show up in the image...(read more)

How to load mismatched PDBs in Platform Builder

deanmel — Fri, 21 Sep 2007 03:23:00 GMT

I don't know how many times I wished Platform Builder had an option to load PDB files with timestamps that do not match my executable. This option was available for desktop users for decades. Windbg and other desktop debuggers can do it without problems....(read more)

How to get work done while building the tree

deanmel — Tue, 01 May 2007 02:45:00 GMT

Over past several months I've noticed that when our partners come here onsite. They like to build Windows Mobile images on their laptops while trying to get other work done at the same time. The problem with that is when compiler, linker, etc. do their job they slow down the computer significantly because there is a lot of compiling and disk I/O happening.

Interestingly Microsoft engineers that worked on the build process foreseen this problem. There is "secret" switch that is actually described in the documentation but not many people know about it. It runs all build processes in an idle priority. All you need to do is add "-m" to the blddemo command.

The pros of doing this you already know -- you can work without significant slowdowns while your image is building. The cons, however, are insignificant -- it might take 5 minutes longer to build the image.

KITL timeouts with USB Serial Part 2

deanmel — Sun, 14 Jan 2007 08:18:00 GMT

In the first part we talked briefly about KITL timeouts. In this article we are going to give you a little bit more details and try to explain why it behaves the way it does.

How did we run into this issue?

It all started with Hopper (apparently Hopper is also good at finding KITL problems as well as all the other problems it usually finds :-). When we ran Hopper on one of the devices, it would behave in a very strange way. Sometimes it would start spewing a lot of data into the debug output and stop completely, the other times it would keep going, sometimes it would stop for 20 minutes and then continue like nothing happened. And when debug output stopped we could still type in Shell window and get the response back from the Shell commands. Most of the time, however, we would see a small hick up (sometimes less then a second) and debugging would continue after that. As you see it was completely random. Obviously if the only thing we saw were hick ups we would have never even noticed the problem.

Everyone knows how frustrating these ambiguous bugs could be when you don't have a consistent repro. We got lucky because we saw that it would happen when Hopper tried to print out a 27 character string.

Why 27 characters? As it was mentioned in the previous post the way USB Serial KITL indicates the end of the transmission is by sending a partial packet (less then max packet size). However, if the data sent over the wire happens to be a multiple of max packet size then a zero length packet needs to be sent to indicate the end of the transmission. The default packet size that most manufactures use right now is 64 bytes. When you send 27 character text you will get exactly 64 bytes. Let's look at the debug message packet to see why it is 27 character string. Here is how the Debug Message packet looks when sent over the USB Serial:

If you look at the header file you'll see the following sizes:

sizeof(OAL_KITL_SERIAL_HEADER)

sizeof(KITL_HDR)

sizeof(KITL_DBGMSG_INFO)

That's 34 bytes plus 27 byte string. Which is 61 bytes. Where is the other 3? Let's look at the packet that was sniffed from the line:

There are 3 extra bytes at the end: 0x0D, 0x0A and 0x00. They are \n, \r and \0.

So that's how the mystery of 27 byte string was solved.

The Explanation

We know that this hick up or hang happens when the data sent by the client is aligned to exactly max packet size. But why in some cases do we see a hick up and in the others a hang? And why would we still be able to type shell commands? That didn't make any sense.

Why would we still be able to type shell commands? The answer lies in the KITL Architecture. KITL has a full duplex architecture meaning it has separate channels for sending and receiving data. It also has 16 separate clients (see the picture blow). Debug Message is client number 0 and Shell command window is client number 10 then you can bring down client number 0 but client number 10 would still work. The problem we were having was on the Transport level which is below the Client level but since the Transport was still Polling the data underneath it was able to respond to our Shell requests and only the Debug Message Client was stuck.

But why would it sometimes stop just for a split second and other times it would hang? The answer is simply luck! Let's say the device was trying to send 4 packets which were exactly 64 bytes each. The host would receive all 4 packets however on the last packet it would just stop because it is waiting for a transmission end packet which never comes because there is no more data in the queue on the device. That is the case when the Debug Message output would get stuck. If the device were to send something else at that moment then the data flow would resume and packet number 4 would be flashed out.

There are other ways that would allow the data flow to be resumed. When the host waits for the packet to come but it never does for any reason then the host would send a packet back asking the device to retransmit the data. However if the only thing the device does is retransmit the last packet it will still be aligned at 64 bytes and the host would still think that there is more data. In most cases though the device send something else with the retransmit (echo, acknowledgment etc.) which will generate a partial packet in the end and the data flow will be resumed. I hope now you see why I said "luck" in the beginning.

The Conclusion

In the end this was a very challenging issue and I'm glad that we had USB sniffer that allowed us to look at the raw packets. Some of you might not have the USB sniffers that's why we thought it is important for us to share our finding with you.

KITL timeouts with USB Serial

deanmel — Sun, 24 Dec 2006 01:49:00 GMT

Recently the JDP team investigated a group of KITL timeout failures. Each timeout occurred under similar conditions and seemed to be associated with a single bug. At first we were puzzled as to why we were seeing these failures on different devices with different OS versions. As the investigation progressed it became clear that the bug was located in a section of BSP code that handles KITL packets sent over the USB Serial transport. To indicate a transmission is complete the USB client sends a partial packet (i.e. a packet smaller then the max packet size) to USB host. This means that if the data being sent is equal to the exact size of the USB max packet size, the USB client must send a zero length packet at the end of the transmission. On the BSPs we were using, this zero length packet was never sent, so in some cases KITL would think it was hung waiting for a transmission to complete; in other cases, the debug message flow would simply stop for a split second and then continue. To test for this bug you can write a very simple application that will send 64 byte packets to the debug output. Keep in mind the standard KITL header is 34 bytes (do not worry about these details if they do not make sense right now, I will explain KITL header and transmissions in more detail in a follow-up blog). Also, do not forget that debug messages are sent in ASCII therefore each character will take 1 byte. The following code will detect the KITL packet bug if the bug is present, a hang or a pause will be shown in the debug messages when this code is executed:

char strng[101];

for(int a = 0; a < 100; a++)

{

memset(strng, 'k', a);

strng[a] = '\0';

printf("%s\n", strng);

//!!If the bug exists, the second printf should cause a hickup or hang!!

printf("%s\n", strng); //Remember '\n' is compiled into two characters (CR and LF).

}

The Fix
This bug has a simple fix. Find the KITLSend function in your KITL implementation, (it's called USBKitlTxIntHandler in some BSPs), then find the spot right before a packet is sent to USB and insert these 3 lines:

// here we split the last trunk of data into 2 short packets(EP1Len - 4) and 4 bytes

if (len == EP1Len)

{

// if the requested send size happens to be multiple of EP1Len,

// we should send a zero length packet to indicate end of data,

// here we split the last trunk of data into 2 short packets(EP1Len - 4) and 4 bytes

//, so we never need to send a zero length packet

lastPacketLen -= 4;

}

How to setup AppVerifier for "Always running" processes

deanmel — Tue, 26 Sep 2006 02:49:00 GMT

AppVerifier is a very useful tool when it comes to finding memory leaks. It is relatively easy to configure if you have some understanding of how AKU's and PB work. There are pretty comprehensive instructions on MSDN found here. However if you need to use AppVerifier on something that loads during boot time, you need to perform extra steps that are harder to find. The purpose of this article is to explain in a greater detail what you need to do to setup AppVerifier for processes that load at boot time.

Here are simple steps that you need to follow to make AppVerifier work for "Always Running" processes:

1. Install Platform Builder (tools only install is enough because CETK is included with that install), install AKU if you haven't already
2. Open your AKU build window where you build images
3. (Optional) Removed/reset all the security or make sure it's configured properly
4. Copy all the AppVerifier binaries to your %_FLATRELEASEDIR% by running getappverif_cetk.bat from your wcetk directory.

a. Default is "c:\Program Files\Platform Builder for Windows Mobile\5.00\CEPB\wcetk\DDTK\DESKTOP\getappverif_cetk.bat"
b. Or copy them manually. Here is the list of files needed (AppVerif* shim_*.dll symhlp.dll vlog.dll lmemdebug_autoshim.dll shimeng.dll)

5. Set the environment variables for the modules that need to be shimmed in our case device.exe and the OEM dlls that you want to check

a. Run SET VERIFY_MODULES=device.exe <list the rest of your dlls and exes delimited by spaces>. For example: SET VERIFY_MODULE=device.exe atidrv.dll

6. Set IMGSHIMENABLE=1 if you have set IMGPERSISTENTSTORAGE=1
7. Set VERIFY_OPTIONS=EnableFanOut (read more about options later ion the article)
8. Run makeverifyfile.bat from your WCETK directory (default is at "c:\Program Files\Platform Builder for Windows Mobile\5.00\CEPB\wcetk\DDTK\DESKTOP\makeverifyfile.bat")
9. Run makeimg.exe
10. Make sure that all shim dlls got signed properly
11. Flash the image on to the device
12. Connect the device to KITL and boot it (you don't need KITL but it's much more convenient to have it)
13. To make sure that KITL is connected you should see things like

a. TID:<address> ++Shim autoloader
b. Found shim 'shim_heap.dll'
c. And so on

14. Unfortunately there is no magic bullet that would just show all the memory leaks. The best way to find a leak is to get a snapshot then run user scenarios then get a delta. Here is an example of what can be done:

a. Boot the device
b. Run 'heapverif device.exe chk' (this will create a check point on device.exe)
d. Run 'heapverif device.exe delta' (this will report the potential leaks since the last check point on device.exe)
e. Repeat these steps to find all the leaks

Obtaining the log file

If you are connected with KITL the log file with AppVerifier information can be found in your release directory it should be called something like this
AppVerifier_<ProcessName>_<ProcessIdentifier>.log older versions of AppVerifier might call it VerifierLog_<AppName_<ProcessIdentifier>.log

DevHealth's mysterious dup(d) type explained

deanmel — Wed, 20 Sep 2006 00:56:00 GMT

If you run DevHealth on your devices you probably saw dup(d) on the page summary for a process or a dll. It looks something like this:

=== lap_pw.dll page summary: code=0[rom(C):0 ram(c):0] data r/o(R)=0 r/w=0[ro(r)=0 rw(W)=0 exe(E)=0 dll(D)=0 heap(H)=0] page(p)=0 stack(S)=0 dup(d)=0 unknown(?)=0 obj(O)=0 reserved(-)=23

A lot of times it equals 0 but sometimes you see other values. MSDN and Platform Builder documentation has some of the information about DevHealth which you can find here or here on MSDN. Scroll down to "The following table shows the memory information definitions that are included in the page summary" table it has some of the definitions that DevHealth uses in its reports.

In the past we received several emails from OEMs concerned that these might be memory leaks because the name kind of suggests it. Please do not be alarmed. It will become clear why in a few moments.

First thing that DevHealth does is it checks what’s called a “Secure Section” of the OS. "Secure Section" is a special 32Mb section that lives in Kernel space Slot 93 (c2000000). This section is designed to keep all the DLLs and read/write code that processes load into Slot 0. To put it in other words, if a DLL ever shows up in Slot 0 then it will show up in Secure Section.

This is a housekeeping mechanism which allows the loader to only load a particular DLL once greatly increasing the loading speed. When the next process asks for the same DLL, since the OS has it loaded in Secure Slot, it just references that DLL and increments the ref count. In the end Secure Slot is an ORed (logical OR) version of all the DLLs that ever show up in Slot 0.

Let's look at an example which will hopefully clarify some things. If we have two Processes. Process 1 which has loaded DLL A and DLL C and Process 2 that has loaded DLL B and DLL C.

As I mentioned previously DevHealth reads the “Secure Section” first and when it sees DLL A, B and C in it (remember all the DLLs that show up in Slot 0 show up ORed in the "Secure Section"). Then it reads Section 0 and sees that Process 1 has DLL A and C loaded. As it prints this information out it will print them as dup(d) type meaning that it already saw this DLL pages. Same thing happens for the Process 2.