Sue Loh's blog : CE Performance Tools

Lock Convoys and How to Recognize Them

sloh — Fri, 27 May 2005 18:39:00 GMT

The problem: When using a particular application on one specific device, users noticed very bad UI response for a period of a few seconds. After those seconds, UI responsiveness went back to normal.

The investigation: I used ActiveSync to drop celog.dll and oscapture.exe onto a standalone device. I could have hooked the device up to Platform Builder and run Remote Kernel Tracker over KITL, but in this case the device I had was configured to run without KITL. I just didn’t bother reconfiguring it.

I made a .lnk file to start oscapture, and a second .lnk file to trigger it (run a second instance of oscapture with the “-c” parameter). I dropped these onto the Start menu of the device. I ran oscapture, and played around with the errant application until the UI slowdown occurred. As quickly as I could (due to the UI slowdown it took a few seconds) I brought up the start menu and clicked the link to trigger oscapture. OScapture saved its data to a file. I then hooked up ActiveSync again, and copied the log file off the device. Actually now that I think of it, I probably could have just connected Remote Kernel Tracker to the device over ActiveSync, but you know I’m not that smart.

Here is what I saw when opening up the file in Remote Kernel Tracker:

(scroll down if you can't see anything)

The main thread of pimg.exe is taking all the CPU time, and a few other threads (notably a couple in telshell.exe, which is responsible for the Smartphone UI) took little blips of time here and there, contending over a critical section. I know they are contending because CeLog only logs critical section entry and exit events when there is contention. Critical sections are acquired and released far too often to log uncontended operations. You don’t see any critical section exit events in this screen shot, but they’re there; they happen so close to the entry events that the “exit” icons are hidden underneath the “entry” icons. If I had zoomed in some more, you’d see them resolve into distinct events. All of these threads are at the same priority, 251 (THREAD_PRIORITY_NORMAL). I know the thread priorities because I hovered over the thread lines and saw the priorities in the resulting tooltips. But despite the equal priorities in this case, only one thread, that from pimg.exe, is getting very much CPU time. It is running to the completion of its quantum. I know that because I set the “time marker 1 & 2” in the time scale bar – you can see them in the screen shot. In the bottom right of the Remote Kernel Tracker window, where it says “Marker Time Delta,” is the time between the two markers. That time is about 100ms, the default thread quantum in Windows CE. Each time the pimg.exe thread runs for about the same length of time, about 100ms.

I immediately recognized this as a lock convoy problem. Let me describe what that means. A lock convoy is a situation which occurs when two or more threads at the same priority frequently (several times per quantum) acquire a synchronization object, even if they only hold that object for a very short amount of time. It happens most often with critical sections, but can occur with mutexes, etc as well. For a while the threads may go along happily without contending over the object. But eventually some thread’s quantum will expire while it holds the object, and then the problem begins. The expired thread (let’s call it Thread A) stops running, and the next thread at that priority level begins. Soon that thread (let’s call it Thread B) gets to a point where it needs to acquire the object. It blocks on the object. The kernel chooses the next thread in the priority-queue. If there are more threads at that priority which end up trying to acquire the object, they block on the object too. This continues until the kernel returns to Thread A which owns the object. That thread begins running again, and soon releases the object. Here are the two important points. First, once Thread A releases the object, the kernel chooses a thread that’s blocked waiting for the object (probably Thread B), makes that thread the next owner of the object, and marks it as “runnable.” Second, Thread A hasn’t expired its quantum yet, so it continues running rather than switching to Thread B. Since the threads in this scenario acquire the synchronization object frequently, Thread A soon comes back to a point where it needs to acquire the object again. This time, however, Thread B owns it. So Thread A blocks on the object, and the kernel again chooses the next thread in the priority-queue to run. It eventually gets to Thread B, who does its work while owning the object, then releases the object. The next thread blocked on the object receives ownership, and this cycle continues endlessly until eventually the threads stop acquiring so often.

The real performance killer about a lock convoy situation is that every time the threads involved need to acquire the synchronization object, they block and take at least a couple of context switches. Since the premise of this situation is that the threads acquire the object frequently (several times a quantum), they are running for much less than a quantum between each context switch. The less time they run between acquires, the more context switches involved – and remember that this is all regardless of how long or short of a time they spend owning the object.

The other performance killer in a lock convoy situation is that it dramatically increases the percentage of CPU time non-contending threads at the same priority level get to run. If all threads at that priority level are contending over the object, what you see is very frequent context switching between them all. If one or more threads at that level are running continuously without blocking on any objects, those threads get to run for a full quantum each time the contending threads loop on the object. That is what you see in the screen shot above; the pimg.exe thread is not involved in the lock convoy, but since it’s at the same priority level as all the threads in the convoy, it gets all the CPU time. In this case, the UI threads only ran very tiny periods once each quantum of the pimg.exe thread execution, so the UI was very slow to respond to user input. Eventually the pimg.exe thread completed its work, the UI threads completed their convoy, and the UI response time went back to its snappy old self.

The solution: Lock convoys are not always easy to solve. In this case, I don’t know the solution for sure, but I believe they either lowered the priority of the pimg.exe main thread or made it do its work at another time. While that doesn’t actually get rid of the lock convoy, it speeds it up to the point where it is unnoticeable to the user.

Another work-around might have been to make the pimg.exe thread yield frequently, by calling Sleep(0), to allow the other threads to do their work more frequently than once per quantum. Note, however, that this isn’t a very ideal work-around, because it’d lead to very frequent thread switching. Context-switching very often is a problem in itself; you spend all your time making the kernel reschedule threads instead of letting those threads run.

A different work-around is to reduce contention over the object by allowing it to be shared more often. For example, suppose the object is a critical section protecting some data. You don’t want threads to read the data while it’s being written, or write concurrently, etc. If reads are relatively frequent while writes are relatively rare, then a way to reduce contention over the lock is to convert it to a reader / writer lock. If writes are frequent, the improvement of using a reader/writer lock is minimal, but otherwise such a change can allow threads to read at the same time without serializing over a lock.

Sometimes you can work around the problem by acquiring the lock at some other time. For example, I note that there’s a thread in filesys.exe involved in the convoy. So I suspect the critical section that was undergoing contention was inside filesys.exe. (While critical sections cannot be shared between processes, the non-filesys.exe threads may have used its critical section during calls into filesys.exe APIs.) Maybe the UI threads were reading from files to draw icons, or reading from the registry, or writing to a log file. In the read case, perhaps the UI could have kept the objects in a cache so as to avoid entering the file system. In the write case, perhaps it could queue up file system writes to execute at a later time. Either solution amounts to caching data and performing operations asynchronously, instead of during time-critical UI processing.

While lock convoys are also possible in Windows XP, they’re a little less problematic because of the priority boosting and aging that the Windows XP scheduler does. If a thread blocks on an object, then when it becomes un-blocked the XP scheduler temporarily boosts its priority a little. When a thread runs to the completion of its quantum, the XP scheduler ages its priority a little. In this manner, threads that block often end up at higher priority than threads that run continuously without blocking. Then the lock convoy isn’t eliminated, but there is less chance of another unrelated thread skunking up the works by running a full quantum each thread switch.

A better solution, if you can change all the code that acquires the contentious lock, is to use TryEnterCriticalSection to acquire the lock only when it’s uncontended.

retries = 0;

while(!TryEnterCriticalSection(&CritSec)) {

if( retries++ < SpinLimit ){

Sleep(0);

continue;

}

EnterCriticalSection(&CritSec);

break;

}

This code will use TryEnterCriticalSection a number of times, yielding the processor on each failure until it reaches the SpinLimit and then uses regular EnterCriticalSection. The SpinLimit is necessary in case the critical section is owned by a lower-priority thread which would not run and release the critical section while the current thread is looping. In that case the EnterCriticalSection call is required to make the kernel invert the priority of the owning thread briefly, so that the higher-priority thread can acquire the critical section.

You may expect this code, which basically polls for the critical section, to hurt performance. But believe it or not, it can greatly improve it. The difference between this loop and a regular EnterCriticalSection call is that threads do not enter the queue to acquire the critical section if some other thread owns it. So in the example above, when Thread A releases the critical section, Thread B does not immediately become the owner. Thread A is free to continue running and even acquire and release the critical section a few more times before Thread B comes back and tries to acquire it again. Even with a SpinLimit of 2 (= 1 Sleep() call) two threads can recover from critical section contention without creating a convoy.

Alas, this solution is unavailable if you cannot change the code which acquires the critical section.

To those of you who might ask whether we intend to make changes to Windows CE to eliminate lock convoys, I’m not at liberty to discuss what we may or may not do in future products. I can only say the standard non-informative spiel: Microsoft is committed to improving the quality of its products and considers all improvements in the context of other improvements we wish to make, to provide our customers the biggest benefits possible in the time available blah blah blah. (By the way that spiel, while hokey, really is true.)

In the meantime I hope I’ve provided you with some information you can use to identify and escape this problem if you encounter it.

UPDATE 26 June 06: Discussion of whether the TryEnter/Sleep solution hurts perf. I don’t really think the Sleep() will hurt perf at all. Here is what would happen:

Thread 1 quantum expires while holding the CS (which is what starts a convoy)
Thread 2 does TryEnter, fails, and sleeps
Thread 1 runs again, acquires and releases the CS a bunch of times, and quantum expires while not holding the CS
Thread 2 runs again, does TryEnter again, succeeds. Now it can acquire and release the CS a bunch of times.

So the sleep causes a thread switch, but it’s a thread switch you’d have taken anyway if you were using regular EnterCriticalSection. And now that I think more about this, I think your change to hold the CS for a longer period would negatively impact the perf of the TryEnter solution, because it’d increase the chances of expiring your quantum while holding the CS. Though the impact would still not be bad because in general the threads would run to quantum completion.

Windows CE eHow-tos and Tutorials! WOW!!

sloh — Wed, 25 May 2005 21:49:00 GMT

I saw a blip of this at MEDC, and just checked it out myself. How cool! It's like a DevCon for all to share!

http://msdn.microsoft.com/embedded/getstart/basics/tutorialsce/default.aspx

Most of it is pretty basic stuff, but if you're new to Windows CE, you should check some of these out! (If you are new to performance tools like Remote Kernel Tracker which I've been posting about, take a look at the Microsoft Windows CE 5.0 Advanced Lab that's on that site.)

Sue

Recognizing Excessive Thread Switching

sloh — Mon, 23 May 2005 17:16:00 GMT

Sometimes the source of a performance problem is not that code is too slow, but that threads switch too frequently. When the system switches between threads, the system consumes extra cycles in scheduling activities such as selecting the next thread to run, or switching to the new thread, rather than spending the time executing application code. In general, the less time threads run before switching out, the more total time the system spends scheduling new threads to run.

The default amount of time that a thread runs before switching to another thread is called the Default Thread Quantum, dwDefaultThreadQuantum. For example, if the default thread quantum is 100 milliseconds, then a thread would run continuously for 100 milliseconds before another thread ran, unless preempted by a higher-priority thread.

In a system where threads run to the end of a 100-millisecond quantum, there is one thread switch per 100 milliseconds, or 10 thread switches per second. In contrast, in a system where threads run only 50 microseconds before switching, there is one switch per 50 microseconds, or 20,000 thread switches per second. Switching at such an elevated rate has a dramatic performance impact.

You can use Remote Kernel Tracker to identify these situations. Remote Kernel Tracker depicts thread switches as vertical green lines.

The following screenshot shows an example of an application with excessive thread switching. In this picture, I’ve set two markers in the time scale at the top of the display. The difference in time between the two is displayed in the status bar on the bottom right, so from time marker 1 to time marker 2 is 10 milliseconds. As you can see, there are numerous vertical green lines; each represents a switch from one thread to another.

(scroll down if you can't see anything)

You also can see above that the threads seem to be continuously repeating a pattern. The next picture zooms in on that pattern:

(again, scroll down if you can't see anything)

As you can see in the preceding picture, the “OpenDeviceKey” thread runs the longest continuous time, about 0.6 milliseconds. The next longest run-time in this screenshot, “RndisSendRndisMessage,” is 0.16 milliseconds. The greatest switching activity, between “UsbInterruptThreadStub” and “DmacIntrThread,” occurs at a rate of one switch approximately every 50 microseconds. One thread switch per 50us is a rate of 20,000 thread switches per second. This means that thread switching occurs at a rate that is about 2000 times higher than the ideal.

Let me describe where these screen shots came from:

The problem: Poor USB throughput. On particular hardware we expected to get about 50Mbps throughput, but were getting only about 23Mbps.

The investigation: We recorded a Monte Carlo profiling trace, with these results:

HITS PER SYMBOL

Hits	Percent	Address	Module	Routine
6557	7.3	0x8012C8FC	NK.EXE	IDLE_STATE
4643	5.2	0x8010E330	NK.EXE	INTERRUPTS_ENABLE
2882	3.2	0x02FEA884	tcpstk.dll	tcpxsum
2166	2.4	0x80110894	NK.EXE	ObjectCall
2123	2.4	0x03FA37B8	coredll.dll	memcpybigblk
2064	2.3	0x8010DB7C	NK.EXE	PrefetchAbort
1732	1.9	0x80110EC0	NK.EXE	ServerCallReturn
1518	1.7	0x801491D4	NK.EXE	OEMWriteDebugByte
1468	1.6	0x8011A094	NK.EXE	SleepOneMore
1460	1.6	0x8011BCDC	NK.EXE	WaitOneMore
1418	1.6	0x80119360	NK.EXE	KCNextThread
1405	1.6	0x8011C154	NK.EXE	SC_WaitForMultiple
1404	1.6	0x8010E00C	NK.EXE	IRQHandler
1255	1.4	0x8010DEC0	NK.EXE	SaveAndReschedule
1212	1.4	0x03F73CA0	coredll.dll	xxx_CeLogData
1145	1.3	0x03F68258	coredll.dll	xxx_GetTickCount
979	1.1	0x8010EF34	NK.EXE	KCALLPROFON
908	1.0	0x8010EFE0	NK.EXE	KCALLPROFOFF
865	1.0	0x017A4544	pxa27x_dma.dll	InterruptDone

The lines in green are all related to making API calls. The ones in blue are all involved with thread rescheduling. I was a little suspicious about the number of API calls and the amount of thread rescheduling that was going on here. KCNextThread is the primary kernel function which selects the next thread to run, and having that show up 1.6 percent of the time is pretty significant. The API call time might also be partly blamed on calling into the kernel to reschedule.

Then we looked at a Kernel Tracker trace for the same scenario, and saw the screen shots above. I realized how much thread switching was going on, and brought it to the attention of the folks who owned the code involved.

The fix: They changed the design so that the work done between the two most active threads (UsbInterruptThreadStub and DmacIntrThread) was all on one thread instead of shared between two. As a result, the USB throughput jumped from 23Mbps to almost 41Mbps.

Remote Kernel Tracker: More things you should recognize

sloh — Sat, 21 May 2005 08:06:00 GMT

I have a few more examples of patterns you can recognize when you’re looking at CeLog threading data in Remote Kernel Tracker. Note: these images are wacky heights because I tweaked them to cut off white space that didn’t contain any CeLog data.

Thread Migration

I realized that in my previous Kernel Tracker post I meant to add a screen shot of thread migration, and forgot. This is an example of what thread migration looks like in Remote Kernel Tracker. There’s a raised dashed line which represents that the thread is still running, but inside a different process.

(scroll down if you can't see anything)

Screen Shot 1: Example of what thread migration looks like.

KITL / debug messages

I like this example which hammers home how much work is involved in heavy KITL traffic. I run a program called enumfiles.exe which enumerates all of the files in the device file system. The KITL traffic is due to a few different things: mainly the heavy debug messages, and the enumeration of files inside the release directory via RELFSD. (The release directory file system driver, which mounts the release directory from the desktop PC as \Release on the device.) In this case I also loaded the program over KITL from my release directory, so some of this traffic is the work copying code onto the device.

Screen Shot 2: For a program that’s just enumerating the file system, that’s a lot of time on the KITL thread.

As you can see, when the enumfiles.exe program starts, a lot of other activity begins too. The KITL interrupt (25 on this device) and the KITL IST start working overtime. I like this example because it shows that your code doesn’t always live in the vacuum you might expect. The system calls you make, or the pure existence of your program, can fire off other activity in the system. Does the system really behave as you expect? Tools like CeLog and Remote Kernel Tracker are the way to find out for sure.

Mike Hall uses an older version of my enumfiles program (walktree.exe) when he gives labs to teach people how to use Remote Kernel Tracker, so if you’ve taken any labs from Mike at DevCons or other events, you may have played with this same program yourself!

Flushing

It’s good to know what activity represents side effects of data gathering itself. Whether you have Remote Kernel Tracker actively connected to a device and collecting data, or you’re running CeLogFlush.exe to save data on a standalone device, the thread that manages the data impacts the system a little itself. First, Screen Shot 3 shows CeLogFlush.exe flushing periodically. In this case I specified that CeLogFlush.exe should flush at 3-second intervals, so that more than one flush would show in the same window.

Screen Shot 3: CeLogFlush data flushing at 3 second intervals when data’s not being logged heavily.

As you can see above, when there’s not a lot of activity, flushing can be a pretty fast operation. The vertical lines showing a lot of thread switching aren’t very wide; the flush is over quickly.

Incidentally, you can see that the switching is between the CeLogFlush.exe thread and the KITL thread – OK now tell me where I was storing the log file? That’s right, in \Release which required KITL to transport the file data to the release directory on the PC. Otherwise there’d be less thread switching because it could write to local storage on the device without switching to the KITL thread.

In contrast, Screen Shot 4 shows how much of a load flushing can be on a busy system. There are a lot more events being logged, so there is more data to flush and flushes take longer. The RAM buffer that the data is temporarily stored in fills more quickly than 3 seconds, so the flushes are more frequent.

Screen Shot 4: CeLogFlush flushing during heavy logging activity.

For completeness I’m including Screen Shot 5 which shows what flushes look like when you’re using Remote Kernel Tracker to transport the data to your PC, instead of using CeLogFlush.exe. You can see that RTH.EXE does the flushing, and that it flushes over KITL about once per second.

Screen Shot 5: Remote Kernel Tracker data flushing.

Data Loss

I wrote up a couple of sections about data loss in our recent newly-released CeLog documentation:

· Analyzing Data Loss with Remote Kernel Tracker

· Analyzing Data Loss with Readlog

But I could not include screen shots. Here is a screen shot to show you how to recognize data loss when you’re looking at it in Remote Kernel Tracker.

Screen Shot 6: The signs of data loss.

The three characteristics I describe in the Kernel Tracker article are visible in Screen Shot 5. First, it pops up a dialog box telling you how many bytes of data have been lost over time in the log. Second, the interrupt activity appears to stop completely, and one thread appears to be running for a long period. In this screen shot I didn’t expand explorer.exe to show that thread, but it was a single thread and it had no icons showing events occurring during the thread run-time. Finally, when activity picks up again, the flush thread (RTH.EXE) is busy working. All of these characteristics are strong hints that the apparently “inactive” period is really a period of time from which data was lost.

More Tips

While I was poking around building up screen shots for this write-up, I caught one view that surprised me a bit. Look how much priority inversion is going on here! This is a database test program spewing lots of debug output. KITL isn’t something that actually requires “real” real-time, but perhaps I should get someone to examine why we hit priority inversion so regularly.

Screen Shot 7: Debug output leading to lots of priority inversion.

Let it be a lesson that while you’re testing real-time response in your system, you should disable KITL. And that while you’re running performance tests, for heaven’s sake don’t write a lot of debug messages.

Another little exercise. On the left and right of Screen Shot 7 are some periods where activity drops dramatically. Do those periods represent data loss?

The answer is no, because of two clues. First, SYSINTR 0 is still going strong. Keep an eye on that whenever you’re running a profiling build. (I’d say all the time but only profiling builds record interrupts.) SYSINTR 0 occurs frequently enough you should always be able to find it. Second, there are lots of events occurring on the testdb.exe thread during those periods. There are so many icons drawn on that thread that they don’t show up clearly in this screen shot. If it was data loss, there’d be no such activity. Since this particular test program was writing a lot of debug output; the less busy periods probably represent time when the test was doing solid periods of work instead of printing debug messages. When you’re not running a profiling build, look for events on the thread that was supposedly executing during the quiet period, to see if that really is a period of data loss.

Introduction to Remote Kernel Tracker

sloh — Tue, 17 May 2005 17:52:00 GMT

When you connect to a device with Remote Kernel Tracker, or when you open a log file that has been collected by one of the other CeLog tools, you’ll see a lot of data you may not understand. At MEDC this past week, a few people asked for a better explanation of what they’re seeing. So let’s go over it. Screen Shot 1 shows a sample of what you might typically see when you connect Remote Kernel Tracker to a device:

(scroll down if you can't see anything)

Screen Shot 1: Typical idle behavior, no processes expanded.

Someone asked me what all the information in the key means. I’ll go through them a bit at a time. First, the thread and process state lines.

You can see examples of the process state lines in Screen Shot 1 above. Each process in the system has a horizontal line. When one of the threads owned by that process is running, the process line is wide and dark green. Otherwise the process line is thin and black. (In case it’s not clear, only one thread ever runs at a time, so only one process ever runs at a time too.) So in the example above, threads owned by kernel (nk.exe) are running almost all the time. The vertical lines you see are due to threads in other processes running for very short periods of time. Remote Kernel Tracker draws vertical lines to represent switches between threads, to make it easier to see which threads ran before and after the thread switch.

Underneath each process are all the threads owned by that process. Processes can be expanded and collapsed to show and hide the threads, for easy viewing. There are no thread lines in Screen Shot 1, because none of the processes have been expanded to display their threads yet. Screen Shot 2 shows the same log, with some of the process lines expanded and with the key hidden for better viewing.

Screen Shot 2: Typical idle behavior, interrupts and some processes expanded.

In Screen Shot 2 you can see some thread lines too. Like the process lines, the thread lines are wide and light green whenever the threads are running, and thin when they are not. Most of the threads, except for “0x00000000,” run for such short times that at the zoom level shown in this screen shot, that you can’t see the horizontal thread-run lines, but you can tell that they ran because you can see the vertical thread-switch lines. On the threads you can also see some other icons representing the actions taken by the threads. More on these later.

You can also see that interrupts are represented as lines too, with one line for each SYSINTR value to show when interrupts occur.

What you’re seeing in these screen shots is typical idle activity on a device. The thread with handle “0x00000000” is not really a thread, but the placeholder to represent when the CPU is idle. Most of the other threads that run now and then are interrupt service threads (ISTs).

Every process will have one thread which has the same name as the process; that is the process’s primary thread, the one which starts running at main() or WinMain() or whatever flavor of entry point the process might use. All of the other threads in the process are given the name of the thread’s start routine, such as KITLInterruptThread under the process “NK.EXE” (the kernel).

An important detail to note at this point is that some of the features you can see here are due to the fact that you are running a profiling build. First, to minimize the impact of the logging hooks which record this data, interrupt data is only available on profiling builds. When the profiler is not present, the “Interrupts” line will not appear. Second, the thread names are also only available on profiling builds. Remote Kernel Tracker does not look up thread names from the symbol files in the release directory. Instead, on profiling builds the CeLog event logging hooks obtain thread names via the special symbol table that is included in ROM for use by the profiler, to represent all the code that is stored in ROM. The profiler uses that symbol table to look up symbols for profiling hits, and the logging hooks take advantage of the symbol table in order to provide thread names. Threads from processes and DLLs that are not stored in the “modules” section of ROM will not have names, since they will not be listed in the profiler’s symbol table. And on non-profiling builds, no threads will have names.

There is also a trick you can use to make the desktop tool “readlog.exe” look up symbols from the release directory and build a new log file containing the thread names on any build and regardless of where the thread’s code is running from. I don’t quite have documentation to point you to about that yet, but open up a release directory build window and type “readlog -?” on the command line to find out how to use readlog.

The other thread line I haven’t discussed yet is the Thread Migrate line. In Windows CE, when a thread calls a system API that is implemented by a different process, the thread moves from its owner process into the system process for the duration of the call. The Thread Migrate line in Remote Kernel Tracker represents the period of time during which the thread is executing inside a process other than its owner. You can hover over the line to learn which process the thread is executing inside.

In my opinion, this representation of thread migration is not terribly useful. It helps you see when a thread is running inside a process other than its owner, but it does not help you see the entire set of processes the thread transitions to. For example if your thread calls an API, and that API calls a different API, your thread may actually transition through multiple processes, but it is not easy to see if or when those transitions occur. Nor does it help you understand why those transitions took place (which APIs the thread was calling).

Back to the event types in the key. The main events I pay attention to when I’m looking at a log in Remote Kernel Tracker are a subset of those related to thread scheduling:

If you are familiar with the Win32 API set, you probably can guess what these mean. The critical section events represent when threads call the EnterCriticalSection and LeaveCriticalSection functions, and encounter contention over the critical section object. Critical sections are used so widely that logging every entry and exit would generate too many events and bog down system performance. So only those calls which block the current thread or unblock another thread are logged. The Set Event, Reset Event and Pulse Event events represent actions taken on Win32 “event” objects. Similarly the Release Semaphore and Release Mutex events represent actions on Win32 semaphore and mutex objects. The wait event represents a call to the Win32 WaitForSingleObject or WaitForMultipleObjects APIs, regardless of whether the thread blocks on those objects or not.

Finally, the Sleep Event event represents when the thread calls the Win32 Sleep API. In general I am not a big fan of Sleep, at least non-zero Sleep periods. Often it represents lazy programming, and results in polling that churns CPU time and drains power. For example, observe the behavior of the selected thread in Screen Shot 3 below. This thread is polling at 500ms intervals.

Screen Shot 3: Using Sleep to poll at regular intervals consumes CPU and drains power.

If you hunt around most logs, you’ll find cases where one thread will call the SetEvent API, then another thread will start running, do some work, and then block again in WaitForMultipleObjects on the same event handle.

Screen Shot 4: A waiting thread is awakened when another thread signals an object.

In Screen Shot 4 you can see that the threadsample.exe main thread set an event (the “flag” icon) that the last thread in the same process was waiting for. In this screen shot I double-clicked on the WaitForMultipleObjects event and the Set Event event to show their data. This often makes it easier to compare handle values between events. I know these two events are related because they both apply to handle value “0xA3E0FF96” in this case.

There are a few other events I also look at when I’m examining logs.

The interrupt event you’ve already seen above, representing when interrupt service routines (ISRs) execute. It’s worth mentioning that you can see relationships between interrupt service routines (ISRs) running – the “interrupt” events you see in Kernel Tracker – and their corresponding interrupt service threads (ISTs) waking up to do further interrupt handling. In Screen Shot 5 below, you can see the ISR for SYSINTR 1 (the timer interrupt, I believe?) run, and then two different ISTs follow up. You can also see the ISR for SYSINTR 30 (the KITL interrupt) run, and the KITLInterruptThread wake up to do some work. If you look back at Screen Shot 2 above, you can also see this relationship between the KITL interrupt and KITLInterruptThread when scaled back to a larger zoom level. Note that no IST runs for SYSINTR 0, which is a no-op.

Screen Shot 5: The relationship between ISRs and ISTs.

The Invert Priorities event is shown on a thread when the kernel temporarily changes the thread’s priority because it holds a resource that some other higher-priority thread needs. For example, if a low-priority thread holds a critical section that a high-priority thread tries to enter, the kernel temporarily boosts the low-priority thread to a high priority, to execute until it releases the critical section. Then the low-priority thread returns to its original priority. For real-time behavior, you never want real-time threads to block on resources owned by lower-priority threads, since those threads may do an undetermined amount of work before releasing the resources. Screen Shot 6 is an example of a case where a higher-priority thread ends up boosting the priority of another thread over and over again.

Screen Shot 6: Recurring cases of priority inversion.

The icons in Screen Shot 6 are a little hard to see – sorry – but what’s going on is that a high-priority IST in the kernel enters a critical section that is held every time by a low-priority thread from testdb.exe. The kernel thread logs a priority-inversion event representing the priority-boost given to the testdb.exe thread. Then the kernel thread blocks on the critical section. The now-high-priority testdb.exe thread runs until it leaves the critical section, then logs a priority-inversion event representing its return to its regular priority. The kernel thread then begins running again, now that it owns the critical section and is the highest-priority thread. You can see in this log that the inversion happens regularly in this particular case.

The page fault event represents a costly page fault operation occurring on a thread. In CE 4.2 and earlier, a page fault event was logged at the beginning of a page fault; in CE 5.0 and later, an event is also logged at the end of the page fault so that you can tell how long the page fault took. I don’t see this event often, but sometimes I see cases like this, where a thread incurs many page faults:

Screen Shot 7: A thread incurring many page faults. This log is from CE 5.0 so the fault markers are in begin/end pairs so that you can tell how long they took.

When I see many page faults like this, I run the readlog.exe command-line tool to get a summary of the faults, to determine where they are occurring. In general the important thing to check is whether the fault address is paged more than once. Code must be paged once in order to execute at all, but if it is paged more than once, that means the kernel’s page pool is not large enough to hold all the executing code. In that case the page pool may need to be enlarged. If data such as memory-mapped files is paged more than once, then the data may not have good page locality, or the system may be in a low-memory state.

You can also log your own events from your code. Since Remote Kernel Tracker does not have the ability to interpret your data blobs, you may find it useful to log your data using a set of event types that Remote Kernel Tracker does understand:

When Remote Kernel Tracker sees one of these event types, it expects the data logged for that event to be an array of the given type. For example, for the “Raw ulong” type Remote Kernel Tracker interprets the data as an array of unsigned long integers (DWORDs).

For the “Raw wchar” type the data is an array of Unicode characters, and Remote Kernel Tracker will display it as a string.

To see the data corresponding to one of these event types, hover over the event to see the tooltip, or double-click on the event to get the data in a separate window. Note also that the data is displayed in the bottom-left status bar when you click on an event.

Screen Shot 8: Viewing “raw” data in Remote Kernel Tracker.

In Summary

I hope I have helped you understand what the tool is showing you, and see some of the things you can use CeLog and Remote Kernel Tracker for. This is a very versatile tool. It might take some practice to recognize these relationships between events, but a basic understanding of what thread behavior looks like in Kernel Tracker is the first thing you need when you’re troubleshooting a wide range of performance problems. Remote Kernel Tracker enables you to track down bugs related to thread interactions that are difficult to track with traditional development tools, including deadlocks, race conditions, excessive context switching or priority problems.

I have some more posts coming in the near future, about more scenarios you can recognize when you get used to Remote Kernel Tracker.

CeLog documentation

sloh — Mon, 16 May 2005 16:55:00 GMT

At long last, a big chunk of work I did to add documentation of CeLog & Remote Kernel Tracker has finally shown up in a form you can view.

http://msdn.microsoft.com/library/en-us/wcedebug5/html/wce50conEventTracking.asp

I did a huge brain dump and tried to get all of my tips & tricks down on paper. There are descriptions of how to use tools to gather data on standalone devices, how to log your own data and how to customize the tools. There is a little bit more work to go -- some information really required Kernel Tracker screen shots, and right now we can't include screen shots in our documentation. We've been discussing other formats to release the information in, but for now I am planning to post a bunch of it on my blog. I have it mostly written up, but am trying to set up a web share to hold the pictures. (I don't have any storage space to use with my blog, so I can only post text at the moment, but that should change soon.)

So look for more in the coming days! I got to show off Remote Kernel Tracker to a lot of folks at MEDC, and got some good questions. I hope that between the documentation posted above and the material I'll add to my blog soon, you'll get to be as much an expert in using CeLog and Remote Kernel Tracker as I am.

Happy logging!

Sue

What WINDOWS CE debugging tools would you like to see?

sloh — Tue, 31 Aug 2004 20:09:00 GMT

Just an informal survey -- I use some of our tools all the time, some rarely. I have my own nitpicks I'd like fixed and features I'd like to see added. But there's no guarantee I see the world the way you do. What are your opinions?

Here are some ideas of my own, to get the conversation started:

Remote regedit: Export/import data under a key. Track and report changes under a key.
Remote performance monitor: More counters!
Kernel profiler: UI in the remote tools menu in addition to the on-device control we already have.
Remote kernel tracker... don't even get me started, I love this tool but have lots of features I want added. What would YOU ask for?
Does anybody use Remote Heap Walker??
I'd say we need to add a memory visualization tool that shows you what parts of the VM space are used -- something like "mi full" from the target control window.
I also think we need to add a handle enumerator -- eg. to find out what file handles are open (and by whom), window handles, etc.

Tell me what you think is most important! What's the #1 debugging tool you wish you had? What's the #1 feature our existing tools are missing?

Thread Lab materials are posted for EDC participants

sloh — Tue, 06 Jul 2004 16:59:00 GMT

They've finally posted my thread lab handouts and code up on www.edccontent.com. So if you went to DevCon and need that material, you can get it now. The handout contains everything I talked about in my hands-off lab, but also has some details I didn't, mostly samples of other threads (explorer.exe, cmd.exe) interfering with program behavior. Also check out the explanation of how to recognize data loss in section 5.

If you didn't attend EDC and don't have access to the materials, I don't think I am supposed to give them out. But I will be working to make them generally available one way or another -- hopefully to integrate them into Platform Builder help.

Sue

I survived!

sloh — Fri, 02 Jul 2004 21:49:00 GMT

Hooray, I survived DevCon! The kernel talk and perf talk were ready to go by Monday, and after many short nights last week and this week, my lab was ready “enough” by Wednesday morning. The handouts are missing some screenshots and a lot of polish. The program I demo'd with was not quite as complete as I had hoped. But overall I think the content was good.

Sounds like I need to work on my presentation skills :-( so my apologies to any attendees who went to my talks. Even though I know the material I guess I still get too nervous speaking about it. The kernel talk was really too much for one session; it should have been a 100-level basics talk plus a 200- or 300- level OAL & driver development talk. I goofed my lab and ran 15 minutes beyond time, because I had the wrong end time in mind, and a lot of material to cover. I should have skipped going over how I built my image, and just dived right into the meat of things.

Ahh well. Hopefully someone benefited. I'll be cleaning up my lab materials & demo code shortly, and hope to get it all into our SDK documentation or something. I'm pretty excited about my lab handouts because I think they contain a lot of information people haven't easily found on their own before.

Sue

Another DevCon talk

sloh — Tue, 18 May 2004 01:59:00 GMT

Looks like I'll also be giving the “Inside the Windows CE Kernel” talk at DevCon. It means a little bit of studying for me but it shouldn't be a disaster. It's a lot of content for an hour though.

I think my Hands-On Lab with CeLog/Kernel Tracker was turned into a Hands-Off Lab. Which is too bad, so many people don't use our tools because they're intimidated, when it's actually very easy. I would like to develop the content in a way that it can be converted to a hands-on lab anyway, say with a typical CEPC, so that people could go through it separately on their own time.

Get excited about DevCon! I am!

Sue

See you at DevCon!

sloh — Tue, 11 May 2004 02:43:00 GMT

Sorry I haven't been posting any content. Mainly I expect content to come out of newsgroup interaction and general customer questions, and I have been a slacker about being on the newsgroups. Too many high-priority things going on right now.

I'll be giving a talk (“Tools and Techniques for Investigating Performance Problems on Windows CE-based Devices“) and a hands-on lab (”Tools and Methods for Examining Thread Behavior in Windows CE”) at the Windows Embedded DevCon in June. So this will be your chance to pick my brain for how to use our profiling and threading tools. Suggestions and questions about my talks are more than welcome, they're encouraged!

You can find abstracts for these and all the other DevCon content at: http://www.windowsembeddeddevcon.com/tech_content.asp

[Yikes, why did I make the titles so long?]

Sue

This posting is provided "AS IS" with no warranties, and confers no rights.