Mike Dodd's blog

Masking bugs with dual processors

A while back, I had to track down a bizarre bug in a test build. A tester reported that on his machine, some code that ran to grab a decompressed video image took several seconds to run. On my machine, it was near instantaneous.

My machine was certainly faster than his, but it didn't seem like it should be that much faster. I turned on some tracing, ran it against the code on my machine, and looked to see what the code was really doing. Nothing looked out of the ordinary at first glance.

Then I collected the same traces on the test machine. The trace looked completely different -- it was decompressing hundreds of frames more than it was on my machine. Not only was his machine slower in general, it was also having to do far more work.

One interesting difference -- my machine was dual proc (and hyperthreaded), and his machine was single proc (with no hyperthreading). I used Task Manager to bind the test app to a single processor on my PC. Sure enough, now my machine took several seconds, and the trace looked a lot like that of the slower machine.

A quick review: most A/V engines (such as DirectShow) involve at least a couple of threads. Typically, the app calls into the A/V engine on one thread and issues commands (such as pause, stop, run, seek). The A/V engine will, on the calling thread, do some setup, and then kick off another thread to pump the media samples through the various components (parser, codec, renderer). That way, the app can get control back and go do something else (like process user input) while the A/V engine plays audio and video.

With DirectShow, a seek command first issues a flush command. The idea is that since any component in the graph can store samples, you want to clear all those out before you jump to the new time. So the app thread calls seek, which turns into a flush, which goes through the various filters in the graph.

The base renderer class responds to the Flush command by setting an event which unblocks the rendering thread. That way, if the renderer was blocked waiting for it to be time to render a sample, or because the graph was paused, it will unblock so that the streaming thread can be released. In this case, the graph was paused, so the renderer was blocking the streaming thread.

There were two bugs: the first was that one of the upstream filters was calling the downstream filter's input pin's BeginFlush() method, and after it returned, setting the event that would cause the renderer thread to stop pushing samples. The second was that a codec filter, between the first filter and the renderer, was returning S_OK instead of S_FALSE from its IMemInputPin::Receive call.

And here we get to a design decision that exacerbated the bugs: the app thread was running as low priority.

The BeginFlush comand flowed through the graph, on the low-priority app thread, until it got to the renderer, which set the event to unblock itself. The rendering thread was running as normal priority. The renderer dropped the sample, knowing it was in the middle of a flush, and returned back up the stack to the buggy filter -- which hadn't yet gotten the command to stop sending samples, and, since it got S_OK from the codec filter, didn't realize that there was a flush going on. So it sent another sample, which was decompressed and dropped by the renderer -- and so on.

The only thing that stopped it was the NT thread scheduler, which, if a thread has been blocked for three seconds, boosts its priority to prevent it from being completely starved. So the app thread eventually got boosted and then got to send the command to the renderer to stop, and everything finished.

Two observations from this: a few years ago, the problem with doing multi-threaded programming was that almost no one had dual proc machines, and there was a whole class of bugs that were masked by single processor machines. Now almost everyone in development has a dual proc (or at least a hyperthreaded -- kind of like proc and a half) box, so now you can get bugs masked by that.

The second is that, in this case, there was really no reason for the calling thread to be running as low priority. (Of course, if it hadn't, these two bugs probably would have been masked.) It was running as low priority to try to avoid blocking the UI and other tasks from the relatively unimportant task of grabbing a bitmap from a video file. But it didn't matter -- that low priority thread wasn't actually doing any of the heavy lifting. All of the video decompression and other hard work involved in processing the video was running on other threads, created by the various DirectShow filters, and those were all running at normal priority.

So, try code on both multi-proc and single-proc boxes, and verify that you're actually accomplishing what you think you are by setting thread priority.

Ballmer's enthusiasm

As always seems to happen around the time of Microsoft's annual company meeting, it's relatively easy to find various people complaining about how Steve Ballmer's semi-public displays of enthusiasm for Microsoft seem silly or unprofessional.

Before joining Microsoft, I worked at Apple for several years. And one thing I learned there is that it's no fun at all working under a CEO that doesn't seem to have any passion for the company. Spindler never seemed to have any particular interest in the technology that Apple was producing, and Amelio, who should go down in history as one of the worst CEOs of all time, never displayed the slightest understanding of or enthusiasm about what the company was doing. (Jobs, to his credit, was very different. The man is brilliant, and clearly cares passionately about the technology and what Apple does.)

So Ballmer is enthusiastic about Microsoft? Thank goodness.

More Zune Info

David Caulton is blogging about Zune at http://www.zunester.com/. He's posted about codec and formats in Zune, topics I spent much of my time working on. His most recent post is on "Clearing up content formats."

Also check out his picture of "Zune Central Command" - I spent a while shortly after I moved up here working with a bunch of other developers in a conference room off of that giant empty room in his picture. After that, we worked for a while in a big empty space at the other side of the building because they were going to renovate the room we were in. Then most of the team moved to a new building, and my group moved back to our original home on the main campus.

Demonstrating just how frustrating space planning can be sometimes (and how crowded the campus is right now), I had yet another move after that within my building. So in the six and a half months I've been here, I've moved my primary workspace four times, not counting the initial move up here from California.

Creating write-only code: the Reference (&) Operator

Write-only code is a term used to describe code that can easily be written, but that no one but the original author can easily read. Since invariably far more time is spent reading code than writing it, this is NOT a positive term.

Recently, while participating in a code review, I was reminded of one of my least-favorite C++ features -- the ability to use a reference declarator in a function. Most code that needs to change a variable of the caller is written like this:

void foo(long *a)
{
    *a = 3;
}

void caller()
{
    long var = 0;
    foo(&var);
}

However, by using the syntax "type-id & cast-expression" in the function declaration, you can write this:

void bar(long &a)
{
    a = 3;
}

void hard_to_read_caller()
{
    long var = 0;
    bar(var);
}

If you look at the assembly code generated, it's exactly the same in both cases. The (x86, unoptimized) calling code looks like this:

lea         eax,[ebp-8]
push        eax
call        @ILT+459(?foo@@YGXPAJ@Z)

So what's the difference? In the first example, it's obvious from the calling code that the variable 'a' can get modified. You don't know for sure that it will, but since you're obviously passing the address of the variable, you know it's a possibility. Now you can go check the definition of 'foo' to see how it might change your variable.

In the second example, there is no clue at all from the calling code. The call to bar looks no different than if bar were defined as 'void bar(long a)'. Without looking up the function prototype for bar, you'd assume that var won't be modified. In short, this is write-only code.

This is one of the (many) things that the C# team got right. You must use the ref or out keywords from the calling code when passing variables to a function that can change them (ref for input/output variables, out for output-only variables).

Move to Zune team

I clearly haven't written anything in here in forever. In March of this year, my family and I moved from California to Washington, and I now work on Zune on Microsoft's main campus in Redmond. (The picture to the right is of my kid playing with my Zune prototype.)

I'll try to start posting here again.

Reserved fields

Via Larry Osterman's recent series on concurrency, I ran across an older post from Raymond Chen on why you shouldn't use reserved fields in internal Windows data structures.

About ten years ago, when I was working on the QuickTime team at Apple, we got a report from a beta tester about some third-party application that was completely broken with the new version of QuickTime. Several hours of debugging in MacsBug later, I discovered that the problem came from that application using a reserved field in the heap data structures. In that version of QuickTime, we had included a patch to the memory manager to work around a problem in older OS versions. That patch needed to use the reserved field, which obviously conflicted with the third party app's use of that field.

When I sent the vendor an email about this, the response came back: "But you can't use that field; it's reserved!"

Umm ... yes.

New baby

This past week was my first week back to work following my paternity leave for my newly adopted son. Nathan Michael was born on December 17 in Salt Lake City. My wife, Meredith, and I got to be there at the birth. The next day, the birth mother signed the papers placing him with us, and late on December 23, the paperwork between Utah and California (our home state) finally cleared letting us fly home on Christmas Eve.

It’s an open adoption, which means that we stay in contact with the birth mother, sending her letters and pictures, and Nathan will always know that he was adopted. It’s supposed to be much better for the child that way, since there are never any sudden surprises (‘what do you mean I was adopted!?’) later on in life for the kid.

I’ve been very grateful for the Microsoft benefits. I got four weeks paid paternity leave, and Microsoft will even reimburse us for $5000 of the adoption expenses for Nathan. It was wonderful to have January off with Nathan.

There are, of course, way more stories and pictures of Nathan on my personal blog.

And now, back to work.

Frustrations with AIM and Trillian

For a couple of years, I’ve used Trillian for my instant messaging client. I’ve stopped recently, though, because of problems with Trillian. I wrote a longer post about this on my personal blog last month, but basically, my two big complaints are that Trillian doesn’t support the Tablet Input Panel (TIP) on Tablet PC’s, and there’s no support for MSN Alerts. Even the new release of Trillian 3 doesn’t support TIP or MSN Alerts, and there’s no indication I could find as to when, if ever, it will.

Because of the lack of MSN Alerts support with Trillian, I’ve been using MSN Messenger for the last few months, and recently started using the beta. And I’ve been really happy with it. But most of my friends are using AOL Instant Messenger, so for that, I switched to AOL’s AIM client.

AIM does support TIP, so at least I can use it on my Tablet PC in slate mode. But much of the UI is rather frustrating. The most annoying thing it does is pop the Buddy List and ‘AIM Today’ windows to the front whenever I sign in. This means that I open my laptop, start using it, and a couple of minutes later, when it decides to notice that I have a network connection, I suddenly get these two extra windows popping up. I’d love to find out how to get rid of that feature.

Two features I miss from Trillian:

• The ability to rename contacts. I have a hard time remembering who owns the screen name “koswrgex793”. With Trillian, I could rename the contact to something easier (say, “Bob”).
• Logging. Trillian could log conversations; I’ve found no way to do it with AIM. I found a few references to logging programs for AIM on Google, but most of what I found indicated that it was for earlier versions of AIM and was no longer compatible. On the other hand, I remember hearing that Google’s desktop search tool could search AIM conversation logs, so it seems like there must be some way to do this. Maybe I’m just missing something.

Reloading the .NET performance counters

On my laptop running XP SP2, I noticed that Process Explorer wasn't highlighting .NET processes anymore. Not only that, when I went to the options panel to configure the process highlighting, the .NET option was grayed out.

I sent a note to the tool's author, Mark Russinovich, who was nice enough to respond within a few hours, saying that his tool checks for the presence of .NET by looking for the .NET CLR Memory performance object.

unlodctr .NETFramework
lodctr c:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\corperfmonsymbols.ini

TV as relaxation

Many people, when they come home from a long day at work, sit down in front of the TV to unwind. It is, for example, what my wife is doing right now.

I came home today from a long day at work where I spent several hours staring intently at a TV, turning aside only occasionally to make some adjustment or other to various video settings. The thought of coming home and sitting in front of a TV isn't all that appealing at this very moment.

This happens a lot with test content, too. There are some video clips that I've seen so many times that I can practically see them in my sleep. Years ago, I worked on the QuickTime team at Apple, where I saw the music video to Sarah McLachlan's "Building A Mystery" innumerable times. Even now, when I hear that song on the radio, I can see each frame of the video in my head -- including the compression artifacts of the video codec we were using at the time.

Another IPTV partner

Yesterday at IBC, Microsoft announced another partner for our IPTV services: Telecom Italia.

The list now includes Bell Canada, Reliance (in India), SBC, and Swisscom.