Nightmare Invisible Repaint MouseMove Bug on Elms Street
Over this last week I had to debug two issues. If you’re a college student and read this blog, you would have discovered the problems in no time.
MessageBox from Nowhere
As a warm up, the easy one and a new Visual Studio feature: Break All.
The bug is you try and delete this special object which can’t be deleted and it pops up a message box “You must select a shape first.”
It doesn’t happen in the shipped product, so it’s a regression. But this isn’t a message box I engineered. Where the hell is it coming from?
Debug -> Attach to process -> application.exe
Debug -> Break All
Now I have a call stack of who is creating the message box. The problem is there’s none of my code in sight. But! I know where we’re supposed to get when we try and delete a shape. Place a breakpoint, find where we handle keystrokes to begin with (because I didn’t know, learn something every day). Put a breakpoint there, try and delete the special shape.
A-ha! We get there and no message box yet. Some fast F10’s to step through some lines and low and behold I find the problem:
bool fKnowHowToHandle = false;
if( command.Enabled() &&
command.CanGetState(fKnowHowToHandle) )
{
…
}
return fKnowHowToHandle;
We always say we don’t know how to handle something if the command is disabled because of short circuiting. In reality we do know how to handle it, we just don’t allow it to execute at this point. The bug was harmless until someone changed some code way off in another DLL which exposed it.
Nightmare Invisible Repaint MouseMove Bug on Elms Street
This is what I call a “nightmare bug”, mainly because it took me two days to solve.
I develop the feature, it works great. I give it to the tester, no dice (see Dev Magic). Turns out I had done all my work over terminal server but on the console things don’t repaint correctly.
Great, so I add a monitor, keyboard and mouse to the machine and get to work. Finally, I can repro the problem and I immediately see that everything in my code is fine. The bug is then “invisible” as all data points to things should work. Note that this is in shared code and it works great with all the other applications. But I discovered that terminal server rendering uses a different code path in this application.
So I set out on trying to piece together what was so different about this app on console. I discovered that if I moved windows over the area or created a “selection rectangle” as I clicked and dragged the mouse, everything repainted fine in that area. Hooray, different behaviors depending on what I do with the application.
I add some break points and notice that the “good” drawing uses a couple different code paths, where “bad” drawing always came from one particular code path, but sometimes the drawing was “good” too. So all I needed to do was trace through the code and see what in the above functions was different between “good” and “bad”.
Ever try getting a breakpoint to set while you’re dragging out a selection rectangle? Not going to happen. Enter remote debugging [ninja]. There I sat, with one hand on one mouse, dragging out selection rectangles, the other hand on a different mouse placing a breakpoint.
A few hours later, I knew I was close. I had commented out huge pieces of code, the app didn’t redraw very well, but it still had the damn “bad” drawing for my feature.
“What the hell is ‘fCacheDrawing’?” I said out loud.
Placed a filter breakpoint to print out its value every time we run by. Sure enough, “bad” drawing this is true and “good” drawing this was false.
(Side note: is there a way to get Visual Studio to change a variable every time it passes a breakpoint? It would have been nice here to say “Ok, once we get here, always set fCacheDrawing to false” instead of having to recompile).
The hunt was on, F11 to trace through the cache functions and I hit the jackpot: during cached drawing, the application calls SetWindowOrgEx on the HDC that I end up operating on. I then call into someone else's code to do the actual drawing and it apparently draws itself based on the window origin.
L
Back to my code, add GetWindowOrgEx, offset my drawing, the day is saved and its Friday at 10 PM and time to go home.
