The Old New Thing

It seems unlikely to me that a really old program would rely on invalidating the whole desktop. I can only imagine that it would be using that trick to repaint all of its windows. Why can't the new DWM simply take that as a cue to repaint all of the windows belonging just to that thread or process that called Invalidate?

If I remember correctly, when you talk about compatibility hacks, Raymond, people often say "do it but make it very slow" to which you reply "slower programs don't sell the OS"

Still true, but in that case maybe doing it would have given a clue to the authors of the "many programs written in years past [that] also rely on that behavior" and microsoft could have gotten rid of the hack eventually.

Maybe be making it *noticeably* (by the author) slower would have been out of the question (it would have to be very slow to be noticed) but, I don't know, making it blank-then-refresh the desktop three times would have been enough to make the author wonder about it ("Why three times?? I told only once you stupid Windows! Let's see, what do they say about that buggy function... Oh-oh... looks like my instinct wasn't right and NULL doesn't mean what I thought!").

And saying that "now it's too late" isn't right either, since this bad behaviour has been noticed loooong ago (when changing the API behaviour in the case of invalid parameters) and if the counter-measure had been implemented *then* instead of saying "oh it's too late", maybe *now* microsoft could have moved to a cleaner code base!

The DWM team has made the same mistake and won't be able to get rid of it for DWM2.

Well... it's too late now... ;)

if ((hwnd == NULL)&&(DebuggerAttached())

{

 OutputDebugString("NULL hwnd to clear desktop is undocumented, unsupported and deprecated and will not be supported anymore. Drop it. We are tired supporting people like you. Next time a relay will close the contact between your keyboard and the 220v wall plug. Remove this bug or die.\n");

 DebugBreak();

}

this would not solve the problem today, but if it was done let's say in 2001 in XP(maybe with *slightly* different words), today we would have not these problems anymore.

Or you can made the trick work in Win32 but not on Win64 and wait on natural platform selection to evolve..

From the original post:

> Changing nearly anything in the window manager

> raises a strong probability that there will be

> many programs that were relying on the old

> behavior, perhaps entirely by accident, and

> breaking those programs means an angry phone

> call from a major corporation because their

> factory control software stopped working.

The Linux folk are having the same deliberations. Should they throw out the Metacity window manager and go with compiz/beryl or should they extend Metacity keeping all the compatibility hacks in place?

Luckily, the Linux development process has so many levels (unstable WM, stable WM, WM integrated into unstable desktop env, WM integrated into stable desktop env, desktop env integrated into unstable distro, desktop env integrated into stable distro) that there's lots of chances for new systems to be tested and hacks put back in, prior to arriving on a user's desktop.

I notice that the heap memory allocation function, HeapAlloc, detects that a debugger is attached and fills the allocated memory with 0xBAADF00D, which it doesn’t without a debugger. I imagine many compatibility hacks could be disabled in this mode and replaced with warnings of varying visibility (ranging from OutputDebugString to message boxes to blue screens).

This is a job for a shim.  Indeed, most perpetuation of crap decisions should be dealt with by using a shim.  It stops new programs being written to demand the broken behaviour, but lets old programs work.

"But what about applications we won't know to put on the list of programs to be shimmed?"

Obvious answers (to be taken together to provide a satisfactory solution):

1) Make shim administration easier for people to use.  ACT is annoying, to say the least, and not widely used.  Give it some love to make the UI nicer (in particular, allow it to work "live", rather than having to have the silly patch files that then have to be merged locally; that's appropriate for some situations, but totally non-obvious for a user trying to fix a program)

2) Make shim diagnosis easier.  The ACT has bugger all information about what its various hacks do, and there's no easy way to find out what shims might fix a given situation.  Yet this could be done.  It already has the API profiler/logger shim.  Make it process the output of that.  This way it can actually see "hmm, that's invalidating a lot of null windows" or "that's dereferencing a lot of freed heap pointers" (etc.) and propose appropriate fixes.

3) Update the list of needed-shims regularly

4) Make the list of needed-shims community-based (much like certain spyware programs allow what is essentially "voting"; if lots of people report program X needs shim Y, it gets put on the list)

5) Don't allow any shimming or workarounds in 64-bit APIs.  There's just no excuse for it.  If a developer's recompiling then they have to retest the entire app anyway, so they can damn well fix their problems.

Raymond, I guess my question is just why an app would rely on repainting the whole desktop? If it has multiple windows, I can see where somebody would just invalidate the whole desktop rather than enumerate all their windows. But what sort of app would require every window on the desktop to repaint?

DrPizza, you can't take backcompat out of 64-bit Windows. If people can't just recompile and get it to work, they'll simply say that a 64-bit version is unsupported. All you'd end up doing is reducing the number of 64-bit apps.

Here is a question ... when is compat broken? Just if security?

Gabe: One may need to repaint more than just an application's windows. Those conglomerate apps so popular a decade ago provide an example. Third party controls or groups of apps stitched together with DDE might not update in a timely fashion. They will always repaint when the entire desktop repaints.

Like any clever trick, Null window got overused. Once many applications all try to repaint the desktop, video performance has to suffer.

Raymond: "You have this program from 1998 that your company relies on for its day-to-day operations, and now it runs three times worse"

Since "their original plan was to get rid of the compatibility hack", those programs would have been dead anyway.

But I agree that my proposition (and therefore theirs ;-) ) is not great, because it still means that the users will notice the change when they upgrade the OS and not when they buy the product.

Anyway, if I were in this situation, I would think hard to find a solution that prevents new buggy programs to be released, and only if none is sensible would I keep the compatibility hack in place and invisible.

For unknown applications:

some have suggested debuggers detection. I think it is a good idea, since I don't know many programmers that would release an app they couldn't run in the debugger.

If it were possible, maybe activating the hack depending on the app build date: if it is an old app then do the magic, if it is a new app (ie: being developped and run) then do what the doc says (eg: NULL is invalid)

Detecting the application build date could be accomplished by reading the PE header (if such an information is present in it), by looking at the target architecture, the DLLs wanted (some didn't exist in 1998), the type of resources etc... The loader would check that and set up a flag somewhere to activate the compatibility hacks.

(For known applications, the necessary compatibility hacks are already known and used)

Last thing:

"You'd be surprised how many programs crash when you run them under the debugger because of this"

Are you talking about new programs or old apps you just debugged to know why they crashed on a new OS? Is there still programs that are released without having seen a debugger?

"When you see this, are you going to say, 'Aha! I bet a shim will fix this!'"

No, I'm going to do like anyone else and ask google.  Google will tell me that a shim will fix it.

I think Microsoft needs to take responsibility the design decisions they made 10 years ago and drop these kind of hacks from their code.  Every library is going to have some number of bugs or glitches that programmers will take advantage of, but Microsoft makes the problem much worse by effectively condoning the abuse of these glitches.

How about having new versions of windows output nasty error messages to the system log.  "Program XYZ has invalidated a null window in system call Blah - please notify the program's vendor and see if a fix is available."

Now, lots of people don't know the logs exist, but perhaps enough do that support calls, complaints, and the like will pressure the vendors into fixing the problem in future versions.  No, it's not a fix for orphaned software but maybe it'll allow you to get rid of these shims in another decade.

Robert:

>>

this would not solve the problem today, but if it was done let's say in 2001 in XP(maybe with *slightly* different words), today we would have not these problems anymore.

Or you can made the trick work in Win32 but not on Win64 and wait on natural platform selection to evolve..

[Why would this have been a valid solution in 2001 or XP? Who's going to fix that program from 1998? -Raymond]

<<

Nobody's going to fix that old thing, but they will stop (or retard) the use of it today. I personally favor the shim approach, since nobody's going to stop doing this thing if it still works.

And the "major vendor" in this case was M$ :)

"DrPizza, you can't take backcompat out of 64-bit Windows. If people can't just recompile and get it to work, they'll simply say that a 64-bit version is unsupported. All you'd end up doing is reducing the number of 64-bit apps."

Then when the world goes increasingly 64-bit consumers will put them under pressure to shape up or die.

If its not already there, the checked build of windows should display or log (by default or with a special option) any time that an application uses any kind of backcompat hack (either one thats globally enabled or via a shim). Basicly, if you use something that microsoft considers "depreciated", the checked build and/or some kind of application verifier would tell you "dont do that" and you could change the code.

I dont believe something like this already exists but I could be wrong...

> They contacted the vendor who agreed to fix the bug, but the fact that a modern program still relies on this ancient behavior gave them pause.

http://msdn2.microsoft.com/en-us/library/c8xdzzhh.aspx

Why they need to contact in person if there should be compiler warning for such cases??

Nekto2: This is a run-time thing. InvalidateRect(someRectPtr) isn't deprecated, InvalidateWindow(NULL) is. They're the same function. You can't deprecate one without deprecating the other.

Gee, when I connect my home computer to my office computer using Remote Desktop, the icons start to paint, then they all flash, then they continue painting, then they all flash again... they flash three times in total before being completely painted!

I wonder if Nawak is on to something there...

[Okay, I guess I still don't understand. Fire up the time machine, let's go back to 2001. You're at a design meeting for Windows XP and somebody asks, "So what should we do about this null window issue?" What's your answer? -Raymond]

Shim the known applications, remove the general fix for new apps, and die messily in the checked build. If you want to kill this, there will be some pain involved. Maybe I don't get to make that call, but the tradeoff should be clear.

[The world won't go 64-bit until there is a critical mass of 64-bit apps. If you make it hard to port to 64-bit, you won't have many 64-bit apps. See OS/2. -Raymond]

Sure it will - look at linux. It went 64 bit, and now you can run 32 bit apps as well as 64 bit ones. Why would you run notepad as a 64b app, anyway? Only to ditch a dependency, so far as I can tell - the early adopters are CAD, databases, and other big memory apps. You bitch about app compatibility headaches, but when someone proposes a sensible break point (no, raymond, you can't just rebuild. You know better), you shoot it down.

I don't see any good reason to consider to plan to get rid of *any* compatibility hack. You either care about backwards compatibility or not. In the latter case, you should better rename/rebrand the software.

'They contacted the vendor who agreed to fix the bug,...'

I've just checked a 4 year old Win32 SDK help file and this is well documented (If this parameter is NULL, Windows invalidates and redraws all windows,...), I don't think invalidating the null window with InvalidateRect can be considered a bug at all.

In any case, correct way to handle this shouldn't be altering current behavior but introducing a new function (InvalidateRectEx), and then depreciate the original function 20 years later.

I wud believe there is tons of code out there which gets copy pasted or re-used when an application's new version is revved since it just works. These code traces when encountered should either be removed or commented to be incompatible going forward.

What surprises me is that more games arent 64 bit, I would have thought that with gamers being more likely to have 64 bit CPUs and with games being performance hungry, there would be advantages to porting games to x86-64...

DrPizza,

>>

ACT is annoying, to say the least, and not widely used.  Give it some love to make the UI nicer (in particular, allow it to work "live", rather than having to have the silly patch files that then have to be merged locally; that's appropriate for some situations, but totally non-obvious for a user trying to fix a program)...

>>

What version of ACT are you referring to?  Tell us what we can do to make it better.  It might not be too late...

Having bene the victim of my whole desktop flashing randomly in the past, I have very little sympathy for this backcompat fix.

esp. with a multi monitor system the visual jarring and general system slowdown incurred by a desktop repaint is bad enough to warrant seriously considering simply breaking apps that rely on it.

Perhaps a new approach is needed here. Certainly all of us non-MS'ers DO NOT UNDERSTAND the magnitude of testing and decisions that have to be made. What if all applications that are found to break obvious rules were documented? Then, at least us IT'ers could bug the vendors to fix, at least for the apps that are still "alive".

>> [Why would this have been a valid solution in 2001 or XP? Who's going to fix that program from 1998? -Raymond]

Noone, but it would have been a good time to begin inspiring developers that the hack is.. an hack and not the correct way to do things.

Either you do something like that or you're doomed to have random developers around the world use the hack again and again (most of them do not even figure out it's not the right way).

2001/XP was the release date of the most popular OS version. You can bet most developers are running on XP machines and will run on them for a few years even after Vista is released (especially since the market of new machines is slowing down - a 2003 machine is still quite good despite its 3 years age). If it was introduce in 2001 you could safely assume very few applications were developed or maintained from 2003-2006 with the hack in it and you could end up removing the support in say 2010 (or leave it as opt-in in the compatibility page).

Does Microsoft publish a list of app compat hacks in Windows?  It should, under a heading of "Don't do these things!".  Each hack should include a pointer to the documentation on the proper way to accomplish the task.

"[The world won't go 64-bit until there is a critical mass of 64-bit apps. If you make it hard to port to 64-bit, you won't have many 64-bit apps. See OS/2. -Raymond]"

64-bit apps will happen no matter what.  It might take a while, but it'll happen.  There's no need to worry about it.  Just make Win64 "clean" and let the developers do the rest.

Does the App Compat Toolkit just examine your system? Perhaps I am wrong with my quick look. What I meant was a huge list of apps that do weird stuff, that MS has ever found. Perhaps MS can't do that without wrath of the app vendors.

I'm with raymond on the compatibility stuff, but I'd like to know what that beta app was accomplishing?

I can understand invalidating a region of the display with NULL. For example if you've drawn something like a selection rect XORed basically on top of everything (which you should not do since it gives the wrong result with always-on-top windows).  It could be useful for drawing splitter bars interaction, which need to draw on top of the child windows of the splitter view.  Personnally I create a window on top to draw those things into though, it's the only way to get proper clipping - but I can easily imagine people could not think of doing that.

I could see people easily passing a NULL window handle by calling InvalidateRect are the wrong time.

If that's not the kind of use it is, I'd like to know more example.

My suggestion is to keep the workaround (hey, it's already documented in MSDN anyway) but if they're running under the debugger, paint every window in their process a nasty color like bright pink or green before you redraw.

[I'm sure the people who bought 64-bit Alpha machines shared your enthusiasm and look where it got them. -Raymond]

Never mind that Alphas cost more (generally) and ran existing apps slower, while the current 64 bit stuff is cheap and runs fast. Also see pentium pros.

You see the trap you've laid yourself, don't you? You hate having to support brokenness, but aren't willing to change anything.

Anyway, the InvalidateWindow(NULL) thing is something to catch in a check build.

Well, the 64 bit upgrade is going to require testing and code changes anyway - it looks like a great time to drop some of the app compat things.

"Anyway, the InvalidateWindow(NULL) thing is something to catch in a check build."

The last time I used a checked build was in Windows 3.1!  I don't know anyone using these anymore.  (I also used to use codeview a monochrome monitor with an hercules card!  Those were the days.. I miss them :( )

Bounds checker it is for the most developers who actually look for problems .. (if the  company that owns it now hasn't killed it yet..  MS should buy it)

If you want people to use Application Verifier, include it in Visual Studio and have it run automatically in Debug mode (optionally, but by default). This would make an enourmous difference.

Checked builds, and application verfier are two things that I, as a developer, just dont know how to use.

I mean, these products are not exactly integrated into the default development environment. Sure, im my case I *should* know better. But most developers dont even know these tools exist at all.

"Complaining in a checked build is not going to help. I would hazard that 99% of software developers don't run a checked build, and I'm probably underestimating. We can't even get people to use Application Verifier for goodness's sakes."

But I'd hazard that 99% of developers do use the debugger and that 99% of end-users don't.

Now, I'm rather more certain of the first of those 99% figures, so if anyone can point to a body of users who habitually run programs under some sort of watchdog process that uses the debugging APIs, I won't be entirely amazed, but in the absense of such users, Robert's suggestion looks like an efficient way of punishing the programmers whilst letting the end-users off the hook.

You may be tempted to put in a plea for clemency on behalf of programmers using broken third party libraries. Don't bother. Ultimately, they chose the bad vendor. Either they are willing to revisit that decision or they should pay the price!

Would a checked build of NT3.5 have ensured that the application written in 1994 would be compatible with Common Controls v6? Would running the application though the App Verifier under Windows 2000?

Applications written in pre-Delphi 7 are not at all compatible with the Common Controls version 6. Would the Application Verifier have caught it? Would a checked build? i don't know what Borland did, but it did it wrong - inadvertently.  Is Borland not running a checked build? Is Borland not using the App Verifier?

In the same way a web programmer puts a strict DOCTYPE at the top of their page; i like the idea of an API DisableAllCompatibilityHacks(). Or make it a PE flag. Make it a side-by-side assembly manifest. At least this way when i run my program, i'll see problems; and be able to correct them.

If the program doesn't fail, and if i haven't yet come across an article by Raymond explaining how it should be done (as opposed to the docs explaining how you can do it), then i'll rightly assume that i got it right - until some later version of Windows comes along and breaks it.

Although, i guess (and someone could confirm) that a DisableAllCompatibilityHacks() call is too difficult to implement; it would require a whole different build of Windows (aka a checked build.)  Perhaps if checked builds were free, and didn't cost a license, they'd get used more. They're slow enough to be their own deterrent to piracy.

NOTE: ActiveX controls written in Delphi are incompatible with ie7. It has to do with the browser now being in one thread, and the ActiveX in another. Is there anything Borland could have done to have seen it coming?

Application Verifier doesn't complain on

 InvalidateRect(NULL, NULL, True);

And the SDK says, " If this parameter is NULL, the system invalidates and redraws all windows, and sends the WM_ERASEBKGND and WM_NCPAINT messages to the window procedure before the function returns. "

It's not an undocumented feature - it's a documented feature. Woe to those who use documented features.

There is a way to fix at least one part of this -- the one where people use wrong parameters intentionaly.

It would require some work on a compiler but it is doable and IMO not too hard.

Build a database of API calls with invalid parameters and let a compiler check the code it is generating against it. If it detects that it is pushing NULL onto a stack before call to InvalidateRect() then it should stop compilation with an error saying that NULL is not a valid parameters. Of course that wouldn't catch runtime assigned values (i.e. non-constants) but it would help tremendously to prevent new errors.

"[Are you saying that this behavior should not have been documented? -Raymond]"

Yes.

Or at the very least a comment about "this feature is non-standard, unsupported and may be removed in future versions."

Instead, it sounds just as valid as calling:

InvalidateRect(dc, NULL, False);

Unless passing a null rect is also a compatability hack.

My point is, the only source for valid SDK documentation seems to be Raymond's blog. Which isn't good for all the API calls he hasn't blogged about yet.

How about a message box saying:

This program has requested that all windows be invalidated. What would you like Windows to do?

[Bug ISV to fix it] [Block always] [Block this time] [Allow this time] [Allow always]

[Bug ISV to fix it] should be the default and other buttons should be grayed and only unblocked one by one in 5 second intervals. So taking the "easy way out" with [Allow Always] would take 20 seconds giving user enough time to think about what they are doing.

Of course, the developer of the appication would also get the same prompt while testing so the new stuff would be fixed immediately.

Let's get one thing straight: Using InvalidateRect intentionally with the NULL window handle is NOT a bug. It's documented, legal, part of Win32 API. It may have been a 'compatibility hack' but it's not now and it cannot be considered as such anymore. Ian Boyd is right, you shouldn't document 'hacks' like normal behavior, if you do so, it becomes the normal behavior. BTW, I wonder if using MessageBox with a NULL window handle was a 'compatiblity hack' too, was it?

Best regards,

Burak

>Does Windows use its psychic powers to figure out the ISV's email address?

Why not use some cool programming powers for that?

For example some GetVersionInfo() accompanied with Uninstall registry key search for vendor contact information? Perhaps even some database query with exe name and checksum to one of your servers?

>Do you think many ISVs will sign up for this?

Why should they sign up for this? You just make an application database, get support contact from their website and send an email with error details on behalf of the user. It is not exactly impossible thing to do considering that even major vendors have those issues and all of them have support emails (plus you most likely already know who they are).

>Do you think users will say...

I don't know for sure, but instead of just seeing unexplainable flickering each time some program calls InvalidateRect() with NULL window pointer they would at least have an option to complain to the vendor, or even to prevent the nasty program from doing that again.

Whichever way you turn it -- it is bad the way it is right now.

Moreover, I am sure you are aware that people can learn?

When they notice that certain program misbehaves, they usually go to Control Panel / Add/Remove Programs and get rid of it and find an alternative which works better.

BUT... you need to make them notice that the program is misbehaving, not the OS.

Otherwise you will get the blame as in "Gosh, the desktop flashes randomly for no apparent reason. Must be Explorer acting crazy again." <- those are your words Raymond.

On the interface design, this is interesting read:

http://www.joelonsoftware.com/printerFriendly/uibook/fog0000000249.html

i still have to wonder, how am i (a developer) supposed to know if i am using an API wrong? i don't want to use it wrong, i don't mean to use it wrong, i don't try to use it wrong.

But the docs don't say it's wrong. App Verifier doesn't say it's wrong. Return error codes doesn't say it's wrong. Event log error messages doesn't say it's wrong. Exceptions don't say it's wrong.  It is not wrong - except an MS guy blogs about it, and tells us how people are using it wrong.

An excellent example is InvalidateRect with a null DC. There is nothing i can find anywhere that says passing a null DC is not the intended use, and you shouldn't do it. i can't find that information anywhere - except here.

How can anyone ever know that it's wrong?

Help me help myself use the API properly.

People have come up with all kinds of ideas that Microsoft could choose from to address this problem. Sure they have problems - but at least something would be better than nothing.

[Are you saying you're passing NULL intentionally? Why are you invalidating the entire screen on purpose? That's the real question. -Raymond]

i can tell you what the person who is passing NULL was thinking - or hoping to achieve.

"i want a way to invalidate all windows in my process."

Turns out they can. i suppose they *could* keep a list of all windows created by their application, but Windows already has this information; so i'll just ask it to do it.

And perhaps the DWM guys that do that instead; go through their list of windows and only invalidate ones owned by the process calling InvalidateRect(NULL)?

But the real info i'm begging for, is how to know what things are bad.

<joke>

Like the other little known compatility hack in SHFileOperation. You're supposed to triple null terminate the list of filenames. But a well-known piece of software from a big-name company instead only double null terminated the filenames. Another compatibility hack was born.

</joke>

It may be a joke, but for all i know it could be real.

i don't want to be another barrier to innovation by relying on compatibility hacks. But how can i find out if i am in fact relying on one?

Ian, you should never, ever try to invalidate all windows. They don't belong to you or to your application. Period.

PingBack from http://www.eriknovales.com/blog/index.php/2007/12/20/even-when-you-wrote-the-operating-system-you-still-have-to-work-around-other-peoples-bugs/