What can I do with the HINSTANCE returned by the ShellExecute function?

The Old New Thing : How do I test that return value of ShellExecute against 32?

The Old New Thing : How do I test that return value of ShellExecute against 32? — Wed, 08 Nov 2006 18:00:11 GMT

PingBack from http://blogs.msdn.com/oldnewthing/archive/2006/11/08/1035971.aspx

DDE ..? How does it matter..?

Pranav ... Blogging — Thu, 11 May 2006 10:21:45 GMT

When you are talking about office programs what happens when you open a new document seems to depend...

re: What can I do with the HINSTANCE returned by the ShellExecute function?

Norman Diamond — Tue, 09 May 2006 03:58:02 GMT

> Cast it to int, cast it to INT_PTR, it won't
> make any difference.

That is obvious only to someone who either knows or has reverse engineered the code. MSDN isn't quite sure about that, your colleagues seem sure about the opposite, and even you didn't quite seem so sure a few replies back.

> What error codes are born as long longs?

I don't know, but the responses of both you (prior to this one) and your colleagues made it obvious that there are some.

In a 64-bit program, HINSTANCE is 64 bits. Now consider a case where ShellExecute has set that field to the value 2**33 + 1. If the program casts this to type INT_PTR then the result of the cast will be greater than 32 and the program will think that it represents success. If the program obeys MSDN and your colleagues, casts this to type int, and sees the result of 1, then the program will observe that it represents failure. Since your company is so firm about the incorrecness of casting this to INT_PTR, it must be because some failure returns will be misinterpreted as successes unless they get cast down to int.

re: What can I do with the HINSTANCE returned by the ShellExecute function?

Raymond Chen - MSFT — Mon, 08 May 2006 07:54:25 GMT

Cast it to int, cast it to INT_PTR, it won't make any difference. (I'm not sure what you're referring to in that last sentence. What error codes are born as long longs?)

re: What can I do with the HINSTANCE returned by the ShellExecute function?

Norman Diamond — Mon, 08 May 2006 04:43:25 GMT

> When I wrote, "You can check if it is greater
> than 32," I mean "You can check if it is
> greater than 32 after casting it to an int"
> [...] Do I have to spell out very last
> excruciating detail?

In this case there was a reason. In the base note you also wrote this:

> GWLP_HINSTANCE (or GWL_HINSTANCE if you're
> still living in the unenlightened non-64-bit-
> compatible world).

As stated, I thought it might be appropriate to be 64-bit aware and cast the HINSTANCE to an INT_PTR instead of to an int. Your employer said no, and here you agree with your colleague, but it's still not completely obvious. It is good that MSDN is literally correct here. In fact if MSDN were literally correct in enough other places then I wouldn't have wondered whether to doubt this one. But your base note didn't say int, and any 64-bit-aware person who reads your base note might not be aware that they should only test the low-order 32 bits.

Saturday, May 06, 2006 6:45 AM by KJK::Hyperion
> Rigorousness OK, but it doesn't take a lot of
> psychic skills to understand what goes on
> behind the curtain. The return value is
> obviously born as a positive int and casted
> into an HINSTANCE with IntToPtr().

Um, since the HINSTANCE used to be an address sometimes, it's not obvious to me that it's always born as an int. I only would have guessed that error returns were born as ints ... sometimes. Considering that 64-bit-aware programs must cast it down to 32 bits, obviously even some error returns are born as long longs but have the error value shown in the low-order 32 bits.

re: What can I do with the HINSTANCE returned by the ShellExecute function?

Steven Edwards — Sun, 07 May 2006 12:18:27 GMT

To go even more off topic.

bviksoe: Some programs that try to be compatible on the oldest Windows version possible such as Win9x still use DDE. I tracked down a bug in firefox recently that was causing a hang during DDE communication.

re: What can I do with the HINSTANCE returned by the ShellExecute function?

bviksoe — Sat, 06 May 2006 23:23:43 GMT

Oh no, you said: DDE.
And here I thought you weren't allowed to swear or use obscene words on this blog! Well, good thing the topic was filed under Historic. So many bad memories...

Could it be that Windows Explorer is the only application on the face of this earth to use this technology still?

re: What can I do with the HINSTANCE returned by the ShellExecute function?

Raymond Chen - MSFT — Sat, 06 May 2006 18:52:18 GMT

It's okay to cast to an int because, as both the linked article and the documentation for ShellExecute note, the error code is an integer, cast to HINSTANCE for compatibility. The return value will never be 2**33+1 since that doesn't fit in an int.

When I wrote, "You can check if it is greater than 32," I mean "You can check if it is greater than 32 after casting it to an int" - but I assumed that was obvious since I'm assuming people reading the entry are already familiar with ShellExecute. Do I have to spell out very last excruciating detail? Do you spell out every last excruciating detail when you talk with your friends?

"When you get to the gas station, turn left."

"Even if the traffic light is red?"

"Sigh. Turn left *when safe and legal*."

re: What can I do with the HINSTANCE returned by the ShellExecute function?

KJK::Hyperion — Sat, 06 May 2006 13:45:16 GMT

and, Norman, the result cannot "happen" to be larger than that. Rigorousness OK, but it doesn't take a lot of psychic skills to understand what goes on behind the curtain. The return value is obviously born as a positive int and casted into an HINSTANCE with IntToPtr(). And int is always 32 bit, even on Win64, so there's no kind of trickery that could make it overflow

re: What can I do with the HINSTANCE returned by the ShellExecute function?

KJK::Hyperion — Sat, 06 May 2006 13:40:16 GMT

Norman: just ditch ShellExecute and use ShellExecuteEx, then. Has nicer error handling and generally nicer everything. As for INT_PTR, I'll just say beware, beware sign extension