Just because you're a control doesn't mean that you're necessarily inside a dialog box
Prerequisites: Moderate to advanced
understanding of the window and dialog managers.
When you're implementing a control,
you need to be aware that you aren't necessarily being hosted
inside a dialog box.
One commenter suggested handling WM_KEYDOWN and closing
the dialog box
as a way to prevent multi-line edit controls from eating the
Enter key.
But the edit control can't do that because people create edit controls
outside of dialog boxes.
How do you "close the dialog box" when there isn't one?
This leads to a related topic brought up by another comment:
Doesn't ES_WANTRETURN do exactly this?
The MSDN states the following (emphasis mine):
"ES_WANTRETURN:
Specifies that a carriage return be inserted when
the user presses the ENTER key while entering text
into a multiple-line edit control in a dialog box.
Without this style,
pressing the ENTER key has the same effect as
pressing the dialog box's default pushbutton.
This style has no effect on a single-line edit control."
I remarked that ES_WANTRETURN is a messy subject.
Now I'm going to show you the mess.
It's sort of like visiting your friend's house when they're
not expecting you and wandering into their bedroom where they
haven't tidied up and there's clothes all over the floor.
The authors of the edit control back in 1981 didn't follow the above
guidance.
Probably¹ because back in the days when the edit control
was first written,
the window manager was still in a state of flux and its design
hadn't settled down.
You can't blame the edit control for not following guidance
that didn't exist.
The edit control implements ES_WANTRETURN
as you might expect:
It include DLGC_WANTALLKEYS in its response
to WM_GETDLCODE, which causes all keys,
including Enter, to go to the edit control.
What's more interesting is how the edit control implemented
the absence of ES_WANTRETURN:
It still includes DLGC_WANTALLKEYS,
but when it receives the Enter key, it first attempts to detect
whether it's inside a dialog box, and if so, it tries to mimic
what the dialog box would have done:
It asks its parent dialog box for the default ID,
sets focus to the corresponding control,
and
simulates input via PostMessage
to make that control act as if the user had pressed Enter.
Since only button controls can be the default ID,
the edit control "knows" that the recipient of the simulated
input is the button control.
The author of the edit control then went in and modified the button
control so that it didn't rely on virtualized input state when
handling the WM_KEYDOWN message.
This is ugly no matter how you slice it,
and it violates so many principles of control design it isn't funny.
For one thing, the way it detects whether the control it hosted
inside a dialog is fragile and can be tricked into guessing wrong.
Next, its mimcry of the IsDialogMessage function
is incorrect.
When it wants to invoke the default button, it does so by simulating
input, which we already know is wrong.
And before it does so, it sets focus to the control,
which is also wrong;
the IsDialogMessage function generates a
WM_COMMAND message without changing focus.
And finally, it totally misses the boat if the edit control
is inside a nested dialog.
As I noted, all these mistakes are obvious in retrospect,
but when the control was first written, these mistakes might not¹
even have been mistakes.
(For example, nested dialogs didn't appear on the scene
until Windows 95.)
Why haven't these mistakes been fixed?
Well, how can you prove that there aren't any programs that
rely on the mistakes?
One thing you quickly learn in application compatibility is that
a bug once shipped gains the status of a feature,
because you can be pretty sure that some program somewhere relies on it.
(I've seen a plugin that relies on a memory leak in Explorer, for example.)
This goes doubly true for core controls like the edit control.
Any change to the edit control must be taken with a great deal of
trepidation,
because your change affects pretty much every single Windows program
on the entire planet.
With that high a degree of risk,
the prudent choice is often to let sleeping dogs lie.
Nitpicker's Corner
¹Note weasel words.
This is my educated guess as to what happened based on personal observation
and thought.
It is not a
statement of the official position of Microsoft Corporation,
and this guess may ultimately prove incorrect.