The Old New Thing

If you're running XP/2003, there is the <a href="http://msdn.microsoft.com/en-us/library/ms681952(VS.85).aspx">AttachConsole</a> function.  Couldn't this be combined with a method like <a href="http://support.microsoft.com/kb/105305">this</a> or <a href="">http://www.halcyon.com/~ast/dload/guicon.htm</a> to achieve the desired affect?  I've used the last example a few times, but it allocates a new console instead of using the parent's console.

>>To do this the kernel would have to have psychic powers<<

I don't see why that is necessary.  Why can't there be a third type - "dynamic" - where creating or attaching to the console and whether to release the CMD prompt to accept the next command is made by the application at run-time through API calls.  Many of the API calls are actually already available (AttachConsole, etc.).

This type of usage is actually quite common.  Microsoft has a number of executables that emulate this behavior (e.g., Visual Studio), and if you search the 'net, there are lots of people trying to do the same with their apps.

> These are programs that want to decide at run time whether they want a console or not.

Isn't that exactly what AllocConsole and AttachConsole are there for?

http://msdn.microsoft.com/en-us/library/ms681944(VS.85).aspx

But maybe the point is that there are actually programs that want decide during run time whether a console should have existed during startup?

And yet Unix has no problem doing this.  All programs are effectively console programs, but they can choose not to use stdin and stdout.  In fact, there are ways to determine at runtime if a program is connected to a terminal and perform different actions.  So if a program is started from the command line, stdin would be connected to a tty and the program can take a different path than if stdin was connected to /dev/null, or if it was connected to a file, or whatever.

I like the devenv solution. The problem with the ildasm solution is that the application will never see WM_QUERYENDSESSION messages.

ildasm may have been originally written with an older compiler/linker which didn't automatically work with both (w)WinMain and (w)main entry points, which explains the use of editbin.

Junfeng's description of how devenv.com works is incorrect.

devenv.com is a general purpose console-mode stub application.  When it runs it creates three pipes to redirect the console's stdin, stdout and stderr.  It then finds its own name (usually devenv.com), replaces the ".com" with ".exe" and launches the new app (i.e. devenv.exe) using the read end of the stdin pipe and the write ends of the stdout and stderr pipes as the standard handles.  Then it just sits and waits for devenv.exe to exit and copies data between the console and the pipes.

Thus even though devenv.exe is a gui app it can read and write the "parent" console using its standard handles.

And you could use devenv.com yourself for myapp.exe by renaming it to myapp.com.  But you can't in practise because it belongs to MS.

I've tackled this a little differently in the past. I wrote a small, generic "GUI runner" console exe. You give that the path and arguments of the GUI program you want to run.

It runs the GUI program, hooking its IO handles to pipes which connect to the console's IO handles. It waits for the GUI program to exit so that the command-line launch doesn't run in the background.

The nice thing about this is it's a small, generic exe which you can use with any GUI app. The GUI app doesn't have to do anything special, except ensure that it doesn't display any UI if given a command line.

As someone who came from the Amiga world, where there was no distinction between a command-line app and a GUI app, I still find just about everything related to the console subsytem in Windows to be horrible.

The past cannot be changed but a new, good, well-designed console / command-line shell could be added to Windows. I hoped that was what PowerShell would be but it turned out to run within the same old console subsystem and thus solved nothing. Ah well!

Based on laonianren's description (which I think was posted as I wrote my previous post), I guess my method isn't that different after all. :)

If we could just set some new flag to suppress the creation of a *new* console on launch, everything would be fine. At present, a process either *always* has a console (even if that means creating a new one) or *never* has it (even if that means detaching from the existing one) - an intermediate option, inheriting an existing console if present but never creating one, would solve the last irritating glitch of a console appearing then disappearing immediately.

(Alternatively, having CSRSS.EXE delay opening the new console window until it's actually needed would solve this even for existing applications: discard it unopened upon a FreeConsole call, avoiding the transient appearance where it's unwanted.)

How does the shutdown command work? It has an option "/i" that displays a GUI.

It doesn't create a new process (my first guess), the command prompt window stays there waiting, and pressing CTRL + C in the command window closes the program, and therefore the GUI window.

Dunno, but aren't such "workarounds" potential backward compatibility problems in future Windows  versions?

It is nice to hear from an authori^H^H^H^H^H^H^H guy who usually seems to know what he's talking about that there is no easy solution for this one.

It appears that it would ALMOST make sense to "inherit a console if present" -- but how is cmd.exe then going to know whether it should wait for the process to terminate before emitting a new command prompt? If the process decides that today it wants to be pure GUI, the user might want his shell back before it terminates.

Then one would need to kludge on some ad-hoc APIs to let CMD know whether and when the process has closed its console handle (which I can imagine would not integrate nicely with how the kernel otherwise handles handles).

The reason why this is not a problem on Unix is that Unix shell users accept that they have to predict the behavior of their application and tell the shell what to do (by adding an ampersand on the command line for GUI programs). Starting to require this in Windows would probably be perceived as a compatibility break by many.

Henning: The application would call FreeConsole if it wanted to detach itself from the console, as it does already.

Still, I don't think this is a must-have feature, since the "console flash" thing you get with the solution used by ildasm isn't *that* horrible. I don't think many non-technical people would even notice it (though I guess dual-mode applications would be targeted to technical people...)

The distributed.net client (dnetc) is another example of something that uses the devenv.com method.  It includes a dnetc.com frontend (see w32cuis.c) that creates pipes before launching dnetc.exe to do the real work.  If dnetc.exe is launched directly, you get a custom GUI that happens to resemble a console-like application but with pull-down menus other GUI enhancements.

I don't know whether it's appropiate to do this way, but in my previous company, I've written a C# program typed "service" and detect whether it's launched from console in the main() function. If it's not attached then run Application.Run(new frmMain) from there... And it does seems to work.

Does anybody know how to generate a ".com" output file using VS2008?  I'm trying to work up a solution similar to what devenv.com/devenv.exe does.

This "magic" can be done by using masm (and specifying wingui in the exe header). There's even a hello world app (wingui) that writes to the console it is started in.

@Tom: I doubt that devenv.com is really a DOS .COM file - COM files lack the PE header. Instead it's a renamed standard PE EXE. The only reason to call it devenv.com is because Windows will run a .COM file first, rather than a .EXE if it finds both in a directory.

The quirk is that a) Windows inherits the DOS prefer-.com-over-.exe, and b) that the Windows image loader doesn't inspect extensions when loading - it'll see if a file has a valid header first, before trying to set up a DOS COM environment and running it that way.

I think Raymond did an article about this... apparently it dates back to DOS too.

http://blogs.msdn.com/oldnewthing/archive/2008/03/24/8332730.aspx

Just that devenv.com is a console Win32 app with a false extension. Or how screensavers with .scr are really also .exe in disguise.

Now the question is, if you create a console app... can it call all the GDI functions as normal (including message pump, etc)? I think you can, and I believe I have, but I'm not sure - it was a while back.

If so, then the only real reason to have the com/exe thing was to hide an "ugly" console window...

BTW, Exchange 2007's setup.com uses the same trick as devenv.com. But why use this trick when there is AttachConsole() and AllocConsole?

AttachConsole is only available starting with XP, and AllocConsole does not attach to the parent console (which is the whole point of making it run as a console application).

@Henning

Most unix GUI programs need '&' in that usage, because their authors don't think they need to detach from the console.

It is possible (under unix) to close you stdio/err/out, and respawn yourself via fork. Then exit the original process - this is normally called daemonising.

Moving back on topic, this is akin to want the Windows kernel does for GUI Subsytem programs. (There are differences though). Unfortunately it depends on the fork() syscall for which windows has no alternative - although I believe Cygwin can fake it somehow.

The nearest equivalent for this trick under windows is the ildasm solution. The reason unix apps don't suffer from the windows flicker issue - is that the kernel doesn't set an environment up for new processes in unix in the same way. They just inherit form their parent.

The downside of this is starting a console app in unix from inside a windowing context is that the consoles output may never reach the user since stdout may only be directed to /dev/null. (stderr is often direct to a hidden file). Of course if the GUI launcher knows an application is console app it can start a console/terminal window for it.

It an good example of the different approaches taken.

Leo,

I may be remembering wrong, but in the Amiga world I seem to remember that it was similar to the UNIX model where a GUI app is just like any other app but if you launch it from Workbench its output handles aren't attached to anything.

This means that if you wanted to launch a command line app from Workbench you needed a wrapper that would open a console and then spawn the command line tool... which is basically what Windows is doing, but doing it in the kernel rather than needing the intermediate app.

UNIX desktops generally follow a similar principle to how command-line tools were launched in Workbench, but desktop environments usually make life a little easier by providing a "Run in Terminal" checkbox that launches your default terminal.

One thing that was neat on the Amiga though was that they took "everything is a file" to an even greater extreme than UNIX, with the CON: device that would open an on-screen console, and the SPEAK: device that would route anything written to it through the text-to-speech engine. This meant that the Amiga equivalent of AllocConsole was just fopen(f, "CON:").

> Consider the app that wants to run either in GUI mode or console mode (creating a console if necessary) based on a command line switch.

It's just up to the app to let go of the console if the command-line switch is missing or create one if needed. At this point, I think that would mean another devenv.com-style trick.

> But why use this trick when there is AttachConsole()

In addition to the OS support John mentions, you also want cmd to wait for your program to finish (when it's run from cmd.)

> The application would call FreeConsole if it wanted to detach itself from the console, as it does already.

Unfourtunately, cmd will still wait for it to finish.  I don't think there's any extant mechanism for cmd to tell the FreeConsole has happened.

So we have two feature requests: An "inherit" application mode and something in the console subsystem that cmd would use to tell when the child calls FreeConsole.  And this "inherit" application mode can't break backwards compatibility with older OSes, debuggers, 4nt, etc.  While I've wanted this power for ages, I'm not holding my breath :)

(I seem to be having trouble submitting this.  I hope I don't end up being "that guy" with five copies of his post.)

Try this (VB code):

 ' Hide the console window if there are no command line args

 Dim myHandle As IntPtr

 myHandle = Process.GetCurrentProcess.MainWindowHandle

 Dim hwnd As Integer = myHandle.ToInt32

 Dim nCmdShow As Integer = SW_HIDE

 Dim bShow As Integer = ShowWindow(hwnd, nCmdShow)

 myFrm.Show() : Application.DoEvents()

That's a great post. I found MSDEV is actually loading msdev.com .... I think it's loading the COM file because of the priority it get from .COM over .exe extension. Am I correct? Otherwise how it's possible to determine to launch .com or .exe program?

PingBack from http://sarathc.wordpress.com/2009/01/05/how-you-can-write-code-both-works-with-console-and-windowed-mode/

[The flag would solve many cases but not all. Consider the app that wants to run either in GUI mode or console mode (creating a console if necessary) based on a command line switch. -Raymond]

if (console_mode_selected)

 AllocConsole(), puts("problem solved");

You need a better counter-example.

I done that for few applications at work. I used C#.NET, where I create a windows app looking for number of arguments. If argument exists (proper ones) then run it as console. Otherwise follow the normal route and bring up the GUI.

Assumption is that the argument I am looking for should be provided only from the console mode.

In a similar vein, is there no way to make Visual Studio route the I/O of a console app to an "Console Output" window like IDEs on Mac OS X and Unix do, (rather than starting a console window...)?