Welcome to MSDN Blogs Sign in | Join | Help

Suppressing Console Windows for Custom Actions

If you have authored a custom action into your installer package and are annoyed by the console window that always pops up when the custom action runs, it's because the custom action executable is running under the console subsystem.

When you link your object files with the VC++ linker you can specify the subsystem using the /SUBSYSTEM switch. If you're using the C# or VB.NET compilers the /target switch allows you to specify which subsystem to use for the executable. However you specify the subsystem or however it's determined depending on which entry point you use, you shouldn't use the console subsystem for custom actions if you don't want the console window to appear, unless you have a good reason for it. For example, maybe you just want to reuse a console application that you're installing such as for custom action type 18 or any custom action using msidbCustomActionTypeExe (0x01).

In most cases you should consider running under the Windows subsystem. No window is created unless you create a window yourself and you can still allocate a console if necessary.

If you already have a console EXE you want to use, you can cause the window not to be displayed by passing certain parameters to APIs like the CreateProcess function. When calling CreateProcess pass CREATE_NO_WINDOW (0x08000000). WiX defines the CAQuietExec custom action function that does this and redirects standard handles as well.

Published Monday, January 23, 2006 2:25 PM by Heath Stewart
Filed under:

Comments

Monday, January 23, 2006 7:29 PM by C++ guy

# re: Suppressing Console Windows for Custom Actions

From the msdn docs for Process Creation Flags:

CREATE_NO_WINDOW
0x08000000 This flag is ignored.
Monday, January 23, 2006 7:45 PM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

That's interesting. Both WiX and the .NET Framework via System.Diagnostics.Process.Start() use the CREATE_NO_WINDOW flag. Sounds like a documentation bug. I will look into it.
Wednesday, January 25, 2006 1:36 PM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

In MSDN Online in http://msdn.microsoft.com/library/en-us/dllproc/base/process_creation_flags.asp it does indeed say that the flag is ignored, but in local MSDN content like that installed with the Platform SDK it says the flag is ignored under 9x. It is used under NT. I am working with the kernel team to make sure this gets updated.

In 9x you can't hide a console, it's true, which is why you should run under the Windows subsystem.
Thursday, January 26, 2006 8:23 PM by C++ guy

# re: Suppressing Console Windows for Custom Actions

Thanks for the thorough investigation. I've been wondering about this flag for a while...
Friday, February 03, 2006 4:02 PM by Madnad

# re: Suppressing Console Windows for Custom Actions

What about if you want to run a built in command like net.exe or sc.exe? I'm using WiX's CAQuietExec, but get an error 0x80070002 in the log when the command runs.
Friday, February 03, 2006 4:23 PM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

Madnad, HRESULT 0x80070002 is a Win32 error (uses FACILITY_WIN32) for ERROR_FILE_NOT_FOUND. The command you're running doesn't specify the appropriate path to executables such as net.exe or sc.exe. Are you specify a full path and if so are you using folder properties like [SystemFolder]net.exe?
Friday, February 03, 2006 7:18 PM by Madnad

# re: Suppressing Console Windows for Custom Actions

heaths, yes I am using exactly what you specify: here's a snippet:

<CustomAction id=startsvc.command Value="[SystemFolder]net.exe start myservice" Property="QtExecCmdLine" />
<CustomAction id=startsvc Return="check" Execute="immediate" DllEntry="CAQuietExec" BinaryKey="wixca" />
...
<Custom After="InstallInitialize" Action="startsvc.command">NOT Installed</CUSTOM>
<Custom After="startsvc.command" Action="startsvc">NOT Installed</Custom>

Friday, February 03, 2006 8:22 PM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

Madnad, there's several things wrong with that snippet.

First, if you're installing a service you should use both the ServiceInstall and ServiceControl tables not a custom solution with a command-line like this. It presents serviceability problems where if you use the ServiceInstall and ServiceControl tables you can properly reinstall and patch your service.

Second, immediate CAs should never modify system state.

Third, you scheduled your immediate CA after InstallInitialize. The file hasn't even been installed yet. InstallFiles is a deferred standard action that only gets scheduled between InstallInitialize and InstallFinalize and doesn't get processed until after InstallFinalize or any occurance of InstallExecute or InstallExecuteAgain. Windows error 2 would happen if the service doesn't exist. Try this: go to a machine where your product or service isn't installed and type "net start myservice". Now type "echo %errorlevel%" and you'll see 2 returned - the same error as you got back from the MSI custom action server in your log (wrapped in an HRESULT using the FACILITY_WIN32 (7) facility).

I highly recommend that you read all of the topic "Custom Actions" at http://msdn.microsoft.com/library/en-us/msi/setup/custom_actions.asp and child topics. For WiX information see http://wix.sourceforge.net. Look up the <ServiceInstall> and <ServiceControl> elements. Using these will schedule everything correctly and will help produce a more serviceable MSI.

Friday, February 03, 2006 8:40 PM by Madnad

# re: Suppressing Console Windows for Custom Actions

heaths, problem is i'm trying to start a driver that i'm going to be installing (type = kernel). does not handle this, so i am using "sc create" to register the driver and "net start" to start it up. It all works when i do it using standard CA's, but a command window pops up which is ugly. I'm simply trying to do it without popping up a command window. I'll read the MSI topics mentioned and play around with the scheduling and immediate/deferred. I'm on a very tight deadline (tomorrow) so any extra help will be appreciated.
Friday, February 03, 2006 8:48 PM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

If it's a kernel driver then yes, as you said, it's not supported using the ServiceInstall table.

Still, you're CAs must be deferred CAs that run after the service is installed. You can't start a service before it's installed and it won't be installed until after the standard InstallFiles action is processed, which actually won't be until after InstallFinalize (since it's a deferred CA).

Finally, the other problem I meant to mention last time (sorry) was that the Id of the property you set must be the same as the name of the custom action or be set immediately preceding the CA as the CustomActionData property. So, the propery should actually be set to "startsvc" since that's what your other CA is called, or use the property "CustomActionData" but make sure no other CA sneaks in between those two scheduled actions in your InstallExecuteSequence table.

Look up "Quiet Execution CustomAction" in the WiX.chm help file and look toward the bottom of the help topic for the section entitled "Deferred execution".
Friday, February 03, 2006 9:27 PM by Madnad

# re: Suppressing Console Windows for Custom Actions

heaths, thanks for the continued replies. I tried what you suggest but still get the 0002 error. I think it has to do with net.exe because when I change it to something like "notepad" it works. I took out all the params to net.exe so my code now looks like this (still does not work):

<Property id="QtExecDeferred" Value="c:\windows\system32\net.exe"></Property>
<CustomAction id="QtExecDeferred" Return="check" Execute="deferred" DllEntry="CAQuietExec" BinaryKey="wixca" />
...
<Custom After="InstallInitialize" Action="QtExecDeferred">NOT Installed</Custom>

So now I am not even referencing any of the files I am installing and still I get the error. I'm going nuts :)

Saturday, February 04, 2006 11:48 AM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

Madnad, you're still not scheduling it in the right place. You must schedule your deferred CA that starts your service after the service file is installed or you will continue to get the error. You should schedule it at least after the InstallFiles action.

Also - as I mentioned before - you should not hard-code a path. Set your property to "[SystemFolder]net.exe start myservice".

Finally, you should expect an error if you don't pass net.exe any arguments since it will return an error in such a case. Why don't you try starting a service in this way using a service that's already installed on your machine and stopped (you could stop it then try installing your app that would start it).

In order to start your service, however, you MUST schedule it after InstallFiles otherwise your service executable will not be installed when your CA runs and, of course, net.exe will return an error. Don't forget to run your CA to call sc.exe to install your service before trying to start it either. You can't start a service if the service doesn't exist. So, your relative order should be:

...
InstallFiles
...
sc.exe create ...
net.exe start ...
...
Saturday, February 04, 2006 5:11 PM by Madnad

# re: Suppressing Console Windows for Custom Actions

heaths, thanks for the reply. I've already tried all this :) many times... still nothing. I'll keep you posted on any developments...
Friday, February 17, 2006 4:34 AM by Varsik

# re: Suppressing Console Windows for Custom Actions

I want a little function to be run as an castom action. What kind of project shall
i create to suppress the console window
when installing. I change /subsystem as you saggest, but it doesn`t help. May be the problem in kind of a project?
Friday, February 17, 2006 5:03 AM by VHR

# re: Suppressing Console Windows for Custom Actions

I use Setup Wizard .NET.
I want a little function to call
as a castom action. Which kind of
project can I create for executable
to suppress the console window. I
change "/subsystem switch" as you suggest
but it doesn`t helps. May be the problem is
what kind of project to choose for executable.
Friday, February 17, 2006 12:17 PM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

Varsik, I'm going to assume you're also VHR since the content of the post is the same and answer this once for both. Either way, it should equally apply.

The "little function" is your custom action - you don't call a custom action from a custom action (well, you can, actually, but that's typically not necessary). It would help to know what you're trying to accomplish, though. You should almost always try to use a DLL custom action server because it lets you log to the same MSI log (logging goooood!) and can be a little easier to debug.

Typically the only good reason for an EXE is because you're installing an EXE that already does what you need it to do, and it would be cumbersome to reimplement that in a DLL just for install/patch/repair scenarios. Another reason is that an EXE exists on the system already and its difficult or seemingly impossible to implement it in a DLL, like the ngen.exe executable for generating native images from .NET assemblies. Generally, try to avoid EXE custom actions.
Wednesday, April 19, 2006 12:24 PM by Jeremy

# re: Suppressing Console Windows for Custom Actions

so have you really provided a way to suppress the console on an .exe and using custom action type 18 or 2?

all i have read is use wix or change flags passed to createprocess.

neither of which are available to me.
Wednesday, April 19, 2006 12:34 PM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

Jeremy, why isn't writing your own custom action that calls your EXE using the CreateProcess function not available to you? If you can't write such a CA, you can use the custom action server DLL from WIX without having to use WIX to create your Windows Installer packages.

If your EXE runs under the console subsystem, there's no flags you can pass to type 2, 18, 34, and 50 custom actions to prevent the window from being displayed. For this reason you need to use a CA that will create the process silently.
Saturday, August 12, 2006 6:52 AM by Nick Heim

# re: Suppressing Console Windows for Custom Actions

Hi Heat,
is there a way to call an exe from a CA with this technic, which is stored in the binary table and not on the HD itself?
Sunday, August 13, 2006 2:39 PM by Heath Stewart

# re: Suppressing Console Windows for Custom Actions

Nick, you would need to first extract the EXE, but then you'd be responsible for cleaning it up in all scenarios (rollback, commit, uninstall). It would be better to let Windows Installer handle that.

You should also avoid using EXE custom actions anyway, since they can't log to the same file nor can they access the MSI database. If you're not planning on installing the EXE (for example, maybe it's a useful command-line tool after install) you'd best stick with a DLL.
New Comments to this post are disabled
 
Page view tracker