Suppressing Console Windows for Custom Actions

Suppressing Console Windows for Custom Actions

  • Comments 20

If you have authored a custom action into your installer package and are annoyed by the console window that always pops up when the custom action runs, it's because the custom action executable is running under the console subsystem.

When you link your object files with the VC++ linker you can specify the subsystem using the /SUBSYSTEM switch. If you're using the C# or VB.NET compilers the /target switch allows you to specify which subsystem to use for the executable. However you specify the subsystem or however it's determined depending on which entry point you use, you shouldn't use the console subsystem for custom actions if you don't want the console window to appear, unless you have a good reason for it. For example, maybe you just want to reuse a console application that you're installing such as for custom action type 18 or any custom action using msidbCustomActionTypeExe (0x01).

In most cases you should consider running under the Windows subsystem. No window is created unless you create a window yourself and you can still allocate a console if necessary.

If you already have a console EXE you want to use, you can cause the window not to be displayed by passing certain parameters to APIs like the CreateProcess function. When calling CreateProcess pass CREATE_NO_WINDOW (0x08000000). WiX defines the CAQuietExec custom action function that does this and redirects standard handles as well.

Leave a Comment
  • Please add 8 and 1 and type the answer here:
  • Post
  • From the msdn docs for Process Creation Flags:

    CREATE_NO_WINDOW
    0x08000000 This flag is ignored.
  • That's interesting. Both WiX and the .NET Framework via System.Diagnostics.Process.Start() use the CREATE_NO_WINDOW flag. Sounds like a documentation bug. I will look into it.
  • In MSDN Online in http://msdn.microsoft.com/library/en-us/dllproc/base/process_creation_flags.asp it does indeed say that the flag is ignored, but in local MSDN content like that installed with the Platform SDK it says the flag is ignored under 9x. It is used under NT. I am working with the kernel team to make sure this gets updated.

    In 9x you can't hide a console, it's true, which is why you should run under the Windows subsystem.
  • Thanks for the thorough investigation. I've been wondering about this flag for a while...
  • What about if you want to run a built in command like net.exe or sc.exe? I'm using WiX's CAQuietExec, but get an error 0x80070002 in the log when the command runs.
  • Madnad, HRESULT 0x80070002 is a Win32 error (uses FACILITY_WIN32) for ERROR_FILE_NOT_FOUND. The command you're running doesn't specify the appropriate path to executables such as net.exe or sc.exe. Are you specify a full path and if so are you using folder properties like [SystemFolder]net.exe?
  • heaths, yes I am using exactly what you specify: here's a snippet:

    <CustomAction id=startsvc.command Value="[SystemFolder]net.exe start myservice" Property="QtExecCmdLine" />
    <CustomAction id=startsvc Return="check" Execute="immediate" DllEntry="CAQuietExec" BinaryKey="wixca" />
    ...
    <Custom After="InstallInitialize" Action="startsvc.command">NOT Installed</CUSTOM>
    <Custom After="startsvc.command" Action="startsvc">NOT Installed</Custom>

  • Madnad, there's several things wrong with that snippet.

    First, if you're installing a service you should use both the ServiceInstall and ServiceControl tables not a custom solution with a command-line like this. It presents serviceability problems where if you use the ServiceInstall and ServiceControl tables you can properly reinstall and patch your service.

    Second, immediate CAs should never modify system state.

    Third, you scheduled your immediate CA after InstallInitialize. The file hasn't even been installed yet. InstallFiles is a deferred standard action that only gets scheduled between InstallInitialize and InstallFinalize and doesn't get processed until after InstallFinalize or any occurance of InstallExecute or InstallExecuteAgain. Windows error 2 would happen if the service doesn't exist. Try this: go to a machine where your product or service isn't installed and type "net start myservice". Now type "echo %errorlevel%" and you'll see 2 returned - the same error as you got back from the MSI custom action server in your log (wrapped in an HRESULT using the FACILITY_WIN32 (7) facility).

    I highly recommend that you read all of the topic "Custom Actions" at http://msdn.microsoft.com/library/en-us/msi/setup/custom_actions.asp and child topics. For WiX information see http://wix.sourceforge.net. Look up the <ServiceInstall> and <ServiceControl> elements. Using these will schedule everything correctly and will help produce a more serviceable MSI.

  • heaths, problem is i'm trying to start a driver that i'm going to be installing (type = kernel). does not handle this, so i am using "sc create" to register the driver and "net start" to start it up. It all works when i do it using standard CA's, but a command window pops up which is ugly. I'm simply trying to do it without popping up a command window. I'll read the MSI topics mentioned and play around with the scheduling and immediate/deferred. I'm on a very tight deadline (tomorrow) so any extra help will be appreciated.
  • If it's a kernel driver then yes, as you said, it's not supported using the ServiceInstall table.

    Still, you're CAs must be deferred CAs that run after the service is installed. You can't start a service before it's installed and it won't be installed until after the standard InstallFiles action is processed, which actually won't be until after InstallFinalize (since it's a deferred CA).

    Finally, the other problem I meant to mention last time (sorry) was that the Id of the property you set must be the same as the name of the custom action or be set immediately preceding the CA as the CustomActionData property. So, the propery should actually be set to "startsvc" since that's what your other CA is called, or use the property "CustomActionData" but make sure no other CA sneaks in between those two scheduled actions in your InstallExecuteSequence table.

    Look up "Quiet Execution CustomAction" in the WiX.chm help file and look toward the bottom of the help topic for the section entitled "Deferred execution".
  • heaths, thanks for the continued replies. I tried what you suggest but still get the 0002 error. I think it has to do with net.exe because when I change it to something like "notepad" it works. I took out all the params to net.exe so my code now looks like this (still does not work):

    <Property id="QtExecDeferred" Value="c:\windows\system32\net.exe"></Property>
    <CustomAction id="QtExecDeferred" Return="check" Execute="deferred" DllEntry="CAQuietExec" BinaryKey="wixca" />
    ...
    <Custom After="InstallInitialize" Action="QtExecDeferred">NOT Installed</Custom>

    So now I am not even referencing any of the files I am installing and still I get the error. I'm going nuts :)

  • Madnad, you're still not scheduling it in the right place. You must schedule your deferred CA that starts your service after the service file is installed or you will continue to get the error. You should schedule it at least after the InstallFiles action.

    Also - as I mentioned before - you should not hard-code a path. Set your property to "[SystemFolder]net.exe start myservice".

    Finally, you should expect an error if you don't pass net.exe any arguments since it will return an error in such a case. Why don't you try starting a service in this way using a service that's already installed on your machine and stopped (you could stop it then try installing your app that would start it).

    In order to start your service, however, you MUST schedule it after InstallFiles otherwise your service executable will not be installed when your CA runs and, of course, net.exe will return an error. Don't forget to run your CA to call sc.exe to install your service before trying to start it either. You can't start a service if the service doesn't exist. So, your relative order should be:

    ...
    InstallFiles
    ...
    sc.exe create ...
    net.exe start ...
    ...
  • heaths, thanks for the reply. I've already tried all this :) many times... still nothing. I'll keep you posted on any developments...
  • I want a little function to be run as an castom action. What kind of project shall
    i create to suppress the console window
    when installing. I change /subsystem as you saggest, but it doesn`t help. May be the problem in kind of a project?
  • I use Setup Wizard .NET.
    I want a little function to call
    as a castom action. Which kind of
    project can I create for executable
    to suppress the console window. I
    change "/subsystem switch" as you suggest
    but it doesn`t helps. May be the problem is
    what kind of project to choose for executable.
Page 1 of 2 (20 items) 12