A lot of folks are told to connect a kernel debugger (over a serial calbe) to their systems if it is constantly blue screening or if there are suspected issues in the kernel or a loaded driver. Most of these folks do not have the skills to debug the issue themselves, they are just setting up their machine so that once the problem shows up, somebody else can debug it. One thing that happens is that the person whose machine is being debugged looks in device manager and sees one of two things...

  1. The COM that they are using to debug the machine is yellow !'ed out and non functional.
  2. The COM port does not appear in device manager at all.

...and then summarily freak out. Reactions range from wondering if the kernel debugger is working at all (how could it? the port it using appears to be non functional) to anger that the COM port that they wanted to use is now gone.

So what exactly is happening? First and foremost, in both situations the kernel has assumed ownership of the COM port's resources and is using those resources to directly manipulate the port. The kernel is not opening the port by name and using the serial driver (serial.sys) to control the hardware, it is direct manipulation. This has an important side affect...there must be real UART hardware for the kernel to access; a USB to serial adapter will not work on the target side of a kernel debug session because there are no real UART registers to touch! When debugging over a serial cable is enabled, the kernel export KdComPortInUse contains the starting address of the UART resources being used by the kernel debugger.

So why are there different visual results to enabling a kernel debugger? Well, the results depend on if it is an ACPI machine (and chances are, any machine built in the past 5 years is ACPI enabled by default). If ACPI is enabled, ACPI will look at the address in KdComPortInUse and see if a device in the ACPI table starts with that address. If a match is made, ACPI does not even enumerate the COM at all. This is why you don't see a !'ed out device. Looking at ACPI.sys's imports, we see that it does reference the export:

C:\WINDOWS\system32\drivers>link /dump /imports acpi.sys | findstr KdComPortInUse
                   38 KdComPortInUse

The !'ed out device occurs when ACPI is not enabled on the machine. In this case, the BIOS reported COM is enumerated, the serial driver is attached to the stack, and an IRP_MN_START_DEVICE is sent to the stack. In the process of starting the device, serial looks at the assigned resources to the device and if they match KdComPortInUse, it will fail the IRP_MN_START_DEVICE, resulting in a failed start which shows up as a !'ed device in device manager. Since the serial driver is a DDK/WDK sample, we can see what it does:

NTSTATUS
SerialInitController(
    IN PDEVICE_OBJECT PDevObj,
    IN PCONFIG_DATA PConfigData
    )
{
   [...]

#ifdef _WIN64
   BOOLEAN DebugPortInUse = FALSE;

   //
   // First check what type of AddressSpace this port is in. Then check
   // if the debugger is using this port. If it is set DebugPortInUse to TRUE.
   //
   if(PConfigData->AddressSpace == CM_RESOURCE_PORT_MEMORY) {
        PHYSICAL_ADDRESS  KdComPhysical;
        KdComPhysical = MmGetPhysicalAddress(*KdComPortInUse);
        if(KdComPhysical.LowPart == PConfigData->Controller.LowPart) DebugPortInUse = TRUE;
   } else {
        if ((*KdComPortInUse) == (ULongToPtr(PConfigData->Controller.LowPart))) DebugPortInUse = TRUE;
   }

   if (DebugPortInUse) {
      [...]
      return STATUS_INSUFFICIENT_RESOURCES;
   }
#else
   //
   // This compare is done using **untranslated** values since that is what
   // the kernel shoves in regardless of the architecture.
   //
   if ((*KdComPortInUse) == (ULongToPtr(PConfigData->Controller.LowPart))) {
      [...]
      return STATUS_INSUFFICIENT_RESOURCES;
   }
}