A customer asked of me...

.NET Event Tracing for Windows

dixi — Thu, 23 Apr 2009 21:17:00 GMT

I didn’t find a good step-by-step guide on how to create a .NET Event Tracing for Windows (ETW) provider with a manifest. And there are some caveats I would like to share with the community so here I go.

The very first thing I recommend is to familiarize with ETW concepts like the manifest, session, provider, events, channels, etc. And the second thing is to install the latest version of the Microsoft Windows SDK (v6.1 as I am writing this) because several utilities ease working with ETW. Those tools are the Manifest Generator (ecmangen.exe), the Windows Message Compiler (mc.exe) and the Resource Compiler (rc.exe).

Now you need to create the XML 1.0 document known as the ETW manifest. Although you can use any text editor, I suggest to use the Manifest Generator utility (ecmangen.exe). And here it comes the first caveat: although the documentation states that the SDK Include directory must be part of the INCLUDE environment variable, I am probably doing something wrong because no manifest file could be opened or created successfully; the Manifest Generator fails with the following error: "Center window failed to initialize. Error: 0x80070006. The handle is invalid.". What I did was to point the Manifest Generator "Start in" shortcut property to "C:\Program Files\Microsoft SDKs\Windows\v6.1\Include\" (the Microsoft Windows SDK default path installation) and it started to work.

With the Manifest Generator utility and according to your needs for providing ETW information, create your manifest file. To demo the next steps I supply a sample manifest file (Simple.Manifest.xml) that you can see here.

The Simple manifest file defines the SimpleEtwProvider with ID {d9b453b9-6230-486c-8dec-c7c5a2230d04}. The .NET initialization should look like this:

using Etw = System.Diagnostics.Eventing;
//
// Code omitted intentionally...
//
Guid etwProviderId = new Guid("{d9b453b9-6230-486c-8dec-c7c5a2230d04}");
Etw.EventProvider provider = new Etw.EventProvider(etwProviderId);

Also, the Simple manifest file declares a SimpleEvent (ID and version), a SimpleOperationalChannel, a SimpleTask, a SimpleOpcode, and a SimpleKeyword. These must be used to create a ETW event descriptor as following:

Etw.EventDescriptor descriptor;

unchecked
{
  descriptor = new Etw.EventDescriptor
  (
    0x3e9,                    // Event ID
    0x1,                      // Event Version
    0x10,                     // Operation Channel ID
    0x2,                      // Level
    0xa,                      // Opcode
    0x1,                      // Task
    (long) 0x8000000000000001 // Keywords
  );
}

The constituent parts of the event to be published are determined by the SimpleTemplate element in the manifest file: a StringData and an IntegerData. Data types and order must be observed when the event is published as shown below:

// We know that the event has two parts: a string and an integer
// Be careful to provide the right types in the right order...
object[] payload = { "An event with manifest...", 98 };
provider.WriteEvent(ref descriptor, payload);

A recommended practice is to check if the provider is enabled (at least one session like Performance Monitor) and listening to what you publish. This can be accomplished as following:

if(provider.IsEnabled())
{
  // We know that the event has two parts: a string and an integer
  // Be careful to provide the right types in the right order
  object[] payload = { "An event with manifest...",  98 };
  provider.WriteEvent(ref descriptor, payload);
}

That's it for the development part. Now let's go for the compilation part. First, for the manifest file you need to use the Windows Message Compiler (mc.exe) like this:

MC Simple.Manifest.xml

And I found another caveat here. It happens that both utilities, Manifest Generator and Windows Message Compiler, depend on the winmeta.xml to be located through the INCLUDE environment variable. But even when everything seems to be in place, MC fails with error: ".\winmeta.xml(0) : error Failed trying to parse file. result=0x800c0005: At Line=0, Column=0, The system cannot locate the resource specified.". I even tried the MC –W command-line switch but that didn't work either. I'm probably doing something wrong but to make it work, I had to copy winmeta.xml to the same location of Simple.Manifest.xml.

A successful MC compilation generates four files: MSG00001.bin, Simple.Manifest.h, Simple.Manifest.rc, and SimpleTEMP.BIN. The only interesting and useful, at least from the .NET developer perspective, is Simple.Manifest.rc. This is to be compiled with the Resource Compiler (rc.exe) to generate the corresponding Simple.Manifest.res as shown below:

RC Simple.Manifest.rc

The last part is to embed the Simple.Manifest.res file into the ETW provider binary image. Let's assume that the .NET code shown above is a console application EtwProvider.cs, the compilation would look like the following:

CSC /win32res:Simple.Manifest.res EtwProvider.cs

From inside the Visual Studio 2008 IDE, the .res file can be specified through the Resource File field in the Project Properties Application tab.

That's it for the compilation part. Now let's go for the deployment part. The full path from where the application will be executed in the target machine must be supplied in the messageFileName and resourceFileName attributes of the <provider> element in the manifest file (Simple.Manifest.xml). If, for example, the EtwProvider.exe is installed and executed from "C:\Program Files\" the manifest file should be modified to look like the following (some attributes and declarations were omitted intentionally for clarity sake):

<?xml version="1.0" encoding="UTF-16"?>
<instrumentationManifest>
  <instrumentation>
    <events>
      <provider messageFileName="C:\Program Files\EtwProvider.exe"
                resourceFileName="C:\Program Files\EtwProvider.exe">

The modified manifest file must be registered in the target machine through the Windows Event Command-line utility (wevtutil.exe) as shown below:

WEVTUTIL im Simple.Manifest.xml

Now we're ready to start publishing ETW events. In my next article I will show how to create an Event Trace Session through Performance Monitor.

Application Recovery and Restart

dixi — Wed, 19 Nov 2008 21:59:00 GMT

I just faced an interesting scenario. It happens that a .NET 3.0 WinForms-based application running on Windows Vista stops responding. Of course, the common sense would say "Find out the root cause and fix it". It turns out that in this case, it is not that easy. The application depends heavily on threading and asynchronous calls and needs to interact with several devices (which leads directly to establish hypothesis on device drivers) and external systems by using products from several vendors. The underlying root cause might be in any or a combination of those elements. As I am writing this, the problem had not been identified.

But that's the technical aspect of the scenario. The real problem is with the end-user experience. The application is used to enter a lot of information but, as I stated above, the application stops responding. This leaves no option to the end-user but to kill the application (she was trained to do that) losing all of the information entered up to that moment. The impact on end-user productivity caused by reentering the information is the real problem.

While working on finding the technical root cause, a customer asked of me if there is something he could do in order to minimize the impact on end-user productivity. Incidentally, Windows Vista (and Windows Server 2008) provides a new set of APIs collectively known as Application Recovery and Restart and the Windows Error Reporting Service (WER). In a nutshell, an application registers itself as a recoverable application which means that in case it stops responding (or receives an unhandled exception) the OS will give it a chance to do some processing prior to being terminated. Usually this processing involves saving a copy of its internal state. The second part, Application Restart, allows the user to launch another instance of the application, restore its state, and get to the point it was when the undesirable condition appeared.

I am not entitled to show the real application and what I did in it. But to demo the Application Recovery and Restart concepts I created the following form:

The TextBox Text property is all the state I am interested in. The button's task is easy: a while(true); loop caughts the UI thread making the window a Not Responding one as shown in the next figure:

At this moment, everything the end-user is able to do is to click on the Close button, restart the application, and cry for the information gone. So the first thing I do is to declare the Application Recovery and Restart Windows APIs to be invoked from within the C# demo application:

namespace Win32
{
  public static class ApplicationRecoveryRestart
  {
    public delegate Int32 ApplicationRecoveryCallbackDelegate(RecoveryInformation information);

    [DllImport("kernel32.dll")]
    public static extern Int32 RegisterApplicationRecoveryCallback
    (
      ApplicationRecoveryCallbackDelegate recoveryCallback,
      RecoveryInformation information,
      UInt32 pingInterval,
      UInt32 flags
    );

    [DllImport("kernel32.dll")]
    public static extern Int32 RegisterApplicationRestart
    (
      [MarshalAs(UnmanagedType.LPWStr)]
      String commandLineArgs,
      UInt32 flags
    );

    [DllImport("kernel32.dll")]
    public static extern void ApplicationRecoveryFinished(Boolean success);

    [DllImport("kernel32.dll")]
    public static extern Int32 ApplicationRecoveryInProgress(out Boolean canceled);

    [DllImport("kernel32.dll")]
    public static extern Int32 UnregisterApplicationRecoveryCallback();

    [DllImport("kernel32.dll")]
    public static extern Int32 UnregisterApplicationRestart();
  }

  public class RecoveryInformation
  {
    public string Parameter;
  }
}

I am only using a subset of the Application Recovery and Restart Windows APIs but you can go to the MSDN and see the rest. To register the application for recovery, the Main() method is a good place to start:

[STAThread]
static void Main(string[] args)
{
  string restartFile = Guid.NewGuid().ToString("N") + ".dat";
  RegisterApplicationRecovery(restartFile);

  Application.EnableVisualStyles();
  Application.SetCompatibleTextRenderingDefault(false);
  Application.Run(mainForm = new MainForm());
}

private static void RegisterApplicationRecovery(string restartFile)
{
  RecoveryInformation information = new RecoveryInformation();
  information.Parameter = restartFile;

  ApplicationRecoveryRestart.ApplicationRecoveryCallbackDelegate callback;
  callback = new ApplicationRecoveryRestart.ApplicationRecoveryCallbackDelegate(ApplicationRecoveryCallback);
  Int32 result = ApplicationRecoveryRestart.RegisterApplicationRecoveryCallback
  (
    callback,    // Function to be called to store application state when the recovery starts
    information, // It could be anything useful for a restart. In this case is the file holding the state stored during the recovery
    3000,        // Recovery ping interval. It also has to do with ApplicationRecoveryInProgress. See MSDN documentation.
    0            // Reserved for future use
  );
}

public static Int32 ApplicationRecoveryCallback(RecoveryInformation information)
{
  bool canceled = false;

  // Let Windows knows recovery is starting and check if the user has cancelled the recovery...
  ApplicationRecoveryRestart.ApplicationRecoveryInProgress(out canceled);
  if(canceled)
  {
    Trace.WriteLine("User canceled application recovery...");

    // Let Windows knows recovery is complete...
    ApplicationRecoveryRestart.ApplicationRecoveryFinished(false);
  }
  else
  {
    Trace.WriteLine("Application is trying to save data and state information before the application terminates...");
    using(StreamWriter writer = new StreamWriter(information.Parameter))
    {
      writer.Write("{0}", theTextBox.Text); // The TextBox Text property is all the state to be saved...
    }

    // Let Windows knows recovery is complete...
    ApplicationRecoveryRestart.ApplicationRecoveryFinished(true);
  }

  return 0;
}

During startup, the application lets Windows knows that in case of becoming a "Not Responding" one, its ApplicationRecoveryCallback function is to be called. Any useful state will be saved inside this function. Before running a quick test of the code and concepts covered so far, it is important to have a couple of caveats in mind:

If the application is started inside the Visual Studio hosting process, it won't work because it is the latter the one to be recovered and restarted. Test the application through the Debug->Start without debugging menu item or press <CTRL>+<F5>.
Only applications running for at least sixty seconds are considered by the Application Recovery and Restart and the Windows Error Reporting Service.

Once the application has entered into the "Not Responding" state, the end-user, unable to do anything else, tries to close it. In response, Microsoft Windows shows the following message box:

When the application is doing a lengthy operation, usually selecting "Wait for the program to respond" could be enough; once the lengthy operation is done, the control returns to the application and the user will be able to continue. In this case, while(true); won't relinquish control ever so the only thing the user is able to do is to click on "Close the program". This is where the ApplicationRecoveryCallback function is called by the Windows Error Reporting Service and the application state is saved. However, no provisions were made to restart the application. This could be accomplished through the following code:

[STAThread]
static void Main(string[] args)
{
  string restartFile = Guid.NewGuid().ToString("N") + ".dat";
  RegisterApplicationRecovery(restartFile);
  RegisterApplicationRestart(restartFile);

  Application.EnableVisualStyles();
  Application.SetCompatibleTextRenderingDefault(false);
  Application.Run(mainForm = new MainForm());
}

private static void RegisterApplicationRestart(string restartFile)
{
  string commandLineArgs = restartFile;
  ApplicationRecoveryRestart.RegisterApplicationRestart(commandLineArgs, 0);
}

Notice that the restartFile created during the ApplicationRecovery episode will be used as the commandLine argument for recovery. Therefore the application must determine during startup, if it is being started or restarted. The following is the modified Main function:

[STAThread]
static void Main(string[] args)
{
  // If the a command-line argument is supplied then the application is being restarted and
  // the supplied argument is the name of the file containing the state to be restored...
  string restartFile;
  if(args.Length > 0)
  {
    restartFile = args[0];
    using(StreamReader reader = new StreamReader(restartFile))
    {
      theTextBox.Text = reader.ReadToEnd();
    }
  }
  else
  {
    restartFile = Guid.NewGuid().ToString("N") + ".dat";
    if(File.Exists(restartFile))
    {
      File.Delete(restartFile);
    }
  }

  RegisterApplicationRecovery(restartFile);
  RegisterApplicationRestart(restartFile);

  Application.EnableVisualStyles();
  Application.SetCompatibleTextRenderingDefault(false);
  Application.Run(mainForm = new MainForm());
}

Under these new circumstances, every time the application hangs and its Close button is clicked the following message box will be shown:

Now there is the "Restart the program" option which explains it by itself. If the user select such option, the application hanging instance is terminated and a new instance is launched as the following message box is shown:

If everything works well, the application is restarted and its state is restored. That's it.

One final note. I have noticed that after a number of times the application is started, hung, and restarted, a point is reached where the recovery/restart functionality stops working. The RegisterApplicationRecoveryCallback returns E_FAIL and I have found that the reason is that the Windows Error Reporting Services runs out of some internal resource that keeps track of every application registered for recovery/restart. Like a good citizen, the application should unregister from the Windows Error Reporting Service through the UnregisterApplicationRecoveryCallback and UnregisterApplicationRestart API but at the time of this writing and after several tests, I have not been successful to find the correct place to unregister which leaves me no option but to restart the Windows Error Reporting Service to make everything works again (yes, I know it is not very end-user friendly).