Goals of this post:
  • Example for writing a custom trace listener which outputs real-time information over the network, which can then be consumed by a remote client.
  • Example for writing that remote client, which functions much like FileMon in showing real-time data from the trace listener:

The whole solution is attached to the post. Written [hurriedly] in VS2008 Beta2, but it will probably work just fine in VS2005. It's no where near pristine code; more a proof-of-concept, so take what you will from it.

The Problem:

One of the apps we developed internally consists of many instances of a service running on multiple machines, talking to many external services and then to a central DB, and doing so at a very high frequency with many threads.

We find that it's difficult to profile and debug such services due to both the complexity introduced by the multiple threads, and due to the fact that it's difficult to replicate the behavior in a test environment since the live one offered so much more data and opportunity for failure.

What we ended up doing in the latest incarnation was throwing in Trace.Write() statements everywhere, to allow the service to communicate not only what it was currently doing (as in the traditional "printf debugging"), but to also throw in occasional useful measures (as in a standard windows performance counters).

The problem is, how do we view these traces? The built-in eventlog and logfile trace listeners weren't suitable for us because they inundate their respective targets with an ever-increasing amount of data; which requires maintenance.

The Solution:

What we wanted was something exactly like the SQL Profiler, or ProcMon (previously "FileMon" + "RegMon"), something that would let us view what's happening as it happens, without being too intrusive.

The solution consists of two parts:

  1. The LiveTraceListener. This is a very simple class derived from the built-in TraceListener, but instead of writing to a file or to the eventlog, it writes to a network port.
  2. The LiveTraceViewer. This is a Winforms application that can connect to the above listener, read from that network port, and output the data to the screen.

The Listener

The listener is fairly simple. All you need to do in order to implement a custom listener is inherit from the System.Diagnostics.TraceListener class, then implement the Write() and WriteLine() methods. Here is a good article explaining it.

In this particular case, the complication comes in the constructor, where we perform some intialization to bind to the TCP port and get a stream to write to.

In the Write() methods, we check if there's someone connected to us. If there is, we write the string to them. If not, we discard it.

This way, we have a lightweight tracing solution that only really does anything when you're watching. Dare I say, a Schrödinger Listener? :)

The Viewer

The core of the viewer is just opening a TCP connection and reading strings from it. Everything else is window dressing. It could be a console app, an MMC plugin, a WPF app - it really doesn't matter.

What could be done better?

  • Well, for one, there are nicer ways to serialize data over the wire instead of reading/writing individual bytes over a TCP port. I was just in a hurry, and I happened to be comfortable with this method :)
    • Ideally, it would be changed to use proper serialization, so we could send trace events with more data, such as severity.
  • Both sides of the solution could use better error checking/handling.
  • It really should be an MMC plugin, so that we could monitor multiple machines concurrently with ease, and because it's the Right Thing To Do.

I'm sure there's plenty more that could be done to this to make it more useful. If you have suggestions, feel free to post them here as comments.

Avi