Angry Richard's WebLog : Profiling

Off-road Profiling Windows Services

AngryRichard — Thu, 28 Jul 2005 22:36:00 GMT

I've just posted an article on the pitfalls of profiling services with the Visual Studio profiler. It includes a sample service with a quick walkthrough. Enjoy.

Profiling Windows™ Services with the Visual Studio Profiler

Limiting the Data Crunch from Trace Profiling

AngryRichard — Mon, 17 Jan 2005 03:41:00 GMT

Typically, one can use the sampling profiler to nail down the hot spot in an application. Having done that, what does one do when the sampling data doesn't provide enough information? The trace profiler can offer up more detail, particularly if the issue revolves around thread interaction. However, if you profile a heavily CPU bound application, you may find that you are getting huge trace files, or that the profiler is significantly impacting the performance of your application. The VisualStudio profiler offers a mechanism to stem the avalanche of data.

I'll illustrate the general idea with an example.

Suppose we'd like to run trace profiling on the following highly useful piece of code. We've decided that we only care about profiling in the context of the function "OnlyProfileThis()"

using System;

public class A
{
private int _x;

public A(int x)
{
  _x = x;
}

public int DoNotProfileThis()
{
  return _x * _x;
}

public int OnlyProfileThis()
{
  return _x + _x;
}

public static void Main()
{
  A a;
  a = new A(2);

  Console.WriteLine("2 square is {0}", a.DoNotProfileThis());
  Console.WriteLine("2 doubled is {0}", a.OnlyProfileThis());
}
}

The VisualStudio profiler provides an API for controlling data collection from within the application. For native code, this API lives in VSPerf.dll. A header (VSPerf.h) and import library (VSPerf.lib) provided in the default Team Developer install allows us to use the profiler control API from native code. For managed code, this API is wrapped by the DataCollection class in Microsoft.VisualStudio.Profiler.dll. We can update the example to the following to control the profiler during our run:

using System;
using Microsoft.VisualStudio.Profiler;

int x;
Console.WriteLine("2 square is {0}", a.DoNotProfileThis());

  DataCollection.StartProfile(
      DataCollection.ProfileLevel.DC_LEVEL_GLOBAL,
      DataCollection.PROFILE_CURRENTID);

x = a.OnlyProfileThis();

  DataCollection.StopProfile(
      DataCollection.ProfileLevel.DC_LEVEL_GLOBAL,
      DataCollection.PROFILE_CURRENTID);

Console.WriteLine("2 doubled is {0}", x);

}
}

We still need to instrument the application as normal. We also have one additional step. When running the code above, data collection will be enabled by default, so the API won't appear to do anything beyond stopping data collection after the call to OnlyProfileThis. We need to disable data collection before running the application. The profiler control tool,
VSPerfCmd, has options to do this.

To run this scenario from the command line:

csc /debug- /r:Microsoft.VisualStudio.Profiler.dll Sample.cs
VSInstr Sample.exe

VSPerfCmd /start:trace /output:SampleTrace.VSP
VSPerfCmd /globaloff
Sample.exe
VSPerfCmd /shutdown

VSPerfReport /calltrace SampleTrace.VSP

There are several control functions in the DataCollection class. Here is a brief rundown.

StartProfile(ProfileLevel level, UInt32 id)
StopProfile(ProfileLevel level, UInt32 id)

This unconditionally starts or stops profiling. The level parameter indicates whether profiling should be stopped for the current thread, process, or globally. For thread and process level changes, the id specifies thread id or process id to control (respectively), or PROFILE_CURRENTID to control the current thread or process.

SuspendProfile(ProfileLevel level, UInt32 id)
ResumeProfile(ProfileLevel level, UInt32 id)

This works very much like StartProfile and StopProfile, however, calls to these functions are reference counted. If you call SuspendProfile twice, you must call ResumeProfile twice to enable profiling.

MarkProfile(Int32 markId)
CommentMarkProfile(Int32 markId, String comment)
CommentMarkAtProfile(Int64 timeStamp, Int32 markId, String comment)

Inserts a 32-bit data value into the collection stream. Optionally, you can include a comment. With the last function, the mark can be inserted at a specific time stamp. The id value and the optional comment will appear in the CallTrace report.

The profiler control tool provides similar functionality through a command line interface, though the notion of a "current" process or thread id is obviously not relevant.

VSPerfCmd /?

[...]

-GLOBALON Sets the global Start/Stop count to one (starts profiling).

-GLOBALOFF Sets the global Start/Stop count to zero (stops profiling).

-PROCESSON:pid Sets the Start/Stop count to one for the given process.

-PROCESSOFF:pid Sets the Start/Stop count to zero for the given process.

-THREADON:tid Sets the Start/Stop count to one for the given thread.
Valid only in TRACE mode.

-THREADOFF:tid Sets the Start/Stop count to zero for the given thread.
Valid only in TRACE mode.

-MARK:marknum[,marktext]
                Inserts a mark into the global event stream, with
                optional text.

[...]

If you find yourself buried under a ton of trace data, investigate the profiling API to help focus on the important parts of your application.

Trying out the profiler on the VPC

AngryRichard — Fri, 06 Aug 2004 21:19:00 GMT

I've frequently heard the question asked, "Can I use the profiler on a Virtual PC?" It has even come up on the blog feedback a few times. My answer has always been, "Theoretically, yes." I didn't want to post this answer externally until I'd actually gotten around to trying it myself.

I've finally been nagged into it.

In my limited experience with our VirtualPC product, it has quite impressed me with its functionality. However, it does not emulate the hardware performance counters upon which the profiler implicitly depends. For this reason, you can not run the sampling profiler using a performance counter based interrupt. My collegue Ishai pointed out^* that you should be able to use page-fault or system-call based sampling, but the VPC has a different problem with these modes that is still under investigation.

Instrumentation based profiling will work on the VPC. However, as I've already mentioned, there is a bug check issue with the driver when it unloads. Fortunately, instrumentation based profiling doesn't rely on the presence of the driver.

By renaming the driver to prevent the profiling monitor from installing it at startup, I was able to use instrumentation based profiling on the VPC. This is obviously just a workaround, but I hope this will allow you to investigate some of our tools in the comfort of a VirtualPC environment.

Here is how you can prevent the driver from loading on your VPC installation.

Use regedit to delete the key HKLM\CurrentControlSet\Services\VSPerfDrv.
Find C:\Program Files\Microsoft Visual Studio\Enterprise Developer Tools\Performance Tools\VSPerfDrv.sys, and rename it to something silly like __VSPerfDrv.sys__, or delete it.
Reboot your VPC to ensure that the driver isn't installed

Happy hunting!

^*I'm pretty sure Ishai was hired to point out things I do wrong. Fortunately, he usually points out solutions as well.

VPC and the BSOD (part 2)

AngryRichard — Fri, 04 Jun 2004 03:00:00 GMT

I had a nice long email chat with members of the Virtual PC team.

The good news: The Virtual PC emulates the host processor well enough that our kernel-mode driver can detect what features are enabled.

The bad news: The Virtual PC does not emulate an APIC or performance counters.

So, if you were planning on running the profiler inside a Virtual PC, the best you can hope to do is get function trace data on an instrumented app. Sampling will not work at all, and collecting perfomance counter data in the instrumentation will fault the application.

Bummer. I wish I had better news.

VPC and the BSOD

AngryRichard — Wed, 02 Jun 2004 03:14:00 GMT

I'm so pleased. Someone did something exciting and dangerous with the profiler. In case you're not reading the newsgroups, an intrepid customer tried to profile on a Virtual PC, and discovered that it only leads to pain and misery via the BSOD.

So don't do that.

Seriously, is this something people want to do? I mean, VPC is about the coolest thing ever, but we do use hardware performance counters by default, and VPC is not exactly a real life environment for performance analysis and measurement. Still, maybe y'all have good reasons for this.

At the very least, we'll fix that BSOD thing. I mean, how 1990's is that?

This is an excellent time to point out that the profiler does in fact install a kernel-mode device driver in order to play with the hardware counters on your Intel and AMD processors. There are some fun implications from this:

You need to be running as a user who can install a kernel-mode device driver. We currently install the driver dynamically when starting the profiling monitor, and we uninstall it when the monitor shuts down.
The kernel-mode device driver is not signed. If you're one of those cheeky folks who has unsigned drivers shimmed through verifier, you may discover the joy of a forced BSOD if you exercise the profiler in new interesting ways we hadn't thought of. This is not all bad; just make sure you send us the error report after you've rebooted.
If you're running on Intel Pentium I or AMD K6* (or anything older), expect no love from the sampling based profiler, and some uphill battles with instrumentation. I may post hints on this latter point if a few people speak up about legacy hardware.

C'mon, all your Friends are Doing it

AngryRichard — Thu, 27 May 2004 22:15:00 GMT

A bunch of the guys on the team I work for have been starting up blogs. I started feeling left out,
which made me very angry.

It appears all blogs start with "Hi, I'm a developer who does X and I'm going to talk about Y and maybe Z."
It's all part of Microsoft's new image -- we're transparent now.

Transparency's good, right?

Go down to your local German car dealer, or, if you own a newer German car, go out to your driveway. Shiny, pretty, isn't it? Open the hood. Look at that -- a big, fat, intake pipe that goes into a big box that says "BMW" in a 4" Century Gothic font. Cool. Now pop that plastic thing off the top. Go ahead, I dare you. Not so pretty now. Look at all those wires and tubes. Look at all that stuff you could cut yourself on. That's why that plastic thing is there; it makes owning all that power a little less scary.

Some of us spend all our time under the shiny plastic thing, hands full of wires and tubes and spark plugs. That'd be me and some of my less 'transparent' friends, working on the instrumentation and data collection engine in the profiler.

Of course, with great power, comes great potential for disaster. Some day you'll have your turn with the profiler. Trust me, something's always too slow. If it works, great. If you find yourself upside down in a ditch, tell us, we want to know where we need to cover up the sharp edges.