Bill Wert's weblog

Tip: Don't rely on environment variables for critical decisions

billwert — Mon, 25 Feb 2008 05:35:49 GMT

I saw a mail this week asking how you can determine what bitness your process is running in, and one of the suggestions was to check for the PROCESSOR_ARCHITEW6432 environment variable. If that variable exists, you know you're on 64 bit Windows in a 32 bit process. It's likely that the questioner needs to know which bitness the application is running in so the correct bitness DLL can be loaded, since by default native code is processor agnostic.

This is one example, but the broader point really is that you shouldn't rely on environment variables as proof of things important to your application. It is too easy for someone to intentionally or accidentally corrupt your environment and cause your program to function incorrectly. Please don't do this in shipping code.

Use of SGEN.exe to avoid common XmlSerializer performance pitfalls

billwert — Sat, 23 Feb 2008 22:01:00 GMT

This week, I encountered a test scenario which makes use of an XmlSerializer on it's startup path. Unfortunately, MSDN's documentation for the type is not clear on the performance impact of using an XmlSerializer. The documentation on introducing XML serialization, found here, actually mentions the solution, but it also does not state what the problem is.

When you use an XmlSerializer, a helper assembly(1) with a custom type for serializing your type is created. The C# CodeDom is used to generate this code, which then is compiled by the C# compiler. The runtime must then load this generated assembly, and JIT the code. Additionally, a separate assembly is generated for each type you create an XmlSerializer for, each of which requires an invocation of the C# compiler.

As you can imagine, running the C# compiler potentially several times to generate these assemblies can cause a large performance hit, especially on your startup path. (On your startup path, since the application blocks while this is happening, your user will think they didn't click the right thing, or that your app is broken because it isn't loading fast enough.)

The really great news, though, is that you can avoid all of this. Starting with version V2.0 of the runtime, shipped with Visual Studio 2005, there is a new SDK tool called Sgen.exe. This tool works much like NGEN, to create serialization helper assemblies ahead of time. You can do this as a post build step, and then ship these binaries. You can (and should!) also NGEN them. When you sgen MyLibrary.dll, you'll get a file called MyLibrary.XmlSerializers.dll. This DLL will serializers for all of the types in the assembly, thereby giving you some consolidation over the non-SGEN route.

So, then, how does one use SGEN as a post build step? In your VS project's Build Events tab, you can add a line like this to the post build steps:

"<PathToSdk>\sgen.exe" "$(TargetPath)"

Where of course <PathToSdk> is whatever is correct for your environment. (If you have an SDK path environment variable, you can use it here.

Now you have a DLL you can ship and NGEN as part of your install process! Your customers will thank you.

Please post in the comments if I can clear something up.

1: You can see this happen in the debugger. You'll see a module loaded with a temp file name, like 9rg4jqp1. This assembly is typically generated in memory, but if you want to look at the generated code, add this to your application config file:
<system.diagnostics>
        <switches>
           <add name="XmlSerialization.Compilation" value="4"/>
        </switches>
</system.diagnostics>
The dlls, pdbs, and source will appear in your %TEMP% directory.

How often should we be measuring?

billwert — Tue, 13 Apr 2004 15:45:00 GMT

Testing performance is tough. As a product is being developed, you need to track the performance of what you're building. The question is how often?

It turns out it's a very difficult and time consuming thing to identify the cause of failures in a performance suite when the amount of change to the product is large. You spend a lot of time profiling something, fixing a bug, and re-running your profiles. Say, for example, that there are 15 changes in the build you're testing, base lined against the last one. You end up with 43 performance degrades. What caused what? Finding out is going to be a long process. Your options, essentially, are to profile each individual degrade and find the hotspots in the new code, or to profile a sample of them and hope the fix for those problems fix the rest. The first option is time intensive to the max: Running the tests again under the appropriate profiling tools, finding the bugs, fixing them, and validating the work. The second approach is a crapshoot. What do you do the next day when your tests haven't come back? Now you have to ask yourself the question “Did the fix I made really fix this problem, and this is caused by something new? Or did I not fix it?”

A good solution to the problem is serious performance testing on each and every change to the product. This is not to say that you must run days of testing for each change, but some testing is prudent. The problem with this approach is that it costs some infrastructure time. As we discussed earlier, performance testing can't be done on any old machine sitting around the office. It should have dedicated hardware and as static an environment as possible, to avoid putting noise into the process.

On my team, there's a system which produces a build of the product for nearly every change. This is to facilitate running automated check in BVTs. As a nice side effect, we on the performance test team can grab those builds and test them. We have a bank of machines set up and some automation in place to monitor the drop shares for the builds and start tests automatically. The results are then pumped into a database and viewable through a webpage.

This system is new in the last six months. The old process was to test a build or so a day, and write bugs in hopes something could be done. Today, we're able to write much more targeted, actionable bugs. The result is that problems are fixed much faster, before they turn into “Death by paper cuts“, which we'll explore soon.

The best laid plans...

billwert — Wed, 07 Apr 2004 19:10:00 GMT

I started this, as most do, with the best of intentions. A long list of topics to discuss, potentially interesting things to say, and so forth. Then, as is usual, life interrupted.

So, I put forth the question to the proverbial teeming masses of the readership: What do you want to know about testing performance? Some thoughts:

Workingset measurement
Scalability measurment
Startup time
?

Let me know what you're dying to know.

Timing your code

billwert — Thu, 11 Mar 2004 21:51:00 GMT

Timing your code

One of the fundamental aspects of performance testing is identifying how long it will take you to execute the code you’re writing. There are a few ways you can measure this: measuring time elapsed while your code executes, computing how many instructions it may take and extrapolating time from there (based on target hardware), or I suppose you could guess. I can’t recommend the last option, however.

Hopefully, you’re reading Rico. He has a lot of information to share, but qualifies it by saying you should almost always follow it, except when you shouldn’t. I’m even less firm that that, even – all I wish to offer at this point is one method for accomplishing this task. It is by no means the “correct” way, nor is it necessarily the correct way for you.

A useful way to measure the product as it’s going along is to write micro benchmarks which exercise the hot code paths. A micro benchmark, essentially, is the distillation of the meat of a piece of functionality, stripped down to its core. With the application separated in this fashion, it’s easy to identify what’s slowing down an overall scenario. Once you’re done writing it, make sure to spend a little time thinking about what kind of results you expect to see. Depending on your requirements, of course, examples of good goals for a micro benchmark can be like “we need to be able to execute ShoppingCart::Add 300 times a second”, or “We need to be 5% faster than the last release.”

With that in mind, let’s first discuss some common pit falls of attempting to measure managed code execution speed.

Warm vs. Cold code paths

The first time a managed method runs, it’s going to go through a JIT prestub. The first time the method is called, it’s JITed. It also eliminates the prestub call, replacing it with the actual address of the freshly jitted code. Calling your method once outside the loop ensures you are not timing JIT compilation, or timing replacing the prestub.

Run your code path more than once

It’s a good idea, when designing a micro benchmark, to run many, many iterations. The point, essentially, is to discover the answer to the question “How many times can I do this, in how long a time span?” As you can’t really control the thread scheduler and other aspects of the system, a small amount of variance can creep into your measurement. The solution is to run a benchmark for around the same number of time, increasing or decreasing the number of iterations as necessary. On the CLR Performance test team, we target five second runs for most of our micro benchmark tests. We then track how many iterations it takes to accomplish that, and we can use the metric of iterations per second as the benchmark number.

Ensure the environment is right

There are plenty of things you can do to the machine to ensure your performance measurement won’t be impacted. It’s preferable, for example, to start from a clean state if possible. This can be accomplished thorough the standard means of imaging, using good uninstall / reinstall steps, or what have you. It’s also a good idea to disable things like the screen saver, anti-virus software (carefully! Make sure you’re in a safe environment), and any scheduled tasks that might run in the background. It also can be useful to disable any services that might start using the CPU or disk unexpectedly, if possible.

Tools, tools, tools

Of course, you have to have a method with which to time code. Classically, one method for doing so has been QueryPerformanceCounter in the unmanaged world. However, no facility exists in the released versions of the framework which implements this functionality. The solution is to use interop to access this functionality. At the least, you need a start, stop, and result reporting method. I’ve written a simple example of this, called SimpleTimer, for which the source is at the bottom of this post.

Writing a benchmark

Now, let’s spend some time thinking about our benchmark. I’m going to demonstrate a fairly contrived example here. Let’s suppose we’re implementing the ShoppingCart::Add method I’ve mentioned a couple times. This is a fairly perf critical place in our application, and we want to make sure that changes to it in the future don’t impact it heavily. Pretend with me that someone’s given us a DLL containing a ShoppingCart class, which we can test.

Our benchmark code might go something like this:

using System;

using Perf; // The namespace of our timer class, SimpleTimer

public class Test

{

public static void Main(string[] args)

{

SimpleTimer timer = new SimpleTimer();

long nIters = Int64.Parse(args[0]);

ShoppingCart sc = new ShoppingCart();

sc.Add(); // Warm up that code path

timer.StartTimer();

for(long i = 0; i < nIters; i++)

{

sc.Add();

}

timer.StopTimer();

timer.Result(nIters); // the Result method prints the results of the test to the screen.

}

Essentially, when this is compiled to an executable, you’d execute it passing in the number of iterations desired. Timer.Result will then print out the results of the test: number of iterations, time elapsed, and iters/sec.

That’s a pretty good start. There’s a potential problem to think about though: The overhead of the for loop might be more costly than what you’re attempting to time. In that case, the trick is to execute your code path multiple times within the loop.

What to do with the results?

Congrats! You now have a fully functional and useful benchmark for your project. A final tip is to run your benchmark a few times, and average the results together. This will go a long way to eliminating any further noise that may be creeping into your results.

Wrapping up

This is getting a bit long, so I think I’ll leave it there for now. The key takeaways are:

Identify and isolate your hot code paths – these can become your micro benchmarks.
Write your benchmarks with what you’re timing in mind – ensure you’re not timing your loop or other extraneous code.
Warm up the code paths you wish to test
A uniform environment is key to repeatable results
Report the result of multiple runs of a benchmark

SimpleTimer source

using System;

using System.Runtime.InteropServices;

using System.Security;

namespace Perf

{

[StructLayout(LayoutKind.Sequential, Pack = 8)]

public class large_int

{

[MarshalAs(System.Runtime.InteropServices.UnmanagedType.I8)]

public long value;

}

[SuppressUnmanagedCodeSecurity()] // This is to eliminate the security check calling down to QueryPerformanceCounter. Be careful where you use this!

public class SimpleTimer

{

[DllImport("kernel32.dll")]

public static extern bool QueryPerformanceCounter(large_int count);

[DllImport("kernel32.dll")]

public static extern bool QueryPerformanceFrequency(large_int freq);

private large_int m_Freq = new large_int();

private large_int m_Start = new large_int();

private large_int m_Stop = new large_int();

public SimpleTimer()

{

// It's a good idea to run your timer code path once to warm it up

StartTimer();

StopTimer();

}

public void StartTimer()

{

QueryPerformanceCounter(m_Start);

}

public void StopTimer()

{

QueryPerformanceCounter(m_Stop);

}

public void Result(long nIters)

{

QueryPerformanceFrequency(m_Freq);

double nResult = ((double)m_Stop.value-(double)m_Start.value) / (double)m_Freq.value;

double nItersSec = nIters / nResult;

Console.WriteLine("{0} iterations took {1} seconds, resulting in {2} iterations per second", nIters.ToString(), nResult.ToString("F3"), nItersSec.ToString("F3"));

}

[This post is provied AS IS, implying no warentees and conferring no rights.]

What To Watch

billwert — Tue, 09 Mar 2004 22:25:00 GMT

[Housekeeping stuff. I added categories. Neat stuff.]

RichB asked me yesterday in the comments to my first post:

“What are the range of metric values I should expect to see for a given scenario. For example, how many gen1 collections per minute is OK for a heavily loaded web app?“

This, unfortunately, is not the kind of advice I can give you. Rico or Jan are far more likely to have useful answers to that kind of question - they're design kind of guys. Being a tester, my angle is more from the “how do I measure and keep track of this stuff“ point of view.

On with the show...

When I said I'd point you at some metrics to keep in mind, I was talking about these things:

Working set
Startup Time
Code execution speed (Throughput)

There are other considerations, but these are the big concerns. Let's define, shall we?

Working Set

Working set can be defined as the number of pages your process is keeping in RAM. This includes the space required for all the code and data you have loaded. It also includes any dynamic data that your process allocates, including the GC heap. As a rule, measuring your working set does not require any changes to your code.

Startup Time

This is the time required to bring your app to a useful state. Depending on what your particular scenario is, this is going to be a different measurement. If you're concerned with time to Main(), as we are with the CLR, you'll write a test scenario that exits as soon as possible. More likely, however, you'll find yourself wanting to time to when your app is loaded and responsive to the user. This type of testing requires some instrumentation.

Code Execution Speed (Throughput)

When we're discussing Code Execution Speed, we're asking ourselves how long an arbitrary bit of code takes us. Often times, it's useful to create benchmarks for your high priority code paths so you can track them independent of the entire product. For example, if you're concerned about the execution speed of submitting an item to your shopping cart, you might write a console application which exercises the ShoppingCart::Add method.

As a note, if you have a modification or clarification for these definitions, by all means let me know! Going forward, these are what I'll be using.

That's all nice: So what?

Having laid some foundation, we can start discussing how to measure these things. I realize that this may be starting slow for some of you. I'm starting from the absolute ground up to ensure that I don't leave anyone behind. If you find this stuff useful, great! Please drop me a line in the comments. If you're reading this and yawning, great! You're ahead of the game on perf. Drop me a line too. Once I gauge who's reading and where the right level is, I can narrow in on more useful stuff. Until then, I'm going to keep going along the basics for now. Tommrow (Or maybe later tonight), we'll get into some useful tools for measuring the stuff I'm discussing.

Managed Code Performance

billwert — Tue, 09 Mar 2004 02:13:00 GMT

I never quite know how to start something like this without sounding a bit goofy. “Hi there” seems odd, seeings as I have no real idea who I'm talking to yet. Ah well.

My name is Bill Wert. I'm a Software Test Engineer on the CLR Performance team. I started this blog in hopes that I can fill what I see as a niche. People like Rico talk about what kind of design patterns can lead to bad performance. The message at the end of the day is that until you measure your own scenario, you won't really know. What I'm hoping to provide through this blog are tips on measuring your performance. There are a number of gotchas that you may or may not be aware of. So, over the next N weeks as time allows, I'm going to cover topics in this neighborhood:

How to time code
What metrics should you be tracking, and how often
Useful tools for measuring performance
Suggestions?

I'm sure there are things that you might be curious about which I haven't listed here - please let me know in the comments of this post.

Also - Thanks to Scott!