Vance Morrison's Weblog : Perf

Slides for a talk on Performance

vancem — Tue, 21 Jul 2009 18:05:00 GMT

I am giving a .NET Performance talk for the users group at the Royal Bank of Canada, and I am posting the slides for the talk here for future reference. (see power point document attached to this entry below).

Anyone who has heard me speak will see recognise the themes: Good perf does not just happen, you have to plan, and to plan you have to measure. I touch very briefly on some ways of measuring (Stopwatch, MeasureIt, XPerf)

As with all talks, the goal is not so much to teach something, but rather to MOTIVATE you to learn more. Thus I want it to be very easy to follow up on the topics I so briefly touched upon. Thus I have pulled the bibliography out of the deck and placed it here so you can quickly research more.

Articles

Blogs

Tools

I did not have time to more than just mention it, but I want to emphasize here that Event Tracing for Windows (ETW) and its viewer (XPERF), is the most useful general purpose tool for doing performance analysis on windows, period. It is well worth investing time in learning about it more. Currently it has some limitations associated with diagnosing managed code, however rest assured that we are addressing these issues.

An important take away here is that learning about XPERF is a great investment. It is THE tool that the entire Windows team uses for all of its performance diagnosis, and we are investing in it for the future too.

Slides for our All Day PDC 2008 talks on: Performance By Design

vancem — Thu, 30 Oct 2008 07:34:00 GMT

Every year or so, when Microsoft believes it has something useful to say to develoeprs it hosts a Programmer Developer's Conference. It is doing so this year from 10/27 through 10/30. Between Rico Marani, Mark Friedman and myself, we gave a all day talk on writing high performance software for the .NET Runtime.

I am posting the slides here for reference. I had to break the slides up into the sessions we gave them to fit within the size restrictions of the blog. I also had to post them as seperate posting (since the blog software seems to only support one attachment per blog entry). However I have put the quick links to them here.

PerfByDesign.Intro.pptx - talk given by Rico Mariani on the importance of thinking about performance early in the design cycle.
PerfByDesign.CPU.pptx talk given by me on measuring time (particularly CPU time). I talk about MeasureIt here.
PerfByDesign.Memory.pptx talk given by me on measuring memory in managed applications (Good Stuff)
PerfByDesign.Parallel.pptx talk given by me on using our new Parallel Extentions Framework (PFX) to exploit multi-procs.
PerfByDesign.aspnet.pptx talk given by Mark Friedman on tuning ASP.NET applications.

These slides are of course more useful to those who actually attended the talks, however I think there is a lot that can be learned from them even without having heard us talk. I am happy to answer any questions that anyone might have after viewing the slides.

You can view the other talks and slides from the PDC by going to there web site http://microsoftpdc.com/Agenda/. This will certainly work for those attending the PDC, but might work for everyone (I don't really know).

Slides for PDC 2008 Talk: Performance By Design: Measuring CPU Time

vancem — Thu, 30 Oct 2008 07:12:00 GMT

These are the slides for the talk I gave on 10/26/2008 at the Programmer Developer Converence (PDC) on the basics of performance investigation

Links to MSDN articles I have written on designing for performance

vancem — Wed, 22 Oct 2008 17:46:00 GMT

I just happen to notice that I don't have any links from my blog to some recient MSDN articles I wrote on performance. I want to quickly correct his with this posting.

There is actually a very nice summary page that MSDN created that gathers together all the articles I have written over time. The link is here (a web search of MSDN Vance Morrison will also turn it up).

The two articles I want to call your attention to however are these

CLR Inside Out: Measure Early and Often for Performance, Part 1

CLR Inside Out: Measure Early and Often for Performance, Part 2

These articles are about how most perf problems start very early in the design, and that there is no substitute for knowing how much things cost when doing your design.

There is a nice companion tool that I wrote callled 'MeasureIt' which you get download (free), that allows you to

Get a bunch of useful benchmarks on how much various .NET operations cost
Measure other stuff that you happen to be interested in easily (eg how fast is that collection class you are using?)

In fact, I intend I just recently used MeasureIt to measure how expensive the various levels of the CPU cache hierarchy are. (that is how much does a fetch from L1 cost? How much if you miss the L1 Cache but hit the L2 cache? How much more expensive is it when you miss both L1 and L2 and have to go to main memory?). If there is interest I can write a quick blog about that.

Giving Performance Talk at Programmer Developers Conference (PDC) 10/26

vancem — Thu, 25 Sep 2008 19:21:00 GMT

This is a quick plug for a pre-conference session I am giving on 10/26 at Microsoft’s Programmer Developers Conference (PDC) 10/26 held this year in Los Angeles. My talks are part of a all day session that I am giving along with Rico Mariani and Mark Friedman entitled

Performance by design using the .NET Framework

In the morning Rico will be starting us off with some basics: having a performance budget, designing for performance and the basic of measuring. Mark will speak on the performance characteristics of important parts of the framework. In the afternoon, I will be doing two sessions, the first is on memory performance in .NET: when you should care, how to measure it, behavior of the GC, and best practices. I will follow up on multi-threading performance in .NET. This session will overlap a bit with the talk on multi-core programming, but I will focus on the .NET aspects only try to have a very practical perspective. Finally Mark will finish off with two talks, one on ASP.NET performance, and another on Silverlight performance.

I am looking forward to it. If you are interested, you can learn more about registration here. I am also interested in talking to customers and getting feedback, so if you want to talk to someone who has been on the .NET Runtime team 'forever', come to the PDC and we can talk.

To Inline or not to Inline: That is the question

vancem — Wed, 20 Aug 2008 06:41:00 GMT

In a previous posting, I mentioned that .NET V3.5 Service Pack 1 had significant improvements in the Just in time (JIT) compiler for the X86 platform, and in particular its ability to inline methods was improved (especially for methods with value type arguments). Well now that this release is publically available my claim can be put to the test. In fact a industrious blogger named Steven did just that and blogged about it here. What he did was to create a series of methods each a bit bigger than the previous one, and determined whether they got inlined or not. Steven did this by throwing an exception and then programmatically inspecting the stack trace associated with the exception. This makes sense when you are trying to automate the analysis, but for simple one-off cases, it is simpler and more powerful to simply look at the native instructions. See this blog for details on how to using Visual Studio.

What Steven found was that when tried to inline the following method

public void X18(int a)

{

if (a < 0 || a == 100)

{

Throw(a * 2);

}

It did not inline. This was not what Steven expected, because this method was only 18 bytes of IL. The previous version of the runtime would inline methods up to 32 bytes. It seems like the JIT’s ability to inline is getting worse, not better. What is going on?

Well, at the heart of this anomaly is a very simple fact: It is not always better to inline. Inlining always reduces the number of instructions executed (since at a minimum the call and return instructions are not executed), but it can (and often does), make the resulting code bigger. Most of us would intuitively know that it does not make sense to inline large methods (say 1Kbytes), and that inlining very small methods that make the call site smaller (because a call instruction is 5 bytes), are always a win, but what about the methods in between?

Interestingly, as you make code bigger, you make it slower, because inherently, memory is slow, and the bigger your code, the more likely it is not in the fastest CPU cache (called L1), in which case the processor stalls 3-10 cycles until it can be fetched from another cache (called L2), and if not there, in main memory (taking 10+ cycles). For code that executes in tight loops, this effect is not problematic because all the code will ‘fit’ in the fastest cache (typically 64K), however for ‘typical’ code, which executes a lot of code from a lot of methods, the ‘bigger is slower’ effect is very pronounced. Bigger code also means bigger disk I/O to get the code off the disk at startup time, which means that your application starts slower.

In fact, the first phase of the JIT inlining improvement was simply to remove the restrictions on JIT inlining. After that phase was complete we could inline A LOT, and in fact the performance of many of our ‘real world’ benchmarks DECREASED. Thus we had irrefutable evidence that inlining could be BAD for performance. We had to be careful; too much inlining was a bad thing.

Ideally you could calculate the effect of code size on caching and make a principled decision on when inlining was good and bad. Unfortunately, the JIT compiler has does not have enough information to take such a principled approach. However some things where clear

1. If inlining makes code smaller then the call it replaces, it is ALWAYS good. Note that we are talking about the NATIVE code size, not the IL code size (which can be quite different).

2. The more a particular call site is executed, the more it will benefit from inlning. Thus code in loops deserves to be inlined more than code that is not in loops.

3. If inlining exposes important optimizations, then inlining is more desirable. In particular methods with value types arguments benefit more than normal because of optimizations like this and thus having a bias to inline these methods is good.

Thus the heuristic the X86 JIT compiler uses is, given an inline candidate.

1. Estimate the size of the call site if the method were not inlined.

2. Estimate the size of the call site if it were inlined (this is an estimate based on the IL, we employ a simple state machine (Markov Model), created using lots of real data to form this estimator logic)

3. Compute a multiplier. By default it is 1

4. Increase the multiplier if the code is in a loop (the current heuristic bumps it to 5 in a loop)

5. Increase the multiplier if it looks like struct optimizations will kick in.

6. If InlineSize <= NonInlineSize * Multiplier do the inlining.

What this means is that by default, only methods that do not grow the call site will be inlined, however if the code is in a loop, it can grow as much as 5x

What does this mean for Steven’s test?

It means that simple tests based solely on IL size are not accurate. First what is important is the Native size, not the IL size, and more importantly, it is much more likely to be inlined if the method is in a loop. In particular, if you modify Steven’s test so that the methods are in a loop when they are called, in fact all of his test methods get inlined.

To be sure, the heuristics are not perfect. The worse case is a method that is too big to be inlined, but is still called A LOT (it is in a loop) and calls other small methods that COULD be inlined but are not because they are not in a loop. The problem is that the JIT does not know if the method is called a lot or not, and by default does not inline in that case. We are considering adding an attribute to a method which gives a strong hint that the method is called a lot and thus would bump the multiplier much like if there was a loop, but this does not exist now.

We definitely are interested in feedback on our inlining heuristic. If Steven or anyone else finds real examples where we are missing important inlining opportunities we want to know about them, so we can figure out whether we can adjust our heuristics (but please keep in mind that they are heurisitics. They will never be perfect).

What's Coming in .NET Runtime Performance in Version V3.5 SP1

vancem — Tue, 13 May 2008 02:14:00 GMT

What's Coming in .NET Runtime Performance in Version V3.5 SP1

It certainly has been a while since I last blogged. Most of this is laziness on my part, but I can truthfully say that it is partly because I have been busy trying to get the next servicing release of the .NET framework (called Version 3.5 Service Pack 1) out the door. Part of the framework (runtime + libraries) servicing is an updated version of the runtime DLL (which we call the CLR).

How do you know if you have an updated CLR? If the file

c:\windows\microsoft.net\Framework\V2.0.50727\mscorwks.dll

has a version number greater than 50727.3000 then you have the changes I will be talking about. Because this is just a servicing of the runtime, we had to confine ourselves to changes that we felt had a very low chance of breaking existing applications. Nevertheless we were able to add substantial performance value. There are two changes I would like to talk about in particular.

Improved Cold Startup performance

Cold startup time is the time it takes an application to start when all the data needed to run has to be fetched from disk (rather than from the operating system disk cache). The second time an application starts (Warm startup) is substantially faster because very few (slow) disk reads are needed to get the application running. It is not unusual for the warm startup time to be good (< 1 sec), but for the cold startup time to be quite bad (5-10 sec). In the new runtime we go to a lot of effort to pack all the code and runtime structures (for now only in DLLS associated with the framework itself), to reduce the amount of Disk I/O needed. While the improvements benefit all code that uses the framework, it helps the most on code that has been precompiled using NGEN.exe and the effect is more pronounced on larger applications.

Inlining of value types (C# structs). (X86)

C# structs are what the .NET runtime calls value types. They are called this because when you have fields or arrays of such data structures, the value is embedded directly in the field (not referenced through a pointer). All the primitive types (int, char, bool) are value types, as well as a few types defined in the base class library like DateTime, Decimal, Point and Rectangle. The previous version of the code generator (for X86) did almost no optimization on value types. This was unfortunate, because although value types are not common (most types are classes, not structs), when they are used, they can be used heavily, and so optimization is important. In fact the biggest single piece of feedback we got on our feedback site related to performance was concerning more agressive inlining of value type methods. The runtime’s 64 bit code generators could already optimize value types, but the code generator for X86 could not. Since X86 is still the dominate platform there was an unmet need. In this servicing we included the work we did to enable this for X86. Your own code often may not benefit from this improvement (because it does not use value classes much), but if you do use them, you tend to use them heavily, and the value could be substantial.

64 Bit machines: X64

Much of the cold startup work was rearranging the data in precompiled framework executables to pack heavily used items together, and as such is applies equally well for 64 bit platforms like X64. However there are places where 64 bit machines are quite different (how exceptions are delivered by the operating system) and we did not invest heavily in tuning (packing) these places. The result is that cold startup wins will not be as dramatic on 64 bit machines. There were improvements to the 64 bit code generator, but since this code generator could already inline value types, we did not need to do work in that specific area.

64 Bit machines: Intel Itanium

Most users of the .NET runtime run it on the X86 processor, however beginning with Version 2.0 of the runtime also supports the Intel Itanium architecture. In fact, internally the Itanium version of the runtime came on line well before the X64 version did. The Itanium processor tends to be used in high end server applications, where scalability is a great concern. In the next major release of the Framework, we are continuing to investigate and improve our performance on the Itanium architecture. Microsoft is a charter member of the Itanium Solutions Alliance (ISA) and we are excited to work with the other members of ISA to further enhance the performance of the .NET Framework on Intel Itanium Architecture. Check out the CLR Performance Team’s blog on investigating Itanium perf for more on this work.

Assumptions used in the cold startup formula (when is it accurate).

vancem — Sun, 15 Apr 2007 18:03:00 GMT

After my last blog entry on cold startup a reader (dimkaz) worried that the formula would not be accurate in many cases.

This topic is worth discussing in some detail because it pretty common to apply formulas outside the assumptions implicit in them, and simply get wrong conclusions. The most fundamental part of the formula is

Cold Startup = WarmStartup + DiskIOTime

Technically this is only accurate if WarmStarup time and DiskIOTime don't overlap in time. For example, if DiskIOTime and WarmStartup time could be done completely concurrently then ColdStartup time would be the maximum, not the sum of the two times, which could be quite different. So the question is how much overlap is there between the disk and the CPU?

Well in one very common case the answer is simple. If the application only has one thread of execution (eg. you did not use System.Thread, or System.Threadpool in your startup path), and you don't do any significant explicit file I/O in your application on startup, then the times will be additive (the formula above is correct). In this very common scenario, the vast majority of disk time is time spent fetching the program itself from the disk. The operating system does this by doing what is called 'demand paging'. The idea is that in one operation (the API is called LoadLibrary), the entire file is read into memory, however all the data is not actually read in at that time, but is done 'lazily' on demand. Basically whenever the CPU tries to access memory that was supposed to be read in, but is not actually been read in, the OS takes a 'hard page fault', stops what it is doing and fetches the needed page from disk (on x86 this happens pages are 4096 bytes).

The upshot of this is that during cold startup the CPU is constantly trying to access some new part of the program which needs to be paged in (it takes a hard page fault). Each time it has to stop, and get data from the disk. Since it really needs that data (typically the next instruction to executed) to proceed, so it simply has to wait until the I/O is done. Thus there is no overlap between the CPU and disk I/O for that thread of execution. Hard page faults typically take 4-8msec to service, which translate into 8 to 16 million instructions on a 2Ghz machine (so a hard page fault is quite significant), which h is why cold startup times can easily be many seconds long.

It is possible that if there were multiple threads of execution during startup, that while one thread is waiting for I/O to complete it can be doing something useful on another thread (thus CPU and I/O can overlap), so this may invalidate the formula above, however as we will see, the even in this case the windows prefecher keeps the formula working.

Even single threaded programs might be able to overlap I/O and CPU if they do a lot of explicit, sequential file reads during startup. The reason is that when a Stream.Read is executed the operating system knows that the program is very likely to want to continue reading even after it has return the data that is actually requested. Thus it issues the disk requests to prefech the next chunks of data from the disk and allows the disk to fetch them while the program operates on the last chunk that it was given. If the program does a lot of computation and explicit sequential reads, then the CPU tends to completely overlap the I/O and the formula is wrong (it is too pessimistic). This can happen for example in compilers, where allot of explicit reads as done as well as lot of computation (although even there, the bulk of the computation happens later, after all the data is read in, and thus does not overlap with I/O).

Finally there is one more interesting case of overlap. In the formula above all disk I/O is assumed to not overlap with itself. This is correct if there is only one disk involved, but if multiple disks are involed obviously overlap is likely to happen. Thus the formula is for the 1-disk case and is only an upper bound for the mulit-disk case. In practice, few applications have control over the hardware configuration of the system, and in fact most client machines have only one disk, so the formula covers what is likely to be the most interesting case, but the assumption should be called out.

The upshot is that the formula is valid for programs single threaded programs that don't do allot of sequential file I/O. It is an upper bound for all programs, and frankly, tends to be pretty accurate even in those cases, because events tend to conspire to prevent CPU and I/O from overlapping effectively.

In my next blog I will talk about the windows prefetcher, which is some very cool technology that has been in the OS since windows XP, and happens to make the cold startup formula accurate for pretty much any program.

A Model for cold startup time of an application on Windows.

vancem — Mon, 09 Apr 2007 07:56:00 GMT

Well it is has been a while. I see now that it is been getting close to 1/2 a year since I last posted. Sigh. It is so easy for other things to get busy and not to blog. I have resolved to try to be a bit more methodical about and insure that I write something every week or so.

Over the past year I have been doing quite a bit of work with the Common Language runtime trying to improve its cold startup time. This has been a common complaint, and we knew when we shipped version 2.0 of the runtime that there was a lot we could do. Unfortunately, it may be a while before the wider world see the benefit of this work (our ship cycle is quite long, and the next public release of the runtime has no major changes to the 'core' part of the runtime that I work on). Nevertheless, I learn some useful things so far in this work and I thought I would write some of them down.

When we first seriously looked at cold startup time we had two problems

Our measurements of cold startup were VERY noisy. This makes work quite difficult.
We did not have a clear idea how to prioritize work. We knew Disk I/O was important, but how important? Were disk seeks or data transfer time most important? We needed to have a good model of what contributed to cold startup time before we could come up with an attack plan and prioritize the work properly.

The good news is that we came up with simple model that we are pretty happy with that helped solve both of these problem that I would like to share with you.

First: What is cold startup time?

Cold startup time is defined to be the time it takes an application to start up (be responsive to user input), on a machine that has just been rebooted. The important characteristic of cold startup time is that any data on the disk (mostly the program itself), needs to be read off the disk because it is not in the operating system 's disk cache. Thus cold startup has a very strong component of Disk I/O time associated with this data fetch. In contrast, warm startup time is defined as application startup time on second launch. Under normal circumstances warm startup has almost NO disk I/O because literally everything the program needs is in cache, and thus the program never needs to go to disk to get anything.

This leads to a very nice formula

ColdStartup = WarmStartup + DiskIOTime

This formula is accurate to a very good approximation. Moreover DiskIOTime can be broken into two parts: The time it takes to move the disk head and for the disk to rotate so that the correct data is under the head (Seek Time), and the time it takes to actually transfer the data. Thus

DiskIOTime = DiskSeekTime + DiskDataTransferTime.

These quantities can also be modeled well. Disk Seek time does depend on how far the disk head needs to travel, however disks are now quite large (100s of Gig), and the Operating system does try to keep files that are used together close together on the disk (More on that on a later blog if there is interest), so the amount of head movement is typically very small as a % of the total possible disk travel. Moreover, whenever there is ANY movement of the disk head, it takes a significant amount of time for the head to 'settle' so that an accurate read can happen. Thus to a good approximation, the seek time is independent of how far the disk head moves, and thus can be accurate modeled by a constant. Some empirical data gathering shows that for the machines that we typically use this time was about 4msec per movement. Thus

DiskSeekTimeMsec = NumberOfSeeks * 4

The DiskTransferTime can also be accurately be modeled. It is completely determined by the rotational speed of the disk, the number of read heads, and the density of the bits on the disk (all of which are constant for a given disk). Typical disks that we use give about 50MBytes / sec transfer speeds. This means that it takes 1/50 of a second (20 msec) to transfer 1MByte of data. Thus

DiskTransferTimeMsec = NumberMBytes * 20

Putting this all together gives the formula

ColdStartupTimeMSec = WarmStartupTimeMsec + 4 * NumberOfReads + 20 * NumberMBytes

This formula shows that you can attach cold startup one of three ways: Either reduce the warm startup time (computations done at startup by the app), the number of data read from the disk (typically dominated by the amount of UNIQUE code used at startup), or the number of disk reads (typically by packing the code used in the startup path so that it is all together and can be read from the disk in one read).

Now the formula does depend on disk technology, but typically will not vary in huge ways (by more than a factor of 2) from one disk to another (assuming a certain quality in your hardware). In fact one of the nice benefits of the formula is that it is MACHINE INDEPENDENT (assuming the warm startup time is hardware independent (at least approximately)). Thus measurements on one hardware can be compared to that on different hardware (you can't do that with time). Effectively the formula calculates the cold startup time on a theoretical piece of hardware (that happens to have the specs of the machine I used to derive the 4 msec and 20 msec/MB numbers.

This solved both of our original problems (noise and prioritizing dev work). The main reason cold startup is noisy is that the disk is being shared by ALL apps in the system, and thus their activity can perturb the time. However the operating system already tracks the metrics above (number of read I/O and the number of I/O bytes), and breaks it down PER PROCESS, which means these I/O measurements have NO NOISE associated with other processes activity. This is GREAT and reduces the noise SUBSTANTIALLY. It solved the prioritizing problem because it tells us the first level of breakdown of what contributes to cold startup and thus we know which one more dominate for a given scenario (typically none of the terms can be ignored), and thus where we can get the best benefit (for a given cost).

Next time I will talk about how these numbers play out in a real scenario, and where that led us to work prioritization.

Its good to be back!

Vance

Drilling into .NET Runtime microbenchmarks: 'typeof' optimizations.

vancem — Sun, 01 Oct 2006 16:04:00 GMT

In my last blog entry showed how to use a simple simle class called MultiSampleCodeTimer to measure the performance (time), of pretty much any fragment of CLR code quickly and easily. That entry had a solution packaged up as a zip file TypePerfMeasurement.zip that you could use to duplicate the experiment yourself. However I did not have time to show you what is particularly interesting about those paricular performance measurements (I pciked them for a reason). Here I wish to do complete the explaination.

When I unpack and run TypePerfMeasurement.zip (please do it yourself, detailed instructions at the end of my last post), I get the following output.

Data units of msec resolution = 0.279365 usec
10 typeof(string)                       : count: 10000    7.677 +- 3%    msec
10 typeof(string).TypeHandle            : count: 10000    0.000 +- >400% msec
10 anObj.GetType() == type              : count: 10000    7.393 +- 41%   msec
10 Type.GetTypeHandle(obj).Equals(tHnd) : count: 10000    4.554 +- 9%    msec
10 anObj.GetType() == typeof(string)    : count: 10000    0.103 +- 7%    msec
10 (anObj is string)                    : count: 10000    0.594 +- 12%   msec

Please take the time to run it yourself. You don't need Visual studio do so (see readme.txt file, it is as easy as running 'buildit, and running the resulting exe). Note that if you are running under visual studio, and you get very differrent numbers based on whether you run it under VS or outside VS, you did not set your VS settings as I describe in this post. Go and do that now. Note that your numbers are going to be different than mine, but the relative sizes between the various rows should be reasonably close.

What do these numbers mean? We need some background first. The .NET Runtime defineds a type called System.Type, which is a representation of a type in the system. EVERY class that has been defined in the system has a cooresponding System.Type. The System.Type object is the gateway to exploring at runtime all the data that the users typed when he defined the type. You can get superclasses, methods, properties fields, etc by calling the appropriate operations on System.Type. This ability to inspect and invoke characteristics of the source code is called Reflection, and System.Type is your gateway into this capability.

While the System.Type class is very powerful, that power comes at a cost. Also most programs doe NOT need this power and should NOT use it (I need to blog about that too) becuase of the perf cost. Thus the .NET runtime has tried to provide alternatives to using System.Type for some common operations that CAN be implemented efficiently.

One such operation is testing if an object is of a EXACTLY a certain type. While the 'is' operator in c# can often be used for this (and is the preferred method), however that mechanism will also match any subtype (eg ("foo" is object) returns true) and also only works for literal types (if the type was a variable of type System.Type), you could not do this. For example

    System.Type myType = typeof(string); // in real life not a literal
    // and then in other methods (and maybe in a loop)
    if (anObj.GetType() == myType) { /* some op */ }

In this example we use the 'GetType() method on System.Object to see if 'anObject' is a string. Unfortunately both the operations used above (typeof, and GetType()), are relatively expensive (as shown in my data above 5-10ms instead of .7ms for 'is' operator).

Type checking operations like the one above are common enough that the .NET runtime has added a special type called 'RuntimeTypeHandle' to make them fast. RuntimeTypeHandle is a less user friendly but 'fast' alternative to System.Type. It happens to be simply a managed wrapper that represents the internal pointer (to a structure called a MethodTable), that the runtime uses internally to represent a type. As mentioned it is not very friendly, however it can handle the scenario above, and is very fast. The code using RuntimeTypeHandles looks like

    RuntimeTypeHandle myType = typeof(string).TypeHandle;
    // and then in other methods (and maybe in a loop)
    if (Type.GetTypeHandle(anObj).Equals(myType)) { /* some op */ }

This code is significantly faster than the previous code becuse no System.Type object needed to be created, only RuntimeTypeHandles, wich is simply a value type wrapping an unmanaged poitner, and thus is very lean (The code above sadly is slower than it should be because some JIT optimizations where not done in time for V2.0, but is still significantly faster than using System.Type. The first operation

    RuntimeTypeHandle myType = typeof(string).TypeHandle;

Is blazenly fast, as shown by our measurements (it is slow small it is in the noise). This is suprising because this line does a 'typeof(string)' and we know that that operation is relatively slow. Setting a breakpoint in this code shows why. That line of code compiled down to

    00000000  mov         dword ptr [ecx+10h],790FA3E0h

Basically the JIT compiler is smart enough to realize that to fetch the RuntimeTypeHandle, you don't need to fetch the System.Type and fetch its TyepHandle, you can just lookup the value at compile time and emit code to generate the literal. This is very fast.

Sadly the other part of the operation

    if (Type.GetTypeHandle(anObj).Equals(myType)) { /* some op */ }

Should be just as fast (fetch the MethodTable poitner from the the object, and test for pointer equality), but is not due to some JIT optimizations that did not kick in appropriately (you will see that we make real calls for both GetTypeHanlde and Equals). Even so, it is faster because we avoid creating a System.Type object.

As data point of just how fast the code above coudl be (when the JIT is fixed), consider the final experiment in the code

    result = anObj.GetType() == typeof(string);

Given what we know already, we would expect this to be slow, and yet, the measurements above show it to be FASTER than using the 'is' opertator. How can that be? The answer is that the JIT recognises this sequence and knows that while it seems like two System.Type object need to be created and compared, all you really want is a yes-no answer on a type question that can be answered using RuntimeTypeHandles. It thus substitutes that code

    0000000f  cmp         dword ptr [edx],790FA3E0h 
    00000015  sete        al   
    00000018  mov         byte ptr [ecx+0Ch],al

EDX holds 'anObj' and the method table for an object is the first field of the object, and 790fa3e0h is the method table pointer (RuntimeTypeHandle) for string. (See previous post on using !DumpMT to determine this). Thus in one instrution we have tested whether 'anObj' is string. The 'sete' instruction converts the procressor condition flags set by 'cmd' into a boolena value in register AL, and the last insturction sets 'result' to this value. This is pretty lean an mean!

Summary:

In this entry, we have taken a look at some code generation for doing 'type reflection'. We measured its performance and got some anomolies (some operations were much faster than we expected. We looked into the disassembly for those operations and determined that the JIT compiler was doing some non-trivial optimizations that made certain operations very fast (in one case faster than the 'built in' 'is' operation).

We have learned that a System.Type object is relatively expensive compared to RuntimeTypeHandle, and we have used techniques from the last few perf blog entries to help dig into exactly why.

I again encourage you to experiment with the TypePerfMeasurement.zip example yourself and hone your skills in measuring and investigating .NET Runtime performance. In my next blog entry I will be doing 'inventory' on the performance characteristics of the 'basic' operations of the runtime.

Measuring managed code quickly and easiliy: CodeTimers

vancem — Fri, 22 Sep 2006 06:16:00 GMT

My performance blog entries to date have been 'foundational'. In entries so far, I talk about how to use Visual Studio to look at the native code generated for the runtime. With this foundation, we can now start exploring what the native code for managed code looks like and what optimzations the runtime does on your behalf.

However before we do that, we need one more foundational piece. You really can't do performance work, without measuring, and so we need tools to help us measure. The simplest of these is the stopwatch. Thankfully Version 2.0 of the rutnime provides a nice, high resolution class that does this called System.Diagnostics.Stopwatch, which is ideal for measuring short intervals accurately.

As nice as System.Diagnositcs.Stopwatch is, I found myself wanting more. I was repeatingly typing code to start and stop the Stopwatch, subtracting the difference, subracting out the harness overhead, and doing statistics on the resulting numbers. This led me to write two classes to automate this, call CodeTimer, and MultiSampleCodeTimer. I have included the source for these in the Zip file below. Simply open it, copy the directory to your local drive, and open the .SLN file using visual studio. You can even compile the code without having Visual Studio. Simply run the buildIt.bat batch file The only prerequisite is that you need V2.0 of the .NET runtime installed (available from windows update, among other places).

The basic usage pattern for CodeTimer is

    CodeTimer timer = new CodeTimer();
    timer.Measure("Measurement Name", delegate {
       // Code that you want to measure
    });

Basically I am making use the 'anonymous delegate' feature of c# to allow the client to specify the code that he wants measured inline. One of the nice feature of anonymous delegates is that they 'capture' any local variables that are in scope. Thus you can write things like

   
    CodeTimer timer = new CodeTimer();
    string myString = "aString";
    string outString;
    timer.Measure("Measurement Name", delegate {
       outString = myString + myString;  // measuring concatination.  
    });

To measure the time it takes to concatinate two strings. What is actually going on 'under the hood' is that the C# compiler generates a class with two fields 'myString' and 'outString' (a field for every local variable referenced in the body of the delegate code). An instance of this class is created and initialized to the values of locals and this object is used to create a delegate to pass to 'Measure'. Thus the locals are 'captured' and passed to the code that 'Measure' will ultimately call.

This detail is imortant if you are measuring very small operations, because field access generates different code than local variable access. Thus in the example above, I am not just measuring string concatination, but also fetching an instance field (mystring) twice, and setting an instance field (outstring). Generally these fetches are small, and so they don't purturb the data much, but you should be aware of them.

When you are measuring very short times (like the case above), you are likely to wan to run the test in a loop. CodeTimer has built-in support for this. I can run the example above 1000 times in a loop by doing

   
    CodeTimer timer = new CodeTimer(1000);
    string myString = "aString";
    string outString;
    timer.Measure("Measurement Name", delegate {
       outString = myString + myString;  // measuring concatination.  
    });

Which tells 'timer' to run any measurment 1000 times. This is great for measuring micro-operations.

While 'CodeTimer' is great, I want still more. I don't want just one sample, but 10 or more, so I can take the average and standard deviation and understand how 'noisy' my data is. That is exactly what MulitSampleCodeTimer does. What is nice is that only the defintion of timer changes.

   
    MultiSampleCodeTimer timer = new MultiSampleCodeTimer(10, 1000);
    string myString = "aString";
    string outString;
    timer.Measure("Measurement Name", delegate {
       outString = myString + myString;  // measuring concatination.  
    });

Which asks that the code be run 1000 times per measurment and that 10 measurments be taken and used to compute min, max, mean, median and standard deviation. The results are then printed (we are reallly starting to add value).

I find I use MultiSampleCodeTimer alot. I use it in the'TypePerfMeasurement' project below to measure the speed of some reflection type constructs. More on that in my next blog entry.

SUMMARY:

In this blog entry, I have introduced to two classes 'CodeTimer' and 'MultiSampleCodeTimer' that can be used to meause the performance (time), of a wide variety of code. You are encouraged to open, copy and build the attached sample below (you don't need VS to do it just the .NET runtime), and try it for yourself. Measuring performance has never been easier.

Next time we will talk about the numbers that the TypePerfMeasurement experiment actually produce and talk about why the performance is the way it is.

HOW TO USE THE DEMO:

I want my blogs to be 'hands-on' which means that there should always be something you can do after reading to experiement with the concept I talked about (that is there is homework!). To make this easy I have attached a ZIP file that is a archive of a Visual Studio Solution for code to experiement with. Note that you do NOT Need Visual Studio to compile and run the demo, so don't let that stop you. To unpack this simply

Click on the ZIP file attachement at the bottom of the post. This will bring up a dialog asking if you wish to open the file.
Click on the 'Open' button. This will bring up an explore window displaying the contents of the ZIP file, which is a single directory. If you just want to look at files, you can browse the files directly from this window.
Open a target folder on your local computer. This can be anywhere, but if you have Visual Studio installed 'My Documents\Visual Studio 2005\Projects' is a good place
Drag the directory from the ZIP file window into the target folder. This copies the directory and all its contents to the local hard drive.
Once copied, you can close the ZIP window, and click into the target directory. There you will find a .SLN file, which you can double click on to bring up the solution in visual studio.
If you don't have Visual Studio, you can still build the solution, there is a one line batch file called 'buildIt.bat' file that does this (it simply turns around and calls MSBuild to do its work). You do need Version 2.0 of the .NET Framework installed. If you don't have it you can get it from www.windowsupdate.com (custom downloads), or here.
All the samples build and run and do something interesting 'out of the box', They are meant to be easy to understand, commented, samples so start looking around and experimenting (Hitting F10 to start stepping into the code is a good place to start).

Low-Lock Techniques in action: Implementing a Reader-Writer lock

vancem — Tue, 28 Mar 2006 18:40:00 GMT

In my article What Every Dev Must Know About Multithreaded Apps I discuss the fundamental principles of using locks correctly. In that article I strongly encourage the use of reader-writer locks because these locks create the protection you need (insuring that readers and writers don't conflict), while potentially allowing significantly more concurrency to take place (by allowing multiple readers to enter the lock simultaneously). I did not have space in that article to discuss reader-writer locks in detail however. I would like to start correcting that here.

The .NET Runtime implementation of a reader-writer lock is System.Threading.ReaderWriterLock. It is really a pretty simple API at its heart (ignore stuff about LockCookies, it is there to support nesting of locks, which you should probably avoid anyway). Here is a sample use

MyReaderWriterLock myLock = new MyReaderWriterLock();

myLock.AcquireReaderLock(Timeout.Infinite);
// No writers can be in here (AquireWriterLock will block)
myLock.ReleaseReaderLock();

myLock.AcquireWriterLock(Timeout.Infinite);
// No readers or other writers can be in here (Aquire*Lock blocks)
myLock.ReleaseWriterLock();

It is pretty straightforward to convert a program from using ordinary locks into using reader-writer locks (for those blocks that only read use AcquireReaderLock, otherwise use AquireWriterLock).

This is all very nice. Unfortunately, there is a problem. The current .NET Runtime implementation of ReaderWriterLock is about 8 times slower than System.Monitor (normal locks), in the common case where there is no lock contention. This is quite unfortunate, as it acts as a disincentive for doing the right thing (and using reader-writer locks). Microsoft is likely to fix this in the next release, however what do you do in the mean time?

Implementing a Reader-Writer Lock

The obvious answer is to implement an efficient 'drop in' version of ReaderWriterLock that you can use until Microsoft provides an better one. This is what this article is all about. Sadly, if our goal is to be as efficient as System.Montor, we can't use the standard locking protocol to implement a Reader-Writer Lock since every operation would have to lock and unlock, making it at least twice as slow as System.Monitor. Thus we are forced to look into 'Low Lock Techniques', for an answer.

In my article Understand the Impact of Low-Lock Techniques in Multithreaded Apps I list the techniques that if used very carefully, can sometimes be applied to do very efficient multi-threaded synchronization. I also mention that these techniques should only be used when the use of standard locks are problematic (usually because of performance issues). Reader-Writer locks fall into this problematic cataegory.

Possible Low lock techniques to use

In the article the techniques are listed in order of decreasing utility, and are

Avoid locks on reads
Spin Locks
Raw Interlocked operations
Lazyinit

Because the important APIs for a reader-writer lock always do an update, technique 1 is not applicable. Thus our best bet is Spin locks (note the later two techniques are not NEARLY as useful as the first two).

Implementing a spin locks

It turns out that spin locks have some very nice properties. Namely they ARE locks, so all the nice properties of locks apply (once you take the lock you have mutual exclusion and reasoning about the program gets MUCH easier). They are also efficient (one interlocked operation per Enter-Exit pair). Thus we should be able to implement a Reader-writer lock with only one interlocked operation per method call, which is the same as System.Monitor (which also has one interlocked operation per method call).

Please refer to the code attached to this post readerWriterDemo.cs. The first class is the spin lock. It has an 'Enter' and 'Exit' API. For more information on spin locks see my low-lock article or Jeff Richter's article Concurrent Affairs. The implementation in readerWriterDemo.cs deals with all the subtle details you need to worry about. It might need some tuning on how long it spins, but otherwise it is a very respectable implementation and you should feel free to use it if you need a spin lock yourself. The only caveat I curretly have is that I have commented out calls to Thread.BeginCriticalRegion and Thread.EndCriticalRegion because they are relatively expensive (they make my perf numbers look bad), and is only needed in environments like SQL Server where threads may be aborted at any instant. If a thread is aborted while the spin lock is held, the SQL Server host wants to know so that it can kill the entire appdomain instead of just the thread. If you are not in SQL Server you can get away without this.

Before using the spin lock, we need to confirm that our read-writer lock is a good candidate for its use. To be a good candidate is must be the case that the time the lock is held is very short (measured in at most dozens of instructions). We believe that this is the case since all of our API for the reader-writer lock do little and are perf critical, so the code paths NEED to be short. However after we get our implementation we need to confirm that we don't have rare paths that might hold the lock for a long time. It turns out with a bit of work we can make certain this is the case.

Implementing a reader-writer lock with the spin lock

With a spin lock in hand, writing the reader-writer lock is not too much code (~ 150 lines, with comments), and hopefully is easy to understand. Every API takes a lock, does work, and leaves. To deal with the case where you must wait, we need OS structures to wait on (eg ManualResetEvent, AutoResetEvent). When a ManualResetEvent is 'Set' all threads waiting will wake up. This is appropriate for readers. For writers, it makes more sense to only wake up the first thread, and this is what AutoResetEvent does. The code has a lot of comments, so please take a look. There are actually a lot of subtleties to think about (what happens in various error conditions, what APIs can fail, how efficient the lock is in various scenarios, how long we are holding the spin lock ...). I point out some of these subtleties in comments, however the really important point is that the code itself is short and straightforward. This is CRITICAL because the more code you have, the more difficult it is to confirm the code works in all the subtle cases. It pays to keep things simple.

Performance Results of the reader-writer lock

If you run the program, it will measure the performance of an Enter/Exit pair. Here is what I got.

10Meg Enter/Exit 371.6953 msec = 59.61 CPU clocks per Enter/Exit
10Meg RW Locks (.NET library) 3175.8594 msec = 8.5X monitor time
10Meg New RW Locks 428.8234 msec = 1.2X monitor time

As you can see, on my machine, the .NET library is 8.5X slower than System.Monitor, but the new RW locks are only 20% slower. With some more tuning I suspect I can get even closer, however it will make the code more complicated, so I don't want to do that in this published code (cleanliness is worth something).

As written, the lock has the property that if a writer is waiting for the locks, new readers can't come in and block it indefinitely. Instead the readers wait until all the other readers drain, and the writer gets its turn. The lock is unfair in the sense that when a lock is released it is not guaranteed that the first waiter gets the lock. Thus starvation is possible (however unlikely). It turns out that fairness causes performance issues (I will discuss in a later blog), so the current unfair protocol has better performance characteristics in almost all cases. It is not too hard to change the implementation to be fair, and I might take that up in a later blog (please request it if you are interested).

The lock as written is also not reentrant. It is relatively easy and efficient to add this (although it is not clear this feature is a good idea), and I show how to do that too (please request it if you are interested).

The code is really worth looking at carefully and I encourage you to do so. It is small enough that you can understand the intent quickly, but there are many scenarios to think about.

Recap

What are the take-aways?

Use reader-writer locks as suggested in my first article. If lock perf is a problem, take the implementation here as a 'drop in' replacement for the .NET System.ReaderWriterLock. Feel free to use this implementation until Microsoft fixes the version in its library.
Spin-locks are a really valuable low lock technique. There were used here to make a clean, simple but efficient reader-writer lock written competely in managed code. This lock is significantly simpler than most implementations I have seen, yet is clost to optimal. The reason for this is that the use of spin locks GREATLY simplifies the design and analysis of the lock, without sacrificing performance. Feel free to use the Spin lock implementation show here to make other high performance concurrent constructs.
Even something as simple as a Reader-Writer lock has subtleties about its behavior (especially when you factor in performance concerns). Keeping it simple really pays off here.

How to use Visual Studio to investigate code generation questions in managed code.

vancem — Tue, 21 Feb 2006 01:35:00 GMT

Introduction: What does ‘foreach’ actually do?

It is not uncommon for a new group to want to use manage code to pepper the CLR team with performance questions. They want to know how expensive ‘foreach’ is, or whether certain methods get inlined or a variety of other questions about the code quality of.

Sometimes, people look at the IL in attempt to answer questions about the code quality of managed code, however this approach is misguided. The JIT compiler transforms the code tremendously going from IL to native, and thus for most performance questions the quality of the IL does not correlate strongly with the ultimate performance of the code. For codegen performance questions, what matters most is the quality of the JIT compiler.

Often internal partners will simply ask us how questions about what code gets generated for a particular construct, and every time they do this I think ‘Why should I have to do this? Run it under a debugger and see for yourself!’ This blog is the step-by-step instructions for doing just this in Visual Studio.

Conceptually, the process of determining what Native code is generated by a construct is very straightforward

Find or generate code that has the construct you wish to understand
Compile the code with optimizations turned on
Run it under a debugger, stepping to the code interesting code an view it in the disassembly window.

Unfortunately, while all of these are possible to do in Visual Studio, the defaults work against you and you have to know how to change them.

To make things concrete, I will go through the steps for an example. I happen to interested in looking at the the overhead of a foreach() for List<int>.

[System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.NoInlining)]

static void F(int arg) {}

static void ForEachTest(List<int> list) {

foreach (int elem in list) {

F(elem);

}

Compiling the code with optimizations:

This part is almost straightforward in Visual Studio. You would have liked it to be the case that simply compiling the ‘Debug’ to ‘Release’ in the Solution Configuration window at the top-center of the VS IDE would be sufficient. Unfortunately this is simply not true for C# projects (I have not checked for VB or C++ projects).

The issue is that by default release builds do NOT build the program data base (PDB) files that hold the line number and local variable information. This makes it impossible for the debugger to correlate the assembly code back to line numbers. This makes it much more difficult to read the assembly code for large methods. To fix this going to the properties for the project (right click on the project in the Solution Explorer), select the ‘Build’ tab and click the ‘Advanced’ button at the bottom. In the dialog box that comes up there will be a line ‘Debug Info’. It should be set to ‘pdb-only’. This tells the compiler to compile with optimizations, however also create a pdb file.

With that done, make certain your Solution Configuration window says ‘Release’ and compile your solution (F6).

Running optimized managed code:

Sadly, Visual Studio makes debugging optimized code harder than you would like. First, by default, if you launch a managed program under Visual Studio, it will force the JIT compiler to created non-optimized native code. This improved debuggability, but makes it impossible to see that code that actually runs when the app is not run under Visual Studio. The other problem is that Visual Studio has a feature called ‘Just My Code’ in which the debugger will not step into code that it does not believe is being developed. In particular any optimized code is not considered ‘My Code’ and is skipped. Again this makes it impossible to actually stop and see the optimized code.

To fix both of these problems

1) Go to Tools -> Options -> Debugging -> General

2) Make certain that box labeled ‘Suppress JIT optimization on module load’ is UNCHECKED.

3) Also make certain the box labled ‘Enable Just My Code’ is also UNCHECKED.

Note that these are global settings (the affect all solutions), however, I have found I don’t miss either of these ‘features’ (Code compiled ‘Debug’ is still not optimized by the JIT so you still get good debugging there).

Once you have disabled this ‘features’, set a breakpoint at an interesting spot (F9) and run the program (F5).

Examining the optimized code:

Once your breakpoint is hit, it is a simple matter of switching to the disassembly window (Ctrl D CtrlD, or Debug -> Windows -> Disassembly) to look at your code.

You should not see and ‘nop’ instructions in the code. If you do this means you are looking at a ‘Debug’ build of your code. It is not a bad idea to run a simple test by creating a function that returns a constant integer and confirm that it gets inlined in the resulting assembly code. This confirms that all the preceding steps were successful.

You will notice that the assembly code does not have symbolic names for the call targets. Often this is not a hardship, because you can deduce it from the corresponding source line associated with the assembly code. If for things like ‘foreach’ or other constructs where the IL compiler is doing significant work, it is a hardship not to know what the symbolic name of the target is. In the case of my example above the interesting code is

00000077 mov ecx,dword ptr [ebp-24h]

F(elem);

0000007a call dword ptr ds:[00913028h]

foreach (int elem in list) {

00000080 lea ecx,[ebp-30h]

00000083 call 7878BFF4

00000088 test eax,eax

0000008a jne 00000077

So we are pretty sure that we know that dword ptr ds:[00913028h] is really our F() method, however we don’t know what this 7878BFF4 that we call a bit further down. You also might not know what instructions like ‘lea’ do, or how the ‘elem’ parameter is passed to our ‘F’ method.

Luckily all of these issues (and others) will be the topic of my next blog entry.

Recap:

In this blog entry I have given step-by-step instructions for using Visual Studio to look at optimized managed code. Hopefully you will agree that is really is not that bad at all, and it now gives you the tool to answer a whole host of questions such as

1) Is a particular method inlined or not?

2) Does the JIT perform optimization X?

Note that the example above was for X86, but the technique above works just as well for X64. Note that the X64 JIT compiler and the X86 JIT compiler are completely different, so you if you care about performance on both platforms, you will need to repeat your experiment on both platforms.

Vance Morrison's Introduction to Bloging: Lets talk about Perf!

vancem — Tue, 21 Feb 2006 00:36:00 GMT

This is my first real blog entry, so I would like to introduce myself. My name is Vance Morrison and I am currently a performance architect on the .NET Runtime Team. I am one of a handful of developers who have worked on the .NET runtime from its inception. I did most of my work in the Just in Time Compiler associated with the runtime (for example I was the primary contact for all IL opcode issues back in the time when we were actually adding opcodes). What is wonderful about working in the JIT compiler is that you are at what I consider the 'heart' of the runtime. Generally speaking, if it needs to be fast, the JIT needs to know about it. This gave me a lot of exposure to many of the conventions that are fundamental to the runtime's operation (What is the layout of an object, or array, how do virtual method calls happen? Where to security checks get put? How do you report GC pointers to the GC? When can a GC happen? ....) In reality the JIT has a whole is a just a performance optimization and so it is not so a big leap for me to change from optimizing small sequences of instructions to optimizing the runtime more holistically

In my new role, a bit part of what I need to do is to encourage best practices. This blog is one part of my new role. There has been a lot already written about .NET performance, much of which has been written by my predecessor Rico Mariani. This is all good stuff and I have spent a bunch of time reading about it. My focus is going to be a bit different. I want to focus on TOOLS. .

Part of the reason why performance is hard, is that either there are not good tools out there, or that there is not good guidance about how to use the tools well. For example NT has literally hundreds of performance counters. So many it is very difficult to understand which ones are actually good for what. PerfMon, the tool Microsoft ships to display this information has some useful information buried in its help, but most people give up long before they find it. It is also too terse to really be helpful. It is really just a monitoring tool for evaluating the overall health of your system. Once you determine your system is not healthy and have narrowed it to a particular process, you typically need other tools, and a debugger to drill further. Thus perfmon is useful, but it is not obvious how to use it well, and is only useful for one phase of a performance investigation. Seems like some guidance would be useful, so one of my blog entries will be on when I use perfmon, how I use it, and when I move on to other tools.

While I like to believe that I am an authority on some things (like how the .NET framework generates code), there are many more things where I can not offer authoritative guidance (like what half the performance counters collected by windows are good for). In short, I too, have a lot to learn. What I bring to the table is a passion for performance and a desire for finding and promoting the best tools for the job. Undoubtedly I will learn a lot for all of you along the way, and I look forward to it.