IntelliTrace Info : beta

IntelliTrace iTrace Files

ianhu — Mon, 16 Nov 2009 21:26:00 GMT

If you are not familiar with the new IntelliTrace feature in Visual Studio Team System 2010 than you might want to first check out either my or John Robbins’ introductions to this feature as a general overview of IntelliTrace would be helpful before digging into this article.

What is an iTrace file?

In my introduction article linked above I talked a little about how IntelliTrace captures the current state of the debugger at multiple points during a program’s execution and, when F5 debugging, allows you to debug back in time to previous debug states in your program. This in and of itself is a very handy feature, but in this day and age it’s often hard to have a bug with an easy and consistent repro that you can debug on a local dev box.

The solution to this lack of a local repro is that not only does IntelliTrace enhance your local debugging experience, but it also saves all the collected debugger data points into a trace log file (.itrace extension) that can then be opened and debugged using Visual Studio later and on a different machine. The analogy for this scenario is that of a black box in an airplane in that iTrace files provide a “voice from the grave” from crashed programs that allow for a developer to debug in and around the point of failure after the fact.

Integration with Microsoft Test and Lab Manager

One of the big new testing features being added in Visual Studio Team System 2010 is the Microsoft Test and Lab Manager (more info on MTLM on their blog site here). MTLM is a standalone tool that focuses on the tester role by providing a TFS-integrated UI for managing test cases and lab environments without the overhead of a full Visual Studio installation. Since one of the key focuses of IntelliTrace is to try to eliminate the “no repro” disconnect between developers and testers we knew that we needed to get IntelliTrace integrated with MTLM. This integration is accomplished via a combination of TFS and iTrace files. I’ll detail the scenario more in a future blog post, but at a basic level at anytime during a test run a tester using MTLM can choose to file a bug on a specific test step failure and when that bug is filed an iTrace file of all the recent debugging events and exceptions is automatically collected and attached to the bug. Then, when the developer opens up the bug in Visual Studio, they can just click the iTrace file linked in the bug and be debugging into the exact execution path in which the tester was seeing the failure.

iTrace files collected during debugging

Whenever you are running a normal F5 debugging session from the Visual Studio IDE with IntelliTrace turned on you are collected an iTrace file in the background. Now in this scenario you can pretty easily be using IntelliTrace features like browsing back in debug history without ever noticing that this file exists, especially since to keep your system from getting clogged with iTrace files we clean these files out when Visual Studio is shut down. So if you ran into something interesting while debugging from the IDE with IntelliTrace you will need to copy the iTrace file out from its saved location to keep it from being cleaned up. Just look under the following file path to see the iTrace files that have been collected during the current VS session: C:\Users\All Users\Microsoft Visual Studio\10.0\TraceDebugging. With both the IDE scenario and the MTLM scenario iTrace files will be truncated at a specific size (currently set to 100MB by default) to keep from filling up your hard drive. This truncation value will discard older events from the log and can be changed from Tools->Options->IntelliTrace->Advanced.

Collecting iTrace files from the command line

If you want to collect iTrace debugging files without having Visual Studio up and running we’ve provided the IntelliTrace.exe command line tool. IntelliTrace.exe will get its own blog entry sometime in the future but if you want to try figuring out how to get it running just try starting with the /? command for help. Intellitrace.exe is located in your Visual Studio install at “Team Tools\TraceDebugger Tools.”

Working with iTrace files in Visual Studio

(Note: All screenshots are from my current working build and will look a little different from Beta 2 builds)

Regardless of if you collected your iTrace file via MTLM, Visual Studio or IntelliTrace.exe when you first open it up in Visual Studio you will end up with a document that looks somewhat like the below.

Note that at this point we’ve just opened up the document summarizing the debugging session. No debugging session has been started and the time to open up the document should be pretty minimal. At the top of the document you will see a chart showing all the threads that were running during the life of this debugging session. Below that there are a series of lists containing more information about Threads, Exceptions, Test Events, System Information and Modules for the debugging session. Currently, the Exceptions list is expanded out and showing all the exceptions that were encountered during the debugging run. The exception that is currently selected in this list is represented in the thread timeline by a vertical red bar. This bar helps you match up exactly where in your program’s execution an exception was being thrown. In addition to supplying the thread, HResult and message of the exception we will list out the stack of each thrown exception in the textbox below the exceptions list.

Threads List:

The threads list provides a table view of the threads active during your debugging session. The actively selected thread will be highlighted in the thread chart above, and vice-versa.

System Info:

The system information section contains a set of information about the computer that this iTrace file was collected on. It seems pretty basic, but this info has already come in useful several times for me during my development work. In particular knowing the OS, the number of processers and the CLR version have been useful to me when investigating bugs that QA has provided me with iTrace files for.

Test Data:

I don’t have a picture of this right now, as I’m going to be speaking more about this when I cover MTLM integration in greater depth. When you collected an iTrace file via MTLM (since this file was not collected via MTLM the section is grayed out) the test data section will contain info on all the test steps that were logged via MTLM during the execution of the tests.

Modules:

As expected, this control lists out the modules that were loaded during debugging.

On all of these controls there is a search box above the list. If you are looking for a specific module or exception just start typing into one of those boxes to narrow down the results being shown in the list.

Starting a debug session from an iTrace file

Up until now we’ve been dealing with the information that you can glean from the summary page of an iTrace file. But while it can be quite informative the real point of the summary page is to allow the user to jump into debugging close to some point of interest. Lots of information can be collected during a debugging session and if you were to just jump into debugging an iTrace file blindly it could take a while to get to the correct location to start diagnosing a failure.

From an iTrace summary page you can jump into debugging from a thread, from an exception or from a test event (with a caveat that I’ll mention later for test events). For threads you can double click on a thread in the thread chart, double click on a thread in the threads list or click the “Start Debugging” button. Any of these options will start up the debugger and jump you to the last event that we collected on that thread. We chose the last event as in many cases an iTrace log captured a failure or crash so starting at the end point makes more sense than starting at the beginning of the log when all was running smoothly. For exceptions, you can either double click the exception in the list, or click the “Start Debugging” button below the exception list. Starting debugging on an exception will start the debug session exactly on the exception event selected (see picture below). Note that starting the session is a slightly slow process and it might take a bit for the debugger to get up and running. Test data function much the same as exception with the only difference being that since test events are not represented in our debugging UI debugging context will actually be set to the event in time nearest the selected test event.

This entry is mainly focused on the iTrace file and summary page so I’ll cover more about the IntelliTrace UI during debugging later. But as for now, you can look at the picture above and see that debugging has been started and our context has been set to the exception that I clicked in the summary page by looking at the source location and the autos window.

Up next

Coming up next I’ll be talking more about the various controls that you can use to move around in IntelliTrace debugging data and what data we will be showing in the Visual Studio UI.

Collecting performance counter information with the Visual Studio Team System profiler

ianhu — Wed, 17 Oct 2007 20:21:00 GMT

Collecting performance counter information with the Visual Studio Team System profiler

Introduction

If you have ever done performance work on Windows systems you are probably already familiar with the PerfMon tool. This tool gives you an overview of your system performance and can be an invaluable tool in the early (and sometimes the late) stages of a performance investigation. If you look at either of my two PerfMon links you can see that PerfMon is usually used to pick out what the performance bottleneck is when examining an application. By “performance bottleneck” I mean picking out the resource that is being saturated by your program (CPU, network, disk). PerfMon does this by tracking some subset of performance counters that monitor different aspects of system performace. This is important as you often need to use different techniques and tools for investigating different bottlenecks. For example, using the Visual Studio profiler it’s usually best to use sampling mode to investigate CPU issues and instrumentation mode to investigate memory and disk issues.

With the Visual Studio Profiler we wanted to give customers an easy and integrated way to collect this performance counter information and view it alongside their performance data. This was especially important to us as with this information we could help customers to analyze specific trouble areas of their program or to choose the correct profiling modes based on their performance bottleneck.

Turning on performance counter collection

To access this performance counter collection start out by going to the session properties of your performance session. In the session properties you will want to select the new “Windows Counters” options, shown in the screenshot below.

On this page you can check the box in the top of the screen labeled “Collect Windows Counters” to add windows performance counter collection to your next profiling run. Below that checkbox you see a box to input a value for how often the counters are collected labeled “Collection interval (msecs).” But before we dig into the collection interval it is important to understand just how these counters are collected. With our profiler we have the concept of “marks” that show up in the data collection stream with a comment associated with them. Unlike all other profiling data these marks are not aggregated and show up in chronological order in the “Marks” view of the performance report. These marks are usually added by annotating your code with the data collection APIs (native API here). When we tell the profiler to collect Windows performance counters they get collected every time one of these marks gets hit. The collection interval control allows you to automatically insert marks in your program while running at some specific time interval. That way you can collect performance counter information easily without all the hassle of adding marks to your program manually.

In addition to setting the collection interval you also need to control what performance counters are being collected. In the lower section of the property page are all the different performance counters that you can selected to be collected. We surface all the same counters as PerfMon, so if you have a favorite counter that you like to track you can be sure that we’ll have it. By default we’ve included basic processor, memory and disk usage counters. We picked these as they give a good basic overview as to how your application is taxing the system.

Viewing performance counter data

So to demonstrate how these marks and associated counter values are actually surfaced in the report file we need to run the profiler and collect our data. For this example I’m using the PeopleTrax sample application and running it in sampling mode with the default set of performance counters and the default 500 ms marking interval. For my profiling scenario I’m just going to wait for the application to initialize, hit “get people” button to load the records from the database and then shut down the application. After the application is shut down the performance report will load automatically and then I will jump to the “Marks” view.

In the view above you can see the automatically collected marks. And for each mark you can see the memory, disk and processor usage at that point in time. Now this data can already tell us much about our program. For example we can see that we started out with a brief processor spike between marks three and five closely followed by a memory usage spike between marks five and eight. But how can we actually tie these values into all the rest of the performance data that we collected? After all these are timeline values while the rest of the data in the report is aggregate data from the whole run. Luckily we’ve provided a new filtering feature to help use the performance counter data to guide your performance investigations.

Filtering performance data from the marks view

Looking further into the marks data there was another big memory spike later in my program between marks 129 and 133. But how can we tell what was actually happening in the program during that time? To start the investigation ctrl-click marks 129 and 133 in the marks view so that they are both selected. Then right-click on one of them and select the “Add Filter on Marks” command. When you do this a new filter dialog will appear docked on the top of your performance report.

The filter control has a lot of depth to it, but for now you can just understand that is it currently saying “only show the data collected between mark 129 and mark 133.” To actually apply the filter to the performance report click the “execute filter” button in the toolbar (it’s the button with the green play icon). You will see a progress bar for the reanalysis that is being performed and then the report will pop back up. Only now all the aggregate data for the report is only the data that we captured between those two marks! With just a quick glance at the summary page below you can tell exactly what our application was doing and why we were seeing the memory usage spike at that point (big blue circle added by me for emphasis ;-) ). However it is important to keep in mind that you will need to clear the filter items and rerun the empty filter or to close and reopen the report to get back to seeing all of your performance data.

In addition to filtering on marks the filter grid can so some other very neat stuff such as filtering on threads, by timestamps or by time percentage (to do something like “show only the first 10%” of my run). I don’t have room to cover all the other filtering types here though, so I’ll have to come back and hit them in a later article.

Conclusion

Collecting performance counters is a very cool new feature that we’ve added to the profiler for Orcas. Even before we get into how cool it is to be able to filter down your profiling results based on performance counters it’s nice to just be able to collect and store those values side by side with your profiling data. And when you throw in the filtering features it quickly becomes an invaluable tool for performance analysis.

Comparing performance reports with the Visual Studio Team System Profiler

ianhu — Tue, 03 Apr 2007 20:54:00 GMT

Comparing performance reports with the Visual Studio Team System Profiler

With the recent release of the first Beta for Visual Studio Team System (codename Orcas) customers will get their first chance to see all the great new features that we are adding to the product. For the profiler in particular we’ve added some very cool new features that I’m really happy to finally be able to reveal publically. Remember that the profiler is only included with VSTS: for Developers and VSTS: Team Suite, so make sure that you have one of those SKUs if you are following along.

Comparison Reports from the IDE

The first feature that I want to cover is the new comparison document that we’ve introduced to help users compare profiling data from two different profiling runs. Say that you have just checked in a possible fix for a performance issue and you want to compare a new performance report to one that existed before the change to see if your change really helped. Our new comparison features make these types of questions easy to answer. Comparing two files of performance data is a very common scenario for customers, especially when dealing with regression testing, so this was a priority feature for us in this release. After all, performance data when taken in isolation, without goals to hit or old values to compare with, can be pretty hard to work with. Our goal with this new comparison work is to help customers to make better use of their performance data to achieve the performance results that they desire for their applications.

I would be remiss if I did not take a quick second here to dive a little deeper into the importance of setting performance goals for your applications. Too often developers end up in the situation of closing in on product release and realizing “my app is just way too slow.” Now, it is perfectly understandable that developers want to save performance optimization for the end of the product cycle; after all if the underlying structure is going to change greatly why waste too much time early on trying to tweak things to run as fast as possible? But the real issue with the situation above is in the generalness of the “my app is just way too slow” part. What exactly do you mean by “too slow?” What parts of the app are too slow? What type of performance do customers expect from your app? How long does it take for other similar products to do the same task? For our new comparison features to be really useful you will need to take some time before and during development to get at least basic answers to some of the questions above.

So now that you know to set performance goals for major functionality of your application how can you use the new comparison functionality of the profiler to meet those goals? First off, let’s say that you have just submitted some major code changes to your product and you want to see how those changes have either improved or degraded the performance of your app. Start off by moving to the new “Developer” top level menu in VS. This menu (Figure 1.1) is new, so expect that the names and locations of the items in it may change by the time that RTM rolls around. From this menu select the “New Comparison Report” option to start the process of creating a new comparison file.

(Figure 1.1 – The new VSTS developer menu)

Next, the UI will pop up a dialog that will ask you to select two performance data files to compare. Note that in this screen you can select any combination of VSP and VSPS files provided that both files are either both sampling mode files or both instrumentation mode files. If you have used the VSTS profiler before you will know that VSP files are the report files created by the profiler, but you may not be familiar with VSPS files. I will cover VSPS files in more depth in a following article, but in a nutshell they are VSP files that have been saved after performance analysis, resulting in a much smaller file that can be opened much quicker than a full VSP file. However, after saving as a VSPS file you lose some of the advanced analysis options that you had with a VSP file. For this example, I’ll just be comparing two profiling runs of a simple console application. This application concats a bunch of strings (a common performance issue) and since VSTS code analysis flags this as a possible issue I’ve converted it to use the StringBuilder class instead. The second analysis file is from the performance run that uses StringBuilder and I want to compare it to the earlier run to see if I have made any performance gains.

If you select one sampling mode performance file and one instrumentation mode performance file then you will get an error popup about incomparable file types. Sampling and instrumentation work in totally different ways and with completely different column sets, so we don’t want to allow comparisons between the two types of files.

After selecting the two files to compare and clicking “ok” a new document window will open in the IDE and you will see progress bars for the analysis of both of the performance files. After the loading finished you will see a comparison document like the one below open in the IDE (Figure 1.2).

(Figure 1.2 – The comparison report view)

So there is a heck of a lot of information in that comparison report, so let me see if I can help to break it down. In the top left “Comparison Files” groupbox we see the filenames of the two report files that we are comparing (Concat.vsp and StringBuilder.vsp). In the “Comparison Options” groupbox we specify what values to use for comparison and what magnitude of change we require to actually show items in the comparison report. The “Table” value in this group specifies the type of objects that we are comparing. The default value is to compare functions but you can switch this to compare other values like modules or IPs. The value in the “Column” combo box specifies what report column to use to compare between the baseline report and the new report. The column values available for comparison will change based on the comparison table that you have selected. By default, we are comparing the “Exclusive Samples %” value for the functions table. If you are not familiar with the various performance columns (you can get more info on performance report columns here) “Exclusive Samples %” is the number of total samples that were taken in the specific function and not in any subfunction. Finally we have the “Threshold” value. This value specifies the minimum difference that there must be between new and baseline values for the function to actually be listed in the comparison report. For example, let’s change the threshold value to 1.0 and click the apply button to apply the changes. Now the comparison report has been trimmed to just show functions that changed by one percent of the overall samples (figure 1.3).

(Figure 1.3 – The trimmed comparison document)

Now that we have trimmed our report to a more manageable size let’s take a look at the body of the comparison report. In the “Comparison Column” we see the names of the functions that are being compared between reports. Next we see columns that show the value of “Exclusive Samples %” in both the baseline report, in the comparison report and the delta between the two (comparison value – baseline value). Note that now that we have changed our threshold value to 1.0 only functions with an absolute delta value greater than 1.0 show up in the report. If we were to change threshold to 0.0 we would see every function in both reports, even if the function did not change value at all. Functions that were only present in the baseline report or only present in the comparison report also show up in this report as long as their value (and thus their delta value) is greater than the threshold value.

So what does the comparison report tell us about our code change from using String.Concat to using StringBuilder.Append when constructing our strings? Well it appears that our overall change to system performance was basically a wash, we used to spend 4.4% of our time in the AppendItemsConcat function and now we spend 4.62% of our time in the AppendItemsStringBuilder function. At a lower level we spend less time in the String.wstrcpy function and more time in memcopy, this is to be expected based on how String.Concat and StringBuilder handle combining strings. Also, if we see something in the comparison report that we want to investigate further we can right click on that function and choose “show matching row in baseline file” or “show matching row in new file.” These options will open up (this open should be very quick since we already have it analyzed the file for the comparison) the matching performance report so that you can dig deeper into the calltree for that specific function. You can see the result below of right-clicking the “show matching row in new file” on the String.wstrcpy function. The new performance file has been opened and the String.wstrcpy function has been highlighted in the function view (fig 1.4).

(fig 1.4)

From the investigation above we can see some differences between the Concat implementation and the Stringbuilder implementation but we really don’t see any measurable performance gains. Perhaps instead of saving CPU cycles we actually ended up saving time and space in allocations. If we want to take a look at the different allocation patterns for the two scenarios we can create two memory profiling reports and compare them. Note that memory allocation reports are a separate type of report so you will not be able to compare allocation reports to non-allocation reports. In the example shown below we can see the difference in managed allocations between the String.Concat and the StringBuilder scenarios (Figure 1.5).

(Figure 1.5 – A managed memory comparison report)

In this comparison I’m looking at the “Types” table and checking the difference in bytes allocated for that specific type (Exclusive Bytes). As we can see, for the cost of allocating one StringBuilder and a Char[] I’ve cut down the amount of string bytes allocated in half! Not too shabby. Outside of specific performance investigations the new comparison reports can be a really handy learning tool for when you want to compare a few different approaches to the same scenario.

Comparison Reports from the Command Line

So now that I’ve show you how to work with these nifty comparison reports from the IDE you may wonder how you would integrate this functionality into something like a script driven regression prevention system. We’ve foreseen this need (and we know that not everyone loves the VS IDE like I do) so we added on comparison report functionality to the command line performance report tool VSPerfReport. VSPerfReport can be found in the Team Tools\Performance Tools directory under the main visual studio directory in Program Files.

If you run -? on VSPerfReport.exe you will see the new comparison (called diffing in the command line tool) in the “file diffing” section.

---- File Diffing ----

/diff Activate diffing mode for comparing two vsp files (summary options will be ignored in diff mode)

/diffthreshold:[value] Below this value of difference the differ will regard two values as the same. Also, new data with values under this threshold will not be shown.

/difftable:[tablename] Use this specific table to perform diffing. The default is the functions table.

/diffcolumn:[columnname] Use this specific column to perform diffing. The default is the exclusive samples percent column.

/querydifftables List the valid diff tables and columns for the two vsp files provided.

To perform a simple comparison using all the default values just type something like the following:

Vsperfreport /diff Concat.vsp StringBuilder.vsp

This will compare the two reports using the “Functions” table and the “Exclusive Samples Percent” column with a default threshold level of 0.1. This command will generate a CSV (comma separated value) file that contains the same data that we would see in the IDE comparison report.

Location, Function Name, Old ExclSamplesPercent, New ExclSamplesPercent, Delta

Both,"System.String.wstrcpy(char*,char*,int32)",10.35549,6.076618,-4.278869

In this text report you will see the location of the function (Both, OrigOnly, NewOnly) as well as the name, old value, new value and delta. Remember that by default this has a threshold value set, so if you don’t see all the functions that you expect some of them may be under the threshold value.

Just like in the IDE reports you can tweak the table and column used for the comparison. To see all of your options just run /querydifftables with the /diff command to see the output below (customized for report type of course).

Available Tables and Columns for Diff:

Table: Function

Column: InclSamples -- (Inclusive Samples)

Column: ExclSamples -- (Exclusive Samples)

Column: InclSamplesPercent -- (Inclusive Samples %)

Column: ExclSamplesPercent -- (Exclusive Samples %)

Table: Module

Column: InclSamples -- (Inclusive Samples)

Column: ExclSamples -- (Exclusive Samples)

Column: InclSamplesPercent -- (Inclusive Samples %)

Column: ExclSamplesPercent -- (Exclusive Samples %)

Table: Line

Column: ExclSamples -- (Exclusive Samples)

Column: ExclSamplesPercent -- (Exclusive Samples %)

Table: IP

Column: ExclSamples -- (Exclusive Samples)

Column: ExclSamplesPercent -- (Exclusive Samples %)

Then just use the /difftable and /diffcolumn switches with the diff command to change the table or the column being used. Below I’ve listed an example of doing a comparison of Inclusive Samples on a per module instead of a per function basis.

vsperfreport /diff /difftable:Module /diffcolumn:InclSamples Concat.vsp StringBuilder.vsp

Also, you have the /diffthreshold switch that can be used to set which items will actually get included in the comparison report. From here you should be able to figure out how to generate the same types of reports from the command line as you could from the IDE.

Hopefully this article has given you a good intro to using the new performance report comparison tools in Visual Studio Team System codename Orcas. Before I go I’ll leave you with one last cool feature that we’ve added for Orcas. Say you wanted to show a co-worker the results of your performance investigations into String.Concat and StringBuilder. Just highlight those rows, hit copy and then paste into an Outlook e-mail (fig 1.6).

(Figure 1.6 – Comparison data pasted into an e-mail)

Check out that fancy HTML autopasting! Much nicer then having to look over the unformatted text.