IntelliTrace Info : developer

Historical Debugging in Visual Studio Team System 2010

ianhu — Thu, 14 May 2009 02:56:00 GMT

Historical Debugging in Visual Studio Team System 2010

What is Historical Debugging in a nutshell?

If you’ve been in the development world for any length of time you’ve probably ended up in a situation like one of the following more than a few times.

· You’ve received reports of a crash from a tester, but on your local box you can’t get the bug to reproduce.

· You’ve received a crash dump from the tester along with the bug. But the callstack that actually caused the crash was just a cascading effect and you can’t trace the bug back to the root issue.

· The bug that you are currently working at resolving has an extremely long set of reproduction steps and you just accidentally stepped over the function that is returning bad data.

· You know that some part of your program is hitting the registry way too often, but while stepping through all that you see are .Net framework calls and you are unable to isolate which of them is doing all the extra registry work.

With Visual Studio 2010 Team System Editions we are introducing a new Historical Debugger aimed at getting rid of these developer pain points. The Historical Debugger plays a role similar to that of a black box in a plane. We keep track of important points in your programs execution and allow you to play back what happened at those points at a later time. We’re very proud of the current experience that we offer with the Visual Studio debugger so we’ve worked hard to surface all this new historical data in a way that is both useful and consistent with what you would expect from debugging in Visual Studio.

A little more depth on the Historical Debugger

In the previous section I’ve mentioned some scenarios that the Historical Debugger is meant to help solve but I’ve not mentioned much about how the Historical Debugger actually works aside from the black box analogy. To get an overview about how the Historical Debugger actually works I’ll do a quick little rundown on what the Historical Debugger collects, when it collects it and how you can view this information after collecting it. I’m going to keep to rather general terms for now, but in later blog posts we’ll dig down deeper into each area and talk more about how to configure and use the Historical Debugger from within Visual Studio.

What does the Historical Debugger collect?

Consider what you usually see in Visual Studio when you stop at a breakpoint during your program’s execution. At a basic level, you will probably see a source file with an arrow in the margin indicating where you currently are in program execution. You’ll probably also have access to a window showing your local and watch variables, a window showing your current callstack and some intellisense values for variables in your source code. Also, you might be digging into some more advanced windows like the threads window or the memory window available in the debugger. With a few notable exceptions the basic data that we will be collecting with the Historical Debugger will be a subset of the information normally available to you when doing standard live debugging.

The first thing that might strike you about this is “Isn’t collecting all this information going to slow down debugging my application by some crazy amount?” Trust me when I say that this worry is the proverbial Sword of Damocles that dangles over the head of our team. As big of an issue as this is, I think that we’ve come up with some pretty clever solutions to collect a useful amount of information without perturbing normal debugging patterns (more of these solutions will be mentioned in the “when does the Historical Debugger collect?” section below). A big part of this process was selecting what information is most commonly used during debugging but without picking anything that overly bloats our log files or slows down debugging too much. For starters, every time we stop to collect data we will grab your current code context and the current callstack. Second, we will get the value for any primitive data types that would appear in the locals or watch window at that point, we also get the primitive values for up to one level deep on any objects that would be in the locals or watch windows .Third, to the immense relief of those doing multi-threaded debugging, we will collect the data on the currently active threads.

When does the Historical Debugger collect data?

Above, we’ve talked a little about the debugger information that the Historical Debugger collects. But now we need to address how often this information is collected. After all, even if we are collecting a tiny amount of information collecting it too often will quickly lead to your program being slowed down to a halt and generating massive log files. To make Historic Debugging useful we need to provide the user with some solid default settings for how often they collect data and allow for some tweaking to adjust these default values.

When designing the Historical Debugger we based it around two main default levels of data collection. The first level of collection is based around the concept of collecting debugger data at specific points of interest in your program called diagnostic events. Diagnostic events are locations that we have selected as being common points of interest for customers when debugging a managed application. These diagnostic events are selected by Visual Studio and are meant to cover a broad range of programming types. An example diagnostic event that is included with the Historical Debugger would be RegistryKey.SetValue. If this diagnostic event is enabled you well collect a full set of Historical Debugging data every time that RegistryKey.SetValue gets called. These diagnostic events will act like checkpoints when you go back to examine your historical data. We of course allow for you to tweak which sets of diagnostic events will be enabled anytime you are debugging.

We think that we’ve selected diagnostic events that will be useful across a broad range of scenarios, but there will be plenty of times where the area you are interested in debugging after the fact will not have any diagnostic events of interest in it. For this scenario we’ve added an option to also collect debugging data at all method entry points in your program. Also, in this mode we will collect additional data on the parameters that were passed into each method. As would be expected this mode will increase the overhead that the Historical Debugger creates, so be aware of the effect it will have on your applications performance when debugging. Overhead aside, we believe that the data collected in this mode can be very useful especially in that it give a better idea of the shape of your program as opposed to just diagnostic events.

How do we show the Historical Debugger information?

For the most part the Historical Debugger information will be shown in the normal debugger windows such as the watch window, the locals window and the thread window that you already know and love. Although as mentioned above we do only collect a subset of total debugger information, so don’t expect to see everything when debugging historically. When you are in the middle of a normal debug session you will be able to just step backward to the most recent diagnostic event (or method enter / callsite if you are collecting them) from there you will be able to move about between the various points that we collected data during your current debugging session. As we dig into the new historical debugging UI in later blog posts I’ll talk more about the features that we’ve integrated into VS to help make this navigation easier.

In addition to being able to move back in time from a normal debugging session you can also save off historical debugging data in a separate .tdlog file and open this later or on another computer. This tdlog file is a key component of what we call the “no repro” scenario, the scenario in which testers pass off a bug to developers but developers are unable to recreate and debugger the error condition locally. We’ve provided integration with Camano (our new standalone test case management tool) and TFS to make it super easy for testers to attach tdlog files to any bugs that they file. Now when developers open up a bug they will be able to also open up the attached tdlog file and debug to any point where historical data was collected to help track down the issue.

What’s next?

This goal of this little intro was to tell you the very basics about the how the Historical Debugger works. In the upcoming weeks I’m going to start rolling out more articles that provide in-depth detail about how the Historical Debugger works and about how you can use it from Visual Studio 2010. Expect more pictures of how the UI will actually look and function as well as more samples about how Historical Debugging can help to solve common (and uncommon) programming issues.

Back To Blogging

ianhu — Thu, 14 May 2009 02:49:00 GMT

Back to Blogging

So for the last 18 months or so there has been nary a peep on this blog about anything. One contributing factor to this extended period of silence would happen to be the fact that I now have an 18 month old daughter and I was feeling a bit of a time crunch. But the actual big reason is that I moved teams within Visual Studio and was working on a project that was flying under the radar for about a year. This new product is the Historical Debugger that is going to be shipping with Visual Studio 2010 (I’ll have an introduction post up for this feature later today) and it’s much more out in the open now so I’m ready to start blogging up some hype for it. I’m still a part of the Visual Studio Diagnostics team and just down the hallway from my profiler ex-coworkers so I’m still around for answering / forwarding profiler questions. As before there will be a few odd-ball posting here and there, but in general just expect lots of information, samples and chat about the Historical Debugging feature.

Collecting performance counter information with the Visual Studio Team System profiler

ianhu — Wed, 17 Oct 2007 20:21:00 GMT

Collecting performance counter information with the Visual Studio Team System profiler

Introduction

If you have ever done performance work on Windows systems you are probably already familiar with the PerfMon tool. This tool gives you an overview of your system performance and can be an invaluable tool in the early (and sometimes the late) stages of a performance investigation. If you look at either of my two PerfMon links you can see that PerfMon is usually used to pick out what the performance bottleneck is when examining an application. By “performance bottleneck” I mean picking out the resource that is being saturated by your program (CPU, network, disk). PerfMon does this by tracking some subset of performance counters that monitor different aspects of system performace. This is important as you often need to use different techniques and tools for investigating different bottlenecks. For example, using the Visual Studio profiler it’s usually best to use sampling mode to investigate CPU issues and instrumentation mode to investigate memory and disk issues.

With the Visual Studio Profiler we wanted to give customers an easy and integrated way to collect this performance counter information and view it alongside their performance data. This was especially important to us as with this information we could help customers to analyze specific trouble areas of their program or to choose the correct profiling modes based on their performance bottleneck.

Turning on performance counter collection

To access this performance counter collection start out by going to the session properties of your performance session. In the session properties you will want to select the new “Windows Counters” options, shown in the screenshot below.

On this page you can check the box in the top of the screen labeled “Collect Windows Counters” to add windows performance counter collection to your next profiling run. Below that checkbox you see a box to input a value for how often the counters are collected labeled “Collection interval (msecs).” But before we dig into the collection interval it is important to understand just how these counters are collected. With our profiler we have the concept of “marks” that show up in the data collection stream with a comment associated with them. Unlike all other profiling data these marks are not aggregated and show up in chronological order in the “Marks” view of the performance report. These marks are usually added by annotating your code with the data collection APIs (native API here). When we tell the profiler to collect Windows performance counters they get collected every time one of these marks gets hit. The collection interval control allows you to automatically insert marks in your program while running at some specific time interval. That way you can collect performance counter information easily without all the hassle of adding marks to your program manually.

In addition to setting the collection interval you also need to control what performance counters are being collected. In the lower section of the property page are all the different performance counters that you can selected to be collected. We surface all the same counters as PerfMon, so if you have a favorite counter that you like to track you can be sure that we’ll have it. By default we’ve included basic processor, memory and disk usage counters. We picked these as they give a good basic overview as to how your application is taxing the system.

Viewing performance counter data

So to demonstrate how these marks and associated counter values are actually surfaced in the report file we need to run the profiler and collect our data. For this example I’m using the PeopleTrax sample application and running it in sampling mode with the default set of performance counters and the default 500 ms marking interval. For my profiling scenario I’m just going to wait for the application to initialize, hit “get people” button to load the records from the database and then shut down the application. After the application is shut down the performance report will load automatically and then I will jump to the “Marks” view.

In the view above you can see the automatically collected marks. And for each mark you can see the memory, disk and processor usage at that point in time. Now this data can already tell us much about our program. For example we can see that we started out with a brief processor spike between marks three and five closely followed by a memory usage spike between marks five and eight. But how can we actually tie these values into all the rest of the performance data that we collected? After all these are timeline values while the rest of the data in the report is aggregate data from the whole run. Luckily we’ve provided a new filtering feature to help use the performance counter data to guide your performance investigations.

Filtering performance data from the marks view

Looking further into the marks data there was another big memory spike later in my program between marks 129 and 133. But how can we tell what was actually happening in the program during that time? To start the investigation ctrl-click marks 129 and 133 in the marks view so that they are both selected. Then right-click on one of them and select the “Add Filter on Marks” command. When you do this a new filter dialog will appear docked on the top of your performance report.

The filter control has a lot of depth to it, but for now you can just understand that is it currently saying “only show the data collected between mark 129 and mark 133.” To actually apply the filter to the performance report click the “execute filter” button in the toolbar (it’s the button with the green play icon). You will see a progress bar for the reanalysis that is being performed and then the report will pop back up. Only now all the aggregate data for the report is only the data that we captured between those two marks! With just a quick glance at the summary page below you can tell exactly what our application was doing and why we were seeing the memory usage spike at that point (big blue circle added by me for emphasis ;-) ). However it is important to keep in mind that you will need to clear the filter items and rerun the empty filter or to close and reopen the report to get back to seeing all of your performance data.

In addition to filtering on marks the filter grid can so some other very neat stuff such as filtering on threads, by timestamps or by time percentage (to do something like “show only the first 10%” of my run). I don’t have room to cover all the other filtering types here though, so I’ll have to come back and hit them in a later article.

Conclusion

Collecting performance counters is a very cool new feature that we’ve added to the profiler for Orcas. Even before we get into how cool it is to be able to filter down your profiling results based on performance counters it’s nice to just be able to collect and store those values side by side with your profiling data. And when you throw in the filtering features it quickly becomes an invaluable tool for performance analysis.

Pinpoint a performance issue using hotpath in Visual Studio 2008

ianhu — Fri, 14 Sep 2007 20:34:00 GMT

Analyzing a performance report

When you first open a performance report in Visual Studio 90% of the time the question on your mind is “where is my program spending most of its time?” In Visual Studio 2005 the best way to find this was to move to the calltree view and from there to follow the execution path with high inclusive functions down the tree. If you are not familiar with the calltree view, this view places all the execution paths of your program into one big tree structure. In this tree you can see both inclusive and exclusive values for either samples (in sampling mode) or time (in instrumentation mode). In a nutshell, inclusive means “functions that caused work” so any inclusive time or samples for a function can be attributed to that function or any of its sub-functions. On the other hand, exclusive values can be attributed only to that function and do not include any values from sub-functions.

So with the above explanation the way to find out where your program is spending most of its time is to drill down the tree and whenever you have a choice between children of a function pick the child with the highest inclusive value (meaning that it is doing more work below it then the other children). Eventually you will reach some functions with high exclusive values, which are the functions that are actually doing the most work in your program. At that point you can examine those functions and the path taken to them and start working to fix your performance issue.

Hotpath

The problem with the above scenario is that, quite frankly, it’s a pain in the butt to have to always drill down on the tree to find these hot functions and hot paths. Especially with sampling reports the calltree can be many levels deep and it can take a while to get to the functions of interest. So for Visual Studio 2008 we’ve added a small, but very useful feature called “Hotpath” to automate this scenario. Now when you open a performance report and move to the calltree view you will see a small flame icon located in the toolbar. When you push this button it will expand the hot path from the root following the rules that we mentioned above (following the high inclusive numbers when choosing between children). In the example seen below I’ve run a profiling session on a managed application that calculates and displays a Mandelbrot set. When running Hotpath on the report it jumps me down the tree directly to the DrawMandel and SetPixel functions that are performing the bulk of the work for this scenario. The Hotpath will stop whenever it reaches a function with high exclusive values or when it hits two children with similar inclusive values. Also, it can be run from any point in the tree (just right click on the function) so you don’t have to always run it from the root function.

As you can see, the hotpath feature can save you a lot of time in performance analysis. Already folks who use the profiler internally here at Microsoft have become big fans of this feature. In fact, there has been spontaneous applause at meetings when we demoed the feature for the first time! As of right now it’s the closest that we have to a “show me my performance issue button.” And it’s one of the many ways that we are trying to help users to better understand and take action on their performance information in Visual Studio 2008.

The New Developer Menu in Visual Studio Team System

ianhu — Tue, 07 Aug 2007 19:29:00 GMT

The new Developer menu in Visual Studio Team System

In my recent series of blog posts I’ve been covering some of the cool new features that the Visual Studio Profiler team has been adding to the code profiler that is included with Visual Studio Team System for Developers (and Team Suite edition). With all the features that have required hours and hours of work from us we’ve also made a small change that we think will have a big impact on how customers find the profiler. This change was creation of the new top-level “Developer” menu and moving the profiler and code analysis commands to this new menu.

It was scary for us to find out how many customers were unaware that a powerful profiler shipped with VSTS. Some were even purchasing competing profilers without first checking out the one that they already had! So to make the profiler more discoverable we moved it out from where it was buried in the Tools->Performance Tools menu to the new Developer menu. Now the name of this Developer menu might change, but there will assuredly be a top level menu with both your profiling and code analysis tools available in it.