IntelliTrace Info : 2010

IntelliTrace iTrace Files

ianhu — Mon, 16 Nov 2009 21:26:00 GMT

If you are not familiar with the new IntelliTrace feature in Visual Studio Team System 2010 than you might want to first check out either my or John Robbins’ introductions to this feature as a general overview of IntelliTrace would be helpful before digging into this article.

What is an iTrace file?

In my introduction article linked above I talked a little about how IntelliTrace captures the current state of the debugger at multiple points during a program’s execution and, when F5 debugging, allows you to debug back in time to previous debug states in your program. This in and of itself is a very handy feature, but in this day and age it’s often hard to have a bug with an easy and consistent repro that you can debug on a local dev box.

The solution to this lack of a local repro is that not only does IntelliTrace enhance your local debugging experience, but it also saves all the collected debugger data points into a trace log file (.itrace extension) that can then be opened and debugged using Visual Studio later and on a different machine. The analogy for this scenario is that of a black box in an airplane in that iTrace files provide a “voice from the grave” from crashed programs that allow for a developer to debug in and around the point of failure after the fact.

Integration with Microsoft Test and Lab Manager

One of the big new testing features being added in Visual Studio Team System 2010 is the Microsoft Test and Lab Manager (more info on MTLM on their blog site here). MTLM is a standalone tool that focuses on the tester role by providing a TFS-integrated UI for managing test cases and lab environments without the overhead of a full Visual Studio installation. Since one of the key focuses of IntelliTrace is to try to eliminate the “no repro” disconnect between developers and testers we knew that we needed to get IntelliTrace integrated with MTLM. This integration is accomplished via a combination of TFS and iTrace files. I’ll detail the scenario more in a future blog post, but at a basic level at anytime during a test run a tester using MTLM can choose to file a bug on a specific test step failure and when that bug is filed an iTrace file of all the recent debugging events and exceptions is automatically collected and attached to the bug. Then, when the developer opens up the bug in Visual Studio, they can just click the iTrace file linked in the bug and be debugging into the exact execution path in which the tester was seeing the failure.

iTrace files collected during debugging

Whenever you are running a normal F5 debugging session from the Visual Studio IDE with IntelliTrace turned on you are collected an iTrace file in the background. Now in this scenario you can pretty easily be using IntelliTrace features like browsing back in debug history without ever noticing that this file exists, especially since to keep your system from getting clogged with iTrace files we clean these files out when Visual Studio is shut down. So if you ran into something interesting while debugging from the IDE with IntelliTrace you will need to copy the iTrace file out from its saved location to keep it from being cleaned up. Just look under the following file path to see the iTrace files that have been collected during the current VS session: C:\Users\All Users\Microsoft Visual Studio\10.0\TraceDebugging. With both the IDE scenario and the MTLM scenario iTrace files will be truncated at a specific size (currently set to 100MB by default) to keep from filling up your hard drive. This truncation value will discard older events from the log and can be changed from Tools->Options->IntelliTrace->Advanced.

Collecting iTrace files from the command line

If you want to collect iTrace debugging files without having Visual Studio up and running we’ve provided the IntelliTrace.exe command line tool. IntelliTrace.exe will get its own blog entry sometime in the future but if you want to try figuring out how to get it running just try starting with the /? command for help. Intellitrace.exe is located in your Visual Studio install at “Team Tools\TraceDebugger Tools.”

Working with iTrace files in Visual Studio

(Note: All screenshots are from my current working build and will look a little different from Beta 2 builds)

Regardless of if you collected your iTrace file via MTLM, Visual Studio or IntelliTrace.exe when you first open it up in Visual Studio you will end up with a document that looks somewhat like the below.

Note that at this point we’ve just opened up the document summarizing the debugging session. No debugging session has been started and the time to open up the document should be pretty minimal. At the top of the document you will see a chart showing all the threads that were running during the life of this debugging session. Below that there are a series of lists containing more information about Threads, Exceptions, Test Events, System Information and Modules for the debugging session. Currently, the Exceptions list is expanded out and showing all the exceptions that were encountered during the debugging run. The exception that is currently selected in this list is represented in the thread timeline by a vertical red bar. This bar helps you match up exactly where in your program’s execution an exception was being thrown. In addition to supplying the thread, HResult and message of the exception we will list out the stack of each thrown exception in the textbox below the exceptions list.

Threads List:

The threads list provides a table view of the threads active during your debugging session. The actively selected thread will be highlighted in the thread chart above, and vice-versa.

System Info:

The system information section contains a set of information about the computer that this iTrace file was collected on. It seems pretty basic, but this info has already come in useful several times for me during my development work. In particular knowing the OS, the number of processers and the CLR version have been useful to me when investigating bugs that QA has provided me with iTrace files for.

Test Data:

I don’t have a picture of this right now, as I’m going to be speaking more about this when I cover MTLM integration in greater depth. When you collected an iTrace file via MTLM (since this file was not collected via MTLM the section is grayed out) the test data section will contain info on all the test steps that were logged via MTLM during the execution of the tests.

Modules:

As expected, this control lists out the modules that were loaded during debugging.

On all of these controls there is a search box above the list. If you are looking for a specific module or exception just start typing into one of those boxes to narrow down the results being shown in the list.

Starting a debug session from an iTrace file

Up until now we’ve been dealing with the information that you can glean from the summary page of an iTrace file. But while it can be quite informative the real point of the summary page is to allow the user to jump into debugging close to some point of interest. Lots of information can be collected during a debugging session and if you were to just jump into debugging an iTrace file blindly it could take a while to get to the correct location to start diagnosing a failure.

From an iTrace summary page you can jump into debugging from a thread, from an exception or from a test event (with a caveat that I’ll mention later for test events). For threads you can double click on a thread in the thread chart, double click on a thread in the threads list or click the “Start Debugging” button. Any of these options will start up the debugger and jump you to the last event that we collected on that thread. We chose the last event as in many cases an iTrace log captured a failure or crash so starting at the end point makes more sense than starting at the beginning of the log when all was running smoothly. For exceptions, you can either double click the exception in the list, or click the “Start Debugging” button below the exception list. Starting debugging on an exception will start the debug session exactly on the exception event selected (see picture below). Note that starting the session is a slightly slow process and it might take a bit for the debugger to get up and running. Test data function much the same as exception with the only difference being that since test events are not represented in our debugging UI debugging context will actually be set to the event in time nearest the selected test event.

This entry is mainly focused on the iTrace file and summary page so I’ll cover more about the IntelliTrace UI during debugging later. But as for now, you can look at the picture above and see that debugging has been started and our context has been set to the exception that I clicked in the summary page by looking at the source location and the autos window.

Up next

Coming up next I’ll be talking more about the various controls that you can use to move around in IntelliTrace debugging data and what data we will be showing in the Visual Studio UI.

Adios Historical Debugging. Hello IntelliTrace.

ianhu — Tue, 20 Oct 2009 16:14:00 GMT

It’s been a little while since I’ve gotten a post up here due in part to two main reasons. First off, we’ve been pushing really hard as a dev team in getting Beta 2 polished up and out the door. And secondly, the actual name of our feature has been in flux for a little while and I wanted that to be sorted out before I blogged any more.

So as of this past Monday Beta 2 of Visual Studio 2010 and .NET Framework 4.0 have been released out to customers (MSDN subscribers can grab the download from here). And at the same time we finally got clearance to use IntelliTrace as the official name for our new historical debugging feature. So from here on out expect to see much more of me blogging about the ways that IntelliTrace can help improve your debugging process and also more about how we built V1 of IntelliTrace and what we’re going to be doing for V2.

For an introduction on what Intellitrace is all about check out my intro on the topic or a recent blog post from John Robbins.

Diagnostic Events in Historical Debugging

ianhu — Thu, 18 Jun 2009 20:13:00 GMT

Diagnostic Events in Historical Debugging

If this post is the first time that you have heard about the historical debugging feature in Visual Studio Team System 2010, then you might want to first take a look at my little overview of this nifty new feature before diving into the info below (John Robbins also has an excellent blog post here). But if you don’t want the read that article the gist of the historical debugger is that it records your debugging context at specific points during your program’s execution. Then, while debugging (or after finishing debugging if you load up one of the log files that we collect) you can move back to the locations where we collected data to examine things like locals, parameters and return values. You can set the historical debugger to collect data at every function enter and exit but, to avoid the slowdown that this cause,s by default we just collected data at diagnostic events. This article is here to give you some more information on diagnostic events and on how they are used in the historical debugger.

What is a diagnostic event?

Simply put, a diagnostic event is a specific point during a debugging session that is likely to be of interest to the programmer when debugging. Examples of diagnostic events that we define are things like opening a file, writing to the registry or clicking on a WPF button. Diagnostic events were selected with the duel criteria of being placed at interesting locations, but not being at any locations that would be called too often and create excessive slowdown in your application.

How will I see what diagnostic events are being collected?

Since historical debugging when collecting on just diagnostic events is low overhead it will be on by default with all managed projects in VSTS 2010. So if you just start up a basic C# console application, set a break point and hit run you will see the new Debug History tool window pop up docked with the Solution Explorer. By default this window will show you a list of all the diagnostic events that have been collected during your current debugging session. Below I’ve posted a picture of the Debug History window for a little WinForms application that I created to exercise a few different diagnostic events (file open, button click and registry write). From this picture I’ll provide a brief walkthrough of the Diagnostics Events control (Note: I’m working with the current Beta 2 in progress bits, so please understand both that it will look a little different from Beta 1 and that the final RTM product might also look somewhat different). At the point this picture was taken the program was stopped in the live debugger with a breakpoint at the end of the main function.

The main space in this control is occupied by a flat, chronological list of the most recent diagnostic events hit during your debugging session. If you look at the bottom of the list you will see the “Live Event” event, this entry corresponds to the current location of the live debugger and tells us that we are at a breakpoint at line 26 in the file form1.cs. Moving up from there you can see that we recorded a few registry access events, a few file events and a user input gesture. Each entry in this list represents a specific location where we grabbed much of the current debugger state and stuffed it into our log file. Using this flat list you can jump back in time to any of the diagnostic events that the historical debugger captured. Note that there are page forward and page back buttons at the bottom of the control for if we get to large a number of events collected to browse easily.

If you are looking for a specific event or event type in the list we’ve provided a few browsing aides in this tool window. First off, if you start typing in the search box the diagnostic events in the list will be filtered down to just those events containing your search text, very helpful when trying to track down a specific exception that was thrown. And secondly, if you use the “All Categories” and “All Threads” dropdowns you can restrict the list into showing just diagnostic events from specific categories (see the below tools->options section to learn more about our categories) or just showing diagnostic events from specific threads.

What happens to the debugger when I select a diagnostic event?

In the picture below I’ve selected the “File: Close” event. Also I’m showing the rest of the IDE along with the tool window so you can see how the debugging context changed with the switch over to historical mode.

Notice that in the diagnostic events tool window we’ve selected the event that you are currently at, as well giving some inline expanded information about the event. For this file close event we give a little more information about the event “Close a FileStream accessing the path C:\2xj3hs4l.aox,” we mention the thread the event took place on and we give some quick links to other debugger tool windows that might have some more information about the event.

Aside from the tool window being updated there were some other changes in the IDE when we selected an event in the flat list and moved back to historical mode. First off, if you look at the source editor you won’t see the usual yellow arrow indicating the next statement anywhere in the source file. Instead you will see a curved orange arrow located at the end of the “using(FileStream fs =…)” block of code. The curved orange arrow indicates that the current historical debugger context is somewhere in external code below the given statement, roughly corresponding to the green arrow that the live debugger uses when you are stopped in external code. In general when you move between diagnostic events your debugger context will not be located directly in user code, as the diagnostic events are all defined at specific points in Microsoft framework code.

This can be better illustrated by looking at the callstack window in the picture above. When we moved back to the diagnostic event the callstack updated to the time when we captured that event. The actual code context is down in mscorlib.dll at FileStream.Dispose, but since we don’t have code for that location we show the location in user code that triggered that specific diagnostic event (that being when the end of the using statement disposed of the FileStream). When moving back in history we will also populate the values in the autos and the watch windows with some specific limitations. Those limitations being that we will just capture primitive values (strings, doubles, ints ect…) and two levels of primitive values off of any objects.

How can I change what diagnostic events are captured?

We’ve tried to pick a solid set of diagnostic events that provide coverage of some common problem areas and that are not called too often in a program (in which case collection could slow down your application too much). But in specific cases our default settings might not be the right solution for every user, so we’ve added the ability to trim or add to this initial set of diagnostic events via a tools options page.

If you open up Tool->Options you will see a new option for Historical Debugger in the list. Clicking this item will first off take you to the general settings page shown below.

On this page you will notice that by default (for managed projects) historical debugging is turned on and is set to collect debugging data just at diagnostic events. In a later blog post I’ll cover more about the “Event, Methods and Parameters” setting but in a nutshell turning on that setting makes you collect data at diagnostic events and at all function enters and exits. This gives you much more complete historical debugging data to look at, but comes at a much higher cost in terms of application performance while debugging.

For now, just we’ll just leave the general settings as they are and move on to the “Diagnostic Events” tab shown below.

On this page you can see the selection of diagnostic events that we provide broken down by various framework categories. By checking or unchecking events you will either add them to or remove them from the set of events that we capture while debugging. A example could be if your application is doing tons of file opens and file closes you could turn off the collection of file diagnostic events to keep collection from slowing down too much.

Diagnostic Events Summary

With the above post I’ve tried to lay out a little bit of how diagnostic events work, how you can use them to browse back into debugging history and how to configure what diagnostic events are collected. By collecting data at common fault areas / points of debugging interest we’ve think that they will be useful in solving many debugging issues and the low overhead allows us to turn them on by default for all users. However, there is much more to the historical debugger than just diagnostic events collection. For example, one of the more interesting features is that the debugging log can be saved off and opened in the debugger later on a different machine. You can see how diagnostic events would be highly useful in this scenario, as a tester could collected a low overhead log, attach it to the bug and then the developer could debug back in time to specific diagnostic events on their own machine at a later date. In my next blog posting I’ll be digging deeper into this log scenario and examining how it can help to solve the “no repro” issues that plague developer / tester communication.

Historical Debugging in Visual Studio Team System 2010

ianhu — Thu, 14 May 2009 02:56:00 GMT

Historical Debugging in Visual Studio Team System 2010

What is Historical Debugging in a nutshell?

If you’ve been in the development world for any length of time you’ve probably ended up in a situation like one of the following more than a few times.

· You’ve received reports of a crash from a tester, but on your local box you can’t get the bug to reproduce.

· You’ve received a crash dump from the tester along with the bug. But the callstack that actually caused the crash was just a cascading effect and you can’t trace the bug back to the root issue.

· The bug that you are currently working at resolving has an extremely long set of reproduction steps and you just accidentally stepped over the function that is returning bad data.

· You know that some part of your program is hitting the registry way too often, but while stepping through all that you see are .Net framework calls and you are unable to isolate which of them is doing all the extra registry work.

With Visual Studio 2010 Team System Editions we are introducing a new Historical Debugger aimed at getting rid of these developer pain points. The Historical Debugger plays a role similar to that of a black box in a plane. We keep track of important points in your programs execution and allow you to play back what happened at those points at a later time. We’re very proud of the current experience that we offer with the Visual Studio debugger so we’ve worked hard to surface all this new historical data in a way that is both useful and consistent with what you would expect from debugging in Visual Studio.

A little more depth on the Historical Debugger

In the previous section I’ve mentioned some scenarios that the Historical Debugger is meant to help solve but I’ve not mentioned much about how the Historical Debugger actually works aside from the black box analogy. To get an overview about how the Historical Debugger actually works I’ll do a quick little rundown on what the Historical Debugger collects, when it collects it and how you can view this information after collecting it. I’m going to keep to rather general terms for now, but in later blog posts we’ll dig down deeper into each area and talk more about how to configure and use the Historical Debugger from within Visual Studio.

What does the Historical Debugger collect?

Consider what you usually see in Visual Studio when you stop at a breakpoint during your program’s execution. At a basic level, you will probably see a source file with an arrow in the margin indicating where you currently are in program execution. You’ll probably also have access to a window showing your local and watch variables, a window showing your current callstack and some intellisense values for variables in your source code. Also, you might be digging into some more advanced windows like the threads window or the memory window available in the debugger. With a few notable exceptions the basic data that we will be collecting with the Historical Debugger will be a subset of the information normally available to you when doing standard live debugging.

The first thing that might strike you about this is “Isn’t collecting all this information going to slow down debugging my application by some crazy amount?” Trust me when I say that this worry is the proverbial Sword of Damocles that dangles over the head of our team. As big of an issue as this is, I think that we’ve come up with some pretty clever solutions to collect a useful amount of information without perturbing normal debugging patterns (more of these solutions will be mentioned in the “when does the Historical Debugger collect?” section below). A big part of this process was selecting what information is most commonly used during debugging but without picking anything that overly bloats our log files or slows down debugging too much. For starters, every time we stop to collect data we will grab your current code context and the current callstack. Second, we will get the value for any primitive data types that would appear in the locals or watch window at that point, we also get the primitive values for up to one level deep on any objects that would be in the locals or watch windows .Third, to the immense relief of those doing multi-threaded debugging, we will collect the data on the currently active threads.

When does the Historical Debugger collect data?

Above, we’ve talked a little about the debugger information that the Historical Debugger collects. But now we need to address how often this information is collected. After all, even if we are collecting a tiny amount of information collecting it too often will quickly lead to your program being slowed down to a halt and generating massive log files. To make Historic Debugging useful we need to provide the user with some solid default settings for how often they collect data and allow for some tweaking to adjust these default values.

When designing the Historical Debugger we based it around two main default levels of data collection. The first level of collection is based around the concept of collecting debugger data at specific points of interest in your program called diagnostic events. Diagnostic events are locations that we have selected as being common points of interest for customers when debugging a managed application. These diagnostic events are selected by Visual Studio and are meant to cover a broad range of programming types. An example diagnostic event that is included with the Historical Debugger would be RegistryKey.SetValue. If this diagnostic event is enabled you well collect a full set of Historical Debugging data every time that RegistryKey.SetValue gets called. These diagnostic events will act like checkpoints when you go back to examine your historical data. We of course allow for you to tweak which sets of diagnostic events will be enabled anytime you are debugging.

We think that we’ve selected diagnostic events that will be useful across a broad range of scenarios, but there will be plenty of times where the area you are interested in debugging after the fact will not have any diagnostic events of interest in it. For this scenario we’ve added an option to also collect debugging data at all method entry points in your program. Also, in this mode we will collect additional data on the parameters that were passed into each method. As would be expected this mode will increase the overhead that the Historical Debugger creates, so be aware of the effect it will have on your applications performance when debugging. Overhead aside, we believe that the data collected in this mode can be very useful especially in that it give a better idea of the shape of your program as opposed to just diagnostic events.

How do we show the Historical Debugger information?

For the most part the Historical Debugger information will be shown in the normal debugger windows such as the watch window, the locals window and the thread window that you already know and love. Although as mentioned above we do only collect a subset of total debugger information, so don’t expect to see everything when debugging historically. When you are in the middle of a normal debug session you will be able to just step backward to the most recent diagnostic event (or method enter / callsite if you are collecting them) from there you will be able to move about between the various points that we collected data during your current debugging session. As we dig into the new historical debugging UI in later blog posts I’ll talk more about the features that we’ve integrated into VS to help make this navigation easier.

In addition to being able to move back in time from a normal debugging session you can also save off historical debugging data in a separate .tdlog file and open this later or on another computer. This tdlog file is a key component of what we call the “no repro” scenario, the scenario in which testers pass off a bug to developers but developers are unable to recreate and debugger the error condition locally. We’ve provided integration with Camano (our new standalone test case management tool) and TFS to make it super easy for testers to attach tdlog files to any bugs that they file. Now when developers open up a bug they will be able to also open up the attached tdlog file and debug to any point where historical data was collected to help track down the issue.

What’s next?

This goal of this little intro was to tell you the very basics about the how the Historical Debugger works. In the upcoming weeks I’m going to start rolling out more articles that provide in-depth detail about how the Historical Debugger works and about how you can use it from Visual Studio 2010. Expect more pictures of how the UI will actually look and function as well as more samples about how Historical Debugging can help to solve common (and uncommon) programming issues.

Back To Blogging

ianhu — Thu, 14 May 2009 02:49:00 GMT

Back to Blogging

So for the last 18 months or so there has been nary a peep on this blog about anything. One contributing factor to this extended period of silence would happen to be the fact that I now have an 18 month old daughter and I was feeling a bit of a time crunch. But the actual big reason is that I moved teams within Visual Studio and was working on a project that was flying under the radar for about a year. This new product is the Historical Debugger that is going to be shipping with Visual Studio 2010 (I’ll have an introduction post up for this feature later today) and it’s much more out in the open now so I’m ready to start blogging up some hype for it. I’m still a part of the Visual Studio Diagnostics team and just down the hallway from my profiler ex-coworkers so I’m still around for answering / forwarding profiler questions. As before there will be a few odd-ball posting here and there, but in general just expect lots of information, samples and chat about the Historical Debugging feature.