Shawn Burke's Blog : Source

How to disable optimizations when debugging Reference Source

sburke — Tue, 29 Jan 2008 22:35:44 GMT

When you debug code in the .NET Framework using the newly available Reference Source functionality in VS 2008, you may notice that many variables are not available for inspection.

This is because you're debugging against retail-optimized code. In many cases, since you can still step through, this is something that's manageable.

But what if you really need to get a better idea of what's going on? Fortunately, there is a way.

What you need to do is to tell the CLR not to load the pre-JIT (aka NGEN) images. Here is how to do it.

First, create a CMD file that sets an environment variable, then launches Visual Studio. I called mine "NoOptDevEnv.cmd", and it's contents are as follows:

set COMPLUS_ZapDisable=1
cd /d "%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\ide\"
start devenv.exe
exit

I put this cmd file on my desktop. When I want to disable optimizations, and only when I want to do this, I launch VS from this command.

Once in my Visual Studio project, do the following steps:

1) Right click on your project file and choose "Properties"

2) Choose the "Debug" tab and uncheck "Enable the Visual Studio Hosting Process"

3) Launch your application in the debugger.

Now, you'll see full local and member variable information:

Finally, you may be asking yourself "What is the Visual Studio Hosting Process (aka VSHost), and what happens if I turn it off?" And this is a good question.

For the most part, disabling VSHost won't have any major impacts, but it will disable two features that you may be using.

First, you will not be able to do "Debug In Zone", which allows you to debug a process in the context of a security zone such as "Internet" or "Intranet". That won't work without VSHost.

Second, Design Time Expression Evaluation for class libraries will also not work. What that means, for example, is that if you are developing a Class Library, you won't be able to execute code from it in the Immediate Window while under the debugger.

In general, I recommend re-enabling VSHost (undo step 2 above) when you are finished with your debugging.

Hope that helps!

Configuring Visual Studio to Debug .NET Framework Source Code

sburke — Wed, 16 Jan 2008 17:02:00 GMT

It's finally here - the launch of the .NET Reference Source project. This post (hopefully!) contains everything you need to know. Over the past few weeks, we ran a pilot of this feature and collected lots of great data that helped us work through some issues and understand where people were likely to have problems.

First, though, if you have any problems, please make sure you've followed all of the steps exactly as described. If you're still having problems, please check the FAQ/Troubleshooting section at the bottom. If that doesn't work, post a comment below and I'll look into it.

BASIC SETUP

Note this functionality is not available on the Express versions of the Visual Studio 2008 products.

1) Install the Visual Studio 2008 QFE. This Hotfix just updates a DLL that's part of the Visual Studio debugger that fetches the source files, more details on the download page.

UPDATE: If you get an error installing the Hotfix , try inserting your VS 2008 DVD and then running the Hotfix EXE again. We're looking into the root cause - it's related to having a prior version of VS 2008 (e.g. Beta 2) installed on the machine. But this workaround should allow the Hotfix to install properly.

UPDATE (1/18): There were some problems with the QFE link above that have been addressed, sorry for the inconvenience, it's fixed now.

2) Start Visual Studio 2008 and bring up Tools > Options > Debugging > General. If you are running under the Visual Basic Profile, you will need to check the box on the lower left of the Options Dialog marked "Show All Settings" before continuing (other profiles won't have this option).

Set the following two settings:

Turn OFF the "Enable Just My Code" setting
Turn ON the "Enable Source Server Support" setting

Your settings should be as below:

3) Next, bring up the "Symbols" Page and set the symbols download URL and a cache location. Specifically, set the three settings below:

Set the symbol file location to be: http://referencesource.microsoft.com/symbols
Set a cache location. Make sure this is a location that your account has read/write access to. A good option for this is to place this path somewhere under your user hive (e.g. c:\users\sburke\symbols)
Enable the "Search the above locations only when symbols are loaded manually" option.

When you're finished, the settings should look like the image below:

Setup is done! That's it, really!

DEBUGGING INTO FRAMEWORK SOURCE

For this simple example, we'll start with a blank C# Windows Application project, but it will work the same with a VB, Web, or WPF project. To walk through this, go ahead and create that project.

Set a breakpoint in Form_Load:

Now run your project to hit that breakpoint and go to your Call Stack window (CTRL+ALT+C). In the Call Stack, right click a frame that starts with System.Windows.Forms.dll, and choose "Load Symbols". This will load the symbols for the System.Windows.Forms assembly, which are about 10 megabytes, so the speed of the download will vary according to your connection speed. Note that Visual Studio may be unresponsive during this time. However, this download is a one-time cost for each assembly. The symbols (PDB) file will be cached on your machine, in the directory specified in the steps above.

This will load the symbols for the DLL from the server, and you'll see some information in the status bar to reflect this. Note that when this completes, the call frames will turn black and line numbers will be available. Note you'll need to do the Right Click -> Load Symbols step each time you launch a debugging session (but, again, the symbols will now be cached locally so they won't have to download). For more information on this, see the ADVANCED USERS section below.

You have now loaded the symbols for the Windows Forms DLL and can begin viewing the code. You can view code in any way that you normally would in a debugging session. In this case you can either Step In to the line of code above, or you can double-click one of the frames in the Call Stack Window. For this case, we'll step in (F11).

The first time you step into code, we'll be presented with the EULA for accessing the source code. Please take the time to read this EULA. If you agree to the terms of the EULA, hit ACCEPT, and the source will then be downloaded.

That's it! You're now debugging .NET Framework Source!

Now, for each assembly that you'd like to debug into just repeat the steps above (note you'll only see the EULA once, not for each file).

There are times when the Assembly you'd like to debug into isn't on the call stack, for example in the code below:

Before you step in to Graphics.DrawRectangle, you need to get the symbols for System.Drawing.Dll loaded. To do this, we use the Modules Window (CTRL+ALT+U). This lists all of the modules (DLLs) loaded by the debuggee. Just find System.Drawing.DLL in this list, right click, and choose Load Symbols.

Note that once a symbol file is loaded, the path to the symbol file shows up in the "Symbol File" column.

You can now step into the Graphics.DrawRectangle code above using F11! In this case, note, you'll have to step through the PaintEventArgs.Graphics property code first.

ADVANCED USERS

Normally, each time you launch a debugging session, Visual Studio attempt to download symbols for each DLL that loads into the debuggee process. As part of this process, it asks each path specified in the Debugging Symbols dialog for the corresponding PDB. Some projects load a LOT of DLLs which won't have symbols available, so this process can significantly impact debugger startup time as this probing occurs. It is mainly for this reason we've recommended manual symbol loading in the steps above; we don't want using this feature to degrade the debugging experience across-the-board.

There is, however, a way to allow automatic symbol loading (which avoids the "Load Symbols" step) in a way that minimizes performance impact. This is for more advanced users because it requires regular trips back to this Debugging Symbols Dialog. Note you can quickly get to this dialog by choosing the "Symbol Settings..." item on the right click menus pictured in steps above form the Call Stack or Modules windows.

The key is to get all of the symbols for a given project type downloaded and cached locally, then turn off the automatic symbol downloads. This will prevent the pings at debugger startup as well.

To do this, configure your setup as above with the following difference: Uncheck the "Search from the above locations..." item on the dialog.

Now, launch your project in the debugger. This will cause all of the symbols available for the DLLs in your process to be downloaded at on-demand as the DLLs are loaded into the process. Depending on your connection speed, this could take a while (it's generally about 50MB of symbols), so it's a good idea to hit F5 then go do something else for a while. Again, these symbols are cached so this is a one-time cost. Visual Studio will likely be unresponsive during this download process.

Once that process has completed, stop the debugger, and UNCHECK the the Reference Source Server symbol location, and hit OK:

Now when you launch the debugger, symbols will load automatically and you'll be be able to step in and through call stacks normally. Note if you switch to a different project type (that has different assemblies), or have assemblies that are loaded later by your project, you can just repeat these steps to get any assemblies you don't have cached locally).

FAQ/TROUBLESHOOTING

1) Which assemblies are currently available for symbol/source loading:

Mscorlib.DLL
System.DLL
System.Data.DLL
System.Drawing.DLL
System.Web.DLL
System.Web.Extensions.DLL
System.Windows.Forms.DLL
System.XML.DLL
WPF (UIAutomation*.dll, System.Windows.DLL, System.Printing.DLL, System.Speech.DLL, WindowsBase.DLL, WindowsFormsIntegration.DLL, Presentation*.dll, some others)
Microsoft.VisualBasic.DLL

2) When I choose "Load Symbols" I get an Open File dialog asking for the PDB.

This can be caused by one of four situations:

You didn't configure the symbol location URL properly from basic setup above. Ensure it's http://referencesource.microsoft.com/symbols.
You chose a cache symbols directory that your user account doesn't have write permissions for.
You attempted to Load Symbols for a DLL that's not available in the list above
You have a different version of the .NET Framework on your machine - this can happen, for example, if you're running a Windows Server 2008 Beta. To check this, go to the Modules Window in Visual Studio (CTRL+ALT+U) and ensure that the version number of your mscorlib.dll is 2.0.50727.1433, as below. If not, make sure you install the RTM Version of the .NET Framework 3.5.

Check your "C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE" (or wherever you installed VS) for a file called symsrv.no. If that file exists, rename it to symsrv.yes, and restart Visual Studio.

3) When I try to open or step into a source code file, I get a dialog that says "Source is not available for this location" or I get an Open File dialog for the file.

First, see FAQ item (2) above to ensure the symbols for the DLL were loaded successfully. You can verify this by looking in the Modules Window under the "Symbols Status" column.

If the Symbols Status is "Symbols loaded.", check the following.

If you have configured Microsoft Symbol Server in the past, you may have already downloaded symbols for this DLL that don't contain source information. Try specifying a different cache path or deleting your existing cache path, and then repeating the Load Symbols command. See FAQ #4 for more information on this.
Ensure you have checked the "Enable Source Server" item on the Tools -> Options -> Debugging -> General page
Ensure that your account has read/write access to your cache path
If you have _NT_SYMBOL_PATH set, it will override these settings. Details here.

4) I also use Microsoft Symbol Server to download symbols. What's the difference? Can these two programs co-exist?

Microsoft Symbol Server provides symbols without any source information in them. That information has been removed (sometimes referred to as "stripped") before publishing. The symbols provided on the Reference Source Server are full symbols with debugging information.

The key to using both is to have your Reference Source path above the Symbol Server Path so that those symbols are searched/found first. As described in the ADVANCED USERS section above, you'll likely want to launch your debugger once with this configuration to get all the symbols downloaded, then uncheck both of these paths to avoid debugging launch slowdowns in the future. Also note that this may conflict in the future as more DLLs are added to the Reference Source Project. Meaning, if you've already downloaded the Symbol Server symbol, you'll need to delete that or change your cache path to get the Reference Source one (Visual Studio has no way of knowing which is which).

One final note here, if you have the Microsoft Symbol Server configured via _NT_SYMBOL_PATH, you'll need to add the Reference Source path above to that path as well - _NT_SYMBOL_PATH overrides the above settings.

5) Does this work with 64-bit?

Yes, we've also provided 64-bit versions of the PDBs. Note some DLLs work on multiple architectures, so not all of them need a separate 64-bit PDB.

6) How do I set breakpoints in Framework code?

Visual Studio requires the code to exactly match what is expected by the PDB. The source publishing process, however, makes some small updates to the code, such as pushing a standard copyright banner to the end of the source file. This changes the signature (CRC) of the code file. However, it's still easy to set a breakpoint.

Just set your breakpoint as normal (you'll see it fails with a little warning icon), then right click the breakpoint and choose "Location..."

Then check the "Allow Source to be Different" box, hit OK.

Now the breakpoint will set successfully.

If you find yourself doing this often, you can also disable source matching in general by disabling Tools->Options->Debugging: “Require source files to exactly match the original version.”

7) Why don't some features like "Go To Definition" work?

Browse database information is separate from symbol (PDB) information within the debugger, so that information is maintained by the project system when a project is compiled and is not present within the symbol files. So, unfortunately, that ability is not available here.

8) Why are some member or local variables unavailable? Why can't I step into certain functions or lines of code?

The .NET Framework bits that you have installed on your machine are “retail” bits, which means they are optimized for size and performance. Part of this optimization process removes certain information from the process when it is no longer needed. Debugging retail assemblies reflects this. However, most debugging information is still present in the session, and setting breakpoints earlier in a function often allows it to be visible. Another aspect of retail builds is that some small methods may be “inlined” which means you will not be able to step into them or set breakpoints in them. But for the most part you can step and debug as normal.

9) Why does it take so long to download some source files?

Some source files are very large - nearly 1MB - and unfortunately many of these are the common ones to download. However, most are significantly smaller and download quickly. Source files must be downloaded each time you restart Visual Studio. They are not persistently cached like symbols are.

10) Can I just download all of the code at once?

Not currently, but we are currently working on enabling this functionality in the future.

11) When I debug into VB code, for example in Microsoft.VisualBasic.dll, there is a C# style comment at the bottom, is this a bug?

Not really - we do run a post processing step when publishing the code and it adds a standard copyright banner at the bottom. The processor, at this time, isn't able to support multiple comment formats. Having it in a non-VB format doesn't affect the debugging functionality in any way.

12) I got to a source file and all that downloaded was a blank file?

This is something we've seen intermittently but have not been able to diagnose. If you see this, usually the workaround is to just restart VS, which will force the file to reload. If you observe this behavior, please use the "Email" link on the left to email me the name of the file that failed and about what time the failure occurred.

13) What happens if I download a Hotfix or a Service Pack? Will I be able to get source for that?

We've built a system that allows us to publish any number of versions of source and symbols for a given product. We haven't made any firm decisions on how often we'll publish source and are open to customer feedback on those issues. For example, it's clear that publishing source for each Service Pack makes sense, but it's unclear if we'll be able to do so for each Hotfix. Again, we're looking forward to feedback here.

In the meantime, also note that symbol files may no longer match a module if a framework DLL has been updated via a Hotfix. In those cases, the modules window will indicate that the symbols could not be downloaded for that module. Assuming a new symbol file was published, it can be downloaded and cached using “Load Symbols”.

14) Can I point a web browser at the symbols URL and download the symbols directly?

No, you'll get an HTTP 400 (Bad Request) response.

SUPPORT

If you have any other questions, please visit our new MSDN Forum on the topic: Reference Source Server Discussion.

Thanks!

Making .NET Framework Source Available to Developers

sburke — Wed, 03 Oct 2007 20:30:03 GMT

For any of you that have been following my blog, you may remember that I made a splash with a post about possible ways to release Windows Forms source code. This generated a lot of discussion and was picked up by many of the major tech websites out there. That was in February of 2005.

Between now and then, I've taken a new role in the company and had other things (like shipping VS 2005 and getting the AJAX Control Toolkit off the ground) keeping me busy, but earlier this year I started to get much more focused on this idea and pulling together a plan to make it happen.

Today is an exciting day for us here Microsoft and our developers, see ScottGu's Blog for the big announcement: We will be releasing .NET Framework Source Code as part of the VS 2008 (Orcas) release.

Also check out the Podcast on Scott Hanselman's "Hanselminutes" that I did as well - lots more detail in there.

That's right. Getting code ready for release is a big job, and most teams are still focused on getting Orcas itself released, but the current launch lineup is (in no particular order):

Base Class Libraries (mscorlib.dll)
ASP.NET (System.Web.dll)
Windows Forms (System.Drawing.DLL & System. Windows.Forms.dll)
ADO.NET (System.Data.DLL)
XML (System.Xml.DLL)
WPF (System.Windows.DLL)

There may be others as well, but that's what we've currently got a lock on for the initial rollout. The system is setup so that it's easy for us to publish code, so more will get added as time goes on (working on getting LINQ, WCF, and Workflow in there as well).

Scott's Blog has details and screen-shots, but here's some more detail on how it works.

My main goal had been to enable a scenario that we're all familiar with...

You're up late one night writing code. Maybe it's a technology you're just learning or something you haven't tried before. You're getting an exception from deep in the code. You can't figure out why...the blogs aren't helping, the newsgroups aren't helping, the docs aren't helping. The next day you call product support. Since they don't have your code in front of them, it's also hard for them to triangulate what's going on.

Now, imagine you've got source and symbols available. With that, you can enable "Break on Exception" in your debugger, run your scenario, and have the debugger stop exactly where the exception was thrown. The code gets loaded up and you sniff around a bit and realize - duh! - you forgot to set some other piece of state in an earlier call or a parameter. You fix your code and you're off and running.

Even with one of the decompilers (e.g. Reflector) that you can easily get out there, this can be tricky to solve. With the source it's much, much easier.

One of the big challenges here was building as system that didn't require each team to build their own process, installer, etc. And we also had to account for updated releases (Service Packs, etc) so that this didn't become a big tax over time. On the customer side, we also didn't want developers to have to wade through a bunch of choices to find the "right" source & symbols for their install, then have to manually install it and set it up, etc.

It had to just work.

Fortunately, the technology needed to make that happen already existed within the company. We already have a unified place where all the symbols get pushed, and a unified way to get the source that matches those symbols. What's more is that on the server-side, it's just a plain HTTP file server based on IIS.

So a few months ago I started pulling together resources from across the company to help get this to happen. People really rallied around this and I got great support from teams across the company - on the VS side willing to do the work to make the VS Debugger understand how to get these files, on the Windows Debugger SDK side the technology to actually retrieve the symbols and source from an HTTP location, on the MSDN/Code Center side helping build out the server-side that will make all of this available when we launch later this year, and on the management side getting behind this effort and supporting it. It's been great and those teams really are the ones that made this possible.

The experience really is fantastic. There's a screenshot on Scott's blog for setting the options in the Tools/Options/Debugger dialog in Visual Studio 2008. That's pretty much all you need to do. Please note the URL in there is still TBD - that's an example.

All VS needs to know his where the symbols are and the rest happens automatically. When you launch your process, VS can go look at a variety of locations for the symbols to match your binaries - it knows how to tell. So one of the places it can look will be on the MSDN server for this stuff (note this is separate from the current symbol server that's out there - these symbols will have the full information in them like line numbers and variable names). Once it finds the symbol file, it downloads it to the local cache. Now, the symbol file has the pointers to the source files in it. So when you hit a call frame - the debugger checks the symbols file and gets the path to the exact right source file and retrieves that. Moreover, as we release source and symbols for new parts of the product (or new products - this system works for anything) or product updates, they'll just be automatically available for use in a debugging session. No configuration or extra steps needed. Very nice.

It's just that easy - you can step through, set breakpoints, inspect variables, etc. It's goodness, looking forward to making this available later this year!