Notes from a dark corner

Some ASP.NET, IIS and CLR fixes

My job involves a wide variety of stuff, from investigation, debugging, diagnosing and solving of specific customer problems through passing on product feedback and bug reports to product groups and mentoring other support engineers on their cases to looking at how we can make our support processes and products better.

Many of the customer problems I help with can be resolved by applying an existing fix or service pack or by making some configuration change or code fix to the customer’s application.

However every year there are some cases that need to be solved by getting a new hotfix patch produced for one of our products. These cases are always some of the most interesting problems I work on and there are sometimes challenges in getting such fixes prepared tested and delivered. It’s very important we get these right as those code changes get checked into a common servicing code base and are then included in future hotfix packages and service packs. Once finalised, each problem and fix gets documented in a KB article.

Here are some of the ones I’ve been directly involved in over the last 12 months.

 

ASP.NET

FIX: Error message when you view a page in design mode after you recompile a .NET Framework 3.5 Service Pack 1-based project: "Error creating control – 'ControlInstanceName [text]' could not be set on property 'PropertyName'"

FIX: The pop-out menus are not displayed when you use Internet Explorer 8.0 in Standards mode to view an ASP.NET Web page that has dynamic menus

FIX: Error message when you compile an ASP.NET project that contains many application resource files: "A first chance exception of type 'System.ArgumentException' occurred in mscorlib.dll"

FIX: Code blocks embedded in an ASP.NET 2.0 Web page disappear when you use the Generate Local Resource tool in Visual Web Developer to generate a local resource file

KB967253 - Changes in the hyperlink field from the designer doesn’t get affected

 

CLR

FIX: On a computer that has the .NET Framework 3.5 Service Pack 1 installed, an exception is thrown when an application calls the GetOptionalCustomModifiers method or the GetRequiredCustomModifiers method on a ParameterInfo object of a generic interface

FIX: On a computer that is running the .NET Framework 3.5 Service Pack 1, the JIT compiler and the Native Image Generator (Ngen.exe) that use the Mscorjit.dll file may generate incorrect code

FIX: An application that serializes and deserializes a generic class that has at least one static member fails on a computer that is running the .NET Framework 3.5 Service Pack 1

FIX: A TypeLoadException exception occurs when you run a .NET Framework 3.5 SP1-based application

 

IIS / HTTP.SYS

FIX: You receive a Stop 0x0000007e error message on a blue screen when the AppPoolCredentials attribute is set to true and you use a domain account as the application pool identity in IIS 7.0

FIX: Web sites restart and you receive Event ID 1000 with faulting application svchost.exe_WAS in Internet Information Services (IIS) 7.0

FIX: The applicationHost.config file may be temporarily inconsistent and you receive the "Configuration file is not well-formed XML" error message in a server that hosts IIS 7.0

 

Once a fix has been finalised it then becomes available for any customer to obtain either by calling Customer Service and Support, searching for ‘hotfix’ on the MSDN code gallery or in more and more cases directly downloading the fix by clicking on the “Hotfix Download Available” link within the KB article (not all KB articles provide this option)

Hint: on the MSDN Code Gallery page, if you click on “Sort by” and then select “Current Release” you should see the most recently uploaded fixes listed first.

HTH

Doug

On memory leaks…

If you have a rusty old bucket and it is overflowing because you put too much water in it, then you should certainly think about buying a nice new plastic one at some point. But the new one will still overflow if you put too much water in it.

“THIS DUMP FILE IS PARTIALLY CORRUPT”.

I was investigating an issue today and needed to create a kernel dump on demand in my repro machine, a copy of Windows Server 2003 SP2 hosted in Hyper-V. I successfully blue-screened it (!):

and left it to get on with generating the dump and rebooting. Because I wasn’t watching it I never saw it actually get to 100%.

However when I opened the dump I got a whole load of messages I didn’t like the look of:

and so it went on, and on, and on, page after page of scary looking error messages.

So I tried again a few times but always the same result.

Then I got to thinking about why the dump might not be getting fully written and I thought “pagefile”.

The pagefile settings in the pre-configured virtual image I was using looked something like this:

And sure enough if you try to set such settings you get a warning like this:

 

So I set it back to “system managed size”:

 

and rebooted then crashed the system again and this time the dump was fine with no corruption reports.

Now I had always thought (and thought it had been my experience) that if your pagefile was too small then you simply didn’t get the dump file created when the crash occurred.  I had not realised that you might end up with a “partially corrupt” dump file.

HTH

Doug

Help! Where’s my help?

Once you’ve installed Visual Studio 2010 Beta 1 and read the readme (you do read readmes, now, don’t you?) you find that all the help for this beta release is online. No offline documentation is included. That’s understandable and all very well but what if you want to browse it while you are not online, on a plane or whatever? Also, it makes targeted searching a little difficult as well, for example if you just want to browse and search within the Visual Web Developer and ASP.NET documentation.

Fortunately the MSDN Publishing System Content Service and a cunning utility/sample called PackageThis come to your rescue. This tool talks directly to the MSDNPSCS and finds out what is available. It then lets you select those parts of it that you are interested in and will then package them into a Compiled Help Module (CHM) file:

Note that if you want to select all topics within a node you have to actually right click and select “Select This Node and All Children”. Simply checking the check box next to the node only selects that node.

After selecting the content you want, select Export to CHM from the File menu:

Wait a few minutes and there you have it! Your own fully searchable, fully functioning offline help file:

 

As one reviewer on the CodePlex site comments, it is not perfect but it is perfectly functional in my opinion and very handy.

 

HTH

Doug

Debugging and Profiling API enhancements in CLR 4.0

There’s a very interesting interview been published on Channel 9 about the new and changed profiling and debugging APIs in CLR 4.0.

While I’m mentioning CLR 4.0, here are a bunch of links related to Visual Studio 2010 and .NET Framework 4.0:

Downloads

Microsoft .NET Framework 4 Beta 1
Microsoft .NET Framework 4 Client Profile Beta 1
Visual Studio 2010 Professional Beta 1 (web installer)
Visual Studio 2010 Professional Beta 1 (ISO)
Visual Studio Team System 2010 Team Suite Beta 1 – Web Installer
Visual Studio Team System 2010 Team Foundation Server Beta 1
Visual Studio 2010 and .NET Framework 4 Training Kit
WCF WF Samples for .NET Framework 4.0 Beta 1

Blogging/sites/links

Main MSDN site
The Readme
MSDN article on new features
Beta 1 forum
Data sheet
ASP.NET 4.0 and Visual Studio 2010 Web Development Beta 1 Overview
GS switch enhancements in VS2010
10-4 Episode about installation of beta 1

 

HTH

Doug

Must know info on PDB files…

I don’t think I’ve ever come across anything written by John Robbins that is not worth reading. But in a recent post he talks about a subject that rings particularly true for me – the importance of debug symbols. Symbols are like the Rosetta stone of debugging – they are the key to interpreting debugger output and relating it back to the source code of the application or component. Without it, much of the debugger output is meaningless. Not meaningfree. Just meaningless. It is not impossible to debug without symbols. It is just much harder. Maybe 10 times harder, maybe 100 times harder. And harder means more time and more time means more money.

I have lost count of the number of situations where I have assisted customers with application in production that are dying on their feet due to some crash or memory leak and the conversation goes something like this:

Me: You application is failing/leaking memory from modulename!RandomSoundingFunctionName+0xVERYBIGHEXOFFSET

Customer: My development team doesn’t recognise that function

Me: That’s because it is simply the nearest exported function name that the debugger has found, plus an offset

Customer: Can we get the actual function name?

Me: Unfortunately not without matching symbols

Customer: My development team have just rebuilt the component with symbols and I’ve uploaded the PDB to you

Me: I can’t use that because it doesn’t match the binary that was on the server when the dump file was generated

Customer: My development team says they haven’t changed the source code much since the original deployment

Me: Unfortunately it won’t work. Even with no code changes a PDB can change from one compilation to the next, depending on the compiler. They will need to recompile and then you will have to redeploy, reproduce the issue, gather more dump files and upload them

Customer: That will take a week of dev, a week of test and a week of change control approval

Me: Ok, well, let’s see what we can do in the meantime

 

So my recommendation is – read John’s post and heed his wisdom.

HTH

Doug

MIX09 sessions available online

http://videos.visitmix.com/MIX09/All

Doug

UK IE8 ‘Compatibility Labs’ events

Do you live in the UK near Reading?

Do you manage, maintain or develop a web site and are you working to ensure it is fully compatible with Internet Explorer 8?

There are a few places available on a couple of FREE hands-on lab events to give assistance with Internet Explorer 8 compatibility. The events are on 22nd and 23rd April.  These events are at the main Microsoft UK campus, in Reading. If you are interested in attending, please send email to IE8UKLABS@live.co.uk with your…

Full name

Job Title

Company name

Date you wish to attend (22^nd or 23^rd)

Learning objectives – briefly state what you would like to get from attending the lab

 

If the events prove popular (which they already appear to be) then there may be some additional ones scheduled.

HTH

Doug

Getting started with building your Web 2.0 business

A new started kit “for planning, architecting, and implementing Web 2.0 applications and services using the Microsoft Platform” has been released:

Kobe: Web 2.0 Service Development Resource Kit

There is also a new “Startup Zone” which is “The guide to working with Microsoft for software startups and their investors.”

HTH

Doug

“Web server here” shell integration

This is a very useful and neat little trick:

http://haacked.com/archive/2008/06/24/vs2008-web-server-here-shell-extension.aspx

Doug

Stop 0x0000007e in HTTP.SYS when using kernel mode authentication and you set appPoolCredentials=”true”

We’ve now published the KB article for a recent hotfix that I was involved in.

I’m not going to write much about it here as a lot of my words have already gone into the KB article:

FIX: You receive a Stop 0x0000007e error message on a blue screen when the AppPoolCredentials attribute is set to true and you use a domain account as the application pool identity in IIS 7.0

The main points are however:

• This problem does not occur “out of the box” – a number of factors have to come into play before you hit this issue.
• Kernel mode authentication (new in Windows Server 2008) has to be enabled.
• appPoolCredentials has to be set to ”true”. A reason for doing this is if your application pool is set to run under a domain account and you are configuring your site to use Kerberos authentication
• we’ve only seen this issue so far in relation to Microsoft Office SharePoint Server 2007, but it is not a problem with that product. it is just that the way that product sometimes requires IIS to be configured leads it to fall victim to the bug in HTTP.SYS
  

HTH

Doug

Some recent ASP.NET fixes (March 2009)

[Updated 8th April 2009. I don’t generally like to change posts after they have been published but I wrote this post while some of the KB articles were still being authored. For a number of reasons there have been some changes to which articles document what issue, so I’m updating the below table to reflect the current situation as accurately as possible]

[Updated 27th April 2009. Correcting some build numbers and adding rollup KB969612]

There have been a few ASP.NET issues lately that have had to be resolved by the production of new hotfixes.

These fixes are recent and are not included in either the Microsoft .NET Framework 3.5 SP1 nor the subsequent .NET Framework Family Update.

When we release a number of fixes at around the same time, it can get a little bit confusing as to which fix package you actually need to get hold of and install in order to resolve the problem you are having. In due course this should be clearly documented in the KB article.

The below tables will hopefully help clarify things a bit. The first column has the link to the KB article that documents the issue. The description column is the high level description of the problem. You then have to decide which operating system you need the fix for. The fix packaging technology is different for different families of operating system. Look in the column appropriate to what you need and look down to the correct row. Either click on the link you find there or, if you see a ‘V’ continue looking down until you find a link or information about how to get the fix.

An important point is the build number. Microsoft hotfixes are always cumulative. So if a fix is included with a particular build of a particular set of files then any later build (higher build number) of the same files will include all previous fixes that affect those files.

Note: at the time of writing this blog post, not all the KB articles describing the problems have been published. So the KB article link may not yet work when you read this. The fixes are available however.

The fix packages in the first table update this set of files:

aspnet_wp.exe
system.web.dll
webengine.dll

I think of these as the “core” ASP.NET files.

The file version number is 2.0.50727.xxxx where xxxx is the build number.

KB describing the problem	Description	Fix package for Windows 2000/XP/2003	Fix package for Windows Vista SP1 / Windows Server 2008	Build number
KB961847	FIX: Error message when you view a page in design mode after you recompile a .NET Framework 3.5 Service Pack 1-based project: "Error creating control – 'ControlInstanceName [text]' could not be set on property 'PropertyName'"	V	V	4002
KB961864	FIX: Web clients may have several problems when you access an ASP.NET 2.0 AJAX-enabled Web site	V	V	4002
KB961884	FIX: You experience very slow performance on an initial request to an ASP.NET Web site after you change the App_Code folder, the bin folder, or the Global.asax file	V	V	4005
KB962351	FIX: ASP.NET Dynamic menus do not render correctly in IE8 in non-compatibility view	V	V	4007
KB961902	FIX: Error message when you compile an ASP.NET project that contains many application resource files: "A first chance exception of type 'System.ArgumentException' occurred in mscorlib.dll"	V	V	4011
KB967535	FIX: A hotfix is available to fix two problems in ASP.NET on IIS 7.0 for Windows Vista and Windows Server 2008	n/a	KB967535	4013
KB969612	Description of the hotfix rollup package for ASP.NET 2.0: April 2009	KB969612		4028

The following fix affects this file:

System.Web.Extensions.dll

KB describing the problem	Description	Fix package for Windows 2000/XP/2003	Fix package for Windows Vista SP1 / Windows Server 2008	Build number
KB957661	FIX: Exception error message when you use a virtual path provider to serve a Web page that has a ScriptManager control: "Directory '<file directory>' does not exist. Failed to start monitoring file changes."	KB957661	KB957661	4030

HTH

Doug

Some .NET samples

A bit of a random link this, but I stumbled upon this list of .NET sample applications hosted on the MSDN Academic Alliance site:

http://docs.msdnaa.net/ark_new3.0/cd3/content/Type_Sample%20Applications.htm

Doug

“Failed to load data access DLL, 0x80004005” – OR – What is mscordacwks.dll?

Ever seen this error in a WinDBG/CDB debug session?

This error message is something that often faces people trying to debug dumps of .NET 2.0 applications using WinDBG/CDB using the SOS debugger extension. I’ve been having more than my fair share of issues with it lately and I thought it justified a bit of explanation.

What is mscordacwks.dll?

The Common Language Runtime (CLR) is the core engine of the Microsoft .NET Framework that executes managed code. In simple terms it does this by taking the intermediate language and metadata in a managed assembly, JIT compiling the code on demand, building in memory representations of the types the assembly defines and uses and ensures the resulting code is safe, secure and verifiable and gets executed when it is meant to.  This engine is itself implemented in native code. When we want to debug a .NET application using a native debugger like CDB or WinDBG (which we currently do a lot of if we want to debug it using post-mortem memory dump files) we have to use a “bridge” between the native debugger and the managed world because the native debugger does not inherently understand managed code. It is a native debugger.

To provide this bridge, the CLR helpfully ships with a debugger extension – SOS.DLL. This understands the internals of the CLR and so allows us to do things like outputting managed calls stacks, dumping the managed heap etc.

But from time to time, these internal data structures and details of the CLR change and so it is useful to abstract the interface to the CLR that this debugger extension needs from the actual internal implementation of the CLR that makes .NET applications work. Enter mscordacwks.dll. This provides the Data Access Component (DAC) that allows the SOS.DLL debugger extension to interpret the in memory data structures that maintain the state of a .NET application.

If you look in your framework folder you should always see a matching set of these 3 DLLs:

If you work with 64-bit you should also see a matching DLL set in the Framework64 folder.

 

What does this error message mean?

It means that the SOS.DLL debugger extension has not been able to find the matching mscordacwks.dll that it needs to be able to debug the dump file you are trying to debug.

How do you know I am debugging a dump file?

Because if you were debugging a live application the debugger extension would automatically find and load the mscordacwks.dll from the framework directory.

When am I likely to get this error message and when will I not get it?

If you are debugging a dump file from an application that was using a different build (e.g. different installed service pack or hotfix) of the CLR to the one installed on your local system, or if the .NET Framework was installed in a different location to where it is installed on your system and if the correct mscordacwks.dll is not discoverable by the debugger by some other means.

What “other means”?

Because having the matching mscordacwks.dll is so important for SOS.DLL to work correctly, SOS has a number of tricks up its sleeve to find it. In particular, provided the correct indexing to the symbol server has occurred the debugger will load it from there. The debugger will also look for it in your debuggers directory provided it has been renamed in a special way (see below).

So how do I fix it?

Most of the time, if you have your symbol path set up correctly (which you will need to anyway to make any headway at all with debugging anything, let alone managed applications) then the debugger should be able to get the correct mscordacwks.dll from the symbol server automatically:

What if that doesn’t work?

The simplest thing is to ask the person that gave you the dump file to look at to give you a copy of the mscordacwks.dll. Once you have it, check its file properties for the version number. It should be something like 2.0.50727.xxxx. Then rename it to

mscordacwks_AAA_AAA_2.0.50727.xxxx.dll

where xxxx is the appropriate bit of the version number and AAA is either x86 or AMD64 depending on whether you are dealing with a 32-bit or a 64-bit application dump. (The AMD64 is a legacy thing before we referred to x64). Then put this renamed copy into your debuggers directory (the one where WinDBG is installed). Then, as per the error message, tell the debugger to try again:

Although we try to ensure that every build of the CLR that is released (as a service pack, a hotfix or whatever) has its mscordacwks.dll indexed on the public symbol server, unfortunately it sometimes does not happen. But since it always ships as part of the CLR you always have the option of getting it from the machine the dump came from.

I tried the verbose logging option and it seems to be confused about whether it wants x86 or x64. Now what?

So you ran .cordll –ve –u –l as instructed and got a message something like this:

What this means is that you most likely took a dump of a 32-bit process (running under WoW64) on a 64-bit system using a 64-bit debugger and you are now trying to analyse the dump using a 64-bit debugger. That's why the message references AMD64 and then x86. This is not going to work. Because the SOS.DLL extension actually makes use of the framework while debugging the bitnesses need to match. I strongly recommend always generating the dump using a debugger of the same bitness as the process (so x86 debugger for WoW64 processes even though the system is an x64 system) and analysing the dump with the same bitness of debugger that generated it. And that means of course you cannot debug a 64-bit dump on a 32-bit system. It also means you have to have the framework installed to debug a managed application dump.

Now it’s telling me it sufferred an “init failure”?

You might see this: This also points to a bitness mix up. I've seen this when using a 32-bit debugger to analyse a dump of a WoW64 process generated with a 64-bit debugger.

Isn’t this problem as old as the hills?

I am certainly not the first person to blog about this and won’t be the last either but I thought it was worth me attempting to explain some of these mysteries since it still continues to get people confused. Here are some other posts about this:

Failed to load data access DLL, 0x80004005 – hm
"Failed to start stack walk: 80004005", "Following frames may be wrong" and other errors you may see in windbg
Production Debugging for Hung ASP.Net 2 applications – a crash course
Loading CLR DAC dll from a different path

HTH

Doug

Internet Explorer 8.0 Resources

Internet Explorer 8.0 is currently available as “RC1” or “Release Candidate 1” (which means it is pretty near final version, much more so than, say, a beta version).

This version of IE is a major new version and has lots to offer developers but also has a lot of implications for web site owners and developers due to some of the bug changes it includes.

My colleague Rainer Maurer has pulled together an excellent collection of links to IE8 resources which I am posting here on his behalf:

What's New in Internet Explorer 8

http://msdn.microsoft.com/en-us/library/cc288472(VS.85).aspx

Internet Explorer 8

http://www.microsoft.com/ie/ie8

Internet Explorer Developer Center

http://msdn2.microsoft.com/en-us/ie/default.aspx

Internet Explorer 8 Readiness Toolkit

http://www.microsoft.com/windows/products/winfamily/ie/ie8/readiness/default.htm

IE Blog

http://blogs.msdn.com/ie/

IE8 on Channel 9

http://channel9.msdn.com/tags/IE8/

IE8 Videos

http://www.microsoft.com/windows/internet-explorer/beta/videos.aspx

http://www.youtube.com/profile_videos?user=ie8videos

IE8 Demos

http://www.ie8demos.com

IE8 Add-ons Gallery

http://ieaddons.com/en/

Description of the Compatibility View list in Windows Internet Explorer 8

http://support.microsoft.com/kb/960321/

Some Web sites may not be displayed correctly or work correctly in Windows Internet Explorer 8

http://support.microsoft.com/kb/956197

 

HTH

Doug