25 Feet Below the Surface

Migrating to a "Web Application Project" in 2.0 when you have your 1.1 Application in TFS

Normally within PSS, we have Breadth Queue's and Depth Queue's. I for example work in a Depth Q, what that means is that I only deal with issues related to ASP.NET and IIS and nothing more than that. So if you were running into a problem related to client side JavaScript you would go to the Internet Explorer Team and if you run into an issue related to say invoking a Web Service, you should be talking to the Distributed Services team.

The reason I am trying to explain how we work is related to a problem I ran into recently which had me stumped (bowled out)  for a while. This was one of those problems that falls into that unknown zone, sort of like the "No Man's Land". It was a problem with TFS and migration in ASP.NET.

So here is a brief about the problem. Customer had a huge Web Project and supporting Class Library projects all under one Solution File. They had everything working fine in Visual Studio.NET 2003 and they were using a 3rd party source control. Once TFS came into the picture, they had learnt that they could also use TFS as a source control and migrated everything to the Team Foundation Server. Everything worked fine and no issues so far.

But then the requirement came in to migrate the project to Visual Studio 2005 and ASP.NET 2.0. On evaluation they decided to migrate to a Web Application Project as opposed to a Website model. At this point I guess they might have thought that since we already have it in TFS we wouldn't need to unbind the project or something from source control and just opened the Solution File (in VS 2005) to start of the migration process to WAP. Lets assume that the migration went on smoothly and the project built successfully, since that's not the problem area I am talking about.

Right after the migration they checked in the solution file. But when the next time a developer would get the files and double click the solution file (from within TFS) to open the project, the following message would pop-up.

---------------------------
Source Control
---------------------------

The following Web projects must be converted to the new Web Site format. The conversion process will remove all source control bindings and the project will not be under source control after migration

C:\inetpub\wwwroot\MyTestProj\MyTestProj.vbproj

-----------------------------------------------
OK HELP
-----------------------------------------------

Followed by the message.

---------------------------
Microsoft Visual Studio
---------------------------

Some of the properties associated with the solution could not be read.

---------------------------
OK
---------------------------

Interestingly these message do not effect the projects itself. The projects stay migrated and when they were working with these projects there is no issue. After doing some Research on his end, the customer had found out that by removing the entries for SccEnlistmentChoice in the solution file, the messages would go away as long as the solution file stays checked out, but as soon as he check's in the solution file and opens it the next time, the problem came back.

When one of my team members got a call from the customer, after getting the problem description, we assumed that this is a problem related to TFS and nothing really to do with migration, after all, when we migrate a project that's not a part of TFS from VS.NET 2003 to VS 2005 WAP model, we don't get this message, right? And what about these weird entries, they appear to be source control related. We discussed this with the TFS folks and they agreed to work on part of the case.

Normally when issues like this arise where we have no clue about which team takes care of it, I don't worry too much about who should be driving the case, as long as due diligence is done and the customer goes back happy. So, I was perfectly happy to "bide my wee" and wait for the problem to get solved. But the problem kept dragging on. When I came into the picture we were at a stalemate and no specific direction.

The first thing I did was to request for a lab machine and try to setup a Repro of the problem. I called the customer and asked him to help me reproduce the problem. As all good Repro's this one didn't fail either and I wasn't able to reproduce the behavior. I went back and digged through the documentation, Source Code etc with no luck. It was only later that I realized how stupid I had been, In my Repro I was migrating to a regular WebSite project and not a "Web Application Project"

Once I corrected this error, I could immediately reproduce the behavior. Once a successful Repro was in place, I just had to play around with the SLN files, the Source control files and come up with the solution. The tool I used here was WinDiff, a file comparison tool. The solution was to delete the file VPSCC which gets generated for each project and manually edit the Solution File to remove the entries related to SccEnlistmentChoice. Once I got the solution it was easy to search for the answer. It appears that when we migrate a project which is already in the Source Control. The VPSCC file has an entry in it called "ENLISTMENT CHOICE" and this is set to "COMPULSORY". Since TFS will re-generate this file, its best that we get rid of this file and when we check-in the solution file, it will re-generate it with the correct entries.

WebResource.axd going over HTTP when you are browsing on HTTPS

Sherlock Homes once said "It is a capital mistake to theorize before one has data". Being in the business of troubleshooting, I for one, believe in this. I ran into a similar problem recently where we made assumptions based on some vague facts rather than actual data. The problem was straightforward, customer was using SSL and when he hit the web page using FireFox over HTTPS he would see the following message in the status bar.

Connection Partially Encrypted

Parts of the page you are viewing were not encrypted before being transmitted over the Internet.
Information sent over the Internet without encryption can be seen by other people while it is in transit.

He was using the built in Validation Controls for an ASP.NET 2.0 website and after some research the customer himself had found out that the somehow the WebResource.axd was tied up with the whole thing. The customer had also tested this in Internet Explorer and he wasn't getting any such error message.

So when the Support Engineer who got this call first heard about this problem, he jumped to the conclusion that there was nothing we could do about this since it was something to do with FireFox not trusting the content of the web page. So he called into the IIS team to see if we can do something from the "server side" of the things to fix this problem.

At this point I would like to point out a minor problem which lead us into a completely wrong direction.  The IIS engineer who got the call may have been hitting the www.asp.net site and he found out that when we go to the page https://forums.asp.net/user/createuser.aspx for creating a user account, we get a similar message in FireFox. So he theorized that since this wasn't working on one of our sites, it probably is a problem with FireFox rather than anything to do with the WebResource.axd. So was the message give to the customer.

When I stepped in to look at this problem, the customer had already been provided a workaround where he was asked to force SSL in his website. If you already have SSL certificate installed we can do this by going to the Internet Information Services, go to the properties of the website and click on the "Edit" button. You will see an option in there to choose "Require secure channel(SSL)" and choose 128 bit encryption under it. I talked to the customer and also pointed out that this is not a problem with FireFox, but if we have the setting "Warn if changing between secure and not secure mode" selected in Internet Explorer (under Tools-->Options-->Advanced) we would get a similar kind of message that says

This page contains both secure and nonsecure items.
Do you want to display the nonsecure items?

Honestly speaking I am not an expert on IIS, in fact I would put myself in the group of novice users who just know how to run ASP/ASP.NET on IIS and maybe some minor details surrounding it. I was completely convinced with the idea that maybe we just need to enforce SSL to make everything run over HTTPS. But after I got off the call with the customer I started thinking that maybe there was something we missed out and went back and reviewed some research logs added by another SE and  an Escalation Engineer which basically said that this needs more investigation with some pointers. I then setup a simple Repro where I had a ASPX page with a validation control thrown in and browsed it over HTTPS from a different machine and like I suspected I never got the above message in IE or FireFox. This obviously wasn't something we would call "by design". Just to ensure I was on the right track I checked with the source code of ASP.NET to ensure that we aren't issuing any Redirects from there for caching etc and we weren't, so there was something else that was amiss.

I would like to circle back a bit here, since there is another detail that I omitted on purpose. This was about getting the network traffic. I had previously installed Fiddler2 Tool installed to look at the network traffic and here is how it looked when I see the WebResource.axd in there.

So if you look at the trace above, you can see that there was first a HTTPS request for the WebResource.axd and then there is a 302 indicating a redirect and we have a 200 OK for the resource. I was getting a similar kind of traffic for the customers website as well.

After some additional digging it turned out that the customer had a custom HttpModule that would allow automatic switching between HTTP and HTTPS protocol. It also had a configuration section where we could define the pages that will run on SSL. We added the WebResource.axd to this list and that solved the problem.

The Breakpoint will not currently be hit. No symbols have been loaded for this document.

Recently one of the Engineers on my team ran into a strange problem. Well at least it looked strange when I went by to look at it. So this Support Engineer (prashant) was working with a customer who wanted to setup debugging through Visual Studio 2005 on his local box.

The problem was that we could set a breakpoint in Visual Studio 2005 but for whatever reason, the breakpoint was not getting hit and the page would load as if we had done a "Run without debugging".

So prashant was pinging me on Instant Messenger and I was throwing out a lot of support articles and voice columns etc out to him and saying "Go check this", "Go Check that" etc and he was getting no where towards a resolution. The message that the customer was seeing in Visual Studio 2005 looked like the following.

Usually when we see such a problem, the first thing we do is to delete all the files in the "Temporary ASP.NET Files folder" restart all IIS Services and then close Visual Studio 2005 (just to be on the safe side) and re-open the project. But this wasn't helping for this customer.

Surprisingly when we created a brand new website it still showed the same problem. Now if we reflect on this, the symbols or PDB files are automatically generated as soon as you Build the project taking care that the debug attribute is set to "true" in the web.config file. Now if I do an F5 in VS 2005 and do not have debug set to true, it will automatically prompt me about that.

So I asked prashant if the customer was seeing any PDB files in this folder and he replied in negative, which confused me a bit more. Well there may be other settings in the Visual Studio environment I wasn't aware of. So I went over and got on the call with the customer.

The next thing I did was to verify each and every setting in my VS 2005, Tools-->options against the customers settings, they were all the same.

Checking the Output window in VS 2005 I could see the following as a part of the output.

I went and looked at the output in the temporary ASP.NET files folder and surprisingly there were no PDB files generated and what was more interesting was that the output had files in it which looked as if the DEBUG attribute was never set to true!!!

Normally when we have the DEBUG attribute set to true, we will have a lot of extra files in there like for example the .cs file, .pdb file, .cmdline, .err and .out. Each of them has its own significance around it, but in this case the only files in there was the .compiled, the .dll, hash.web and the .ccu.

By this time I started loosing some focus, since I wasn't sure what to make of it, theoretically when we have the debug set to true in web.config it will override all other settings, but this was like Crazy!!! I started getting Filemon logs to see if we were even loading the web.config file and what not.

Filemon did show me that the .cs file and .pdb files were getting created but devenv.exe (process for VS 2005) was going and deleting them.

While I was troubleshooting the problem I was also explaining as I went along about the temporary files and what creates them and what kind of information is in there. Finally the customer said "you know what, I was reading up on a Scott Guthrie's blog and I found something there, which may be related"

He then told me about the retail = "true" setting. you can read all about it at the following link

http://weblogs.asp.net/scottgu/archive/2006/04/11/Don_1920_t-run-production-ASP.NET-Applications-with-debug_3D001D20_true_1D20_-enabled.aspx

So if you look at the blog above, if you set the <deployment retail = "true" /> in the machine.config file you will disable the <compilation debug = "true" /> switch for every web application on the server.

It looks like the customer was recently planning to do this on one of there production boxes and probably tested this locally and forgot to turn it back to false and that's what was causing the whole problem.

ASP.NET 2.0: Accessing dynamic controls from content page when you have nested master pages.

Recently I got a problem where the customer was trying to access dynamically created controls from a content page while using nested master pages. I thought it would be a cakewalk only to get slightly confused while working out a small sample to demonstrate how to access the dynamic control. So here is a brief overview about the Troubleshooting.

Customer was developing an ASP.NET 2.0 application which had nested master pages. Part of the requirement for this application was that one of the pages was rendering the rows and columns for a table dynamically based on user input on the same page. Also the customer wanted to add a list of radio buttons to the table rows and access the selected item in another button click event.

So if I have a dynamic control called rbtnDynamic and if i need to get the value of this control in another event, this is how i will need to access it.

This may look to be a weird way of fetching the value of the radio button, but take a look at the control tree below and it will make more sense to you.

As you can see the Page itself is the container for the Child Master which in turn acts as a container for the Parent Master, thats why we need to first back track and locate the content place holder for the parent master, then come down to the child master and finally do a FindControl for the control itself.

Hope that makes things clearer :-)

ASP.NET 2.0 Cache Objects get trimmed when you have low Available Memory

In my line of business I normally do a fair amount of debugging but not as much as an Escalation Engineer would do. This was just one of those problems I was running into and since I really didn't have a clue on what would be the next steps, I resorted to gathering dumps (which didn't really help). But in this blog i am going to try and approach this problem from a troubleshooting perspective to understand the issue and maybe give you an insight on how PSS works.

PROBLEM DESCRIPTION
==================
The title would normally tell you where i am coming from, but here is a brief description.
Customer had written a brand new application in ASP.NET 2.0 and while running a stress test he observed that the Cache objects were getting trimmed almost immediately once they got created. Interestingly there really wasn't a "Memory Pressure"

A bit of clarification here. When I say that "Memory Pressure" wasn't high, what I really mean is that the total committed memory for the worker process was showing up as 100 MB ONLY. This does have to do something with memory pressure, but I will open that card when I come to the end of this blog :-)

TROUBLESHOOTING METHODOLOGY
============================
As usual when a customer calls in and says "I am running into such and such problem" and this is my environment, I get skeptical, not because I don't believe the customer, but just because I don't believe anything until I see it. There are too many cases I have dealt with where we were focused on a particular issue and were getting nowhere only to realize later that the information provided was insufficient or incorrect. And a lot of times, its not the customer to blame but someone else down the line who provided the wrong information.

So the first thing i told Punit (the SE who got this case) was to verify what the customer means when he says he is running into a cache problems getting trimmed, what does he really mean? In this case the customer really knew what he was talking about and to isolate the issue he created a simple application which added about 1000 items to the cache and he had implemented a callback method for these cache items. This callback method would then enumerate the items removed from the cache and display the reason for expiration.

On the customers environment we could see that the cache items were almost immediately trimmed in less than a minute and the reason showing up was "Underused". If we take a look at the documentation (http://msdn2.microsoft.com/en-gb/library/system.web.caching.cacheitemremovedreason(VS.80).aspx) this mentions that "Underused" is when "The item is removed from the cache because the system removed it to free memory"

This was a bit surprising to me because when we looked at the worker process in the Task Manager it was only showing up to be around 100 MB in size. We asked for the sample application and tried to reproduce this behavior and it worked perfectly in our environment. With some experimenting and playing around with the config file at the customers end, we found out that if we increase the value of percentagePhysicalMemoryUsedLimit to 90 or above it doesn't trim the cache items.

So the question now would be why is the Worker Process thinking that its under "Memory Pressure" and trimming the cache objects? First thing that came to my mind was that since this is only specific to the customers environment maybe he just doesn't have enough RAM on this machine. This theory fell to the ground when we found out that he has 2 GB of RAM.

OK, so what else? Dependencies? Maybe the cache items we are inserting have dependencies like File Dependencies or SQL Dependencies. Well from theory we know that the dependent item (cache object) will automatically be removed from the Cache if the dependency changes. So if I have a File Dependency set up and if something is modifying this file like for example a Virus Scan it would cause the Cache item to be removed. But this wasn't the case either, there were no dependencies associated with these items.

Application Restart's? This could be another possibility, Cache is global to the application domain itself, so if due to some circumstances like for example the web.config or global.asax is touched by a virus scan, it could cause a App Domain Recycle resulting in the cache getting trimmed. There are various other reasons for this to happen and its beyond the scope of this blog so I will keep it short for now.

So at this point, I was running out of ideas and I did the next logical thing we do here at PSS, take dumps and Perfmon data. The counters in perfmon that i was looking for were around cache, specifically under the "ASP.NET Application Counters". I was also interested in looking at the counters under .NET Memory to see if I can find anything in there.

When i looked at the perfmon data, it just proved what we already knew. The counters i was looking at were, the Cache Total Entries, Cache Total Turnover Rate, Cache API Entries, Cache API Turnover Rate.  Interestingly the turnover was large and the total entries was going up and then suddenly falling down.

Next thing I did  was to look at the memory dumps (we had taken three consecutive dumps and when i looked at the Private Bytes Pressure this is how it looked.

So surprisingly from the above results, it looks like the the memory limit is 1.2 GB. I did a lot more digging around in the dump to make some sense of the results I was seeing, but didnt make headway, so I took the next step at this point and engaged the Product Team by raising an internal bug. I went  back and forth with Thomas from the product team and finally found out the problem was related to the "Available Memory" being low. Now Cache objects are trimmed based on the algorithm

So in the customers case the total memory was 2 GB, Available was 350 MB and this expression evaluates to
(2048-350)/2048 = 0.82 which means 82%.

The customer had set the percentagePhysicalMemoryUsedLimit to 80% and thats ASP.NET was trimming the cache items. As a rule when we go over the _preassureHigh value we drop half the cache entries and if we go over _preassureMiddle we drop a small percentage.

ASP.NET 2.0 attempts to flush roughly 50% of the memory consumed by cached entries (both output cached and objects cached using the Cache object) when memory consumption for a process approached 90% of the memory consumption limits.  On a 2GB 32-bit machine, the default memory limit is 800MB and on a /3GB 32-bit machine (or WOW64) the limit is 1.8 GB.

Using VirtualPathProvider and Server.MapPath??

Recently one of the engineers on my team (Amitkumar Sharma) got this issue reported by the customer where they had there own VirtualPathProvider to serve content from an alternate location rather then the default File System. The customer also wanted to use the Server.MapPath in conjunction but when they tried to return the actual physical path of the directory, it was returning the "Virtual Path"

The first time I heard of this from Amit, my thoughts were to tell him "Hey, that wont work!!!" and i was write, Look at it this way, when we are using a VirtualPathProvider the file could be located anywhere (literally anywhere), you could have the file located in a SQL Database or an alternative location (rather then the physical path) or even for example a zip file!!! so calling MapPath wont really work, would it?

But just to satisfy my curiosity i did take a peek at the source code (for MapPath) and indeed when we are using a VPP the path does get translated. I could talk a bit more about this provider since i have had more than a couple of issues to deal with but there really are enough resources on the web to tap into, so why reinvent the wheel. Once you get the hang of it, its relatively easy to use it. Having said that here are a list of resources that are quite useful if you are planning to use this provider.

VirtualPathProvider Class
http://msdn2.microsoft.com/en-us/library/system.web.hosting.virtualpathprovider.aspx

How to use virtual path providers to dynamically load and compile content from virtual paths in ASP.NET 2.0
http://support.microsoft.com/kb/910441

Overriding ASP.NET combine behavior using a VirtualPathProvider
http://blogs.msdn.com/davidebb/archive/2005/11/27/overriding-asp-net-combine-behavior-using-a-virtualpathprovider.aspx

Virtualizing Access to Content: Serving Your Web Site from a ZIP File
http://msdn2.microsoft.com/en-us/library/aa479502.aspx

New Kids on the Block

Recently a few new folks joined the ASP.NET team and I had the privilege to train them on "Troubleshooting ASP.NET" for a couple of days, but i don't think i did a fair job of training them at troubleshooting.

Honestly speaking i have never really found effective ways to talk about Troubleshooting ASP.NET. More because i don't know where to start. ASP.NET is a technology rather than a product. It runs on top of the .NET Framework and IIS. Visual Studio on the other hand is a IDE that helps us in writing ASP.NET applications in a much more efficient way.

The kind of problems we get are really different. They can vary from troubleshooting an issue related to Performance where customer is reporting that his ASPX pages are taking 25 secs to load to troubleshooting issues related to pure code, where some customer might be wanting to extend the GridView control.

At Microsoft, I believe we do a fair job in training folks, but we do follow the 70/20/10 rule. Where we get 70% of our learning on the Job (that means taking support calls and troubleshooting the issues) and 10% of time in Training. But when i mean on the Job training, that doesnt mean we leave the Support Engineer on his own to figure things out. There are a bunch of people, starting from his mentor, Technical Lead, Manager and Escalation Services that pitch in when the engineer is not sure about how to proceed and give that extra push to get the problem resolved.

By the way, just as a note. The training we deliver in classroom is also available on the Internet at the following address.

Learn With PSS
http://www.asp.net/learn/withpss/default.aspx

Create/Debug ASP.NET Web Application using Visual Studio 2005 on IIS 7.0 (Windows Vista)

I wanted to blog about this since Windows Vista has been released and we will probably see a lot more such issues. This was an issue that one of my colleagues (Pradeep Martin Anchan) worked on and fixed it. Pradeep is one of the few people on our team who has actually completed the IIS 7.0 training and walkthroughs and all that stuff. He was also the first one to install the Windows Vista Beta and start exploring IIS 7.0. Unlike me of course, I still need to get Vista installed but have been putting it off for one of those lazy days. So here is a brief summary of the problem and the Resolution.

PROBLEM
========
Customer had Windows Vista RTM, Visual Studio 2005 PRO, IIS 7.0 and of course .NET Framework 2.0. While trying to create a new web application he was getting the following error.

"Unable to create the Web 'http://localhost/<website>'.
The Web server does not appear to have the FrontPage server extensions installed."

CAUSE
======
The Visual Studio 2005 IDE uses the IIS metabase APIs to create and configure applications in IIS. So, we must install a metabase compatibility component for IIS 7.0.

RESOLUTION
===========

Install the metabase compatibility component for IIS. 

a. Open the Control Panel and click on Programs and Features. In the left pane, click Turn Windows features on or off.
b. Expand Internet Information Services. Expand Web Management Tools and then expand IIS 6 Management Compatibility.
c. Select the IIS Metabase and IIS 6 Configuration Compatibility check box.
d. Click OK to start the installation process.

On a Windows Vista machine we need to take care that Visual Studio 2005 is running in the context of the administrator account. Even if we loging with Admin credentials, Vista will run applications with limited privilages. Make sure that you right click Visual Studio 2005 and choose "Run as Administrator"

ASP.NET 2.0: AcquireRequestState fires multiple times

PROBLEM/TROUBLESHOOTING
=========================
I came across this problem while troubleshooting a "Session Loss" issue. Customer in this case was using the Application_AcquireRequestState method and within this method was trying to use the HttpContext.Current.Session object to do something with it. This was resulting in a NullReferenceException.

While debugging we discovered that the AcquireRequestState method was being called multiple times and the first time it showed up as a valid object, the second time it hit that breakpoint, the value returned was null.

To troubleshoot this issue, i first created a simple sample application which has nothing but a default.aspx page and added the AcquireRequestState event to the Global.asax codefile. The result was that the event was fired only Once as i would expect it.

This would normally indicate that there are multiple resources being requested for in the same page. So i looked at the code of the page and found out that the customer was using some Validation Controls in the page.

This pointed me in the right direction, since i know that in ASP.NET 2.0 we have the WebResource.axd that gets called to retrieve an embedded resource (in this case the WebUIValidation.js from System.Web.dll) which would be retrieved as a separate request. If you look at the Fiddler logs you will see what i am talking about.

RESOLUTION
===========
To workaround this issue we needed to add the following code in the AcquireRequestState event.

if (HttpContext.Current.Request.Url.AbsoluteUri.Contains(".axd"))
{
     return;
}

A Bit about this Web Log

Its been a while since i have been considering writing a blog on ASP.NET. I know there are a few blogs out there which talk in depth about ASP.NET and some of the problems that people run into, but i still believe that there is a lot more information to be shared.

Before we go into depth, make a note of the title, i talk about 25 feet below the surface, thats because, thats where i am. Just below the surface but still not too deep. While i am exploring ASP.NET i will keep going deeper and deeper and hopefully someday touch the bottom :-)

In this blog i am also going to talk a bit about PSS and what we do in our daily lives. Unlikely to popular beliefs we do a lot of stuff and i strongly believe that when it comes to troubleshooting, we can "fix" any problem if we have the determination to do so. Everday at PSS is a challenge and sometimes the kind of issues we get to tackle are really really tough. But the Engineers here never cease to amaze me.