25 Feet Below the Surface

Migrating to a "Web Application Project" in 2.0 when you have your 1.1 Application in TFS

praveeny — Mon, 09 Apr 2007 20:40:00 GMT

Normally within PSS, we have Breadth Queue's and Depth Queue's. I for example work in a Depth Q, what that means is that I only deal with issues related to ASP.NET and IIS and nothing more than that. So if you were running into a problem related to client...(read more)

WebResource.axd going over HTTP when you are browsing on HTTPS

praveeny — Tue, 27 Mar 2007 13:22:00 GMT

Sherlock Homes once said "It is a capital mistake to theorize before one has data". Being in the business of troubleshooting, I for one, believe in this. I ran into a similar problem recently where we made assumptions based on some vague facts rather...(read more)

The Breakpoint will not currently be hit. No symbols have been loaded for this document.

praveeny — Sat, 03 Mar 2007 08:34:00 GMT

Recently one of the Engineers on my team ran into a strange problem. Well at least it looked strange when I went by to look at it. So this Support Engineer (prashant) was working with a customer who wanted to setup debugging through Visual Studio 2005...(read more)

ASP.NET 2.0: Accessing dynamic controls from content page when you have nested master pages.

praveeny — Fri, 23 Feb 2007 13:45:00 GMT

Recently I got a problem where the customer was trying to access dynamically created controls from a content page while using nested master pages. I thought it would be a cakewalk only to get slightly confused while working out a small sample to demonstrate how to access the dynamic control. So here is a brief overview about the Troubleshooting.

Customer was developing an ASP.NET 2.0 application which had nested master pages. Part of the requirement for this application was that one of the pages was rendering the rows and columns for a table dynamically based on user input on the same page. Also the customer wanted to add a list of radio buttons to the table rows and access the selected item in another button click event.

So if I have a dynamic control called rbtnDynamic and if i need to get the value of this control in another event, this is how i will need to access it.

rbtnlNew = (RadioButtonList)Master.Master.FindControl("cphParent").FindControl("cphChild").FindControl("rbtnlDynamic");
Response.Write(rbtnlNew.SelectedItem.Value.ToString());

This may look to be a weird way of fetching the value of the radio button, but take a look at the control tree below and it will make more sense to you.

As you can see the Page itself is the container for the Child Master which in turn acts as a container for the Parent Master, thats why we need to first back track and locate the content place holder for the parent master, then come down to the child master and finally do a FindControl for the control itself.

Hope that makes things clearer :-)

ASP.NET 2.0 Cache Objects get trimmed when you have low Available Memory

praveeny — Tue, 12 Dec 2006 02:01:00 GMT

In my line of business I normally do a fair amount of debugging but not as much as an Escalation Engineer would do. This was just one of those problems I was running into and since I really didn't have a clue on what would be the next steps, I resorted to gathering dumps (which didn't really help). But in this blog i am going to try and approach this problem from a troubleshooting perspective to understand the issue and maybe give you an insight on how PSS works.

PROBLEM DESCRIPTION
==================
The title would normally tell you where i am coming from, but here is a brief description.
Customer had written a brand new application in ASP.NET 2.0 and while running a stress test he observed that the Cache objects were getting trimmed almost immediately once they got created. Interestingly there really wasn't a "Memory Pressure"

A bit of clarification here. When I say that "Memory Pressure" wasn't high, what I really mean is that the total committed memory for the worker process was showing up as 100 MB ONLY. This does have to do something with memory pressure, but I will open that card when I come to the end of this blog :-)

TROUBLESHOOTING METHODOLOGY
============================
As usual when a customer calls in and says "I am running into such and such problem" and this is my environment, I get skeptical, not because I don't believe the customer, but just because I don't believe anything until I see it. There are too many cases I have dealt with where we were focused on a particular issue and were getting nowhere only to realize later that the information provided was insufficient or incorrect. And a lot of times, its not the customer to blame but someone else down the line who provided the wrong information.

So the first thing i told Punit (the SE who got this case) was to verify what the customer means when he says he is running into a cache problems getting trimmed, what does he really mean? In this case the customer really knew what he was talking about and to isolate the issue he created a simple application which added about 1000 items to the cache and he had implemented a callback method for these cache items. This callback method would then enumerate the items removed from the cache and display the reason for expiration.

On the customers environment we could see that the cache items were almost immediately trimmed in less than a minute and the reason showing up was "Underused". If we take a look at the documentation (http://msdn2.microsoft.com/en-gb/library/system.web.caching.cacheitemremovedreason(VS.80).aspx) this mentions that "Underused" is when "The item is removed from the cache because the system removed it to free memory"

This was a bit surprising to me because when we looked at the worker process in the Task Manager it was only showing up to be around 100 MB in size. We asked for the sample application and tried to reproduce this behavior and it worked perfectly in our environment. With some experimenting and playing around with the config file at the customers end, we found out that if we increase the value of percentagePhysicalMemoryUsedLimit to 90 or above it doesn't trim the cache items.

So the question now would be why is the Worker Process thinking that its under "Memory Pressure" and trimming the cache objects? First thing that came to my mind was that since this is only specific to the customers environment maybe he just doesn't have enough RAM on this machine. This theory fell to the ground when we found out that he has 2 GB of RAM.

OK, so what else? Dependencies? Maybe the cache items we are inserting have dependencies like File Dependencies or SQL Dependencies. Well from theory we know that the dependent item (cache object) will automatically be removed from the Cache if the dependency changes. So if I have a File Dependency set up and if something is modifying this file like for example a Virus Scan it would cause the Cache item to be removed. But this wasn't the case either, there were no dependencies associated with these items.

Application Restart's? This could be another possibility, Cache is global to the application domain itself, so if due to some circumstances like for example the web.config or global.asax is touched by a virus scan, it could cause a App Domain Recycle resulting in the cache getting trimmed. There are various other reasons for this to happen and its beyond the scope of this blog so I will keep it short for now.

So at this point, I was running out of ideas and I did the next logical thing we do here at PSS, take dumps and Perfmon data. The counters in perfmon that i was looking for were around cache, specifically under the "ASP.NET Application Counters". I was also interested in looking at the counters under .NET Memory to see if I can find anything in there.

When i looked at the perfmon data, it just proved what we already knew. The counters i was looking at were, the Cache Total Entries, Cache Total Turnover Rate, Cache API Entries, Cache API Turnover Rate. Interestingly the turnover was large and the total entries was going up and then suddenly falling down.

Next thing I did was to look at the memory dumps (we had taken three consecutive dumps and when i looked at the Private Bytes Pressure this is how it looked.

0:000> !do 05597268
Name: System.Web.Caching.CacheMemoryPrivateBytesPressure
MethodTable: 68a16d9c
EEClass: 68a16d2c
Size: 52(0x34) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fed1c 4001345        8         System.Int32 0 instance       81_pressureHigh
790fed1c 4001346        c         System.Int32 0 instance       80 _pressureMiddle
790fed1c 4001347       10         System.Int32 0 instance       79 _pressureLow
790fed1c 4001348       14         System.Int32 0 instance        0 _i0
791240f0 4001349        4       System.Int32[] 0 instance 0559729c _pressureHist
790fed1c 400134a       18         System.Int32 0 instance       12 _pressureTotal
790fed1c 400134b       1c         System.Int32 0 instance        2 _pressureAvg
790fcb80 4001350       20         System.Int64 0 instance 1286602752 _memoryLimit
79105ef0 4001358       28      System.DateTime 1 instance 05597290 _startupTime
79104f64 4001351      808       System.Boolean 0   shared   static s_isIIS6
    >> Domain:Value 0016cc10:NotInit 0019ef10:0 <<
790fcb80 4001352      7f0         System.Int64 0   shared   static s_autoPrivateBytesLimit
    >> Domain:Value 0016cc10:NotInit 0019ef10:-1 <<
790fcb80 4001353      7f8         System.Int64 0   shared   static s_totalPhysical
    >> Domain:Value 0016cc10:NotInit 0019ef10:0 <<
790fcb80 4001354      800         System.Int64 0   shared   static s_lastReadPrivateBytes
    >> Domain:Value 0016cc10:NotInit 0019ef10:26214400 <<
79105ef0 4001355      1c8      System.DateTime 1   shared   static s_lastTimeReadPrivateBytes
    >> Domain:Value 0016cc10:NotInit 0019ef10:055af9e8 <<
79105474 4001356      80c        System.UInt32 0   shared   static s_pid
    >> Domain:Value 0016cc10:NotInit 0019ef10:6104 <<
790fed1c 4001357      810         System.Int32 0   shared   static s_pollInterval
    >> Domain:Value 0016cc10:NotInit 0019ef10:120000 <<

So surprisingly from the above results, it looks like the the memory limit is 1.2 GB. I did a lot more digging around in the dump to make some sense of the results I was seeing, but didnt make headway, so I took the next step at this point and engaged the Product Team by raising an internal bug. I went back and forth with Thomas from the product team and finally found out the problem was related to the "Available Memory" being low. Now Cache objects are trimmed based on the algorithm

Pressure = (Total Memory - Available Memory)/Total Memory

So in the customers case the total memory was 2 GB, Available was 350 MB and this expression evaluates to
(2048-350)/2048 = 0.82 which means 82%.

The customer had set the percentagePhysicalMemoryUsedLimit to 80% and thats ASP.NET was trimming the cache items. As a rule when we go over the _preassureHigh value we drop half the cache entries and if we go over _preassureMiddle we drop a small percentage.

ASP.NET 2.0 attempts to flush roughly 50% of the memory consumed by cached entries (both output cached and objects cached using the Cache object) when memory consumption for a process approached 90% of the memory consumption limits. On a 2GB 32-bit machine, the default memory limit is 800MB and on a /3GB 32-bit machine (or WOW64) the limit is 1.8 GB.

Using VirtualPathProvider and Server.MapPath??

praveeny — Sun, 10 Dec 2006 12:13:00 GMT

Recently one of the engineers on my team (Amitkumar Sharma) got this issue reported by the customer where they had there own VirtualPathProvider to serve content from an alternate location rather then the default File System. The customer also wanted to use the Server.MapPath in conjunction but when they tried to return the actual physical path of the directory, it was returning the "Virtual Path"

The first time I heard of this from Amit, my thoughts were to tell him "Hey, that wont work!!!" and i was write, Look at it this way, when we are using a VirtualPathProvider the file could be located anywhere (literally anywhere), you could have the file located in a SQL Database or an alternative location (rather then the physical path) or even for example a zip file!!! so calling MapPath wont really work, would it?

But just to satisfy my curiosity i did take a peek at the source code (for MapPath) and indeed when we are using a VPP the path does get translated. I could talk a bit more about this provider since i have had more than a couple of issues to deal with but there really are enough resources on the web to tap into, so why reinvent the wheel. Once you get the hang of it, its relatively easy to use it. Having said that here are a list of resources that are quite useful if you are planning to use this provider.

VirtualPathProvider Class
http://msdn2.microsoft.com/en-us/library/system.web.hosting.virtualpathprovider.aspx

How to use virtual path providers to dynamically load and compile content from virtual paths in ASP.NET 2.0
http://support.microsoft.com/kb/910441

Overriding ASP.NET combine behavior using a VirtualPathProvider
http://blogs.msdn.com/davidebb/archive/2005/11/27/overriding-asp-net-combine-behavior-using-a-virtualpathprovider.aspx

Virtualizing Access to Content: Serving Your Web Site from a ZIP File
http://msdn2.microsoft.com/en-us/library/aa479502.aspx

New Kids on the Block

praveeny — Thu, 30 Nov 2006 14:12:00 GMT

Recently a few new folks joined the ASP.NET team and I had the privilege to train them on "Troubleshooting ASP.NET" for a couple of days, but i don't think i did a fair job of training them at troubleshooting.

Honestly speaking i have never really found effective ways to talk about Troubleshooting ASP.NET. More because i don't know where to start. ASP.NET is a technology rather than a product. It runs on top of the .NET Framework and IIS. Visual Studio on the other hand is a IDE that helps us in writing ASP.NET applications in a much more efficient way.

The kind of problems we get are really different. They can vary from troubleshooting an issue related to Performance where customer is reporting that his ASPX pages are taking 25 secs to load to troubleshooting issues related to pure code, where some customer might be wanting to extend the GridView control.

At Microsoft, I believe we do a fair job in training folks, but we do follow the 70/20/10 rule. Where we get 70% of our learning on the Job (that means taking support calls and troubleshooting the issues) and 10% of time in Training. But when i mean on the Job training, that doesnt mean we leave the Support Engineer on his own to figure things out. There are a bunch of people, starting from his mentor, Technical Lead, Manager and Escalation Services that pitch in when the engineer is not sure about how to proceed and give that extra push to get the problem resolved.

By the way, just as a note. The training we deliver in classroom is also available on the Internet at the following address.

Learn With PSS
http://www.asp.net/learn/withpss/default.aspx

Create/Debug ASP.NET Web Application using Visual Studio 2005 on IIS 7.0 (Windows Vista)

praveeny — Tue, 28 Nov 2006 11:42:00 GMT

I wanted to blog about this since Windows Vista has been released and we will probably see a lot more such issues. This was an issue that one of my colleagues (Pradeep Martin Anchan) worked on and fixed it. Pradeep is one of the few people on our team who has actually completed the IIS 7.0 training and walkthroughs and all that stuff. He was also the first one to install the Windows Vista Beta and start exploring IIS 7.0. Unlike me of course, I still need to get Vista installed but have been putting it off for one of those lazy days. So here is a brief summary of the problem and the Resolution.

PROBLEM
========
Customer had Windows Vista RTM, Visual Studio 2005 PRO, IIS 7.0 and of course .NET Framework 2.0. While trying to create a new web application he was getting the following error.

"Unable to create the Web 'http://localhost/<website>'.
The Web server does not appear to have the FrontPage server extensions installed."

CAUSE
======
The Visual Studio 2005 IDE uses the IIS metabase APIs to create and configure applications in IIS. So, we must install a metabase compatibility component for IIS 7.0.

RESOLUTION
===========

Install the metabase compatibility component for IIS.

a. Open the Control Panel and click on Programs and Features. In the left pane, click Turn Windows features on or off.
b. Expand Internet Information Services. Expand Web Management Tools and then expand IIS 6 Management Compatibility.
c. Select the IIS Metabase and IIS 6 Configuration Compatibility check box.
d. Click OK to start the installation process.

On a Windows Vista machine we need to take care that Visual Studio 2005 is running in the context of the administrator account. Even if we loging with Admin credentials, Vista will run applications with limited privilages. Make sure that you right click Visual Studio 2005 and choose "Run as Administrator"

ASP.NET 2.0: AcquireRequestState fires multiple times

praveeny — Mon, 27 Nov 2006 08:43:00 GMT

PROBLEM/TROUBLESHOOTING
=========================
I came across this problem while troubleshooting a "Session Loss" issue. Customer in this case was using the Application_AcquireRequestState method and within this method was trying to use the HttpContext.Current.Session object to do something with it. This was resulting in a NullReferenceException.

While debugging we discovered that the AcquireRequestState method was being called multiple times and the first time it showed up as a valid object, the second time it hit that breakpoint, the value returned was null.

To troubleshoot this issue, i first created a simple sample application which has nothing but a default.aspx page and added the AcquireRequestState event to the Global.asax codefile. The result was that the event was fired only Once as i would expect it.

This would normally indicate that there are multiple resources being requested for in the same page. So i looked at the code of the page and found out that the customer was using some Validation Controls in the page.

This pointed me in the right direction, since i know that in ASP.NET 2.0 we have the WebResource.axd that gets called to retrieve an embedded resource (in this case the WebUIValidation.js from System.Web.dll) which would be retrieved as a separate request. If you look at the Fiddler logs you will see what i am talking about.

RESOLUTION
===========
To workaround this issue we needed to add the following code in the AcquireRequestState event.

if (HttpContext.Current.Request.Url.AbsoluteUri.Contains(".axd"))
{
return;
}

A Bit about this Web Log

praveeny — Sun, 26 Nov 2006 09:14:00 GMT

Its been a while since i have been considering writing a blog on ASP.NET. I know there are a few blogs out there which talk in depth about ASP.NET and some of the problems that people run into, but i still believe that there is a lot more information to be shared.

Before we go into depth, make a note of the title, i talk about 25 feet below the surface, thats because, thats where i am. Just below the surface but still not too deep. While i am exploring ASP.NET i will keep going deeper and deeper and hopefully someday touch the bottom :-)

In this blog i am also going to talk a bit about PSS and what we do in our daily lives. Unlikely to popular beliefs we do a lot of stuff and i strongly believe that when it comes to troubleshooting, we can "fix" any problem if we have the determination to do so. Everday at PSS is a challenge and sometimes the kind of issues we get to tackle are really really tough. But the Engineers here never cease to amaze me.