My ramblings on ASP.NET and IIS

This week I wrote new blog post on our team blog webtopics which talks about above error and reason for it ..

You can visit here : http://blogs.msdn.com/webtopics/archive/2009/06/19/intermittently-system-web-httpexceptions-a-field-or-property-with-the-name-columnname-was-not-found-on-the-selected-data-source-is-thrown-when-the-application-is-under-heavy-load.aspx 

Till then bye bye :)

Working today on my Vista when I Added New Virtual Directory from IIS manager pointing to newly created folder on Desktop and upon browsing it ended with 500.19 on browser

Next thing came in my mind was to compare permission on new folder on Desktop(even though error message does not indicate of that )

Aha there seem to be difference between the two folders. The new folder on Desktop was missing IIS_IUSRS and USERS Group– Read & Execute permissions.

The IIS_IUSRS group is intended to replace the IIS_WPG group. This built-in group has been granted access on all the necessary file(\wwwroot) more…

After adding IIS_IUSR account first and when I tried browsing it and the error message changed to 401.3 – Unauthorized due to ACL. This forced me to add USERS group to resolve this issue completely!!!

Till then bye

PS: on Vista home edition you won’t be able to find Local Users & Group option in Computer management MMC. It has been this way since XP.

While setting up repro for one of the customer on my Vista IIS 7.I ran into couple of issue(i’ll post both) of my own.My application gave up with "Service Unavailable" "HTTP error 503"

No matter whatever I browse .htm , aspx everything ends up with that error 503.Looking into event viewer to see something reported apparently that was clean.

And this issue seem to be happening only with one particular app pool "MyTest".That forced me to try and create new app pool and see if that resolve the problem.

Bingo!!! everything works like charm in the new app pool "MyTest2" .

Obvious thing was to compare both the app pool setting in applicationhost.config

        <applicationPools>
          <add name="MyTest" autoStart="false" />
            <add name="MyTest2" autoStart="true" />

Checking in IIS manager

Aha we’ve autoStart=”false” and notice stop icon against MyTest app pool. This feature is there in case app pool crashes for some reason and Rapid fail protection kicks in app pool should start automatically.

We can either change this option in applicationHost.cofig or Edit app pool properties in IIS manager as follow:

It would have been nice if warning/error reported in Event viewer or some useful message on browser rather than 503.

PS : In this case Failed Request (FREB) wouldn’t have helped as worker process was not initiating.

In 3 years at PSS I've worked on many issues related Dynamic Control i.e

1) Controls not able retain value on postback.

2) Controls getting disappeared on postback

3) Event handler not fired on first click works fine on second click

4) Viewstate issues blah blah...

No matter how complex the issue maybe I got it resolved by sticking to following silver bullet point

• Always add the dynamic control in Page_PreInit()
• Always assign ID when loading dynamically
  

For Example

protected void Page_PreInit(object sender, EventArgs e)
{
        Control c= LoadControl("./WebUserControl2.ascx");
        i=i+1;
        c.ID= i.ToString();
        PlaceHolder1.Controls.Add(c);
}

OR

protected void Page_PreInit(object sender, EventArgs e)
{
         LinkButton button1 = new LinkButton();
         button1.ID = "button1"
         button1.Text = "button1"
         PlaceHolder1.Controls.Add(button1);
}

This way it would make sure Dynamic Controls are added to Page Control tree and ID's are not messed up.

Tip: Best way to troubleshoot is to enable trace=true at page level and track that particular control id's or compare view source for first time page rendering with view source on page postback.

Hope it helps

We can either use ADSI,WMI,JScript etc ... to query IIS 7 config like how many website, virtual directories.Lately we have another option to use Microsoft.Web.Administrator namespace either from Web app or from Powershell.

At the root level we've class called ServerManager which exposes all the functionality you will need for querying/editing IIS 7 config.

First thing is we need to add reference to web application in VS 2008 for  Microsoft.Web.Administrator.dll from c:\windows\system32\intesrv.

C#

Browsing it from VS 2008 cassini we are able to query successfully and list website id and its name.

Publish it under Default website which is running on Network Service in integrated mode and with Anonymous auth enabled. Now here we are expected to fail with Exception for permission denied.

Add Network Service Read & Execute permission on c:\windows\system32\inetsrv\config to bypass security exception.After this we can query IIS config for Non Admin user.

Or other option would be:

1) Use <identity impersonate=true> with specific username and password who has rights to read
2) Run the app pool in the identity of user who has access to read config.

For Powershell it is not possible to query config's unless you're admin.

Till Then

Today while working on vista IIS 7 and I was trying to delete  one of module named as CustomErrorModule for Default Web Site and immediate I was notified with this nice error.  

And this seem to be happening only with Modules which are listed as Native . I was able to delete Managed module without any problem.Nothing was reported under system & application event viewer  logs.

Solution was to go at Root level Modules configurations and select the required native module and Unlock it.

In IIS 7 Native Modules seem to be locked by default and same would be valid incase of appcmd tool.

C:\Windows\System32\inetsrv>appcmd unlock config /section:httpErrors
Unlocked section "system.webServer/httpErrors" at configuration path "MACHINE/WEBROOT/APPHOST".

See Ya!!!

Helping one of mate in our team with issue where Asp.net application was failing with System.Security when contents were pointing to UNC path.We were getting following error in the browse:

Parser Error Message: Request for the permission of type 'System.Web.AspNetHostingPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed. (\\abc02\myapp\web.config line 84)

This application was hosted on windows 2008 - IIS 7 server.

First thing came into my mind is to see whether we've set FullTrust according to KB 320268.
We were using  caspol.exe to set permissions as we were missing Microsoft .NET Framework 2.0 Configuration tool (it gets installed with SDK)  :-)

Running following command in CMD

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727> caspol.exe -m -ag 1 -url "file:////\\computername\sharename\*" FullTrust -exclusive on

Application in question was still failing with same error. As a general troublehsooting step thought to try with new share folder and with simple test.aspx in it. Then added Fulltrust permission for that path and it worked. Now that made me to point finger toward the application compatibility.

Next steps was looking into procmon logs and it seem like .Net was trying to read config file from framework64 folder.That raise eyebrow and apparently application was hosted on win2008 x64 server and was running in native mode.

It turns out that .NET stores its security configuration separately for x64 and x32.  Thatz it. we have to run the command again from the Framework64 instead of Framework folder

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727> caspol.exe -m -ag 1 -url "file:////\\computername\sharename\*" FullTrust -exclusive on

Till then

Worked on second case in a row where Visual Studio was hanging when we click on Design View.It would take 45-60 second to respond.
First thing was to check if VS 2008 SP-1 was installed because they were couple issue related to Design view slow performance which were fixed in SP-1.We were already at SP-1 level.Next thing noticed was Visual studio hangs only with one particular big project and it woks fine with new web projects.

Common thing to check in slow response/performance cases is was look into in task manager and see whether it is because of CPU spike or high memory.Checking task manger we noticed devenv.exe was spiking CPU between 85-90% when we try to switch to Design View.

Decided to capture three set of  manual memory dump with help of Debug Diag  during CPU spike activity.

!runaway in all the dumps were showing some activity is going on Thread 0.

We were trying to load/resolve dll to populate form toolbox on Design view

Top guess was to try setting auto populate property for Form designer to false so that it does not try to load control automatically.

Under Tools->Options -> Windows Forms designer –> AutoToolboxPopulate=false.

Well that seem to have made significant difference.

Later on that control was removed from the project to get permanent relief.

Till then 

Let say you have generated New Certificate request(certreq.txt) from IIS manager and would like process that pending request later when you get back certificate file from the third party vendor Certificate Authority.

Getting back certificate

You were expecting .PFX file which has associated key pair from vendor and actually got back .CER(Base64 encoded) file which does not include a key pair.

Unfortunately if you have deleted pending request (or if it's not there)*1 from IIS and then try to install .cer it won't work as this certificate would not have private keys associated with it and we would end with 'Page Cannot Be Displayed' error on browser.

IIS stores the private key for a certificate as the pending request. Deleting the pending request deletes the association of the private key with IIS, but the private key still exists in the local certificate store on the IIS server.

Now question comes into your mind "How to make this .cer working using private key info stored in the local certificate store ?"

Answer :

• certutil -addstore my c:\certnew.cer (this would add the certificate into following path)

On the IIS or Certificate Server, open MMC (Start->Run and type MMC) and add the Certificates snap-in using the Computer Account.
-- Expand Certificates(Local Computer) -> Personal -> Certificates

Double click newly install certificate -> go into Detail tab, locate Thumbprint , press ctrl + c to copy

• certutil -repairstore my "18 8a 7e 44 0f 43 59 a9 f9 a3 64 3b 84 f8 14 d1 b7 5f a2 60"

If certificate is not added into my store you would end up with following error :
Certutil: -repairstore command FAILED: 0x80090011 (-2146893807) Certutil: Object was not found."

*1 Same stand valid incase the request has been generated using Exchange 2007 Tool

When you create the certificate request, the New-ExchangeCertificate cmdlet creates the private key pair and stuffs them in the Certificate Request store on the computer where you run the cmdlet. Therefore, you must run the Import-ExchangeCertificate cmdlet on the same computer to bind the private keys to the certificate.

till then

Recently worked on a problem where Asmx Post request fails with 'HTTP 400' status code under the following circumstances :

When the size of the string parameter passed from the client application (Content-Length)> 1024 Bytes and the file is set for either 'integrated' or 'basic' auth.

Request goes  like this Client (exe) -> Web service (asmx) hosted on IIS

First stop was to look at HTTPERR log on IIS box,  Invalid verb error was report in it:

2009-03-09 21:30:53 10.1.48.56 42676 10.1.48.53 80 HTTP/0.0 Invalid - 400 - Verb -
2009-03-09 21:32:39 10.1.48.56 42688 10.1.48.53 80 HTTP/0.0 Invalid - 400 - Verb -

Next thing came in mind to try increasing total size limit of Request Line and headers. Setting MaxRequestBytes to 10MB (10485760) following KB-820129. That did not help much.

Decided to collect netmon traces for working and failing scenario. There was only one difference between two traces(sucess/faliure) “Expect: 100-continue“ header.

When the Post is being made with Basic Credentials, we were getting "Expect: 100-continue"  in the request headers along with the data packet.

Http: Request, POST /Hello2.asmx , Using Basic Authorization
ProtocolVersion: HTTP/1.1
Expect: 100-continue
<n3:Body><n:Hello><n:a>aaaaa....</n:a></n:Hello>

Looking at RFC for Expect: 100-continue header

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3

When this header(Expect: 100-continue) is sent on the wire this tells the server that the client is going to delay sending the body of the request for some period of time(How much time is not specified) because it wants the server to give the client the OK (a 100 continue response) to upload the data.

Because the client is sending another packet of data that has an XML document instead on waiting on server to respond back with 100 continue , IIS is forced to interpret that as another request which of course has no valid verb in it and end with 400 bad request

Only viable alternative was to remove 'Expect: 100-continue' from the request header.

Had it been .Net client exe we would have done it like:

HttpWebRequest webRequest = (HttpWebRequest) WebRequest.Create(http://test/asbc.asmx);
webRequest.Method = "POST";
webRequest.ServicePoint.Expect100Continue = false;

But in this case customer was using LibCurl library , counterpart looks like: curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:'));

After removing Expect: header from request we were able to post successfully regardless of size...

.NET CLR Memory\# Induced GC – This is the number of garbage collections that have occurred as a result of someone explicitly calling GC.

By default: 0 is the ideal no. for it.

Here are the possible reasons incase you see increase no. "# induced GC".

1) Explicitly calling GC.Collect()
2) Calling GC.GetTotalMemory() with true i.e System.GC.GetTotalMemory(True)
3) Low on available memory and Asp.net trimming Cache entries and internally making call for GC.Collect()

If you happen to know more reasons for it.I'd be happy to update the list.

Recently worked on interesting case where internal IP address was revealed whenever we try to query through wfetch tool

Request Flow
==========
Client    -> ISA     -> IIS 7
x.x.x.30 ->x.x.x.10-> x.x.x.20

We have DNS installed on ISA server, having A record entry iistest.com pointing to x.x.x.20

First thought was to follow kb 834141 , we ran following command on IIS 7 box (it’s new install)
C:\Windows\System32\inetsrv>appcmd.exe set config -section:system.webServer/serverRuntime /alternateHostName:"iistest.com" /commit:apphost

That didn't help . Thought to capture Netmon sniffer trace for :
1) Request from wfetch
2) Request from IE

Request:
GET /exchange/ HTTP/1.0

Response:
HTTP/1.1 302 Moved Temporarily
Content-Length: 0

Location: http://X.X.X.20/exchweb/bin/auth/owalogon.asp?url=http://X.X.X.20/exchange/&reason=0&replaceCurrent=1

Set-Cookie: sessionid=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: cadata=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Date: Tue, 02 Dec 2008 15:49:46 GMT
Connection: close

Request:
GET /exchange HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)
Host: iistest
Connection: Keep-Alive

Response:
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: http://iistest/exchweb/bin/auth/owalogon.asp?url=http://iistest/exchange&reason=0&replaceCurrent=1

Set-Cookie: sessionid=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: cadata=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Date: Tue, 02 Dec 2008 15:51:50 GMT

The difference between two requests is HTTP/1.0 and HTTP/1.1 protocol. What I can make

“HTTP/1.1 requires requests to include a Host header”

HTTP/1.0 assumed that a GET would be sent directly to the correct server (with a relative path). So this relative path is translating into IP.

Bingo!! now we know whenever we get 302 request on HTTP/1.0 , internal IP address is revealed
Points is how to disable HTTP/1.0 requests from server side:

1) You can write your own ISAPI filter/Module to scan incoming headers and reject it if its on HTTP/1.0 protocol
2) Or Use URL Rewrite module on IIS 7

I followed the later approach and created rewrite rule in web.config for Default website location (C:\inetpub\wwwroot )

    <rewrite>
            <rules>
              <rule name="RequestBlockingRule1" patternSyntax="Wildcard" stopProcessing="true">
                <match url="*" />
                    <conditions>
                        <add input="{SERVER_PROTOCOL}" pattern="HTTP/1.0" />
                    </conditions>
                    <action type="AbortRequest" />
                </rule>
            </rules>
    </rewrite>
    </system.webServer>

This rule blocked requests coming on HTTP/1.0 with page cannot be displayed you can modify rule to show error page stating HTTP/1.0 not allowed. :)

Why you need to know this:

To figure out what kind of browser(IE, Firefox, Safari) requests are coming for your application.So that you can make sure your application is compatible with that browser type.

IIS 6 Log

You can put the following into *.bat file and double click to execute.

Srcnt ==> It Returns the number of occurrences of a substring in a string , if not found then 0 is returned

@echo OFF

"C:\Program Files\Log Parser 2.2\logparser.exe" "select case strcnt(cs(user-agent),'Firefox') when 1 THEN 'Firefox' else case strcnt(cs(user-agent),'MSIE+6') when 1 THEN 'IE 6' else case strcnt(cs(user-agent),'MSIE+7') when 1 THEN 'IE 7' else case strcnt(cs(user-agent),'Safari') when 1 THEN 'Safari' else case strcnt(cs(user-agent),'Opera') when 1 THEN 'Opera' ELSE 'Unknown' End End End End End as Browser,count(cs(User-Agent)) as Hits into c:\pie.gif from C:\LogParserBlog\ex.log group by cs(User-Agent)" -chartType:BarStacked -o:chart -values:ON -view:ON

EXIT

Output:

You can modify above query to include more browser type for example Opera

Else case strcnt(cs(user-agent),'Opera') when 1 THEN 'Opera'  END

Hope this helps :)

Lately, have worked on issue where most of threads were hooked up same call stack .

Question pop up in my mind how many of thread are effected in total or having msxml3!CXMLHTTP::send in it.

MS Excel spreadsheet came to rescue.

From windbg get call stack on all threads in text file
.logpen c:\out.txt
~*kb
.logclose

Copy the content and paste in MS Excel. Nest select ALL content (Ctrl + A)

Go to Home -> Conditional Formatting  -> Highlight Cells Rules

Enter the search Text (msxml3!CXMLHTTP::send ), it would be highlighted where ever it exist across sheet

Data tab -> Sort . in this Dialog for Sort On drop down select "Cell Color" and then select Color -> OK

Scroll to top, all thread having the search text would be put together

Now that tells me we have exactly 100 thread running same call stack out of 131 total threads

100/131

Happy Debugging

In the changing .Net world it's difficult to keep track of version corresponding to framework .

Following is compiled list of .Net Version till 3.5 SP-1

.Net Framework 1.0 RTM                          ===> 1.0.3705.0
.Net Framework 1.0 SP-1                          ===> 1.0.3705.209
.Net Framework 1.0 SP-2                          ===> 1.0.3705.288
.Net Framework 1.0 SP-3                          ===> 1.0.3705.6018

1.1
===

.NET Framework 1.1 RTM                             ===> 1.1.4322.573
.NET Framework Service Pack 1                    ===> 1.1.4322.2032
.NET Framework included with Win2k3 SP-1   ===> 1.1.4322.2300

2.0
===

.Net Framework 2.0 RTM                             ===> 2.0.50727.42
.Net Framework 2.0 Service Pack 1               ===> 2.0.50727.1433

MS07-040 security patch was released in July ===> 2.0.50727.832

.Net Framework 2.0 Service Pack 2             ===> 2.0.50727.2407 

(.Net 2.0 SP 2 is *not* available as stand alone download, it is included into .Net 3.5 SP-1
It installs the .NET Framework 2.0 SP2 behind the scenes )

  <Update>

.Net Framework 2.0 SP2 now comes as a standalone installation and not clubbed with 3.5 SP1. You can download the same here.

This is supported on Windows XP and Windows 2003. For Vista and Windows 2008, the users would still have to install 3.5 SP1.

  </Update>

3.0
===

.Net Framework 3.0 RTM                             ===> 3.0.4506.30
.NET Framework 3.0 Service Pack 1              ===> 3.1.21022

3.5
===

.NET Framework 3.5 RTM                             ===> 3.5.21022.08
.NET Framework 3.5 Service Pack 1               ===> 3.5.30729.1

keep it handy