HOWTO: Basics of IIS6 Troubleshooting

"NetMon"?

Uwe Keim — Sun, 01 Jan 2006 22:22:21 GMT

When refering to "NetMon", do you mean any special application? I searched Google for "NetMon" and "SysInternals NetMon", but found various unspecific items only.

re: HOWTO: Basics of IIS6 Troubleshooting

David Wang — Mon, 02 Jan 2006 21:45:40 GMT

Uwe - When I say "NetMon" I meant "Network Monitor", which I described how to obtain/install. I'll make the association more clear. It's similar to how "RegMon" is "Registry Monitor".

//David

re: HOWTO: Basics of IIS6 Troubleshooting

Paul Wakeford — Tue, 03 Jan 2006 02:09:35 GMT

Hi David,

A useful post, thanks. One minor typo:

> I suggest using RegMon (aka Network Monitor)

I think you mean NetMon there.

Paul

re: HOWTO: Basics of IIS6 Troubleshooting

David Wang — Tue, 03 Jan 2006 07:21:49 GMT

Paul - Thanks. A typo from copy/paste/edit for the prior comment...

//David

re: HOWTO: Basics of IIS6 Troubleshooting

Holger — Tue, 17 Jan 2006 20:33:30 GMT

I'm looking for an answer to the (silly) question, what does a http status code 200 in the iis website log really mean.

You say: "IIS Website Log contains ... It tells me that IIS Core in user-mode got the request and sent some response (no guarantee the client got the response, of course)."

That I was thinking also, but there are guys arguing "if there is an http status code 200 for a GET is in the IIS logs it means >>guarantee of delivery<< to the client, like a fax-reception-o.k."

I know this looks silly to you, but help me anyway ;-) thanx a lot!!!

re: HOWTO: Basics of IIS6 Troubleshooting

Steven — Wed, 08 Feb 2006 06:53:28 GMT

Hi David:
we have lots of 400 error codes register in IIS logs and HTTP.sys logs. But http.sys logs are not that usefull, as they don't enough information about cs-uri, most of the time "-". So wondering is there way to debug the issue behind this error.
Thanks.

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Wed, 08 Feb 2006 14:57:33 GMT

Steven - 400 errors are called "Bad Request" because the request may not have been well-formed enough to be parsed/understood by HTTP.SYS. Hence, there is nothing to log about these data streams, certainly no cs-uri (parsing is not guaranteed) nor would these requests ever make it to IIS in user mode.

And why wouldn't HTTP.SYS log these 400 unparsed data streams to a log file so that you can debug these errors? Because consider that hackers may use this mechanism to send broken requests and write arbitrary data to your server log files, possibly waiting for some file-format viewer bug to exploit hack your server when you first browse the log file. Just a thought - why would you ever log untrusted and unparsed data verbatim?

Now, HTTP.SYS will usually log a "reason" for the 400 error that indicate what is wrong with the request. These problems indicate client-side errors beyond the means of the server to fix.

Maybe your 400s come from some common malformed requests generated by bad clients. The following discussion may help:
http://blogs.msdn.com/david.wang/archive/2005/04/20/IIS6_HTTP_Parsing.aspx

//David

HOWTO: Microsoft.com OPS on Debugging IIS

David Wang — Sat, 04 Mar 2006 14:31:28 GMT

If you are looking for information on how to troubleshoot a variety of IIS-related issues, the following...

Thoughts on IIS Memory Recycling for 3rd party Applications

David Wang — Tue, 11 Apr 2006 12:40:43 GMT

Sigh... it seems that the Application Health Monitoring features added in IIS6 are merely used by VARs...

re: HOWTO: Basics of IIS6 Troubleshooting

Robert — Tue, 02 May 2006 19:09:23 GMT

As i just noticed, IIS6 will also go all "eek!" on multiple dots within abs_path, too.

"http://server/abc+.+def/ghi.aspx" will work perfectly fine, but
"http://server/abc+..+def/ghi.aspx" will throw some "Bad Request" at me.

RFC 2396 saying
> unreserved = alphanum | mark
> mark = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"
would lead me to thinking "abc+..+efg" was a legal segment, though.

re: HOWTO: Basics of IIS6 Troubleshooting

Ed — Wed, 03 May 2006 01:31:26 GMT

BULLSHIT.

Just give us a normal error log that has the errors in it, like competent developers would do.

Jesus Christ, Apache had better error reporting a decade ago, when it was developed as a hobby by volunteers.

FIX. YOUR. PRODUCT. Don't give us bullshit. Give us an error log.

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Wed, 03 May 2006 01:42:21 GMT

Ed - Troubleshooting is a process, not a magical bullet.

IIS6 already provides error logs (depending on the type), just like Apache. I see nothing to "fix", really, other than introuducing people to what's already there... if you'd only look.

//David

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Wed, 03 May 2006 01:56:27 GMT

Robert - You probably have either:
1. something configured on your IIS6 server
2. between your server and client

that is rejecting those requests because I do not see what you claim.

I am able to make your exact failing requests against my default IIS6 installation and get the correct responses (not 400).

Obviously, just because HTTP.SYS/IIS6 allows a request does not mean other applications running on the server or between the server and client likes the request. The observed behavior there is obviously arbitrary, so you must always be clear on WHAT is handling and rejecting the request because it may not be IIS6 at all.

//David

re: HOWTO: Basics of IIS6 Troubleshooting

Robert — Wed, 03 May 2006 12:57:38 GMT

Hi David.
Following your hints I tried using a new virtual folder on the server in question.
As you suggested, it worked well for defaul..t.htm and defaul&t.htm.

As the original scenario involves an asp.net 1.1 application with rewriting happening within the "begin_request" event (which is supposed to map 'virtual' urls to database-driven content).

Trying to reproduce the behavior using a simpler setup I set up a wildcard-handler "C:\WINNT\Microsoft.NET\Framework\v1.1.4322\aspnet_isapi.dll", and unchecked the "verify if file exists" checkbox.

Then I tried opening "default.htm", "defaul&t.htm" and "defaul..t.htm"

"default.htm" still worked.
"defaul&t.htm" and "defaul..t.htm" caused "Bad request"

So, I conclude, it is not the fault of http.sys or IIS6 at all, but all aspnet_isapi.dll's.

Thank you for pointing me to this direction.

Robert

re: HOWTO: Basics of IIS6 Troubleshooting

Robert — Wed, 03 May 2006 13:26:06 GMT

Hi David,
it's me again. There's one last thin i forgot to mention:
One thing, that might point at IIS6 is that the same application works fine under IIS5 (and the same version of the .net framework). But then it might as well have something to do with some difference between Windows 2000 Server and Windows 2003 I do not know about.
Hopefully I'll finde some clue.
Thank you again.
Robert

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Wed, 03 May 2006 18:22:23 GMT

Robert - You've already proven that your issue is not with IIS6 but something ASP.Net specific.

I would start looking at whether they are the same version and running the exact same httpModules.

WS03 (and WS03SP1) contains updated binaries of .Net Framework 1.1. Verify that you have the same updated binaries on Windows Server 2000.

I suspect that one of the versions of ASP.Net is doing path validation/canonicalization and now deciding to fail your requests.

//David

re: HOWTO: Basics of IIS6 Troubleshooting

Robert — Thu, 04 May 2006 13:20:27 GMT

Hi David,

I guess I finally found out, why it would work on IIS5/Windows 2000, and not on IIS6/Windows 2003.

It had nothing to do with the version of IIS nor the version of Windows, nor the versions of the .Net-Framework (1.4322.2032 / 1.4322.2300).

The different behavior was caused by the registry setting "VerificationCompatibility" under "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET" being set to "1" on the 2000-Server and not on 2003.
(This setting seems to be introduced by kb826437).

I think I'd never found out without you pointing me away from my first suspicions, so thank you again for your patience with me.

Robert

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Thu, 04 May 2006 13:45:18 GMT

Robert - thanks for finding the KB. Actually, it does have to do with the "version" of the .Net Framework since it includes ASP.Net.

I didn't have the exact KB#, but I knew the guys that made the change in ASP.NET, so I was trying to say that during one of the ASP.Net 1.1 SPs a "canonicalization check" was introduced (that resulted in the behavior you observed), and the Registry Key from the KB basically turns that check off (it is on by default for security reasons).

So:
1. if you ran the original ASP.Net 1.1, this check would not be present at all and things would work
2. if you ran ASP.Net 1.1 with the latest service packs, the check would be present and enabled by default so things would fail
3. and if you then add the Registry Key to turn the check off, things would "work" again

//David

re: HOWTO: Basics of IIS6 Troubleshooting

Robert — Fri, 05 May 2006 13:38:49 GMT

Hi David.

I see. Thank you for the explanation.

Regarding the KB-Entry, there's one thing that I'd just like to share though:
I think It'd be nice if the KB-entry explained, what consequences, security-wise, setting the option would have.

It reads a bit like "It won't work, but set this setting, and it will." and one gets to thinking "Where's the catch? If there wasn't any, why would they use a registry-setting instead of changing it alltogether?" But there's no hint as to why leaving this setting alone could possibly wiser.
I think that if the consequence of this setting might be decreased security, one should be informed. And if there is no risk at all, it'd be nice to know, too, so you don't get sleepless nights wondering what evil you might have done by changing this setting.

And a possibility to not disable this check for all the server but to override a "failed" result within one's application would be quite nice to have.

But this shall not bother you, I'm very grateful for the time you voluntarily invested helping me with my problem.

Best wishes
Robert

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Fri, 05 May 2006 16:55:51 GMT

Robert - thanks for the feedback.

Unfortunately, KBs are only scoped to tell you you WHAT and HOW to do something that you've searched for. It does not tell you WHY you want to do something, and that is by-design.

Frankly, the reason KBs do not contain WHY is because its authors are often merely technical writers without in-depth knowledgeable of the issue and cannot articulate WHY - they are just technical writers given a whole bunch of raw technical details and instructed to come up with a KB. Since they rarely have real insight about the issue, they certainly cannot synthesize and provide useful "WHY" knowledge. They end up only able to tell you how to detect something and what to do about it.

This is part of the reason I blog - I essentially get to write my own KB articles on my blog. I locate/organize all the information so that you know WHAT/HOW to do something, and I also try to articulate WHY you want to do something because I deem the deep understanding to be important.

The reason why security switches are "global" is because it does not make sense to partially secure a server on a per-application basis -- the attacker simply tries the same attack on a different application and eventually still hack you -- you gain no security benefit by using partial security

And if I had to choose between full security and false-sense of security, I rather have things break due to security than to work and give you a false-sense of security.

//David

re: HOWTO: Basics of IIS6 Troubleshooting

Robert — Fri, 05 May 2006 19:12:50 GMT

David,

I agree on your point about full security being preferable over a false sense of security - It's just that sometimes things look completely harmless, like "defaul..t.aspx". I see the need for preventing canonicalization issues, but I don't see how "defaul..t.aspx" might cause such.

This is probably the reason why I should leave such things to the .Net people, who know what they are doing, as I generally dislike the idea of being the one responsible when something bad happens to the server; and it's just one, wheras they are responsible for thousands of servers.
Hope they can sleep well.

Robert

re: HOWTO: Basics of IIS6 Troubleshooting

HOVIK GHALOOSTIAN — Mon, 08 May 2006 21:51:06 GMT

HI DAVID

I agree on your point about full security. But I seem to have a little problem when. I log on the internet explorer. It give me a error message that says your Internet explorer has encounted a problem and needs to close. We are sorry for the inconvenience.

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Fri, 12 May 2006 15:19:42 GMT

Robert - Canonicalization issues are by-definition arbitrary and usually hard since paths crossing multiple domains can suffer from such naming issues. I am glad someone else is thinking about it. ;-)

However, I agree that I do not see anything wrong with defaul..t.aspx. It seems that the ASP.Net validation is just checking for presence of ".." and not necessarily "/../"

//David

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Fri, 12 May 2006 15:21:23 GMT

HOVIK - sounds like something is crashing Internet Explorer. Just attach a debugger to iexplore.exe, and on the crash, look at the stack trace, determine what is crashing Internet Explorer, and follow up with the appropriate support party for that software.

//David

re: HOWTO: Basics of IIS6 Troubleshooting

Dan P — Sat, 15 Jul 2006 02:55:36 GMT

Hello David,

My apologies if this has already been asked and lost in the archives, but this post looks like the best place to ask. :)

I am stuck with IIS 5 and have developed an ISAPI extension for it. It works fine on IIS 6 when I add it to the list of allowed extensions, but of course that's not available on 5. I placed it in an appropriate dir and set the execution permissions to "Scripts and Executables". Everything works fine, for a while. Then I start getting 405 errors on the client side, and when I go back to check I find that IIS has magically changed the permissions to "None" for me. I set it back to "Scripts and Executables" and it works fine again for a while. Wash, rinse, repeat. I can find no discernable pattern to it so far.

I've done some pretty extensive web searching and all I've found is one other instance of a guy getting the same issue with some third-party package, and he got totally off the wall, unrelated answers about how to "fix" it.

Do you know of anything that would cause this behavior? Is there anything my ISAPI extension should never do (like returning particular error codes or something) to avoid this?

Thanks.

re: HOWTO: Basics of IIS6 Troubleshooting

Dan P — Sat, 15 Jul 2006 03:28:25 GMT

Sorry for the second post right after the first, but I did eventually figure out the root cause. It looks like ASP.NET automatically sets the 'bin' dir of any application to 'None' the first time it executes something there. I had some ASP.NET DLLs mixed in with ISAPI ones. Moving them out to their own, separate, dir made it work right. Probably more secure anyway.

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Sun, 16 Jul 2006 20:12:48 GMT

Dan P - glad you found the answer - that was what I was going to suggest, but you found it first. :-)

ASP.Net will reset the permissions of /bin directory to "None" on its own for security reasons - .Net assemblies are essentially source code, and having that directory readable is like revealing your source code.

To protect on IIS5/5.1, ASP.Net changes the AccessFlags property because it is LocalSystem. To protect on IIS6, it runs a filtering algorithm because it runs as Network Service which cannot change IIS configuration.

/David

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Sat, 19 Aug 2006 02:53:53 GMT

It's a 10K entry!

//David

re: HOWTO: Basics of IIS6 Troubleshooting

anonymous email — Sat, 07 Oct 2006 15:32:33 GMT

This article is great. I always have to lecture my hosting company when I need error logs. They normally dont give me any information or tell me. "We have found nothing".

Now I know what to tell them next time.

Good Job Dave

re: HOWTO: Basics of IIS6 Troubleshooting

MaryJames — Thu, 24 May 2007 02:14:03 GMT

Hola all

How I can change avatar in this forum?

re: HOWTO: Basics of IIS6 Troubleshooting

nufsaidnow — Wed, 10 Oct 2007 15:08:31 GMT

I'm getting sc-win32-status 121 "The semaphore timeout period has expired." about once a minute in the IIS logs. sc-bytes are 0 and time-taken is about 20-200 seconds.

I can't find any related errors in my application logs and application traces shows me that the code gets executed, but the browser doesn't get any response (I would expect a .

This happens for all types of requests. Images and aspx pages.

I'v looked in http.sys error log but unable to relate any of the errors there.

To a simple developer this would seem like some sort of resource lock. I've searched a bit and it seems this error can be just about anything. Network interfaces having out of date drivers, backup software etc.

Is it possible to use adplus in hang mode to catch the 121, to find out which component/system/application generates it ?

re: HOWTO: Basics of IIS6 Troubleshooting

Kumar — Wed, 07 Nov 2007 21:20:56 GMT

Hi, I am involved in migrating all the web sites from a web server running with IIS 5.0 to another web server with IIS 6.0. For a web site, in the source server it doesnt have a default document but, it displays some links which takes to another web site. I have made copies of those folders and also created a web site which is very much similar to that. But, when I tested the new web site in the target server, it gives the following error: "Directory Listing Denied - This Virtual Directory does not allow contents to be listed". I have checked with the permission for all the sub folders including the Virtual Directory. But, the error is displayed. Please give me a valuable guidance to solve this issue.

re: HOWTO: Basics of IIS6 Troubleshooting

George Novac — Mon, 12 Nov 2007 03:27:03 GMT

Hi David

I have IIS 6 on a Win (SBS) 2K3 domain controller (together with ISA 2000, Exchange 2003 and SQL 2000). On IIS there's only the stuff from Exchange installation (web-mail etc). IIS gives me "Service Unavailable" no matter what I browse from it, "DefaultAppPool" crashes after every "browse" command from IIS Administration Console. Any idea ?

I am desperately trying to put a website on that server (web service goes under port 1234 to avoid interraction with ISA on 80/8080)

Pls help me...

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Sun, 18 Nov 2007 12:45:54 GMT

Kumar - Read this blog entry on how requests are handled:

http://blogs.msdn.com/david.wang/archive/2005/10/14/HOWTO_IIS_6_Request_Processing_Basics_Part_1.aspx

It sounds like your problematic website has additional IIS configuration that you did not migrate over to IIS6. I do not believe your problem has anything to do with permissions of the virtual directory, and the "Directory Listing Denied" error is simply a red-herring that is confusing you.

You say that the source does not have a DefaultDocument yet displays links which goes to another website. There is no IIS functionality that works like that on IIS5 or IIS6. Thus, you are likely describing some custom behavior provided by an ISAPI, and you'll need to determine, install, and configure that ISAPI.

There is no tool to do this automatically because it's arbitrary code.

//David

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Sun, 18 Nov 2007 12:55:35 GMT

George - Is your SBS installation the "Premium" edition of SBS2003, or your proprietary combination of software? If it is your proprietary combination, then do you have support statements from Microsoft saying your combination is possible?

If your combination is allowed, then please contact Microsoft PSS for support.

If your combination is not possible, then please read this blog entry to find the blog URL on how to diagnose your "503 Service Unavailable" issue.

Otherwise, you should expect to pay someone to consult on your issues.

I am happy to tell and teach people on how to do things, but if you want me to diagnose and do the work, then I must be compensated.

//David

re: HOWTO: Basics of IIS6 Troubleshooting

mehdi haeri — Sun, 27 Jan 2008 10:28:56 GMT

The Problem IS In Host Header Value In IIS .

Delete It If You Fill A Value

it's Work fine

;)

re: HOWTO: Basics of IIS6 Troubleshooting

Rob Blass — Wed, 20 Feb 2008 22:14:59 GMT

This is a bit dated. Just ran into this. George, your first and foremost problem is running IIS on a domain controller. DC's have alot of integration complexity with DNS and of course security. If you have no other option but to run in a domain than so be it. We now host over 30 websites on our workgroup. Initially we tried to do DC's to provide tighter security. But the trade off was too grand with complexity. Do yourself a favor, leave domain stuff on DC's and IIS servers on as either member servers or workgroup servers.

re: HOWTO: Basics of IIS6 Troubleshooting

handan — Wed, 21 May 2008 06:00:31 GMT

iis can't start,

service no response or control request!

hwo to doing?

re: HOWTO: Basics of IIS6 Troubleshooting

Ronniebatch — Thu, 19 Jun 2008 14:53:46 GMT

When going into a folder on my usb hard drive it a message comes up saying 'Windows Explorer has encountered a problem and needs to close. We are sorry for this inconvenience'. It is ok in other folders

re: HOWTO: Basics of IIS6 Troubleshooting

David.Wang — Sun, 22 Jun 2008 10:48:42 GMT

Ronniebatch - it sounds like you have installed some application which added folder handlers to Explorer (for example, WinZip), and that handler is crashing Explorer on that folder on your USB hard drive.

You'll have to figure out which buggy application extension you installed and get rid of it (or fix it). It may even be a bogus extension left over from a previously installed folder extension.

In any case, this sounds like a misconfiguration of your computer and not a problem with Windows Explorer. For example, many people, including myself, have no problems going into a folder on our USB Hard Drives. So the problem has to be specific to some software you've installed/uninstalled on your computer, which can only be fixed if you determine the broken folder extension and getting rid of it.

//David

re: HOWTO: Basics of IIS6 Troubleshooting

Thierry Beirens — Tue, 12 Aug 2008 23:23:40 GMT

Hello,

here my problem.

We are running a webfarm in an NLB configuration , unicast mode.

The farm has 3 servers windows 2003 standard edition SP1.

Webfarm works flawlessly asp and asp.net 1.1 and 2.0 sites.

Now, we want to add a 4-th server in the farm.

We configured it exactly identical to the 3 other machines in the farm. We tested all webapplications, and all seems normal.

Now, and here is the problem, we upgraded this 4-th machine to SP2, with the idea of testing out the SP2 upgrade on this machine BEFORE we upgraded are 3 other servers in the farm.

AT first, during the test, all seems normal. But then , after a few minutes , the asp pages in the sites are getting TCP errors. Asp.net pages keep on running. What is even more strange is taht we do not see any error message whatsoever in the logfiles, eventviewer, , http.sys errorlog, none nothing!

I can get the applications stable by reducing the nr of worker procesees in the application pool of the site to 1 worker process. Since this server has 4 CPU's, i configured it to having 4 worker processes (as i did on the other 3 server, because they to are or biprocessor machines or 4 way cpu machines).

Any ideas on how i should troubelshoot this? Or any ideas on possible issues with SP2 regarding this matter?

Thanks,

Thierry

re: HOWTO: Basics of IIS6 Troubleshooting

Susan Chen — Fri, 22 Aug 2008 14:49:13 GMT

Hi David,

I need your II6 Guru advice - the W3SVC service won't start - I have tried uninstall / re-install the Web components but still no joy - IISADMIN, SMTP ..etc all start - I get error 87 parameter incorrect -is there a way to completely clean up the W3SVC so I can install clean again e.g. del some registry and system32\inetserv ..etc

Thanks in advance

Susan

re: HOWTO: Basics of IIS6 Troubleshooting

EL KLOON — Tue, 13 Jan 2009 20:42:21 GMT

well thats page dont want to go ahead

it has some problem with Bad Request (Invalid Hostname)

so i need how to fix it

thanks

re: HOWTO: Basics of IIS6 Troubleshooting

ekreger — Thu, 26 Feb 2009 00:13:01 GMT

Hello. I have run debug diagnostics and am a little confused by its output.

[2/25/2009 2:02:23 PM] Thread created. New thread system id - 4728

[2/25/2009 2:02:23 PM] Thread created. New thread system id - 4016

[2/25/2009 2:02:23 PM] First chance exception - 0xc0000005 caused by thread with system id 2448

This is just a sample but from what I understand, First Chance Excpetions are not necessarily a problem. However.... Should I be seeing this many so often on a site that is not being used hardly at all?