You can analyze the responses sent from a webserver to determine the version of IIS running (And in turn the version of the Windows Server).

I have seen a lot of people discuss this. So here is my guide ...

The easiest way is to capture a network trace or Fiddler trace and analyze the Server header.

IIS by default sends a Server header indicating the version of IIS that is running

Server

In the above Fiddler trace you see that the IIS version running is IIS 7.5

IIS is tightly coupled with the version of the Windows Server running and based on the version of IIS you can identify the Server OS version.

IIS Version Windows Server Version
IIS 5.0 Windows 2000
IIS 5.1 Windows XP
IIS 6.0 Windows 2003
IIS 7.0 Windows 2008, Windows Vista
IIS 7.5 Windows 2008 R2, Windows 7

Administrators usually install ISAPI filters like URLScan that remove the Server header. Apart from the above values you might also see the following values being sent for the Server header.

Server Header Value Windows Server Version
Microsoft-HTTPAPI/2.0 Windows 2003 Sp2, Windows 7, Windows 2008, Windows 2008 R2
Microsoft-HTTPAPI/1.0 Windows 2003

When you see a Server header value containing Microsoft-HTTPAPI instead of the IIS version it means the HTTP.SYS driver handled the request and it did not traverse all the way to the IIS user mode process. For example a request that results in a HTTP 400 status. Hence the version of the HTTPAPI on the server is sent in the Server header. Also since the response is now being generated at the HTTP.SYS layer ISAPI filters like URLScan cannot intervene. 

IIS servers usually have the .NET Framework installed. The .NET Framework adds a custom header called X-Powered-By at the global level in IIS.

So by default any website hosted on an IIS server with the .NET Framework installed will have a header X-Powered-By: ASP.NET sent in the response.

Another header that the ASP.NET engine sends (depending on the enableVersionHeader setting) is the X-AspNet-Version header. It indicates the version of the .NET framework that is being used to execute the current website.

Another trick is to analyze the error pages that the server sends. IIS has a set of static html pages that it sends to the client incase of an error. IIS 6.0 uses the files located at C:\WINDOWS\Help\iisHelp\common and in IIS 7+ the files are located at C:\inetpub\custerr. For example when you try to browse a link that is not available you will get a 404 error page. You can analyze the content of the 404 response to figure out the version of IIS.

Administrators can use a feature in IIS that allows them to customize the error pages. ASP.NET also has a feature of custom error pages. You can configure the customErrors element in the web.config so that custom error pages will be sent to the client. You also have to ability to set a generic error page (defaultRedirect) so that for any error (which is not customized) the client will be taken to this page.

The special case of 401.2. If your website is set for any of the authentication schemes like Basic/Windows or Digest an authentication handshake has to happen. When you browse a website that requires authentication for the first time your browser does not know that the website requires authentication. Hence it sends an anonymous GET request to the webserver. The webserver rejects the anonymous request with a 401.2 status code and using the www-authenticate response headers sends the authentication schemes it supports. The browser uses this to generate the required credentials package(either by prompting you or using the logged in users credentials) and sends another request to the server.

In doing so the webserver sends a 401.2 response that is not customized. So even if you have a defaultRedirect IIS sends an uncustomized 401.2 error page. Also you cannot customize the 401.2 page using the IIS Custom Errors feature because in that case IIS will return a 200 status code and that will not help during authentication.

A lot of security scans identify the Server header being sent by IIS and recommend blocking it. A common question that I get is how to block this header so that the identity of the web server is not revealed. But personally I think it is over hyped.