Care, Share and Grow! : Case Study

Case Study: Much ado about Browser's HTTP connection

Saurabh Singh — Fri, 06 Jun 2008 21:48:00 GMT

Here I will be going in-depth on how a browser maintains HTTP connections with the Server. Something which a Web Developer should be aware of. My main aim is to show some of the tidbits of how a connection is made, how it is reset (from client or server) and some of the connection limitations.

Let's say you have a web application set to run with restricted access. Now assuming that you do not want client browser to renegotiate user's credentials every time it accesses a web resource like .css, .js, .aspx etc, you would like to have a seamless one-time authentication challenge when the user visits the page the first time. This is achieved by IE appending the HTTP header 'Connection: Keep-alive' along with the HTTP request to the server. Note that at times a single page may include tens (sometimes hundreds) of images, each initiating a separate request to the server. If the server is set to honor HTTP Keep-Alive's, a client can maintain an open connection with the server, rather than re-opening the client connection every time with each new request. This will boost up the overall performance of the Web application.

So when a client, say IE requests a web page for the first time, it opens a new connection from some random client port to the HTTP port (By default it is port 80) on the IIS (or any) web server. Now a days most modern browsers like IE/Firefox ask the server to keep the connection open across multiple requests. This is what is HTTP Keep-Alive is all about. If we do not have it in the request header, a browser will make multiple connections for various resources. Let's say you have a Web page that has some graphics and you also need to download some JavaScript/CSS files etc. All these are independent components which have to be downloaded from the web server and depending upon the Keep-Alive header, may have to be downloaded to the client on multiple connections which will need re-authentication every time.

Remember, whenever IE opens a new connection to request a web resource it will be challenged by the server (if it is for a restricted access) and it has to be re-authenticated. This is costly and redundant if all the resources are coming from the same host.

In IIS web server you can enable/disable this option as shown below:

Let's look at the network traffic and see how it looks.

Here's how Client sends a web request. Note the HTTP Keep-alive in the request header.

Depending upon whether your web site is designed to provide anonymous access or restricted access, here's how the server responds back. Since in our case IIS Web site is set to restricted access it will send back the response as 401 the first time, with the list of supported Authentication methods.

The client sends back the requested credentials/tokens (depending upon the authentication method), on the same connection this time. Note, had the Keep-Alive not been supported by the IIS web site, it would have sent back a response header ('Connection =close') and would have closed the connection with the client as shown below.

If IIS has Keep-Alives enabled it will not send the 'Connection: close' header in its response leading to the same connection being used for subsequent requests (until there are other factors coming in which I will talk about shortly). Also notice that since the same connection is being used for subsequent requests by the client, the source and destination ports will remain the same.

Client's HTTP Web request:

Server's HTTP Web response:

Here I am not going to delve deeper into how IE request is authenticated by IIS. You can refer to this KB for more information on this topic.

Now any related requests corresponding to the same web page (like embedded Image, css files) etc are made on the same connection by the browser. Notice the ports and the Connection header for further requests on the same connection.

Let's see another trace wherein HTTP Keep-Alive is disabled in the IIS web site.

Client sends a request (as usual with 'Connection =Keep-Alive' in the request header):

Server responds back with a 401 the first time since it's a restricted access. Forget the status code for now.

Notice that this time IIS responds back with a 'Connection: Close' header in its response. So this means that for further requests IE has to initiate a new connection, which means a new source port will be used this time.

Here it is. Notice the source port has changed from 60122 to 60123.

Client sends another request:

Server responds back with 200 status code this time. Notice the 'Connection =close' again this time in the server's response.

So next request for any resource by the browser will again be on a new connection.

Remember an IE browser can open a maximum of two concurrent connections (by default) to a single host. Hence if you have lot of images etc that have to be downloaded as part of the web page, you will see them coming slowly two at a time. What I mean is that an IE client will open two simultaneous connections to the same host, let's say www.abc.com and request for resources associated with the page. This is as per the HTTP specifications. IE will wait until the two connections are released or if they are reset for any reason for any further requests. Check this out.

Firefox by default also opens two simultaneous connections to the same host. You can increase the concurrent connection limit for IE as per this KB. Also check this for tweaking Firefox.

It's the same reason why you may not be able to make multiple AJAX calls from the same client. You can have as many objects open and waiting as you wish, but the browser will only work on two at a time while the others wait for one of the two active requests to finish.

**In IE8, considering the strong increase in popularity of AJAX applications the number of connections per host has been increased to 6 from 2 for HTTP 1.1 server for broadband connections. Check this specification.

What will happen if we have let's say IFRAMES in a Web page. Let's thinks of such a scenario:

Let's say I have an HTML page which uses IFRAME's and each IFRAME makes a call to some host for a resource. And each of these web requests are long-running. It may be a long executing task at the server end or downloading of large image files etc. Now since we now know that IE has a default limitation of 2 concurrent connections to a host (web site) by default, let's see how will the response be.

You will notice that responses from the IFRAMES come two at a time if they are made to the same host. Every individual IFRAME request takes up one connection of the browser. Basically once two connections are already made to the same host, IE has reached its concurrent connection limit for that host, and hence other IFRAME requests will have to wait if they are being made to the same host as the other two connections.

If on the other hand you have multiple IFRAMES and not more than two are requesting resources from the same host, all of them will open new connections simultaneously to their hosts and get the response accordingly depending on other factors like request execution time/network congestion etc.

*Another thing that one needs to keep in mind is that IE6 will reset an existing connection if it received a 404 as a response from the server. IE7 (going forward IE8..) will not show the above behavior.

What I meant can be better understood by analyzing the HTTP traces below:

An IE Client sends a request:

Server responds back with a 404, assuming request is authenticated this time but resource '/Graphics/main.css' is not found.

Now what should happen? Should the same connection be used for further requests? No

IE initiates a fresh connection to the server for the next request (here '/StyleSheet.css') after server has returned 404. This happens even though HTTP 1.1 is being used. Notice the source port is changed this time. New port means a new connection.

What happens is that initially IE6's WinInet socket connection attempts to download the missing file. Server returns a 404 as a response. But due to the nature of WinInet design it will not immediately return the socket back to the keep-alive pool for about 2 minutes. Hence this will spur a new connection attempt on a new socket/port.

WinInet does not reuse the port by default after a 404 if the socket is in a TIME_WAIT state. TIME_WAIT means the port is still half-open. One side hasn't completely terminated the connection. WinInet in IE6 returning used sockets back to the keepalive pool is less than optimal, and it can take up to 2 minutes for a released socket to go back to the pool (and this can cause 401s, client hangs due to max connections reached, etc). WinInet in IE7 went through a rewrite, and this section of the code was cleaned up quite a bit, causing the behavior difference between IE6 and IE7.

So far we had been seeing how the client behaves when it comes to initiating/persisting a connection. what about the server?

When a browser establishes a connection with the server, the connection is open as long as the transaction is going on between them. Once the client has received what it needed, it can close the connection or leave it open for further requests. Most browsers ask the server to keep the connections open using their 'Keep-alive' feature. Now since we know that a connection is a costly affair, it doesn't make sense to keep it idle for a long time. Here in, a Server can pitch in and reset the connection from its end if the connection has remained idle for a long time. This is set at the server side by using the property called "Connection timeout" of the Web site.

Here is how the network trace will look like:

After ~120 seconds idle time (as in the pic above), server will send a TCP RESET to close the inactive connection.

So you see not all reset from the Server side is bad, it helps ensure the server does not get into any DoS attack situation. There is always someone to clean up the litters ;-)

L8Rz,

>SS

Case Study: Host header, IP and Port combinations within IIS

Saurabh Singh — Thu, 02 Aug 2007 03:33:00 GMT

Talking about IIS website configurations when it comes to IP Address, Port, Host headers cause a lot of confusion to many Web Administrators. Here I will be going (a bit of unnecessary details though) to explain how the configuration resolves to Websites when a web request comes to IIS.

Environment

IIS6.0

I have multiple IP addresses for my demo server (assuming you have multiple NIC cards).

Here is a list:

10.0.1.1

10.0.1.2

10.0.1.3

10.0.1.4

Here is a screenshot of how IIS manager section looks like which I will talk about later.

In the IIS manager console you may have the following scenarios:

Scenario 1:

There are multiple Websites (for our example we will take 2 here) running with following configuration:

Website                   Host Header Value                  IP Address                         Port
================================================================================
Test1                                --                                All Unassigned                        80
Test2                                --                                All Unassigned                        80

Scenario 2:

Website                   Host Header Value                  IP Address                         Port
================================================================================
Test1                                --                                10.0.1.2                        80
Test2                                --                                10.0.1.2                        80

Will this Websites run? No, only one of them can be in started mode. Rest of them will be in stopped state.
Reason: For HTTP sites one cannot have multiple sites running with the same IP/Port/HostHeader combination. For HTTPS sites you cannot have the same IP/Port combination. More specifically you cannot have the same port/IP combination for the “SecureBindings” setting (SSL settings). Technically, you could have an entirely different IP address for the SSL portion of a web site. If you mess up the SSL part, the HTTP part of the site will still start.

If you try to start any of the other Websites you will get an error popup like this:

Scenario 3:

Website                   Host Header Value                  IP Address                         Port
================================================================================
Test1                                --                                     10.0.1.1                             80
Test2                                --                                All Unassigned                        80

Test3 -- 10.0.1.3 80

Test4 -- 10.0.1.4 80

Will these Websites start: Yes. All of the Websites will be up and running. Remember you can have only one website which can have "All Unassigned" and port XX (when there are no host headers in picture and you have all the sites running on same port XX). No two website can have exactly the same combination of Host header, IP Address and Port.

Here remember you can access the site Test1 only with the IP address 10.0.1.1. If you try to use http://localhost (or http://127.0.0.1) , it will go to some other site (discussed soon). Now using http://servername is a bit tricky. Request will actually go to the site to which your servername resolves from the user's machine. If you do a ping servername and if the IP that it resolves to is set for a specific site like Test3, Test3 will serve the page. If http://servername doesn't resolve to any of the IP addresses on which some sites are specifically configured to listen to, the request will be served by a site which is listening on "All Unassigned" (I know it may be confusing but I am giving an example before coming to theory). If none of the Websites are listening on the resolved IP or "All unassigned" then you will see a "Bad Request (Invalid Hostname)" in the browser (You may check the httperr log to confirm the same).

Let's take for our example that ping servername resolves to 10.0.1.2

So seeing the Websites configuration of Scenario 3 above, which site will should serve the page for the following URLs:

http://servername (remember servername resolves to 10.0.1.2) ? Ans: Test2

http://10.0.1.2 ? Ans: Test2

http://10.0.1.3 ? Ans: Test3

http://10.0.1.4 ? Ans: Test4

http://10.0.1.1 ? Ans: Test1

http://localhost ? (Remember it refers to 127.0.0.1) Ans: Test2

I assume you are getting the logic.

So here comes the explanation:

IIS by default listens on all the IP addresses on the server on a specific port (By default 80).

If you run the command netstat -ano from the command prompt you will see an entry like 0.0.0.0:80. This means that IIS process listens on all IP addresses associated with the server on port 80. Now when you associate a website with a specific IP address and port combination it will listen only on that IP address on that port. It won't accept any request on any other IP address or port associated with the server. When you use "All Unassigned" for a website it means that this website listens on all the IP addresses associated with the server on the specified port. Hence as you may assume from above scenario any web request that is sent to the server on a specific IP address will be served by the website following the above rule...let me clear it again with the above example:

http://servername (remember servername resolves to 10.0.1.2) ?

Ans: Test2 , Reason: When the request reaches IIS server, http.sys driver redirects the request to website Test2 because all other Websites are listening on specific IP's which do not match 10.0.1.2.

http://10.0.1.2 ?

Ans: Test2, Reason: It will be sent to and served by Test2 because this is the website which is set to "All unassigned" which means Website is listening on all IP addresses associated with the server. Also this request won't be served by any other site because they are listening on IP's which are different from 10.0.1.2.

http://10.0.1.3 ?

Ans: Test3 , Reason: As expected from explanations so far this will be served by Test3 because Test3 is configured to listen to any request directed on IP 10.0.1.3.

http://10.0.1.4 ?

Ans: Test4 , Reason: Again the same reason as above, Test4 is configured to listen on IP 10.0.1.4.

http://10.0.1.1 ?

Ans: Test1 , Reason: Same logic as above.

http://localhost ? (Remember it refers to 127.0.0.1)

Ans: Test2, Reason: Served by Website Test2, because the request is sent on IP 127.0.0.1 and none of the other Websites are listening on 127.0.0.1. Test2 has "All unassigned" which means it is listening on all IPs on the server including 127.0.0.1 and hence can serve the request. If for testing you change the IP configuration for a website to 127.0.0.1 then all new requests like http://localhost will be served by this specific website.

So the above scenario shows how every request will be served by some site provided we have the configuration like "All unassigned" or specific IP for the sites.

Now let's assume what will happen if we have this scenario:

Scenario 4:

Website                   Host Header Value                  IP Address                         Port
================================================================================
Test1                                --                                     10.0.1.1                             80
Test2                                --                                     10.0.1.2                             80

Test3 (Stopped) -- 10.0.1.3 80

Test4 -- 10.0.1.4 80

Here we do not have any site configured with "All unassigned". Now if you try to access http://localhost or http://10.0.1.3 what should happen?

You should see "Bad Request (Invalid Hostname) in the browser. This happens because now none of the Websites are capable of serving your requests http://localhost (127.0.0.1) or http://10.0.1.3

We have just 3 Websites (running) and they are listening on IPs 10.0.1.1, 10.0.1.2 and 10.0.1.4. Had there been a website listening on "All Unassigned" these requests would have been served by this website.

I am pretty sure you might have got rid of some of your doubts by now :) If not it's my bad.

-------------------------------------------------------------------------------------------------------------------------------------------------------

So far I had been talking about scenarios where Host header was not in picture.

Now let's assume you have host headers as well in above scenarios. Let's first see what's the use of a Host header.

IIS identifies Websites on the basis of 3 parameters: IP address, Port and Host header (if present).

Any combination of the above parameters should be able to uniquely identify a website. If there is an ambiguity IIS will throw error.

A host header is of the form www.abcd.com or abcd or abcd.com or abcd.co.in etc....

To create and host multiple Web sites, you must configure a unique identity for each site on the server. To assign a unique identity, distinguish each Web site with at least one of three unique identifiers: a host header name, an IP address, or a TCP port number.

One method for providing each site with a unique identifier is to use IIS Manager to assign multiple host header names.

Typically Host headers are set in IIS manager and its mapping to an IP is set in the DNS entry or hosts file (I assume you know hosts file is a limited simulation of DNS for a client). Users access the sites using their host headers.

A “Host” header is one header out of multiple headers that are included in an HTTP request. The headers are things like User-Agent, Accept, If-Modified-Since, Content-Length, Host, etc. As a request comes in, IIS looks at one particular header… “Host” and uses its value to decide which web site to route the request. “Host Headers” are not an IIS technology. They are an HTTP technology. Browsers typically put into the “Host” header whatever the user put into the “Address” line of the browser. If you put “http://192.25.109.11/Default.aspx” into the address line then the browser will extract the “host name” from that address and put “192.25.109.11” as the value for the “Host” header. Or maybe there is no header at all named “Host” (which makes it an HTTP/0.9 request).

Here is a screenshot of an http request for http://test1.com

and this one for http://10.0.1.1

Scenario 5:

Website                  Host Header Value                  IP Address                         Port
================================================================================
Test1                        test1.com                               10.0.1.1                             80
Test2                        test2.com                               10.0.1.1                             80

and in the DNS or hosts file you have mapping like:

10.0.1.1 test1.com

10.0.1.1 test2.com

Will these two Websites run simultaneously? Yes they will.

They will run even when they have same IP and same port because now IIS has a way of distinguishing the Websites using a 3rd parameter called host header. So it knows that when a request comes for http://test1.com and http://test2.com it can direct them to the right Websites on the basis of host headers present in the request headers.

When you access a site with host header the request is resolved to the website corresponding to DNS/Hosts mapping for the host header and not the IP address mentioned in the IIS manager. This is IMPORTANT. The request then reaches the IIS server and goes to the website which is listening on that resolved IP.

Let's see here.

DNS/Hosts setting:

10.0.1.1 test1.com 10.0.1.3 test3.com

10.0.1.2 test2.com 10.0.1.4 test4.com

Scenario 6:

Website                   Host Header Value                  IP Address                         Port
================================================================================
Test1                          test1.com                              10.0.1.1                             80
Test2                          test2.com                              10.0.1.2                             80

Test3 test3.com 10.0.1.3 80

Test4 test4.com 10.0.1.4 80

Now when you try to browse to http://test3.com which site should serve the page from the above IIS configuration? Obviously Test3, now this is simple:). what happens if you try to access any of these Websites with their IP addresses? You will see "Bad Request (Invalid Hostname)". So use judiciously as to how users should access the site. If you have host header set use it instead of IP address in the URL.

If you really want to use an IP address along with an alias (a friendly name) for a website (like http://abcd.com, or http://wxyz ) , then just don’t assign the host header value in the IIS manager. And have an entry in DNS or hosts file mapping abcd.com or wxyz to the IP address for the website (there are other ways to but this neat).

Scenario 7:

Website                   Host Header Value                  IP Address                         Port
================================================================================
Test1                          test1.com                              10.0.1.1                             80
Test2                          test2.com                              10.0.1.2                             80

Test3 test3.com 10.0.1.1 80

Test4 test4.com 10.0.1.4 80

Which site should serve the content when accessed with http://test3.com in the URL..............?

"Bad Request (Invalid Hostname)" in the browser. Reason being that http://test3.com resolves to IP 10.0.1.3 through DNS/Hosts and then in IIS mmc it is configured to listen on IP 10.0.1.1. Rest of the Websites work fine because both the DNS/host entry and IP in IIS mmc for the Websites match.

Scenario 8:

Website                   Host Header Value                  IP Address                         Port
================================================================================
Test1                          test1.com                              10.0.1.1                             80
Test2                          test2.com                              10.0.1.2                             80

Test3 test3.com 10.0.1.1 80

Test4 test4.com All unassigned 80

What site should http://test4.com resolve to..............? Test4, because it is listening on All Unassigned (which means all the IP addresses on the server) which includes DNS/hosts mapping for the site (here 10.0.1.4).

Similarly if we have

Website Host Header Value IP Address Port
================================================================================

Test3 test3.com All Unassigned 80

It should still work fine for http://test4.com and also http://test3.com

Let us consider another scenario

If you have multiple entries for a website in the IIS manager->Web Site->Advanced, you may have different host headers but should have same IP (and/or All unassigned) and TCP Port.

If you use a different port for the entries you need to add the port entry after the host header like: http://testX:port.

If you use a different IP address for the 2nd host header you may again see the blunt "Bad Request (Invalid Hostname)" error when you use this host header to access the website.

So all these explanations and pain for one reason: Avoid setting a specific IP address for the Websites in IIS mmc to avoid any conflicts or confusion with the DNS/Hosts mapping. Instead use "All unassigned" and if really required to use an IP make sure that IP address in IIS mmc and DNS/Hosts entry is same. There is no benefit to “over configuring” the IP and host header settings. It is “unnecessary administrative overhead”. For any site that don’t use SSL, assign one or more host headers and leave the IP at "All unassigned". For anything with SSL, give it a specific IP and leave the host header field blank. Leave port alone because that confuses most end users.

Hope this helps!