David Wang : HTTP.SYS

HOWTO: Maximize the Number of Concurrent Connections to IIS6

Anonymous — Thu, 13 Apr 2006 09:55:00 GMT

Question:

I'm considering building a realtime ASP.NET application on IIS6, that'll need to support about 3,000 simultaneous users on a server, each with a keep-alive HTTP connection for "server push."

The HTTP connection will need to stay open for the duration of the user's session, to allow updates to be pushed to the browser in near real time.

My question is, is this technically feasible on IIS6, particularly with the large number of connections? If it's possible, are there any special considerations to get it to work?

Thanks in advance.

Answer:

Yup, IIS6 can easily handle 3,000 concurrent keep-alive connections, assuming you:

Have sufficient HW resources (like RAM) on the system
Reconfigure the system (i.e. tweak any applicable limits)
Run scalable user software that handles the user load (i.e. uses asynchronous IO, not necessarily multi-threaded, has appropriate caches, etc)

Windows Server 2003 and IIS6 come configured for security out of the box, whose goal frequently opposes performance/scalability. For example, ability for the system to hold 3,000 connections ties up valuable system resources and can be considered a "security threat" given the right context. Now, one may want to make the tradeoff for functionality... but last I checked, software is neither clairvoyant nor omniscient (well, neither are humans, but that is a separate tangent altogether ;-) )... so one may need to do some tuning.

In your situation, there are a couple of non-obvious limits:

MaxConnections (HTTP.SYS)

Controls the number of simultaneous HTTP connections (and hence limits number of simultaneous connections serviceable by IIS6).

On Windows Server 2003 RTM x86, this comes out to around 8,700
On Windows Server 2003 SP1, the limit has been removed
On Windows Server 2003 SP1 x64, since NPP is bound by available memory, you can increase concurrent connections by merely adding more RAM. To give a sense of scope - I have seen 50K+ concurrent connections to IIS6 on WS03SP1 x64 with 4GB RAM

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters
Type: DWORD
Value: Range from 0 to 2^32-1

MaxUserPort (TCPIP.SYS)

Controls the max port number that TCP can assign. Every unique client making a request to your web server will use up at least one of these ports on the server. Web applications on the server making outbound SQL or SMB connections also use up these ports on the server... so it highly affects the number of concurrent connections. For SMB tuning, read the de-facto IIS6 and UNC Whitepaper.

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Type: DWORD
Value: Range from 5000 to 65536

Conclusion

Incidentally, default value for both are above 3,000, so given sufficient HW resources like RAM and well written application software, IIS6 should just work for you out of the box without any tuning. :-)

//David

IIS6 and HTTP Server API, Part 1

Anonymous — Fri, 07 Apr 2006 14:15:00 GMT

Question:

Hi,

I've found these articles:
http://support.microsoft.com/default.aspx?scid=kb;en-us;q311852
http://support.microsoft.com/default.aspx?scid=kb;en-us;q327611

First of all I've got a question. READ_RAW_DATA can be used to modify the request body not only the headers. Why is it gone? If you don't want to modify its behavior you should add a new notification type (input and output as well) only for modifying the entity body.

But my main question is that this article says that IIS 6 operates quite different in IIS 5 mode.

I section "SSL in IIS 6.0" I've read that in IIS 5 mode IIS 6 hosts SSL and filters can use SF_NOTIFY_READ_RAW_DATA. This can mean one of the two tings:
1. IIS 6 is not using HTTP API at all. In this case a lot of HTTP API based things can be broken, including HTTP API based Windows system services and SQL Server Yukon's web services. If it's the truth is a very strange thing.

2. IIS 6 is using undocumented HTTP API calls that make it possible to process the raw data. In this case you should document these APIs as they could be used by others.

httpapi.dll exports some undocumented functions (without header declarations) that are used only by IIS. But as HTTP API is an operating system service and not part of IIS I think you should document all the functionality of HTTP API.

Sincerely,

Answer:

Thanks for the query. I've actually not seen either of these KBs until now, but I can tell you that the KB is wrong to say that SSL in IIS6 uses the SF_NOTIFY_READ_RAW_DATA filter notification to function. It does not. SSL in IIS6 is not an ISAPI Filter. I'm also going to try to clarify the info in 311852 because I think the usage of the words "READ_RAW_DATA" and "SF_NOTIFY_READ_RAW_DATA" are quite erratic/confusing, and I'm also going to work to get these KBs fixed.

HTTP.SYS, Conceptually...

The crux of what is going on is this (rough conceptual details):

HTTP.SYS basically:

Reads data from the networking layer
Parses it
Does some basic HTTP-level validations according to HTTP/1.1 spec
Routes it to a user mode process for request execution

All of these steps happen in kernel mode.

SSL and ReadRawData, conceptually...

Two concepts muck with this architecture in IIS -- SSL and SF_NOTIFY_READ_RAW_DATA filters. Both concepts require access to the raw data stream prior to it being parsed by HTTP.SYS, meaning that it has to be somehow interjected between steps #1 and #2 above. Since we cannot just execute existing SF_NOTIFY_READ_RAW_DATA filters (which are user mode DLLs) in kernel mode, this means that HTTP.SYS will need to have some "mechanism" to stream raw data from #1 into a user mode process, let it munge the data however it wants, and then return the data to HTTP.SYS in kernel mode for #2 and continue request handling. Needless to say, this is quite a privileged operation.

Thus, both SSL and SF_NOTIFY_READ_RAW_DATA rely on this "mechanism" to stream raw data from kernel mode into user mode and back, prior to step #2. Now, HTTP.SYS happens to only support this "mechanism" for a SINGLE user mode process for various technical reasons (including the fact that host headers are not known [SF_NOTIFY_READ_RAW_DATA could change it as well]).

Therefore, all requests that require SSL and/or SF_NOTIFY_READ_RAW_DATA go through this mechanism -- first, the data is read in kernel mode, then transitioned into user mode to munge/decrypt, then transitioned back into kernel mode to parse, and later shuttled back into user mode to execute. Not very pretty, but it works and is not the default case.

IIS6 Process Model and SSL/ReadRawData

Now, what does this have anything to do with IIS6 modes? Well, in IIS5 Compatibility Mode, the "mechanism" can use inetinfo.exe as the lone user mode process to handle the raw data since both ISAPI Filters and SSL needs to run in that process anyway for compat reasons, and so SF_NOTIFY_READ_RAW_DATA is cleanly supported.

This all breaks down in IIS6 Worker Process Isolation Mode because individual w3wp.exe load their own ISAPI Filters -- so multiple instances of the same Filter DLL can be in memory in separate w3wp.exe -- and what happens if this Filter uses SF_NOTIFY_READ_RAW_DATA ? We now have >1 process wanting the raw data routed to it, but HTTP.SYS can only route to a single process. Now, you may argue that we should have spun up one single surrogate process to host all ISAPI Filters and SSL, but this design destroys Worker Process Isolation Mode -- this surrogate process is once again a single point of failure for all w3wp.exe and is no better than the IIS5 process model (substitute inetinfo.exe for the single surrogate process and you get the same looking picture). All of the other designs at trying to enable SF_NOTIFY_READ_RAW_DATA in IIS6 Worker Process Isolation Mode have other similar but fatal flaws.

The User-Oriented Compromise

So, the whole SF_NOTIFY_READ_RAW_DATA concept just does not work with Worker Process Isolation Mode, and hence we did the next best thing -- disable the broken concept, but give a better alternative that allows 99.9% of people to do what they actually wanted to do and live with breaking that 0.1%:

Let SSL have the lone raw data "mechanism" and host it in lsass.exe to
have only one process involved in decrypting/encrypting data
Disable SF_NOTIFY_READ_RAW_DATA in IIS6 Worker Process Isolation Mode
Make sure that the new IIS6 feature, HSE_REQ_EXEC_URL, can take care of the 99.9% usage case of SF_NOTIFY_READ_RAW_DATA (which is the munge the incoming request, in particular POST entity body from FORMs)

From my perspective, the main users that are left in the cold by this decision are the people who wrote proprietary non-opaque stream filters, such as custom encryption/compression, and want to run in IIS6 Worker Process Isolation Mode. There are no solutions in this case other than IIS5 Compatibility Mode.

Your suggestion of "...If you don't want to modify its behavior you should add a new notification type (input and output as well) only for modifying the entity body." was considered at one point, but we rejected it. We already introduced HSE_REQ_EXEC_URL in ISAPI Extension in IIS6 to handle the 99.9% usage case of modifying the entity body, and ISAPI Filter is not the preferred IIS extension mechanism (ISAPI Extension is far richer and deterministic), so we chose not to add any new filter notifications.

So, if your whole reason for SF_NOTIFY_READ_RAW_DATA is to get access and control of the incoming request, there are better and supported ways of doing this in IIS6. Configure a ISAPI Extension as a Wildcard Application Mapping and use HSE_REQ_EXEC_URL. You will notice that HSE_REQ_EXEC_URL is able to modify the entire request that is processed by IIS, including URL, headers, entity body, IIS impersonation token, and some ServerVariable values (like AUTH_TYPE, AUTH_USER, LOGON_USER, etc), and when configured as a Wildcard Application Mapping, it gets first crack at all the requests and can filter/redirect as it pleases. HSE_REQ_EXEC_URL is even able to consume and otherwise munge the entire entity body without the child URL realizing it, and it is fully compatible with SSL.

This is basically what most people want to do on the server, and IIS6 gives it to you in that manner. We didn't have this in prior versions of IIS, so people have had to hack together Filters using SF_NOTIFY_READ_RAW_DATA to buffer and modify incoming requests (including entity body) -- which doesn't work with SSL, SF_NOTIFY_PREPROC_HEADERS filter and combo ISAPI Extension if you just want to change URL, headers, and fudge some ServerVariable values if you coordinate with SF_NOTIFY_AUTHENTICATION, and so on. HSE_REQ_EXEC_URL has none of those hacky limitations -- you simple change the entire request that gets executed by IIS and then tell IIS to execute it.

You can still do all those hacky things in IIS5 Compatibility Mode (that's what compatibility is for), but we highly discourage it in IIS6 Worker Process Isolation Mode because there are better alternatives. If we didn't have better alternatives, we would have made sure that SF_NOTIFY_READ_RAW_DATA remained the same in all modes.

If you want to treat HTTP logically, such as URL, headers, entity-body, you can either do the hard work of parsing through data stream as with the APIs you're asking about, or you can use the logical API of HSE_REQ_EXEC_URL to modify the exact same thing prior to IIS executing the request. I leave it up to you to decide which is easier and makes more sense.

Conclusion

Personally, I suggest that you don't get hung up on SF_NOTIFY_READ_RAW_DATA nor HTTPAPI -- you're really not "losing" anything important other than a big hairball mess.

As for your statement in #2 about IIS6 using undocumented HTTP API to process the raw data -- I am not a lawyer nor am I authoritative on the subject, but I can tell you that:

The IIS and HTTP.SYS teams spent a lot of time with the lawyers, who grilled us over all of this
Legally, IIS6 is considered a part of Windows Server 2003 OS; thus this is merely a private API between two OS components -- which is perfectly acceptable by the consent decree

//David

HTTP.SYS, IIS, and the 100 continue

Anonymous — Wed, 05 Apr 2006 10:20:00 GMT

Question:

Hi David,

"My Company" is a leading middleware provider for mobile multiplayer games. Cutsomer like Disney, THQ etc.

The backend is built on .NET. We went live in the US with W2K3/IIS6 which is great.

But we have a major IIS6 issue. The handsets connect through HTTP/POST and sometimes the Server answers with "100 continue" and this leads to crashes on certain phones.

In IIS5 we wrote an ISAPI filter - which works well.

Question:

1) Is IIS6 sending the "100 continue". We assume yes ... (due to file must exist)

2) Will an ISAPI Extension be able to fullfill this?

I would really apprciate your help.

thanks,

Answer:

Actually, I would frame it differently - this is probably NOT a "major IIS6 issue".

Clients which advertise to be HTTP/1.1 compliant and then crash on "100 continue" are the real problem (they are not following public specifications), and servers that allow such broken clients are also a part of the problem.

Technically, these phones should just keep crashing until the consumer gets sick of it and switches to another phone that works correctly. This is the way to get the phone manufacturers to write/use properly implemented networking protocol stacks - when their customers hit their pocketbooks. As middleware, it should not differ if the consumer uses one phone or another to run your games... as long as the consumer has *A* phone that runs your games.

On the other hand, if the server keeps hacking to work around client-side bugs, the phone manufacturers never get wind of their problems and have ZERO incentive to fix their phones, leading to accumulation of server-side hacks over time that increase server maintenance costs for you.

Thus, it is in your best interest to notify and get phone manufacturers to fix their buggy software.

I understand that business pressures can force you to compromise and otherwise work-around such issues in the spirit of "making things work", but I just want to remind you of the implications of your actions on your long-term interests.

</soapbox>

On Windows Server 2003, it is a cooperation between IIS6 in user mode and HTTP.SYS in kernel mode that drives HTTP request/response serving. Logically speaking:

HTTP.SYS picks up data off the network, parses it into HTTP requests, identifies which Application Pool it belongs, and places it into its queue.
IIS6 user mode worker process picks up requests from the queue of its Application Pool, processes each by running either a user-supplied ISAPI/CGI or the built-in IIS Static File Handler, and hands the response back to HTTP.SYS to send over the wire.

A "100 continue", like a "400 Bad Request" or a Kernel Response Cache Hit, is special in that HTTP.SYS transparently handles it in kernel mode without notifying user mode of anything. In addition, ISAPI Extensions cannot interact with any response output - they can only generate response output, not see results of response output. Thus, an ISAPI Extension will never be able to interact with requests that generate "100 continue" nor "100 continue" responses themselves to suppress them.

On IIS6, the only way to inject user mode processing into these transparent request handlings of HTTP.SYS is to run in IIS5 Compatibility Mode and use an ReadRawData/SendRawData ISAPI Filter. ReadRawData forces HTTP.SYS to hand the raw data off the network into user mode for filtering PRIOR to parsing that user mode output into HTTP requests to place into queues.

Of course, this method completely defeats the purpose of running IIS6 with Application Pools and process isolation (a single failure in this filtering user mode process halts the entire server)... but such is the server-side compromise when the client is buggy...

FYI: This approach will not work on Vista Server/IIS7. HTTP.SYS will no longer hand raw data off the network into user mode for filtering prior to parsing, so it will be impossible for user mode code to know that a request which triggers the automatic "100 continue" happened.

//David

Does IIS6 truncate HTTP headers at a hard-coded limit?

Anonymous — Sat, 11 Feb 2006 09:50:00 GMT

Question:

Using IIS the HTTP_ACCEPT header server variable received from the client appears to have a hard coded limit of 259 characters. This is causing problems when using a Windows Mobile 5 Smartphone which has a longer string than 259 characters which is getting truncated. The phone then receives a 406 error saying that e.g. HTML content is not acceptable. I can hack the registry on my phone but I would rather there was a fix in IIS to get round this limit so that my site can be used by others with this type of phone. Is there a patch or a hotfix or a service pack level I should know about?

Answer:

I cannot reproduce your claim on either Windows Server 2003 RTM or with Service Pack 1 installed. I think the issue is outside of IIS.

Using WFetch from IIS Diagnostics Toolkit, I am able to successfully send Accept: header of up to ~16K in length (the server-side limit depends on HTTP.SYS settings for MaxRequestBytes [i.e. max size of entire request] and MaxFieldLength [i.e. max size of a given request line, like the Accept: header], both of which default to 16K) as well as retrieve the complete header value using HTTP_ACCEPT in ASP.

Since I accomplished this on a default system, I presume the observed difference is due to non-default configuration of the IIS6 server that you are using. My guess at the issue involves custom ISAPI DLLs running on your server. I tried a couple of popular configurations, like ASP.Net 2.0 ISAPI configured as Wildcard Application mapping as well as my own test ISAPI Filter, and I did not see any issues with large Accept: (or any) header, so I do not believe the ISAPI interface is involved.

Perhaps your managed hosting provider has some 3rd party security ISAPI DLL which intentionally truncates header lengths. For example, ISAPI Filter or ISAPI Extension configured as a Wildcard Application mapping can filter and truncate request headers. Or the intervening network between your client and server imposes a limit on request header length.

At this point, you should probably contact your managed hosting provider regarding their server and network configuration and have them investigate what is going on. Based on my observations and investigation, I do not believe IIS6 truncates HTTP headers like the Accept: header on its own.

//David

HOWTO: Measure Effectiveness of the HTTP.SYS Kernel Mode Response Cache with IIS 6

Anonymous — Tue, 03 Jan 2006 10:03:00 GMT

Hmm... ever since I wrote the referenced blog entry about HTTP.SYS Kernel Response cache and IIS 6.0, I have gotten questions about one thing that I forgot to mention - how to determine the effectiveness of the Kernel Response Cache. Here goes...

Question:

David,

To begin, I happened to come across your blogs on msdn regarding kernel caching http://blogs.msdn.com/david.wang/archive/2005/07/07/HOWTO_Use_Kernel_Response_Cache_with_IIS_6.aspx . It is excellent, but it also gave rise to another question for me:

Do you have a blog or are you planning to blog the details of how to measure the Kernel Cache performance using the Performance Counters for Web Service Cache? I am trying to figure out how I can analyze the performance of non-SP1 systems for window 2003 using perfmon.. Any existing docs/content you can think of would be much appreciated.

Please let me know,

Answer:

The "official" way to determine the effectiveness of the Kernel Response Cache involves PerfMon (aka Performance Monitor). The counters you want to watch are under the "Web Service Cache" Performance Object, and the most useful ones for the Kernel Response Cache are:

Kernel: URI Cache Hits
Kernel: URI Cache Misses

In short, the "Kernel: URI Cache Hits" counter increments every time a request is served from the cache... and "Kernel: URI Cache Misses" measures the other obvious event.

How do these two counters relate to the Cache Insertion policies I mention in this blog entry?

For the IIS Static File Handler, it means that you need to make two requests to the same URL within the ActivityPeriod timespan which pass the HTTP.SYS Cacheability criteria to insert the response into the cache, and the third and subsequent response comes from the cache until flushed via the Cache Eviction policies.

You end up seeing "Kernel: URI Cache Misses" increment twice and "Kernel: URI Cache Hits" increment once for each subsequent request thereafter.

For example, I make three requests within ActivityPeriod to /pagerror.gif and I see two misses initially and one hit on the third request..
For the HSE_REQ_VECTOR_SEND approach, you only need to make the request which causes the ISAPI to invoke HSE_REQ_VECTOR_SEND in the right manner as described in the earlier blog entry, and any subsequent requests with properties that end up matching the value of UNICODE_CACHE_URL of the original cached request will come from the cache until flushed via the Cache Eviction policies.

You end up seeing "Kernel: URI Cache Misses" increment once on the first request, and "Kernel: URI Cache Hits" increment once for each matching subsequent request.

An "unofficial" way that I use to prove to myself that the perf counter actually works and that the response actually came from the kernel response cache, is to attach a debugger onto the w3wp.exe which should service the request in user mode and debugbreak into it. This ensures that this w3wp.exe cannot service the request... so if the response is truly cached in the kernel response cache, the request will succeed and the cache hit counter increments... while if the response was not really cached, the request will simply hang since w3wp.exe is broken into and cannot service the request.

//David

HTTP.SYS rejection of request lines not terminated by CRLF (Redux)

Anonymous — Thu, 13 Oct 2005 09:53:00 GMT

Ok... this is the second day in a row now where I am answering a blog comment where I think the answer already exists in one or more of my blog entries. I am now wondering how to avoid having to make redux posts because I am sure there will be a third, a fourth, etc... hey, I have no problems making posts, but I really do not want to dilute the content/post ratio.

Maybe I need to start a new RSS feed for "previously answered questions" in addition to "Web Q&A" for the first time I answer a question so that readers can discern between news vs repeats. Any thoughts on how to organize/arrange this?

Question:

Hello David,

I found your page regarding http.sys rejection of request lines article very informative. In your Q&A about workarounds, I was curious if any of those entries (or which one) fixes the CRLF problem?

Our evaluation shows that we need to find a fix to the win2003 server for a short term period because of the time, cost and risk in modifying the outdated java client that makes the call.

Thanks!

Answer:

First, let me link back to the referenced original post for the benefit of future readers. Your question would have been a good comment to the original post.

Second, I would like to point out that there is no CRLF problem to fix. HTTP.SYS is simply following the appropriate RFCs to the letter. In other words, just because an application worked before is not an indication that it was correct nor that it will continue to work. In Windows Server 2003, when we had to choose between compatibility or security, we heavily favored security (in the past, compatibility would have reigned supreme). This is just the changing of the times... the software industry is growing up.

Unfortunately, you never specified what your "problem" was, so I cannot really offer a concrete solution. If I had to guess at what your problem was... I believe that if you download the Word DOC file from the link I posted six months ago of SP1 enhancements in HTTP.SYS, you should be able to find an answer to your outdated java client. I believe it is under the section "HTTP API Parser Enhancements".

Now, if you are asking for that "SP1 change" for HTTP.SYS on Windows Server 2003 RTM, without requiring SP1, you will have to make a paid support call to Microsoft PSS to see if that backport is possible.

//David

Kernel Caching in IIS 6 (Redux)

Anonymous — Wed, 12 Oct 2005 07:13:00 GMT

Hmm, I got the following question today, and since I had a blog entry that exactly answered the question already, it made me wonder about the visibility of my blog entries to searching...

Question:

Hi David,

I am sure you have a ton of requests for content on your blog but you never seem to run low on content. However, there is one thing that I have always been curious about and never really seem to find a lot of information about and that is the Kernal Caching that is in IIS 6. Could you maybe blog about this some, how it works how to take advantage of it etc. Improvements in IIS 7

Answer:

Actually, I have blogged about this exact topic in the recent past. Please check out this blog entry, where I focused on how it works and how to take advantage of it from IIS6. Hmm, you do remind me that I have an ISAPI code-sample to finish on how to insert/evict Kernel Response Cache entries via a customizable ISAPI Extension that still sitting on one of my machines that I should release to my Source Code RSS feed... I will update this blog entry with the link when I get that done.

I cannot really make official comments about actual improvements in IIS7, but I assure you that you should like the improvements made in IIS7 over the hacky implementation details of IIS6.

Now, I did observe one interesting detail about how "search" works on Community Server. It seems to be a flat keyword search and the blog entry index seems to run quite infrequently (i.e. I could not locate by keyword any blog entry <30 days old, even if I punched in the exact keyword that is only in that blog entry).

So, you can search for the keywords "kernel cache" from my blog's search and find the correct blog entry, but you can't search for the keywords "session state" and find the recent entry. Hmm, that is horrible... because I tend to have a lot of new content within 30 days, and I often respond to blog comments as posts. Grr...

//David

HOWTO: Diagnose IIS6 failing to accept connections due to Connections_Refused

Anonymous — Thu, 22 Sep 2005 09:55:00 GMT

Recently, I have seen a bunch of questions asking: "IIS6 runs fine for X amount of time [where X varies from days to weeks] and then all of a sudden, it stops accepting all connections. If I restart/reboot the server, then it is fine again... until X amount of time later, when I have to repeat the same procedure."

Sometimes, the user even notices that HTTP Error logs in %SYSTEMROOT%\System32\LogFiles\HTTPERR\*.log mention Connections_Refused for all those requests, but what the heck does that mean?

Ok, ignoring the jokers in the back that are snickering "hmm, this is Microsoft software; what do you expect?" , this issue is actually very diagnosable and should be treatable, as I will shortly show...

Question:

Every four or five days the web site hoste on my machine will stop responding to HTTP requests on port 80.

IIS doesn't crash so I cannot get an IIS State log, just this:

Excerpt from the HTTPErr.Log:

(... working fine up until this ...)
2005-09-02 18:45:12 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:46:07 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:46:27 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:46:42 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:47:02 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:48:22 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:48:57 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:49:02 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:49:42 - - - - - - - - - 7_Connections_Refused -
2005-09-02 18:50:02 - - - - - - - - - 3_Connections_Refused -
2005-09-02 18:50:37 - - - - - - - - - 6_Connections_Refused -
2005-09-02 18:50:42 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:51:02 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:51:47 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:52:02 - - - - - - - - - 3_Connections_Refused -
2005-09-02 18:52:27 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:53:17 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:53:22 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:54:02 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:54:22 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:54:32 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:55:22 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:56:02 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:56:42 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:58:17 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:58:42 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:58:57 - - - - - - - - - 1_Connections_Refused -

I have to reboot the machine to get it to work again.

Does anyone have any ideas?

Answer:

Connections_Refused is actually a very diagnosable failure pattern, and I am going to show you how to diagnose it and what is going on.

What does Connections_Refused Mean?

According to KB 820729, Connections_Refused means that the "Kernel Non Paged Pool memory has dropped below 20MB and HTTP.SYS has stopped receiving new connections". What all this means in English is this:

On Windows NT systems, there are many types of memory with different properties. Kernel Non Paged Pool (NPP) memory is one such type, and the important thing to know is that on x86 machines, the size of this memory is fixed and has a maximum size of 128MB. In other words, you may have an x86 machine with 4GB of RAM, but if you only use 128MB and that 128MB all comes from NPP memory, your system is still "out of memory" regardless of how much is actually in use.
[1/4/2006 - Correction - maximum size on x86 is "around 250MB" and 128MB with /3GB. For Windows Vista, it will be bounded by physical RAM and not arbitrary limit.]
HTTP.SYS, being a kernel-mode driver, uses NPP memory for every single connection that is active, and we made a conscious decision to have it stop accepting connections when NPP memory falls below a threshold, arbitrarily set at 20MB. This is reasonable because lots of bad things can occur if NPP memory is depleted.

For example: HTTP.SYS is likely the most active codebase in a web server, and if NPP memory gets depleted and HTTP.SYS needs NPP memory to accept a connection, the machine will blue screen. Now, regardless of the driver leaking NPP memory, HTTP.SYS looks like the culprit because it is most active. In any case, all of this is not good, so HTTP.SYS proactively refuses to get near the edge and plays it safe.

Hopefully, it is now clear what Connections_Refused means. HTTP.SYS is basically telling you "someone is using up a lot of NPP memory, and for protective reasons, I am going to stop accepting requests". We need to figure out what driver is using up all the NPP memory and address it, and the Connections_Refused should naturally go away because NPP memory will not be under pressure.

How to Diagnose this Issue

One of the nice properties of NPP memory is that a unique tag is associated with every piece of this memory, so we can always track down who is using what and how much of NPP memory. The user-mode tool to do this is called POOLMON.EXE (if you have a Kernel Debugger attached, you can use the !poolused command to get this info within the Kernel Debugger). Here are some additional bits of related information:

KB 177415 - How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks - since Windows Server 2003 has PoolTagging enabled by default, things just work.
KB 298102 - How to Find Pool Tags that are used by Third-Party Drivers - this is basically searching for a NPP tag string within all drivers on the system hoping for a logical match...

I am now going to step through the POOLMON output of a user who had Connections_Refused and comment on what is going on. Here is a sample snippet of running POOLMON.EXE -b :

 Memory: 1048016K Avail:  438396K  PageFlts:   219   InRam Krnl: 3604K P:42372K
 Commit: 538908K Limit:11245092K Peak: 609576K          Pool N:109056K P:43740K
 System pool information
 Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

 ISil Nonp    1408366 (  56)   1327427 (  58)    80939 36678632 (  -872)   453
 I100 Nonp   11048877 ( 217)  10967968 ( 219)    80909 14886928 (  -368)   183
 PcNw Nonp     292427 (   0)     88967 (   0)   203460 12203296 (     0)    59
 SavE Nonp    1596066 (   0)   1595659 (   0)      407 11717016 (     0) 28788
 IAM  Nonp    1946164 (  49)   1904725 (  50)    41439 8519576 (  -208)    205
 tdLL Nonp      74748 (   0)     69830 (   4)     4918 2672832 ( -1776)    543
 LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0) 2576384
 R200 Nonp         22 (   0)         1 (   0)       21 2297816 (     0) 109419
 MmCm Nonp        302 (   0)        48 (   0)      254 2236960 (     0)   8806
 TCPt Nonp     190486 (   4)    190462 (   4)       24 1392264 (     0)  58011
 ULHP Nonp       3127 (   0)        41 (   0)     3086 1047216 (     0)    339
 PTrk Nonp     999750 (   0)    996561 (   0)     3189  956824 (     0)    300
 File Nonp    9628160 (  65)   9623252 (  65)     4908  750688 (     0)    152
 rg81 Nonp       6910 (   0)      5294 (   0)     1616  603864 (     0)    373
 Pool Nonp          6 (   0)         3 (   0)        3  602112 (     0) 200704
 Devi Nonp        680 (   0)       262 (   0)      418  558552 (     0)   1336
 Thre Nonp     417221 (   6)    416457 (   6)      764  476736 (     0)    624
 LSwr Nonp        128 (   0)         0 (   0)      128  416768 (     0)   3256
 Mm   Nonp         26 (   0)         2 (   0)       24  379880 (     0)  15828
 AfdC Nonp      21989 (   0)     19872 (   1)     2117  338720 (  -160)    160
....

Some observations:

Notice that this system has 1GB RAM but still only 128MB NPP memory (109MB has been used, hence tripping the 20MB marker of HTTP.SYS).
The NPP memory tags of HTTP.SYS all start with "UL" (for the trivia-inclined: as with all software projects, HTTP.SYS has gone through some evolution of its own. It used to be called something else - Universal Listener - hence its tag names all started with "UL").
Notice that HTTP.SYS is not even in the top-10 NPP memory usage on the system, and its largest set of allocations is barely using 1MB total. As it turns out on this system, TOTAL NPP memory usage by HTTP.SYS was under 2MB, so clearly, it is not the aggressive user of NPP memory that is now causing issues.
Meanwhile, focus your attention on the top four users of NPP memory, with tags "ISil", "I100", "PcNw", and "SavE". I looked up drivers for those tags, and I only find that "PcNw" comes from Microsoft and is associated with WDM audio. The others are all 3rd party drivers.
Umm, this is a server, so is audio really that important? I checked on my system, and it was at 275KB, so clearly this user is/has done some heavy-duty audio stuff on this server. Thus, the memory may be legitimately used. You know, you might want to reconsider the effects of playing your MP3s on your Web Server's ability to accept connections...
Meanwhile, Isil and I100 have suspiciously similar active allocations and ISIL is the stock ticker for a semiconductor company, so my guess is that I100 is related to a Fast Ethernet card... and since web servers totally rely on the network driver, having a memory-hog for a network driver cannot be a good thing and should be fixed. My guess for SavE is probably anti-virus.
Now, I may be totally wrong with those arbitrary tag names, but one thing is for certain - IIS6 and HTTP.SYS is running just fine on this system. There are some other memory-hungry drivers running on the system, and if you get fixes for them such that your system will stay running longer, IIS6 will likely keep running as well

Note: Don't shoot the messenger (HTTP.SYS)! Always strive to figure out the root of your problem and address that. The rest of the problems should naturally fall away.

//David

Controlling network packet size from IIS/ASP.Net?

Anonymous — Sat, 30 Jul 2005 09:46:00 GMT

I got this question about response packet size control...

Question:

Running a web app - native .NET. 2003 for server. Reports come out as garhugic (66K) packets in testing. Is there a way to control max packet size from within .NET at the program level or within the IIS settings?

Answer:

Actually, IIS6 does not buffer nor parse response data, so no, there is no IIS setting to control, let alone manipulate, network packet size. I am not certain if ASP.Net gives any configuration of response buffer packet size.

However, I do not think you need to control packet size from within any framework. Optimizing packet size for network transmission is a function of the networking layer, not application/framework layer.

ASP.Net 1.1 on Windows Server 2003 uses HSE_REQ_VECTOR_SEND to directly send responses (as well as kernel-response cache them) via HTTP.SYS. In the optimum case, ASP.Net directly sends the response buffers to HTTP.SYS, who fills up network packets according to MTU size and sends them out.

However, HTTP.SYS did not implement efficient network buffering until Windows Server 2003 SP1, so there are a lot of popular, non-optimal cases whose performance suffered tremendously prior to SP1. You may be seeing performance artifacts related to those described in this KB article.

So, I would suggest that you first try this test against Windows Server 2003 SP1. If the SP does not work for you, please feel free to send me a short sample ASPX page (and instructions for content as well as necessary server configuration) which reproduces this issue, and I can try and get it to the right folks to look at it.

//David

HOWTO: Use the HTTP.SYS Kernel Mode Response Cache with IIS 6

Anonymous — Fri, 08 Jul 2005 09:58:00 GMT

Motivation

One of the bigger buzz-word features of IIS 6.0 on Windows Server 2003 is the "HTTP.SYS Kernel Mode Response Cache".

When you do a search against "HTTP.SYS Kernel Response Cache IIS 6", you will inevitably find a large body of literature repeatedly talking about how the kernel mode response cache effectively improves performance and lightens server load by removing the kernel/user mode transitions, improves request/response latency, improves... blah blah blah blah blah... and is the best thing since sliced bread.

Unfortunately, there is not a lot of concentrated, pragmatic information which focuses on the internals of how this cache works so that one can effectively take advantage of this feature. Specially, from within the context of IIS 6.0 and ASP.NET by association since it is an ISAPI Extension DLL.

This lack of consolidated information, along with several newsgroup questions about how the kernel mode response cache works, is what motivated me to write this entry about the background and the raw mechanics of how to leverage this cache from IIS 6.0.

Cache Basics

The best way to understand a cache is by understanding how to:

Place an item into the cache (cache insertion policy).
Remove an item from the cache (cache eviction policy).

Now, how does this translate to IIS 6.0 using HTTP.SYS?

Cache Insertion Policy

There are two ways for IIS 6.0 to insert an item into the HTTP.SYS kernel mode response cache:

The IIS Static File Handler. If you make two requests for the same resource handled by the static file handler within a 10 second contigous interval (configurable via the ActivityPeriod registry key), the resource will be placed into the kernel mode response cache.
ISAPI Extension calls HSE_REQ_VECTOR_SEND to send a complete HTTP response which includes at least the HSE_IO_SEND_HEADERS, HSE_IO_FINAL_SEND, and HSE_IO_CACHE_RESPONSE flags, along with a Last-Modified: and optionally an Expires: response header.

You must realize that just applying the cache insertion policy does NOT ensure that HTTP.SYS actually caches the response. There are many other variables that affect the cacheability of the response, as documented at this URL.

Of course, there are also limits on what can be placed into the kernel response cache. The most interesting variables from an ISAPI perspective are:

Response Size - 256KB by default and controlled by the registry key HKLM\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\UriMaxUriBytes
Scavenger - 120 seconds by default and controlled by the registry key HKLM\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\UriScavengerPeriod

For more info on other registry key tweaks, see:

Personally, I would not tweak with the registry keys unless you know you need to.

Cache Eviction Policy

The HTTP.SYS kernel response cache, as exposed by IIS, has a very simple cache entry eviction policy - whichever of the following criteria triggers first will flush the associated cache entry(s):

TTL (Time-To-Live) - Set by the Expires: header in the response.
HTTP.SYS Scavenger - UriScavengerPeriod controls the frequency with which the scavenger will evict any cache entry that has not been accessed in the last UriScavengerPeriod amount of time, in seconds.
On-Demand - ISAPI Extension can use the HSE_REQ_GET_CACHE_INVALIDATION_CALLBACK ServerSupportFunction to retrieve a pointer to the on-demand URL revocation function, and you call this function pointer with the FULL URL (obtained via the UNICODE_CACHE_URL ServerVariable when you cached that URL's response to begin with) to evict that URL's cached response.

For example, suppose your ISAPI decides to use HSE_REQ_VECTOR_SEND to cache a certain response. If you want to be able to evict it later, using your own cache policy, you want to retrieve and keep the UNICODE_CACHE_URL ServerVariable when you make your HSE_REQ_VECTOR_SEND call. Then, when you want to evict that entry for whatever reason, you call call the cache invalidation callback using that saved UNICODE_CACHE_URL value.

Now, a subtle point with the timing-related eviction policies. HTTP.SYS does NOT immediately evict cache entries when their time is up. Instead, it sweeps through the response cache with a 30 second granularity. What this means is that suppose UriScavengerPeriod is set to 60 seconds. Cache items can between 60 and 89 seconds old, depending on when the cache item was inserted relative to the last periodic sweep.

However, On-Demand invocation is immediate - as soon as you call it, you will get a cache-miss if it used to be a cache-hit.

Conclusion

I hope that my explanation helps clear up the roadmap of how to use this exciting HTTP.SYS feature. If there are any unclear points, feel free to post comments for the benefits of everyone. I will try and follow up with all comments to make this information useful.

Some final observations:

A static file (i.e. .htm, .css, .js, .jpg) is NOT guaranteed to be kernel response cached. You could have an [Wildcard] Application Mapping that applies to those extensions, in which case it depends on the cache insertion policy applied by that Application Mapping (as well as the other cacheability factors that I described earlier) when it handled the request and generated a response.
A dynamic page can be kernel response cached IF the script engine that handled the request uses the ISAPI cache insertion policy mentioned above.

//David

HTTP.SYS rejection of request lines not terminated by CRLF

Anonymous — Thu, 21 Apr 2005 05:35:00 GMT

I recently got the following question about an HTTP.SYS behavior in Windows Server 2003.

Question:

Hi David,

I have a major problem switching to W2K3 and IIS6 currently.

Major handsets in the market (e.g. Nokia) send WRONG HTTP requests - nothing to do against it! Wrong means not seperated with "\r\n".

I've seen your post: http://www.webservertalk.com/archive120-2004-2-132028.html

We are a leading provider for mobile multipayer games - and the only service based on .NET technology. Using IIS6 and W2K3 is important for us.

Is there any solution in the mean time?

greetings from Hamburg/Germany,

Answer:

I hope this illustrates the nasty problem when clients and/or servers do not follow public specifications like HTTP correctly - customers get caught in the middle and usually the server-side has to budge since there are millions more clients than servers to fix. However, realize that budging the server-side simply emboldens the clients to be even more broken in the future, destroying the underlying benefit of a public specification and endangering public communication protocols as we know it.

Now, the behavior being described is the default behavior for HTTP.SYS on W2K3. Due to customer pressure, HTTP.SYS added some features that allowed more relaxed HTTP parsing (i.e. hacks that allowed various broken clients to work). One of them includes allowing request lines to be terminated by only \r or \n instead of the spec-mandated \r\n.

None of these hacks are enabled by default. You must enable them separately yourself. I am inquiring the HTTP.SYS team about any KB articles similar to http://support.microsoft.com/?id=820129 which will describe these new settings and recommendations, and when I get an answer I will post it as follow-up. Stay tuned.

//David

FYI: You can also post questions like this to the microsoft.public.inetserver.iis newsgroup using a standalone news reader or a web news reader like http://www.microsoft.com/communities/default.mspx .