Developers frequently confuse Request Headers, Response Headers, and Server Variables as well as the appropriate syntax to retrieve/manipulate each of them, depending on the API (ISAPI, ASP, and ASP.Net). I am going to clarify all of this right now. :-)

Question:

So I've managed (thanks to your samples) to create a filter that adds a header. I call it SM_USER. When I use ASP to enumerate all the headers I can see it as HTTP_SM_USER in the ALL_HTTP and ALL_RAW header (with a value set).

Unfortunately though when I list out the HTTP_SM_USER variable there is no value for it.

I am setting it in the onPreProcessHeaders with a call sort of like this

fRet = pPPH->SetHeader(pfc, szMyHeader, szMyHeaderValue);

The question is: What's happening here? Why can I see it in the ALL but not when I list out the particular header?

Cheers,

Answer:

Request Headers, Response Headers, and Server Variables represent three totally different logical concepts available to server-side applications (though not all APIs expose these logical concepts), and you manipulate them in different ways.

The main syntactic sugar which exists for legacy compat reasons with CGI is that Server Variables ALSO allow retrieval of certain Request Headers when using the special HTTP_ prefix.

The following is how I treat the logical concepts...

Request Headers

Request Headers represent the header name/value pairs that the HTTP client, such as a web browser, sent to the server. A typical request looks like the following, and request headers are highlighted:

GET / HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n
\r\n

Response Headers

Response Headers represent the header name/value pairs that the HTTP server, such as a web server, sends to the client. A typical response looks like the following, and response headers are highlighted:

HTTP/1.1 200 OK\r\n
Content-Type: text/html\r\n
Server: Microsoft-IIS/6.0\r\n
Date: Thu, 20 Apr 2006 09:00:00 GMT\r\n
My-Response-Header: Dash\r\n
My_Response_Header: Underscore\r\n
Content-Length: 11\r\n
\r\n
Hello World

Server Variables

Server Variables represent runtime state available to the server-side application for request processing. The list of variables and syntax available on IIS is described here. They include state like:

  • SCRIPT_NAME (client-requested URL, such as "/")
  • AUTH_TYPE (type of authentication performed)
  • etc

As stated earlier, it also supports syntactic sugar to access Request Headers via the HTTP_ prefix, in accordance to the CGI 1.1 Specification.

The CGI Specification for Accessing Request Headers

The CGI 1.1 Specification defines the HTTP_ prefix to retrieve Request Headers. Dashes in Header Name transform into Underscores with HTTP_ prefix prepended and make them available as Server Variables (in the CGI context, available as Environment Variables).

In other words, the HTTP_ACCEPT_ENCODING Server Variable name retrieves the Accept-Encoding: Request Header.

The astute reader should notice a quirk between the HTTP and CGI specifications - in particular, how does one retrieve the Accept_Encoding: Request Header in CGI???

Remember, HTTP Request Header names are message-headers, which by definition are (I have collected all the relevant BNF definitions for reference):

       message-header = field-name ":" [ field-value ]
       field-name     = token
       field-value    = *( field-content | LWS )
       field-content  = <the OCTETs making up the field-value
                        and consisting of either *TEXT or combinations
                        of token, separators, and quoted-string>

       token          = 1*<any CHAR except CTLs or separators>
       separators     = "(" | ")" | "<" | ">" | "@"
                      | "," | ";" | ":" | "\" | <">
                      | "/" | "[" | "]" | "?" | "="
                      | "{" | "}" | SP | HT

       OCTET          = <any 8-bit sequence of data>
       CHAR           = <any US-ASCII character (octets 0 - 127)>
       CTL            = <any US-ASCII control character
                        (octets 0 - 31) and DEL (127)>
       SP             = <US-ASCII SP, space (32)>
       HT             = <US-ASCII HT, horizontal-tab (9)>

       LWS            = [CRLF] 1*( SP | HT )
       TEXT           = <any OCTET except CTLs,
                        but including LWS>

According to HTTP specifications, header names can include any CHAR except CTLs or separators. In particular, both "-" (dash) and "_" (underscore) are valid characters in header name.

The problem should be clear now. The CGI 1.1 Specification defines a substitution of "-" in Request Header to "_" in Server Varible names, but what is the substitution of "_" in Request Headers in Server Variable??? It cannot be "_" in Server Variable names, and it is conveniently undefined...

Oops! Did we just spot a flaw in a sacred specification underlying major Internet Applications? You bet. Oh no, heavens forbid that there is a flaw in a publicly implemented specification! The sky is falling!

And you can also bet that it is not the only one in existence... so the next time you think that a peculiar software behavior is due to a software bug and not a specification bug, think again... because believe it or not, even the W3C recommendations are not flawless. Remember, to err is human, and humans write those documents and software.

When troubleshooting, I always recommend to think and evaluate the situation instead of assuming one or another is magically sacred. For example, developers tend to assume that their code is perfect and that the flaws must be from the system.

The Solution in IIS 6.0

Yes, we decided to do something unspecified in IIS 6.0 to resolve this situation. Otherwise, how can you possibly retrieve the SM_USER request header within ASP?

We introduced the HEADER_ prefix for Server Variable names which uses the name as-is to retrieve request headers. In other words:

  • ServerVariable( "HEADER_SM_USER" ) retrieves SM_USER: Request Header
  • ServerVariable( "HEADER_SM-USER" ) retrieves SM-USER: Request Header

It is a neat solution to the problem. Yes, it is unspecified, but I think users are better off with than without it.

Table of Operations

Whew... soapbox aside, the following table categorizes the available syntax to retrieve the three logical concepts of Request Header, Response Header, and Server Variable. Without parsing, of course...

API / Concept Request Header Response Header Server Variable
ISAPI Filter PreprocHeaders:
GetHeader("Header_As-is:")

AuthComplete:
GetHeader("Header_As-is:")

Any event after PreprocHeaders:
GetServerVariable:
HTTP_ prefix
HEADER_ prefix
SendResponse:
GetHeader("Header_As-is:")

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

ISAPI Extension

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

Not Possible

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

ASP

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

Not Possible

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

ASP.Net Request.Headers("Header_As-is")

GetServerVariable:
HTTP_ prefix
HEADER_ prefix
Not Possible

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

The key take-aways:

  • ISAPI Filters can retrieve everything as-is, assuming you are in the right filter event. You just need to remember that header names MUST have the ":" appended, while GetServerVariable() calls using HTTP_ or HEADER_ prefix do NOT have ":"
  • GetServerVariable() calls using HTTP_ or HEADER_ prefix work everywhere (after PreprocHeaders event for ISAPI Filters) to retrieve request headers, especially when using HEADER_ prefix
  • ASP.Net offers the Request.Headers collection to give you parsed access to the Request Headers
  • Notice that ASP and ASP.Net behavior mirror that of ISAPI Extension... because they are implemented as ISAPI Extension DLLs on IIS

ASP Example:

For example, using WFetch, try making the following raw request:

GET /GetServerVariable.asp HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n
\r\n

And have the following source code for /GetServerVariable.asp

<%
headerDash = "My-Request-Header"
headerUnder = "My_Request_Header"
HTTP_headerDash = "HTTP_" & headerDash
HTTP_headerUnder = "HTTP_" & headerUnder
HEADER_headerDash = "HEADER_" & headerDash
HEADER_headerUnder = "HEADER_" & headerUnder

Response.Write( HTTP_headerDash & " = " & Request.ServerVariables( HTTP_headerDash ) & VbCrLf )
Response.Write( HTTP_headerUnder & " = " & Request.ServerVariables( HTTP_headerUnder ) & VbCrLf )
Response.Write( HEADER_headerDash & " = " & Request.ServerVariables( HEADER_headerDash ) & VbCrLf )
Response.Write( HEADER_headerUnder & " = " & Request.ServerVariables( HEADER_headerUnder ) & VbCrLf )
%>

You get the following output:

HTTP_My-Request-Header = Dash\r\n
HTTP_My_Request_Header = Dash\r\n
HEADER_My-Request-Header = Dash\r\n
HEADER_My_Request_Header = Underscore\r\n

Notice that HTTP_ syntax only retrieves the header name with dashes, while HEADER_ syntax retrieves both headers with dashes and underscores individually.

ASP.Net Example

For example, using WFetch, try making the following raw request:

GET /GetServerVariable.aspx HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n
\r\n

Have the following source code for /GetServerVariable.aspx

<%
dim headerDash,headerUnder, HTTP_headerDash, HTTP_headerUnder, HEADER_headerDash, HEADER_headerUnder

headerDash = "My-Request-Header"
headerUnder = "My_Request_Header"
HTTP_headerDash = "HTTP_" & headerDash
HTTP_headerUnder = "HTTP_" & headerUnder
HEADER_headerDash = "HEADER_" & headerDash
HEADER_headerUnder = "HEADER_" & headerUnder

Response.Write( HTTP_headerDash & " = " & Request.ServerVariables( HTTP_headerDash ) & VbCrLf )
Response.Write( HTTP_headerUnder & " = " & Request.ServerVariables( HTTP_headerUnder ) & VbCrLf )
Response.Write( HEADER_headerDash & " = " & Request.ServerVariables( HEADER_headerDash ) & VbCrLf )
Response.Write( HEADER_headerUnder & " = " & Request.ServerVariables( HEADER_headerUnder ) & VbCrLf )
Response.Write( "Header(" & headerDash & ") = " & Request.Headers( headerDash ) & VbCrLf )
Response.Write( "Header(" & headerUnder & ") = " & Request.Headers( headerUnder ) & VbCrLf )
%>

You get the following output:

HTTP_My-Request-Header = \r\n
HTTP_My_Request_Header = Dash\r\n
HEADER_My-Request-Header = \r\n
HEADER_My_Request_Header = \r\n
Header(My-Request-Header) = Dash\r\n
Header(My_Request_Header) = Underscore\r\n

With ASP.Net, the Headers collection is the only way to get all the header values. HTTP_ prefix only retrieves the header name with dashes, as expected.

Conclusion

Whew... a very long winded entry. :-)

But... I wanted to be thorough on the subject of Request Headers, Response Headers, and Server Variables when it comes to ISAPI, ASP, and ASP.Net. So, now you have everything...

//David