HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Jeff Parker — Thu, 20 Apr 2006 16:58:47 GMT

Well I don't know what to say. Here I thought I knew a heck of a lot about the response, request and server variables. I mean I have been doing web dev for about 13 years now. I have read the specs, never noticed the flaw, I just kind of accepted the specs as they have been around for years.

I love it and hate it when do this to us. Love it cause I really learn something new and it makes me think. Hate it because now I will be spending the rest of the day churning over code in my head I have written for several years and processing this info and writing my own samples to play with my new found knowledge.

However good post. I learned a heck of a lot now I just need to digest it all.

Interesting Finds

Jason Haley — Fri, 21 Apr 2006 13:28:48 GMT

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Phylyp — Fri, 21 Apr 2006 14:42:58 GMT

<trivia>
The HTTP_SM_USER header is used by Netegrity SiteMinder, for holding a single sign-on user ID.
</trivia>

In a previous project, we ran into problems where the - to _ replacement was causing a bit of a pain, in our ASP code that needed to parse the user's SSO.

However, our ever-friendly SiteMinder team changed the header to HTTP_SMUSER, thus alleviating our woes.

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Fri, 21 Apr 2006 15:38:25 GMT

Jeff - hehe. My job is done, then. ;-)

I actually started writing more code snippets in ISAPI, ASP, and ASP.Net before I decided to tone it down and just focus on the basics. I'll leave those details to another time.

//David

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Fri, 21 Apr 2006 15:41:25 GMT

Phylyp - well... now you know why all this is going on and how to account for it in various web framework technologies that run on IIS. :-)

I know you all think we are crazy sometimes, but we are not that random. ;-)

//David

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Maurits — Sat, 22 Apr 2006 01:49:47 GMT

... I don't see the ambiguity.

OK, Some-Header: and Some_Header: both map to the same environment variable. Fine. That just means the header-to-environment mapping isn't one-to-one. We knew that anyway... environment variables aren't case-sensitive, for example.

So as far as CGI is concerned, the following are all the same:

Some-Header: a
Some_Header: a
sOmE-HeAdEr: a

Isn't this only a problem when you have two headers, both important to your app, that differ only in case or "- vs. _"?

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Sat, 22 Apr 2006 04:06:46 GMT

Maurits - The classic problem here is when you are an ISAPI, like ASP or ASP.Net, and want to retrieve a Request Header with "_" (underscore) using Server Variable... because you cannot.

For example, SM_USER used by SiteMinder. You will have to parse the ALL_RAW Server Variable to get it in ASP. On IIS6, you can finally use ServerVariable( "HEADER_SM_USER" ) and get it declaratively, without parsing.

//David

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Maurits — Sun, 23 Apr 2006 13:48:18 GMT

This much is clear:

* On IIS, HTTP_SM_USER does not catch the value of SM_User:

You imply that * is a weakness in the CGI spec. In particular, you claim that there is no CGI means to access the value of SM_User: via server/environment variables.

I disagree. I think there is a CGI means, and HTTP_SM_USER is precisely it.

Therefore I consider * a bug in IIS.
See this post:
http://channel9.msdn.com/ShowPost.aspx?PostID=184522
And this repro/analysis:
http://www.geocities.com/mvaneerde/sm_user.txt

Note at the end of the analysis that there is at least one third-party CGI ISAPI that correctly pulls SM_User: from HTTP_SM_USER.

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Maurits — Sun, 23 Apr 2006 14:54:39 GMT

It seems this is a known issue:
http://support.microsoft.com/default.aspx?kbid=294217

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Mon, 24 Apr 2006 13:15:11 GMT

Maurits - Sorry, but you still have not provided any proof that this is an issue with IIS.

You should know by now that just because it is a KB does not mean it is a bug in a Microsoft product... the Microsoft KB system is quite broken, in my opinion - if you only knew how it really works... you'd be surprised that useful information comes from it at all. It's part of the reason that I blog - because I want the KB to go away.

All it means is that someone called in due to some confusion, wanted to do something but couldn't, and someone in PSS wrote a KB article explaining it. The person writing the KB article usually NEVER consults the product team on whether it is right or wrong - they usually just assume the product is wrong and move on. They are paid to create that KB, not whether it is correct. This is why for most interesting keywords, you see DOZENS of KB articles that run the entire gamut of clarity, conciseness, or correctness.

Yes, a major case of left hand not knowing what the right hand is doing.

Anyhow, back to the discussion at hand. I can vent about the MS KB system at another time when they really piss me off. :-)

The argument is very clear in my mind: show me:
1. a CGI 1.1 EXE (not ISAPI) that can
2. correctly and individually retrieve header values for BOTH SM_USER and SM-USER headers
3. from the same request
4. using JUST the HTTP_ prefix syntax

and I will file a bug for IIS7 to get it fixed.

ISAPI can work around the HTTP_ specification flaw by parsing ALL_RAW or ALL_HTTP server variables for headers. That is not interesting to me - the specification flaw is what we are discussing.

Bottom line: the CGI specification is flawed as soon as it gave a mapping that is not bijective. Period. End of discussion.

BTW: None of the "solutions" offered at geocities.com will work for a request with both SM_USER and SM-USER request headers. They work if EITHER SM_USER or SM-USER are given. And since we are nitpicking over specifications, it is the mere possibility, not practicality (I can't see applications intentionally using SM_USER and SM-USER as different things) that matters. Meaning if one gives code snippets, it most likely missed the boat.

//David

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Maurits — Mon, 24 Apr 2006 20:00:27 GMT

> show me:
> 1. a CGI 1.1 EXE (not ISAPI) that can
> 2. correctly and individually retrieve header values for BOTH SM_USER and SM-USER headers
> 3. from the same request
> 4. using JUST the HTTP_ prefix syntax

You have very clearly illustrated the non-one-to-one-ness of the CGI mapping.

This is NOT the bug:

-- REQUEST --
SM_User: foo
SM-User: bar

-- CGI ENVIRONMENT --
HTTP_SM_USER: ? // cannot be both foo and bar -- THIS IS NOT A BUG IN IIS but a FUNDAMENTAL result of the CGI mapping

IIS is free to do what it likes here. CGI does not specify what to do if two headers are redundant.

This, on the other hand, IS the bug:

-- REQUEST --
SM_User: foo

-- CGI ENVIRONMENT --
HTTP_SM_USER: foo // well, the expected result is "foo"; IIS gives "". THIS IS A BUG IN IIS.

Other CGI implementations do not have this bug, and CAN sniff the SM_User header through HTTP_SM_USER -- provided that it is not masked by a SM-User header.

The bug, in particular, lies in ISAPI's GetServerVariable(...)

This blog doesn't allow indentation, so I've posted pseudocode here:
http://www.geocities.com/mvaneerde/sm_user2.txt

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Maurits — Mon, 24 Apr 2006 21:34:51 GMT

There's a bug in the KB article, too. It says "Retrieve and parse the HTTP_ALL header variable" instead of "Retrieve and parse the ALL_HTTP header variable" -- I'll report that through the appropriate method.

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Tue, 25 Apr 2006 13:21:16 GMT

Maurits - I think we are merely arguing whether the glass is half empty or half full.

One can argue that the CGI specification says that HTTP_SM_USER should retrieve SM_USER or SM-USER, whichever one exists, and if both exist, the behavior is unspecified.

Or HTTP_SM_USER only retrieves SM-User, there is no way to retrieve SM_USER, and there is no unspecified behavior.

Yes, the specification did not say "you cannot retrieve headers containing '_'", so you may be inclined to say that IIS interpretation is too narrow. However, it also did not say "you cannot retrieve headers containing CTRL characters", either... so it is a matter of whether one wants unspecified or underspecified behavior.

Neither option is more "correct" than other, so until the CGI specification clarifies the confusion, there is no way for one to say "this is a bug in X". This is the problem with an underspecified specification - things get too open for interpretation. ;-)

//David

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Maurits — Tue, 25 Apr 2006 22:40:19 GMT

FWIW, here's the CGI/1.1 RFC:
http://www.ietf.org/rfc/rfc3875.txt
...
4.1.18. Protocol-Specific Meta-Variables
...
The HTTP header field name is converted to upper case, has all occurrences of "-" replaced with "_" and has "HTTP_" prepended to give the meta-variable name

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Wed, 26 Apr 2006 12:53:52 GMT

Maurits - Good information...

But, this is not a specification... it's more like an "oh, we did it this way; we think it should be codified".

And it still does not deal with what happens with retrieving header values for requests containing both SM_USER and SM-USER headers. tsk tsk.

And it is dated after all versions of IIS under discussion, so kinda moot point except for maybe IIS7, but it's not an endorsed specification, so it is hard to rally behind it

//David

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Maurits — Wed, 26 Apr 2006 18:40:38 GMT

... and both of the authors work for Apache ;)

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Maurits — Wed, 26 Apr 2006 18:58:56 GMT

There's also this Internet-Draft from 1999, by IBM and E*Trade...
http://cgi-spec.golux.com/draft-coar-cgi-v11-03-clean.html#6.1.5
Each HTTP header field name is converted to upper case, has all occurrences of "-" replaced with "_", and has "HTTP_" prepended to form the metavariable name.
...
If multiple header fields with the same field-name are received then the server MUST rewrite them as though they had been received as a single header field having the same semantics before being represented in a metavariable.
</quote>

That last sentence is wonderfully vague :)

This draft hasn't had a great deal of work done on it recently, though :(

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Wed, 26 Apr 2006 22:29:46 GMT

Maurits - hehe... yeah. It's like patting yourself on the back and writing the spec *after* the implementation. ;-)

The vague sentence is dealing with what happens when the browser sends multiple SM_USER headers on the same request (i.e. HTTP spec says to merge their values into one value, comma-delimited)... which is also wonderful information to have in a specification... but sigh, it still ignores the SM_USER and SM-USER ambiguity.

I wonder if people just copy/paste the original sentences of the spec and then just augment/embelish the parts they are interested in...

//David

HOWTO: Run Console Applications from IIS6 on Windows Server 2003, Part 2

David Wang — Fri, 28 Apr 2006 23:18:13 GMT

I finally have enough blog entries about various portions of IIS6 request processing that I can stitch...

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

gaetano giunta — Mon, 08 May 2006 18:02:39 GMT

Sorry, but I think that the spec being vague and broken is a very poor excuse in this case.

As has been stated before, the spec misses the case of two different http headers that will result in the same converted value.

It has no ambiguity whatsoever oh the handling of 'SM_USER': zero, nil, zilch. Anybody doing his best for implementing it, should treat it as valid. And even if the spec had been twice as shady and broken, the main goal for everybody is interoperability, isn't it?

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Fri, 12 May 2006 14:36:16 GMT

gaetano - Good point, but I hope you agree that vague/broken spec is the very bane against interoperability.

As a general rule, when you have communication between two parties, miscommunication and misinterpretations are bound to happen, and having vague/broken specifications (which basically set the foundation of common understanding) does NOT help.

We can argue about relative correctness of miscommunication all day long, but at the end of the day, I think it is the root vagueness that matters and NOT the various instance-manifestations of that vagueness. If you solve the root problem (mapping of headers to server variables), the instance-problems naturally solve themselves. Those are sound design principles.

Now, I caution you against mistaking miscommunication for "not doing his/her best for implementing it". Who says that we are not doing our best?

I mean, you should agree that EVEN IF everyone is doing their best-effort for supporting interoperability, any vagueness in the specification that causes miscommunication destroys interoperability... and you somehow want to imply that someone did NOT do their best effort? Give me a break. That logic is down-right arrogant and insulting.

My statement is that mistakes/ambiguity in specifications happen more than you think, and they introduce mis-communication which can affect interoperability while implying NOTHING about the effort put forth by the implementors.

//David

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Jim Thorstad — Tue, 23 May 2006 00:39:38 GMT

David, I've found your efforts to explain the situation to be very useful. Our company has struggled to support customers doing single sign-on between our product and Netegrity SiteMinder because of the fact that the header containing the SiteMinder Id (by default, without customizations) is either sm_user, or sm-user and we see them (I think correctly) as different.

Thus our customers get frustrated and I had to put a debug page into our product just to show our customers that its not *our* fault they spelled the header wrong.

I've also done some research to educate myself why Tomcat shows headers in one case on WebSphere on Unix shows them in another case (only to later find in the spec that case is not relevant). Your thread here was helpful as you explain why the ASP sample code I downloaded that uses HTTP_ might be contributing to our confusion.

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Arun Chaudhary — Mon, 07 Aug 2006 18:22:34 GMT

Hi,
I am very new in ISAPI but i have my very urgent requiremnet that i have to set a cutsom varibale and then again need to fetch it on next page or any other page like

suppose first i wll create a custom server valriable like "username" and on in other page i will just check that it have any value or not
like
if request.servervariable("http_username")<>"" then
some processing
end if

as im new so please if you explain me step by step how i can create ISAPI as also i dont know much more abour ISAPI please give me full step that how i can craete it and then again how i can access it in my asp page.

Thanks :
Arun Chaudhary

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

David.Wang — Tue, 08 Aug 2006 14:42:15 GMT

Arun - I suggest using Session Variables to accomplish what you want.

Using ISAPI to set custom variables will not do what you describe because such variables do not "carry over" between pages.

I suggest searching the web for basic documents on how web technologies work.

//David

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Ravichandran — Tue, 12 Sep 2006 20:43:15 GMT

I am trying to retrieve an header value and empty it before it is written to the IIS log. Based on the code example given I am able to access the header. But how to modify it. Primarily we are having issue with Site mind assertions being logged in the IIS and we do not want to have it logged.

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Vir — Wed, 30 May 2007 11:32:44 GMT

How to Retreive Request Headers using VB.net.can any one knows the synatx for that.if yes please provide with a sample code for that

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

naveen — Tue, 25 Sep 2007 14:11:41 GMT

How to forward the request headers when we want redirect the request.

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Frankel Lipshitz — Fri, 25 Apr 2008 23:09:51 GMT

Duuuuuuuude.... this was really helpful! Thanks, man!

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Gopi — Tue, 02 Sep 2008 13:00:30 GMT

This is an excellent post, I am surprised to see the IIS behavior and understood now. Parsed all_http to solve my problem. Thanks a lot David. Thanks!

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Paul — Wed, 20 May 2009 21:33:21 GMT

Great information David! Quick question for you. I see that you can set a value for a header in an ISAPI filter using "SetHeader". How can you do this using an ISPI Extension? I've got an existing extension that we're using to block SQL injection. We want to modify it to prevent XSS using header variables like the HTTP_ACCEPT_LANGUAGE header.

re: HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

Paul — Wed, 20 May 2009 21:33:21 GMT