A HTTP Detective Story by Eric Lawrence

re: A HTTP Detective Story by Eric Lawrence

Chris Beach — Sat, 12 Mar 2005 17:13:00 GMT

Nice story -- good to see another anti-IE argument debunked.

Also good to see regular posts to this blog. Looking forward to hearing more about the features of IE7.

re: A HTTP Detective Story by Eric Lawrence

irve — Sat, 12 Mar 2005 17:20:00 GMT

Good work, this is the sort of article that I expect to read here. No hazyness of upcoming things and nice clear adventure with even some morale to it. Now, if someone tried to do the same sort of article about transparent png - it's haze all over again. Stick to the present and past and suddenly everything clicks. Good read.

re: A HTTP Detective Story by Eric Lawrence

Patrick — Sat, 12 Mar 2005 17:31:00 GMT

Great read, thanks!

It's scary what some web developers do in the name of "security".

browser detection sucks

porneL — Sat, 12 Mar 2005 17:38:00 GMT

Gazzilions of sites (or rather uneducated webmasters) use browser detection ;(

These are biggest cause of "bugs" in Firefox and Opera. I didn't suspect that it can occur to IE too :)

Maybe IE7 will have to have User-Agent like:

"Mozilla/4 (MSIE 6.0) Opera/7, but really MSIE/7.0 (not Gecko or KHTML)"

;)

re: A HTTP Detective Story by Eric Lawrence

JD — Sat, 12 Mar 2005 17:44:00 GMT

Great read! :)

Btw, what would you do with Developers who has designed application in a way that a 7 year old can *hack* it. Read http://blogs.law.harvard.edu/philg/2005/03/08

JD

re: A HTTP Detective Story by Eric Lawrence

kip — Sat, 12 Mar 2005 17:53:00 GMT

Thanks for this! What's great is that even though I'm no developer, it reads as a detective story. Keep up this kind of writing!

re: A HTTP Detective Story by Eric Lawrence

AlliXSenoS — Sat, 12 Mar 2005 18:17:00 GMT

This is a great article, it's definitely a *must-read* for any wannabe web developer. haha... brought tears to my eyes (good ol' days of hacking through a friend's web-based game with Proxomitron)

Siti che sfruttano bug di Internet Exploer per funzionare... se i bug vengono fissati... non vanno pi

Lorenzo Barbieri @ UGIblogs! — Sat, 12 Mar 2005 20:33:00 GMT

re: A HTTP Detective Story by Eric Lawrence

Jeremy — Sat, 12 Mar 2005 20:17:00 GMT

Cool!

re: A HTTP Detective Story by Eric Lawrence

Mike Gale — Sat, 12 Mar 2005 21:03:00 GMT

A great article. Thoroughness and engaging the brain instead of the mouth win the day!!

Thank you!

Brion Vibber — Sat, 12 Mar 2005 21:18:00 GMT

Fun story... and *thank you* for informing the developer instead of <a href="http://blogs.msdn.com/ie/archive/2005/02/01/364581.aspx">taking out the bug fix</a>! :)

re: A HTTP Detective Story by Eric Lawrence

Jon Grieve — Sat, 12 Mar 2005 21:43:00 GMT

Great story -- just the kind of content I was hoping to see on this blog. Keep it up!

re: A HTTP Detective Story by Eric Lawrence

Lachlan Hunt — Sun, 13 Mar 2005 00:08:00 GMT

It's good to see you chose standards compliance with HTTP over bugwards compatibility, and it's also good to see the web developer fixed the problem so quickly and easily after being notified. I wonder how many other broken sites could be fixed so easily, if the bugs they depended on in IE were fixed like this one was.

LiveHTTPheaders

Tar — Sun, 13 Mar 2005 00:13:00 GMT

LiveHTTPheaders extension for Mozilla is not such a resource hog:
http://livehttpheaders.mozdev.org/

That Fiddler has some nice features though.

re: A HTTP Detective Story by Eric Lawrence

Brant Gurganus — Sun, 13 Mar 2005 00:40:00 GMT

That's a great story showing the scientific method in action to solve problems.

re: A HTTP Detective Story by Eric Lawrence

Michael Dale — Sun, 13 Mar 2005 05:39:00 GMT

Great post.

When you think about it, this is a great example of why IE should STOP supporting quirks mode. Not that it is a security issue or anything. But some stuff in IE6 (and IE5 etc etc) is broken and web developers should know this.

Backwards compatibility isn't always a good idea.

Keep up the good work though :)

*is hoping IE7 will support standards*

A HTTP Detective Story

Virtuelvis — Sun, 13 Mar 2005 10:25:00 GMT

url=http-detective
via1="Anne van Kesteren: HREF":http://annevankesteren.nl/archives/href/2005/03

re: A HTTP Detective Story by Eric Lawrence

Tim Dawson — Sun, 13 Mar 2005 12:32:00 GMT

I don't understand why the web developer hadn't debugged this himself. He would have found that the reason the logon was failing was the lack of that header, and maybe before contacting Microsoft PSS he would have investigated and found that the RFC said it shouldn't even been there. He would then (hopefully) arrived at the conclusion that MS stopped IE from including that header as a bugfix.

re: A HTTP Detective Story by Eric Lawrence

Markus — Sun, 13 Mar 2005 14:53:00 GMT

It's Firefox, not FireFox.

OdeToCode Links For March 13

OdeToCode Links — Sun, 13 Mar 2005 17:50:00 GMT

re: A HTTP Detective Story by Eric Lawrence

Dave — Sun, 13 Mar 2005 15:53:00 GMT

Best post on this blog ever!

Netscape used to employ an evangelism team where they'd contact webmasters whose sites would not work properly in Netscape or other Gecko browsers and help them get their sites working. It'd be great if Microsoft had a team of people doing what you just did to help web developers write to the standards and then you could trim a lot more of this backwards compatibility.

The fiddler tool sounds very powerful, I find the Live HTTP headers extension in Firefox invaluable and fiddler sounds like it's even better.

I've been spending most of my time in Linux or a Mac lately but will check out that tool next time I'm in Windows.

Well done on your efforts to get to the bottom of this, if it was me doing the investigating I'd have come to the conclusion that the webmaster didn't like IE after it worked when changing the user agent.

re: A HTTP Detective Story by Eric Lawrence

Jonathan — Sun, 13 Mar 2005 17:18:00 GMT

Dave: keep in mind that its a lot more than just web developers who depend on the backwards compatibility. There are probably thousands of programs out there that embed the WebBrowser control or MSHTML and would have potential compatibility issues.

As I've suggested before, MS could keep IE6 components exactly as they are now, and use Side-by-Side components for all future versions, such that apps use exactly the version they need. This would enable them to make radical changes to IE without having these huge compatibility issues.

re: A HTTP Detective Story by Eric Lawrence

ralph — Sun, 13 Mar 2005 18:00:00 GMT

"Wow, that’s really silly. Rather than developing a real security mechanism, they’re using a crusty old bug "

Sure, but what's even sillier: apparently, you are not even surprised or sorry that the bug has been out there for such a long time (Win98 or even earlier? And YOU only were able to fix that in XP SP2??? Geeez, what have you been doing all the time?)

Given these facts, I smell more than just a little bigotry when Microsoft comes along now and points at RFCs and tells all the (good or bad does not matter here) web developers how silly they are after all the years that they have had to live with all those stupid bugs in your engine / browser - you certainly must be kidding if this is all their fault now. Develop your software standards-compliant from the very start and this won't happen, simple as that.

Even the worst web developer would develop standard-compliant sites if he had to. Microsoft, in contrast, has introduced proprietary hacks (NTLM and many more), tolerated stupid HTML and even offered authoring tools for stupid, non-standard HTML (Frontpage). So who are you to point your finger?

re: A HTTP Detective Story by Eric Lawrence

And Clover — Sun, 13 Mar 2005 18:00:00 GMT

Heh. Fascinating and hilarious. I'm sure we've all met horrific hacks like that one!

Fiddler looks really fantastic BTW (although it's a minor shame .NET 1.1 is required).

I am personally sick to death of User-Agent sniffing/faking, and would personally donate UKP100 to Microsoft to simultaneous sabotage and fix it, by making the UA string for IE7 simply "IE/7 (Win32)". :-)

re: A HTTP Detective Story by Eric Lawrence

Ken Kolano — Sun, 13 Mar 2005 18:39:00 GMT

Here here ralph.

Whats worse is when MS claims they can't fix the bugs because too many people use them or have implemented hacks to get around them.

re: A HTTP Detective Story by Eric Lawrence

Will — Sun, 13 Mar 2005 21:47:00 GMT

<<you certainly must be kidding if this is all their fault now.>>

Ralph-- I think you're a bit confused. Sure, Microsoft had a bug... But trying to exploit that bug to build a login mechanism is just dumb.

The website ~must~ have known this behavior was a bug because they disabled this dubious security speedbump for Firefox and Opera.

And then to go so far as to claim there's some mysterious "cookie flaw" in XPSP2 and advise people not to upgrade to SP2 because of it is just nasty.

re: A HTTP Detective Story by Eric Lawrence

Ross — Mon, 14 Mar 2005 07:44:00 GMT

Even though the other browsers still do it wrongly I am glad to see you guys not eating up the Microsoft 'it has to be backward compatible' cake when something needs to be fixed.

re: A HTTP Detective Story by Eric Lawrence

Jim — Mon, 14 Mar 2005 07:55:00 GMT

Nice article, although I echo Tim's sentiment that the web developers should have been perfectly capable of fixing this themselves (and that they shouldn't have been doing this in the first place!).

Please refer to RFC 2606, sections 2 and 3 though, please. You don't own the site.com domain, and example.com etc are created especially for this type of situation. You have broken links in your post - any decent weblog system should be able to avoid auto-linking example URIs like this.

Making up domains for examples instead of using the ones reserved for such purposes only makes it more difficult for automated tools to pick up broken links, both for you and for the owners of the domain you are abusing, since it introduces unwanted noise.

re: A HTTP Detective Story by Eric Lawrence

Robin — Mon, 14 Mar 2005 09:05:00 GMT

Oh, so I should remove the section of my code that checks for IE and serves it text/html instead of application/xhtml+xml? I mean, if I'm not to trust header data :)

re: A HTTP Detective Story by Eric Lawrence

Dave — Mon, 14 Mar 2005 12:06:00 GMT

And Clover:
> I am personally sick to death of User-Agent
> sniffing/faking, and would personally donate UKP100 to
> Microsoft to simultaneous sabotage and fix it, by
> making the UA string for IE7 simply "IE/7 (Win32)".

Of course Microsoft were the ones that seem to have invented deliberately confusing user-agent strings.

The original Netscape UA was like follows:

Mozilla/version (OS; I)

Mozilla was the codename of Netscape.

Then IE decided it wanted to work with all sites that allowed Netscape so they became

Mozilla/x.x (compatible; MSIE version; OS; etc)

The Opera had to work with sites that only allowed IE so it defaulted to
Mozilla/4.0 (compatible; MSIE x.x; OS; etc) Opera/version

Although opera allows the spoofing to be turned off.

Firefox doesn't pretend to be any other browser but it has one of the longest user agents that details the version of Firefox, the build date and the version of Gecko it's based off.

re: A HTTP Detective Story by Eric Lawrence

peterp — Mon, 14 Mar 2005 12:31:00 GMT

Boys and girls ... If you want similair functionality to "fiddler" in Firefox check out: http://livehttpheaders.mozdev.org/

Don't want to be rude by posting this in an IE blog, I'm patiently waiting for IE7 ... IE6 incompatability is HUGE pain, I don't develop in it and thought other people don't aswell.

re: A HTTP Detective Story by Eric Lawrence

EricLaw — Mon, 14 Mar 2005 16:03:00 GMT

>I mean, if I'm not to trust header data :)

I assume you're just teasing, but do note that I was saying that trusting header data ~coming from the client~ when making security decisions is a bad idea.

----
Interesting note on RFC2606. Thanks.
----
Vis-a-vis LiveHTTPHeaders-- Yes, it's an cool little tool, although I haven't encountered anyone who has used both LiveHTTPHeaders and Fiddler that considers their functionality that similar. The original blog posting for Fiddler has a discussion about the additional power in Fiddler.

There are a number of LiveHTTPHeader-like plugins for IE, including ieHTTPHeaders.

re: A HTTP Detective Story by Eric Lawrence

Nektar — Mon, 14 Mar 2005 18:41:00 GMT

"I have the full source code of Internet Explorer on my computer".
Very good! How come you avoid being hacked? Having the IE source code and browsing the dangerous Internet trying out even older IE versions on the same machine. Aren't you afraid? There is always something unpatched in IE, even in the latest version, so one day that source code might be stolen. Don't you protect against this? I thought that the code was safely stored on a special server disconnected from the Internet, not on every developer's machine.

Why one, and not the other?

creativeHavoc — Mon, 14 Mar 2005 19:13:00 GMT

Its great that IE isn't worried about backwards compatability when it comes to things like security. I do wish, however, that this mentality would transfer over to its rendering engine and standards. At some point I would like to see IE say "Ok, well it is time to move forward - sorry all you ancient programs"

re: A HTTP Detective Story by Eric Lawrence

Douglas — Mon, 14 Mar 2005 22:40:00 GMT

> Oh, so I should remove the section of my code
> that checks for IE and serves it text/html
> instead of application/xhtml+xml? I mean, if
> I'm not to trust header data :)

Yes, you should. Use content negotiation instead.

re: A HTTP Detective Story by Eric Lawrence

Jim — Tue, 15 Mar 2005 04:16:00 GMT

> Use content negotiation instead.

Actually, you still need to check for Internet Explorer when doing conneg.

Set your server up so that you prefer to serve application/xhtml+xml over text/html for user-agents that support it.

This means that Firefox etc will get application/xhtml+xml and Internet Explorer will get text/html, right? Wrong.

The problem is that when you hit reload in Internet Explorer, instead of sending its usual Accept header, it sends Accept: */* instead, giving equal weight to each media type. Since text/html isn't mentioned, it doesn't get priority over application/xhtml+xml.

The end result is that you set up your server, check it in Internet Explorer and find nothing wrong, make the updates to the live server, and get bombarded with complaints from end-users, since it breaks for them when they hit the refresh button.

Yet *another* bizarre Internet Explorer bug making things difficult for us!

Hey, Internet Explorer developers, there must be a good reason for you transmitting a dummy Accept header on reload, but I can't think what it is for the life of me. Care to enlighten me?

re: A HTTP Detective Story by Eric Lawrence

gohankid77 — Tue, 15 Mar 2005 05:04:00 GMT

I agree with creativeHavoc. A significant portion of the Internet would probably be "broken", but it would be a push toward proper implementation of Web standards. The current XHTML 2.0 draft seems to be aiming toward Web authors who are serious about following the standards. However, without a *decent* rendering engine, those standards are useless.
Also, why must IE be "hardwired" into the OS? It should be an optional component when installing, separate from the Windows Explorer interface.

re: A HTTP Detective Story by Eric Lawrence

EricLaw — Tue, 15 Mar 2005 06:25:00 GMT

<<Internet Explorer developers, there must be a good reason for you transmitting a dummy Accept header on reload, but I can't think what it is for the life of me. Care to enlighten me? >>

I wondered about this a lot before I joined the IE team. I'll take a look.

re: A HTTP Detective Story by Eric Lawrence

Daniel Martin — Tue, 15 Mar 2005 08:02:00 GMT

I am curious about this line in the story:
“ Opera and FireFox never sent the referer, so any request carrying their user-agent was automatically permitted. ”

Do you mean to say that this site has such horrendous security that anyone going to http://example.com/auth.asp">http://example.com/auth.asp without a Referrer: header is automatically logged in? As in, anyone can fire up any browser they want (before you sent in the fix, only Firefox or Opera, but now, apparently, any browser at all), type http://example.com/ in the browser bar to set the ASP session ID, and then type http://example.com/auth.asp">http://example.com/auth.asp in the browser bar, and be logged in?

I could actually believe this, given some of the horrendous web application security I've seen elsewhere, but if so then there really should be an MSDN article describing how to do this correctly - that is, how to start a session over http, accept the login credentials over https, and then continue a logged-in session over http without requiring a common back-end database for the http and https portions of the website. (I can think of at least one method, but yo no hablo asp. It does require a shared secret phrase between the two halves, but I think that's acceptable.)

re: A HTTP Detective Story by Eric Lawrence

EricLaw — Tue, 15 Mar 2005 16:27:00 GMT

<<Do you mean to say that this site has such horrendous security that anyone going auth.asp without a Referrer: header is automatically logged in?>>

Yup.

There are actually quite a few ways of doing this securely; lacking a common backend does make it more challenging, but certainly not impossible. One thing to keep in mind is that compared to the current approach, pretty much anything is an improvement.

I'm not an ASP pro myself anymore, but the following shared secret approach would seem to work...

When redirecting the user out of the SSL tunnel back to the HTTP site, send along two query string parameters, the username and a hash:

SHA1(Secret + Username + Date)

And when the HTTP page gets the request, rehash the Date, Username, and Secret; if they don't match, then don't log the user in.

The two servers need to share the secret; the date is to make replay attacks less useful.

Of course, you could also just move the login to the HTTP frontend and use JS on the client to MD5 hash the credentials before submitting. Then, use a webservice on the HTTPS site to verify the credential hash. This is how, for example, the Yahoo mail login appears to work.

re: A HTTP Detective Story by Eric Lawrence

Typhoon121 — Wed, 16 Mar 2005 17:37:00 GMT

while you are on the topic of the user string is there any prblem with MOCA (Microsoft Online Crash Analysis) on a WIN 98 machine with IE 6.0 SP 1 NET Passport Unavailable at This Site Help

The .NET Passport service is currently unavailable at this Web site for one of these reasons:

.net passort not avaible on a Microsoft site. HAHA

re: A HTTP Detective Story by Eric Lawrence

Sentinel — Thu, 17 Mar 2005 00:16:00 GMT

Nice article.

However, I am amazed at how you can fix a bug years old in XP2, and then say how people were using that "bug" in order to perform some action? You joking?

So, let's assume that Mozilla/Opera did not implement that 'feature' which was present in IE for ages, and were following RFCs properly (which they usually try to do). Then, magazine site would only work with IE, and Mozilla/Opera would NOT work. What would happen? Everyone would blame Mozilla/Opera for not working properly, even if they followed RFCs. Granted - it's developer error, but you've placed mechanisms (not standard compliant) in IE which allowed him to do that.

Man, how many things did I have to change in my VALID HTML/CSS code in order to make it work under IE...

Now, I can expect SP3 which will fix some more bugs. Then, I will realize those "IE fixed" pages won't display in IE anymore, since you've finally fixed bugs which were years old.

And then you will blame ME for that?

I am sure Microsoft has tons of good coders, and I respect them. But don't give me this crap about you being 'standard compliant' when we all had to break standards numerous times (for YEARS) in order to make your products work properly.

re: A HTTP Detective Story by Eric Lawrence

Meryl — Thu, 17 Mar 2005 13:07:00 GMT

Another site (http://www.meryl.net/css/) for CSS inspiration that lists Big Companies and Non-D (non-designer related sites).

re: A HTTP Detective Story by Eric Lawrence

polemic — Thu, 17 Mar 2005 15:10:00 GMT

I think it's despicable that so called Microsoft developers, having gone through the much vaunted "compulsory security retraining" still advocate the use of techniques like MD5 or SHA1 hashing which are proven to be flawed not only in theory, but in practice also.

No wonder your products are so buggy.

re: A HTTP Detective Story by Eric Lawrence

EricLaw — Thu, 17 Mar 2005 20:58:00 GMT

<<having gone through the much vaunted "compulsory security retraining" still advocate the use of techniques like MD5 or SHA1 hashing >>

1> I'm not a Microsoft developer.
2> What algorithm do you suppose SSL uses for hashing?
3> SHA1 has been "broken" insofar as you can find a collision in 2^69 cycles instead of 2^80. MD5 was broken worse, but it's still non-trivial. We're talking about protecting login to a magazine site; you could buy a subscription for ~way~ less than you could purchase the grid of computers needed to crack the hash. And don't forget, we're changing the hash every day.

re: A HTTP Detective Story by Eric Lawrence

Rob — Thu, 17 Mar 2005 21:37:00 GMT

very scary that there are developers out there (quite possibly developing a site that'll be storing my personal data) that "code" garbage like that. Ugggg... coders should be freakin licensed to do this job.

Interesting finds this week

Jason Haley — Sat, 19 Mar 2005 14:45:00 GMT

Interesting finds this week

Think

Shuai — Sat, 19 Mar 2005 14:31:00 GMT

Very intersting. Hope FF gets a crack at it....

Very nice how you thought of it.

x'SHUAIBAO+gmail=COM`x

re: A HTTP Detective Story by Eric Lawrence

Jim — Fri, 25 Mar 2005 00:26:00 GMT

I don't get it:

> I joined the IE team because...

Okay...

> I recently came across a bug in the bug database for IE7...

Okay...

> I’ve got the full source for IE on my computer...

Okay...

> I wondered about this a lot before I joined the IE team. I'll take a look.

Okay...

All of this I understand. Until I read this:

> 1> I'm not a Microsoft developer.

Huh?

Was somebody else posting as Eric?

re: A HTTP Detective Story by Eric Lawrence

Bruce Morgan [MSFT] — Fri, 25 Mar 2005 01:05:00 GMT

That comment was from Eric and yes, Eric isn't a developer, his position is "program manager".

You can read more about program management at Microsoft at http://www.microsoft.com/careers/careerpath/technical/programmanagement.mspx

re: A HTTP Detective Story by Eric Lawrence

Greg K Nicholson — Tue, 29 Mar 2005 17:35:00 GMT

Hey, InterNet ExPlorEr GuYs*, it's "Firefox", not "FireFox"... nor "FiRefOx", nor "FirEfoX" etc.

(* That there be satire.)

Seems that fiddler is not able to act as a cascading proxy?

Igor Romanov — Sun, 03 Apr 2005 14:27:31 GMT

So, it needs a direct Internet connection?

re: A HTTP Detective Story by Eric Lawrence

Kenneth — Tue, 05 Apr 2005 22:04:49 GMT

I've got a good detective start for you:
what on earth dictates when IE will save my username and password? It seems to happen randomly. Firefox seems a lot easier to use in this regard.

Also, for IE7 wouldn't it be awesome to have multiple-login choices for sites, both for the standard login popup box and on sites?
Often I use 4 or 5 logins, giving me access to different parts of sites.

Huh?

Steve Parker — Sun, 10 Apr 2005 04:17:07 GMT

So FireFox and Opera honour the RFC's by not sending a Referer to an HTTPS site, and MSIE fails the RFC's.

The fix is to make MSIE honour the RFC's, not to workaround it.

If MSIE honoured RFC's, it would not have had these problems in the first place.
The fact that it used to break RFC's in one way, and now breaks RFC's in a different way, is not an excuse. It is not a "detective story". It's a bug.
Fix the bug.

The pleasure about not having to fix the MSIE code I can understand (it must be a nightmare!), but that is what needs to be done.

IE Detectives

MikeKelly's WebLog — Wed, 18 Oct 2006 20:10:24 GMT

Great post on using his tool, Microsoft Fiddler , to track down a web site issue by my friend and colleague,

Fiddler

random ramblings.... — Tue, 30 Jan 2007 19:21:15 GMT

A HTTP Detective Story by Eric Lawrence shows how: 1 - resoundingly useful the Fiddler tool that Microsoft have released is. 2 - resoundingly stupid some people can be. Non-withsthanding the fact that Internet Explorer (pre XPSP2) returned the referer..

Aaron Johnson » Blog Archive » The Referer header, intranets and privacy

Aaron Johnson » Blog Archive » The Referer header, intranets and privacy — Tue, 06 Feb 2007 11:08:00 GMT

PingBack from http://cephas.net/blog/2007/02/06/the-referer-header-intranets-and-privacy/

IEBlog A HTTP Detective Story by Eric Lawrence | Hammock Stand

IEBlog A HTTP Detective Story by Eric Lawrence | Hammock Stand — Wed, 03 Jun 2009 07:19:31 GMT

PingBack from http://hammockstandsite.info/story.php?id=15480

IEBlog A HTTP Detective Story by Eric Lawrence | Green Tea Fat Burner

IEBlog A HTTP Detective Story by Eric Lawrence | Green Tea Fat Burner — Mon, 08 Jun 2009 05:29:56 GMT

PingBack from http://greenteafatburner.info/story.php?id=251

IEBlog A HTTP Detective Story by Eric Lawrence | Cellulite Creams

IEBlog A HTTP Detective Story by Eric Lawrence | Cellulite Creams — Wed, 10 Jun 2009 04:22:50 GMT

PingBack from http://cellulitecreamsite.info/story.php?id=3421