The information published in this post is now out-of-date and one or more links are invalid.—IEBlog Editor, 20 August 2012
Slashdot picked up a story from Bugtraq entitled Web browsers - a mini-farce in which Michael Zalewski talks about feeding a variety of browsers a healthy dose of bad content over 2 hours and seeing what happened. The story also includes pointers to the tools he used for hammering the browsers.
Here is a bit of his report:
6) Pointless rants It appears that the overall quality of code, and more importantly, the amount of QA, on various browsers touted as "secure", is not up to par with MSIE; the type of a test I performed requires no human interaction and involves nearly no effort. Only MSIE appears to be able to consistently handle [*] malformed input well, suggesting this is the only program that underwent rudimentary security QA testing with a similar fuzz utility. This is of course not to say MSIE is more secure; it does have a number of problems, mostly related to its security architecture and various features absent in other browsers. But the quality of core code appears to be far better than of its "secure" competitors. I cannot speak for the other browsers talked about in this report, but I can speak to the IE portion of this report. It is no accident that IE is responding this way to the tests that were run against it because we intentionally take a number of steps to make IE resilient.
At the end of the product cycle for Windows 2000 and as part of the Secure Windows Initiative, Microsoft developed a set of tools called Prefix and Prefast to do dynamic source code inspection, which helps scour the source code for bad code and bad coding practices such as null pointer dereferences. These tools help us find obscure crashing code paths that manual code inspection may miss. For XPSP2, we recompiled with the –GS flag to help mitigate certain classes of buffer overruns. For more information on the Windows Secure Windows Initiative see Michael Howard’s Technet article. These tools are publicly available as well. Prefast is part of the Windows Server 2003 Device Driver Kit [DDK]. There is a code quality tool for managed code called FxCop available on http://www.gotdotnet.com.
In addition to code quality initiatives, there is a very healthy suite of stress or load run against IE that we still use and extend today when we test. We throw a variety of things at the browser, including good HTML, bad HTML, variety of media, and “the kitchen sink” to see if we can get it to hang or crash.
We also utilize the Windows Error Reporting to help understand the causes of IE crashes or hangs in the field. For more information the Windows Error Reporting, see the Windows Quality Online Services web site.
However, despite Zalewski's results and our continued effort with Windows Error Reporting, stress testing and code quality tools I know we can do better as there places where you can crash IE with certain images or HTML. But this is what I relish about my job – continually driving quality up and up over time.