IEInternals

Braindump: Feature Control Keys and URLActions

2012-09-27T17:57:00Z

Note: The “brain dump” series is akin to what the support.microsoft.com team calls “Fast Publish” articles—namely, things that are published quickly, without the usual level of polish, triple-checking, etc. I expect that these posts will contain errors, but I also expect them to be mostly correct. I’m writing these up this way now because they’ve been in my “Important things to write about” queue for ~5 years. Alas, these topics are so broad and intricate that a proper treatment would take far more time than I have available at the moment.

I’m frankly amazed that I’ve made it to the end of an eight-year run on the IE team without having written a general blog about Feature Control Keys and URLActions. It’s time to rectify that oversight, with one day left on the clock before I move along. Both of these topics are covered at length on MSDN, but some of the philosophy behind how they came to be is lacking in the formal documentation.

Feature Control Keys

Feature Control Keys were formally introduced in the huge Windows XP Service Pack 2 update to Internet Explorer 6. XPSP2 massively tightened security for Internet Explorer, but the team faced a big problem—thousands of applications host the Web Browser Control and if the Internet Explorer lockdowns were applied to those applications, they could break, leading users to avoid installing XPSP2 and thus impeding the value of the security push entirely. On the other hand, if the team simply exempted all non-IE applications from the lockdowns, those other applications could be vulnerable to the same attacks forbidden in Internet Explorer, allowing attackers a means to compromise the system by going around Internet Explorer’s defenses into an unhardened Web Browser Control host (e.g. Windows Media Player, for one). In some cases, applications would want exactly the same protections as IE (e.g. 3rd party browsers built on the Web Browser Control, which included at that time Maxthon and Avant Browser). On the other hand, some applications would want none of the protections (e.g. Microsoft Money hosted the Web Browser Control, but only loaded fully-trusted content into it).

Thus were born Feature Controls, a mechanism which allowed an application to opt-in or out to individual security defenses (and later compatibility modes) on a per-process basis. Feature Controls could be set in one of two ways: by manipulating keys in the registry, or by calling the CoInternetSetFeatureEnabled API within the running process. Only a limited number of Feature Controls (often abbreviated FCK for Feature Control Key) can be set via the API for two reasons: First, because some settings must be determined at process startup, before the caller could have called the API, and Second, because the API has generally not been updated to support the many dozens of FCKs introduced after the launch of XPSP2. As a consequence, in almost all cases, FCKs are set via the registry.

You can see the complete list of public Feature Control Keys on MSDN. This list explains how to set the FCK for your process’ name, as well as listing what each FCK does and what its default is. Note: Internet Explorer internally may use some other FCKs that are not documented, but these are not a part of the public API, and can be subject to change in any monthly update.

One significant shortcoming of the registry-based Feature Control Key system is that process names must be distinctive if you hope to apply different FCK settings across versions of the program. For instance, all versions of Microsoft Word are named WinWord.exe, which means that if you have side-by-side versions of Office 2010 and Office 15, the FCKs set for the last-installed version will overwrite whatever FCKs were applied to the previous version. This long-standing limitation is a bit of a hassle, but in practice not a big enough problem to warrant a change since 2003.

Feature Control Keys are sometimes criticized for sometimes prioritizing compatibility over security for 3rd party applications; this seems non-intuitive to some. The apt metaphor I like to use is: “Sports cars go fast because of their brakes.” If sports cars didn’t have excellent brakes to complement their engines, drivers would be unable and unwilling to push them to their performance limits, because the inability to control the speed would be fatal. Similarly, if Internet Explorer adopted a “All security features must be enabled in all contexts at all times” stance, users would quickly stop installing patches in order to prevent the compatibility fallout of such a stance. As a result, enhanced security features would bizarrely result in users becoming less secure.

URLActions

Feature Control keys are applied on a per-process basis, which isn’t sufficiently granular for many needs. Some privileges (e.g. the ability to download ActiveX controls or spawn popup windows) can be safe in a trusted context (e.g. an Intranet site) and unsafe in an untrusted context (Internet pages).

Internet Explorer’s Zone-based security architecture is comprised of two parts:

A set of rules that map content into one of the five security zones, and
A set of security policy settings for each zone that control the privileges (called URLActions) granted to that Zone.

There are nearly 100 URLActions in the browser today, and a half dozen or more are added in each new version. Most of them can be viewed by opening Internet Explorer and clicking Tools > Internet Options > Security, picking a Zone, and clicking the Custom Level… button. However, some “headless” URLActions are not displayed in the UI and can only be controlled programmatically or via the registry.

Internet Explorer (and other programs) determine what privileges should be granted to content by calling the ProcessURLAction API on the IInternetSecurityManager for the process. Applications hosting a Web Browser Control can supply their own security manager for the control to use (using the IServiceProvider’s ProfferService API) but most simply rely on the default security manager. The default security manager gets its settings from four registry nodes: the HKLM and HKCU nodes under \SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Zones, and the associated policy keys.

Aaron Margosis’ excellent Zone Analyzer tool allows you to view your default URLAction settings, including determining the “effective” settings which result from the precedence rules for the four registry nodes.

Working Together

The URLAction architecture predates Feature Control keys, but some security features are controlled using both mechanisms.

For instance, FEATURE_ZONE_ELEVATION controls whether a given process is opted into Zone Elevation Restrictions. If and only if a process has these restrictions enabled does a cross-Zone navigation then consult the target site’s URLACTION_FEATURE_ZONE_ELEVATION setting to determine whether the cross-Zone navigation should be permitted. If the FCK is Disabled, or if it is Enabled and the target zone’s URLAction is Allow, then the navigation is permitted.

This was just a quick summary of these two important subsystems, but I hope you find it useful anyway!

-Eric Lawrence

Braindump: DNS

2012-09-26T18:02:00Z

When you navigate to a web address like www.bing.com/help in your browser, in most cases, the browser (or more specifically, WinINET, the network stack below IE) will immediately convert the hostname component of that URL (in this case, www.bing.com) into a numeric IP address like 165.254.26.34. It performs this magic by sending a query to a Domain Name System (DNS) server that then returns one or more address records for the target hostname. The browser then establishes a TCP/IP connection to the target IP and then uses HTTP (and/or SSL/TLS) to transfer web traffic over that connection.

Address Lists

Many people assume that a hostname will only return one target address, but in practice, it will often return several. For instance, on my PC, a DNS query for www.bing.com returns the following:

2600:1409:1::6010:6119
2600:1409:1::6010:6123
165.254.26.34
165.254.26.43

The first two addresses are IPv6 addresses, while the latter two are IPv4 addresses. Prior to Windows 2000, Windows would randomly select one of the addresses and attempt to use it; after Windows 2000, the first returned address is always returned first. The change was made to support the idea that the DNS resolver will attempt to order the address list by placing “preferred” addresses first. For instance, the list might be sorted based on the resolver’s knowledge of locality or connection throughput to a given server.

Windows Vista and later always sort the IPv6 addresses to the front of the list.

DNS Failover

While WinINET will attempt to connect to the first address, if that connection fails, that entry in the address list will be marked as “bad” and a connection will be attempted to the next returned address. This process repeats until a connection is made, the address list runs out of candidate addresses, or the retry count limit is reached (INTERNET_OPTION_CONNECT_RETRIES defaults to 5). If a connection is not made, the browser will show a “Page could not be displayed” error message.

The linear process of failover can lead to performance problems, if a DNS returns many records which must be tried before a working address is reached. Each connection attempt could take as long as 21 seconds. The most common case where this problem occurs is when a server returns IPv6 addresses to a client that cannot successfully make IPv6 connections due to network configuration problems. A proposal (oddly named “Happy Eyeballs”) was made to address such problems by more quickly attempting to contact IPv4 addresses if IPv6 addresses are slow to resolve. As far as I understand it, Firefox and Chrome utilize a variant of this notion. You can read about how Windows 8 improved IPv6 resiliency and performance over on the B8 blog.

Address List Caching

For performance reasons, WinINET caches address lists using an in-process memory cache; this cache allows reuse of recently used addresses without resolving them again. DNS resolutions are also cached by Windows' DNS Resolver itself, but retrieving them from the Windows cache requires an RPC call, which, while much faster than issuing a DNS request on the wire, still takes some time.

In Internet Explorer 9 and earlier, the per-process cache could hold up to 32 entries; in IE10 and later it holds up to 256 entries. The cache ignores any TTL (time-to-live) specified by the DNS server’s response. Instead each list is stored for 30 minutes by default; the DnsCacheTimeout registry key can be used to adjust this value. (WinHTTP, another Windows HTTP stack used primarily by services, has different defaults; it caches up to 32 records for up to 5 minutes).

It has been speculated that the WinINET DNS cache was created to attempt to prevent DNS Rebinding attacks, but this is a fallacy—the cache exists for performance reasons only, and is trivial to circumvent as a part of a rebinding attack.

What about proxies?

If the client computer is behind a traditional (“CERN-style”) proxy server, WinINET skips the DNS lookup for the target site and instead sends the request to the proxy server, allowing the proxy server to perform the DNS lookup on the client’s behalf. This is useful in some cases, for instance, the case where the client computer doesn’t itself have access to a DNS server that is willing to return records for non-Intranet sites. Of course, the browser may still need to use DNS even in this case—if your browser is configured to send traffic to CORPPROXY:8080, it will need to use DNS to look up the address of CORPPROXY. It’s possible that the DNS lookup for the proxy will return more than one address; if it does, the same pattern described above is used to decide which address to use.

However, in the unlikely event that you’re using a SOCKS-style proxy, WinINET must first perform the DNS lookup locally, sending that address information to the proxy server. That’s because IE only supports SOCKSv4, and the ability to rely upon the remote proxy to do DNS lookups wasn’t introduced until the SOCKSv4a protocol.

-Eric Lawrence

Braindump: ActiveX in Windows 8

2012-09-26T17:18:00Z

There are a few notable changes in Win8/Internet Explorer 10’s behavior when it comes to ActiveX controls.

1. The non-Desktop mode of the browser (let’s call it IEPKaM for lack of a better name) only permits instantiation of controls that are considered to be a part of the web platform. The list of permitted objects is hardcoded into Internet Explorer and consists of:

MSXML DOMDocument	{F6D90F11-9C73-11D3-B32E-00C04F990BB4}
MSXML FreeThreadedDOMDocument	{F6D90F12-9C73-11D3-B32E-00C04F990BB4}
MSXML XMLSchemaCache	{373984C9-B845-449B-91E7-45AC83036ADE}
MSXML XSLTemplate	{2933BF94-7B36-11D2-B20E-00C04F983E60}
MSXML XMLHTTP	{F6D90F16-9C73-11D3-B32E-00C04F990BB4}
MSXML DOMDocument30	{F5078F32-C551-11D3-89B9-0000F81FE221}
MSXML FreeThreadedDOMDocument30	{F5078F33-C551-11D3-89B9-0000F81FE221}
MSXML XMLSchemaCache30	{F5078F34-C551-11D3-89B9-0000F81FE221}
MSXML XSLTemplate30	{F5078F36-C551-11D3-89B9-0000F81FE221}
MSXML XMLHTTP30	{F5078F35-C551-11D3-89B9-0000F81FE221}
MSXML DOMDocument60	{88D96A05-F192-11D4-A65F-0040963251E5}
MSXML FreeThreadedDOMDocument60	{88D96A06-f192-11D4-A65F-0040963251E5}
MSXML XMLSchemaCache60	{88D96A07-f192-11D4-A65F-0040963251E5}
MSXML XSLTemplate60	{88D96A08-f192-11D4-A65F-0040963251E5}
MSXML XMLHTTP60	{88D96A0A-f192-11D4-A65F-0040963251E5}
XMLHTTPRequest	{ED8C108E-4349-11D2-91A4-00C04F7969E8}
DOMDocument	{2933BF90-7B36-11D2-B20E-00C04F983E60}
Scripting.Dictionary	{EE09B103-97E0-11CF-978F-00A02463E06F}
HtmlComponent	{3050f4f8-98b5-11cf-BB82-00AA00BDCE0B}
Scriptlet	{AE24FDAE-03C6-11D1-8B76-0080C744F389}
IE XMLDocument	Not registered – used when hosting XML
IE SVGDocument	Not registered – used when hosting SVG
IE XHTMLDocument	Not registered - – used when hosting XHTML
Adobe Flash	{D27CDB6E-AE6D-11cf-96B8-444553540000}

IEPKaM blocks other forms of extensibility outright: toolbars, BHOs, Pluggable Protocols, MIME Filters, and Namespace handlers will not load in IEPKaM.

2. IEPKaM only permits use of Adobe Flash on sites that are listed in the IE Compatibility List or DebugDomain registry key.

3. When enabled, IE’s ActiveX Filter permits use the controls listed above, except Adobe Flash, which is still filtered. This enhancement makes ActiveX Filtering far more palatable, as it doesn’t block use of legacy objects like the ActiveX version of the XMLHTTPRequest control.

4. Windows RT devices like the Microsoft Surface cannot download or install ActiveX controls.

Windows RT’s ActiveX restrictions are additionally backed by the OS loader, which will refuse to run code that hasn’t been signed with a particular code-signing certificate.
Installed controls that are a part of Windows RT are permitted to run in the Desktop experience.
In the IEPKaM experience, the list above is still consulted before a control is permitted to load.

5. When the Enhanced Protected Mode feature is enabled, controls will not load unless they have been compiled for 64bit (when run on 64bit Windows). When running on Windows 8, there is the additional requirement that the controls are listed in the CATID_AppContainerCompatible component category, indicating that they have been tested to work properly within AppContainers.

For instance, the controls must not expect to perform a non-brokered read of the local disk or registry; instead, such operations must be conducted on the control’s behalf by a registered broker object running at Medium Integrity. In some cases (like writing to a file), the IE Protected Mode APIs will suffice, but IE10 does not include any new Read brokers, so if your control hopes to read an arbitrary file from disk, you’ll need to write your own broker.

6. IE10 enables Enhanced Memory Protections like ForceASLR, which opts all loaded modules into address space randomization, regardless of whether the /DynamicBase flag was set. You should continue to set this flag directly, but be aware that your control cannot take dependencies on fixed module load addresses even if you fail to do so.

-Eric

Understanding Zone Elevation

2012-09-24T16:55:00Z

The security setting “Websites in less privileged web content zone can navigate into this zone”:

... is one that leads to more questions than almost any other. This setting, also known as Zone Elevation protection, was originally designed to prevent navigation from untrusted Internet content into the highly-trusted Local Machine Zone. Prior to Internet Explorer 6 on Windows XP SP2, if an attacker could accomplish such a navigation and inject script into the LMZ, they had found a Remote Code Execution vulnerability and could take over the machine. This was a key source of vulnerabilities in that era, and the threat led to the introduction of Zone Elevation protection.

The Zone Elevation protection feature blocks navigation, either outright (when set to Disable) or via a Prompt:

Regardless of your specific security configuration, for the purposes of Zone Elevation checks, the privilege precedence of Zones is: Local Machine > (Local Intranet == Trusted Sites) > Internet > Restricted Sites. There's a special-case exemption in the logic that allows Trusted Sites to navigate to Intranet sites, even though normally Intranet > Trusted. (One reason for that is that the user may configure an Intranet Site to be a Trusted Site. Most people expect this to increase the site's level of privilege, despite the fact that the opposite is what happens by default, since the Intranet runs at the Medium-Low Security Template while the Trusted Zone defaults to the Medium Security Template. To help match users' expectations, the Trusted Zone is often afforded special exemptions like this one.)

You can explore the Zone Elevation prompt's behavior by adjusting the setting for the Intranet zone to Prompt, then visiting this test page: http://webdbg.com/test/zoneelevation.htm. This page attempts to load a frame at http://localhost, and offers various navigation-triggering methods as well.

In XP SP2, Zone Elevation protection was bolstered by the Local Machine Zone lockdown, such that Zone Elevation into the LMZ would be akin to “Breaking into Jail.” When an attacker Zone Elevates into the Local Machine Lockdown Zone, he ends up with fewer privileges, not more.

However, even after the Local Machine Zone lockdown was introduced, Zone Elevation protection still has some value in that it can prevent Internet Zone content from navigating into Trusted or Intranet Zone content. As such, when set to Prompt or Disable, it serves as a protection against XSS and CSRF attacks against content in those Zones. Without this protection, an attacker with knowledge of your Trusted/Intranet sites and their vulnerabilities could potentially execute a cross-zone attack against those pages from an Internet site. This attack is non-trivial to pull off (since it is, by its very nature, a targeted attack), and it isn’t nearly as scary as a pre-XPSP2 attack against LMZ.

By default, all Zones have their Zone Elevation setting set to Allow except the Local Machine and Restricted Zones, where it is set to Disable. That setting is meaningless for the Restricted Zone because there is no less trusted Zone. It’s set that way because Disable is the default for the High Security template which may be used by other zones. If a Zone’s security level is set to the Low template, then the Zone Elevation URLAction is set to Prompt because this high level of privilege allows download of signed ActiveX controls and one-click access to scripting ActiveX controls that are not marked safe-for-scripting.

Configuring Zone Elevation to prompt or disable access to your Local Intranet and Trusted Sites zones can help protect against CSRF and XSS.

In IE10 Metro Mode, the Private Network Isolation feature more effectively enforces protection of the “Intranet,” as it prevents circumvention of cross-Zone restrictions using the site’s IP address or otherwise causing IE to Zone a “private network” site into the Internet zone.

-Eric

Debugging in IE10 on Windows 8

2012-09-05T21:25:00Z

Emulating the “non-Desktop Experience” in the Desktop Experience

The new full-screen “fast and fluid” experience of IE10 on Windows 8 offers many improvements over Internet Explorer 10 on the Desktop (ranging from UX to Security), but one thing it lacks is the F12 Developer Tools, used by web developers to debug web pages.

While you can use Visual Studio to debug pages, sometimes, debugging with the lightweight F12 Developer Tools on the Desktop can be simpler. However, what if your bug doesn’t repro in the Desktop experience?

In many cases, you can emulate the non-Desktop experience on the Desktop by making simple configuration changes:

Enable Enhanced Protected Mode by clicking Tools > Internet Options > Advanced and ticking the Enhanced Protected Mode checkbox in the Security section near the bottom of the list.
Enable ActiveX Filtering (Tools > ActiveX Filtering)
Press F11 to put the IE Window in full-screen mode

With these configuration changes in place, you can continue to use the F12 Developer Tools while emulating the non-Desktop experience.

Note that one limit to the quality of the emulation is that the ActiveX Filtering feature will filter out Adobe Flash content from all sites. Such content is permitted in the non-Desktop experience only when your site is on the CV List or the DebugDomain registry key is set.

Debugging Local Content

Web Developers often test pages or sites on their local systems before it is deployed to production. In some cases, there are side-effects to debugging local content that developers should be aware of.

When debugging against a dotless hostname (e.g. http://localhost) the content will typically be in the Local Intranet zone. By default, this means:

Security settings (e.g. File download, ActiveX behavior, cookie controls, popup blocker) will be different.
The site will run outside of Protected Mode or Enhanced Protected Mode, which means that extensions will run with fewer restrictions and cookies and other storage will be “partitioned” from Internet Zone storage.
If the hostname is merely dotless and isn’t localhost, the page will run in Compatibility View by default

When debugging against 127.0.0.1 or any other hostname which points to the local computer but is not mapped to the Intranet zone:

Requests from "Windows 8 Apps” and IE running in Enhanced Protected will be blocked due to AppContainer Network Isolation

When debugging content loaded from a file:// URL, other restrictions include:

The content will typically be subject to the Local Machine Lockdown

The IE8/IE9 XMLHTTPRequest object is not able to send requests (fixed in IE10)
The IE8+ XDomainRequest object is not able to send requests
Storage with domain-based security policy (e.g., HTML5 Storage, IndexedDB) is disabled
If the content runs inside Enhanced Protected Mode, subdownloads may fail if they lack a Mark-of-the-Web.

For scenario #1, you can disable the Local Intranet Zone by clicking Tools > Internet Options > Security > Local Intranet > Sites and unchecking all of the boxes. Of course, this will also mean that your content runs inside Enhanced Protected Mode, which can cause the problems listed in scenario #2.

To help address scenario #2, Fiddler4 includes a tool called the EnableLoopback Utility which can be used even when Fiddler isn’t running. It’s simply a graphical version of the CheckNetIsolation.exe command line tool which is used to exempt specific AppContainers from loopback restrictions.

If you’re having issues in scenario #3, you can consider using IIS Express or Fiddler’s AutoResponder tab to serve your local pages from a local web server.

I hope that this post is useful. If you encounter any other “gotchas” or have other tips for local debugging, please sound off in the comments below!

-Eric

Downloading ZIP-Based Formats

2012-08-27T23:03:00Z

More and more file formats are based on the ZIP format. The Open Packaging Conventions use ZIP as a base format, and that means frameworks like .NET’s System.IO.Packaging also generate files that are valid ZIP files. The Office 2007+ formats are ZIP-based, and more personally, Fiddler’s SAZ Format is ZIP-based.

Unfortunately, this trend toward ZIP-based packaging incurs a problem when dealing with file types that are not registered in the server’s configuration. When sending unknown types, a simple server will typically send a Content-Type: application/octet-stream header, indicating very generically that the download in question is of a binary type without providing specific information. Internet Explorer’s MIME-sniffing code kicks in and says, hey, I see that you’ve provided a generic type. Lemme check that content and see if I know what it is.

Now, the sniff for ZIP formats is dead-simple: Does the file start with 0x50 x4B (aka ‘PK’)? If so, then it’s probably a ZIP file. And in the case of ZIP-based formats, the browser’s technically right, but behaviorally wrong. If the server didn’t specify a Filename in a Content-Disposition: attachment header, Internet Explorer will promptly rename the file away from its original extension to .ZIP. The browser will then consult with Windows and determine that the .ZIP file should be opened by a MIME Handler.

For instance, downloading from http://webdbg.com/dl/saz.saz results in the following modal prompt:

If you choose Open, the MIME Handler is invoked and shows the guts of the ZIP file:

If you choose Save, the file will be saved to your downloads folder as a .ZIP. This is generally not what you want.

As a mitigation for this problem, Internet Explorer 9 included an exemption list for the most popular ZIP-based formats of 2010; downloads whose URLs bore the following extensions are not renamed:

.accdt; .crtx; .docm; .docx; .dotm; .dotx; .gcsx; .glox; .gqsx; .potm; .potx; .ppam; .ppsm; .ppsx; .pptm; .pptx; .sldx; .thmx; .vdw; .xlam; .xlsb; .xlsm; .xlsx; .xltm; .xltx; .zipx

To avoid this problem for all ZIP-based types, servers have two options:

Send a specific MIME-type identifying the file’s type
Use a Content-Disposition header to specify the filename

For instance, when the server is reconfigured to send a Content-Type: application/x-fiddler-session-archive MIME, the user gets the expected Download Manager notification, and the file extension is untouched:

The changing web suggests that it probably makes sense to get out of the business of sniffing ZIP files, as such sniffing is likely now causing more problems than it solves.

-Eric

Proxy-Authentication breaks many applications

2012-08-03T18:21:00Z

When I first joined Office, I worked on the team responsible for delivering Help, Templates, and ClipArt into the client applications. As we were testing our work in various simulated customer environments, we found a big problem. At least one big customer (tens of thousands of licenses) had a network environment in which their users were forced to enter a username and password in order to authenticate to the proxy server. Without authenticating to the proxy, all HTTP/HTTPS requests were blocked.

Now, this was a fairly uncommon architecture, even then, and is perhaps more so now. In most environments, either the proxy server doesn’t require authentication, or the proxy relies upon the NTLM/Kerberos authentication schemes which permit users’ Windows logon credentials to be automatically used to respond to challenges from the proxy server. Environments that relied upon BASIC or DIGEST authentication require that the user explicitly submit their credentials, typically once per process (because most networking components, e.g. WinINET would cache these credentials for the lifetime of the process).

The problem with my features in Office was that they all passed the INTERNET_FLAG_NO_UI flag to WinINET, or ran atop WinHTTP, which explicitly doesn’t include any user-interface, including dialogs. The result of this was that in an environment with a BASIC/DIGEST proxy, all requests failed. In order to work properly in such environments, the application must itself supply the needed credentials to the network stack (e.g. for WinINET, call InternetSetOption, passing the INTERNET_OPTION_PROXY_PASSWORD and INTERNET_OPTION_PROXY_USERNAME option flags) to avoid the need to prompt the user.

I added a new rule to Fiddler that made it simple to test products for this problem:

When the Require Proxy Authentication box is checked, Fiddler automatically responds to any request lacking a Proxy-Authorization header with a HTTP/407 response containing a Proxy-Authenticate header specifying the authentication scheme required:

GET /ua.aspx HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host: www.enhanceie.com

HTTP/1.1 407 Proxy Auth Required
Connection: close
Proxy-Authenticate: Basic realm="FiddlerProxy (username: 1, password: 1)"
Content-Type: text/html

<html><body>[Fiddler] Proxy Authentication Required.<BR> </body></html>

A client that supports manual proxy authentication will then prompt the user for the username and password:

The client will then reissue the same request, supplying the provided credentials (base64-encoded) in the Proxy-Authorization header:

GET /ua.aspx HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Proxy-Authorization: Basic MTox
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host: www.enhanceie.com

If the client fails to collect the credentials, it will typically treat the HTTP/407 response as fatal and will show an error message or fail silently.

When you try this, you can find broken scenarios all over. For instance, when I tried to post this blog using Windows Live Writer, the following error message was shown:

Afterward, I was prompted to re-enter my credentials for the web server—there was no way to supply the credentials required by the proxy!

Sometimes, an otherwise failing scenario may pass depending on what happens earlier in a process. For instance, if you enable the Fiddler rule, then launch IE to about:blank you will find that Search Suggestions from the Address bar don’t work, showing “An error occurred.”

Notably, if you subsequently navigate the tab to a web page, IE will prompt you for proxy credentials using the CredUI dialog box shown above. After you supply those credentials, the Search Suggestions feature starts working—that’s because the proxy credentials are cached for the lifetime of the process.

In other cases, failure are silent and there’s no notice to the user. For instance, many background updaters are based on BITS/WinHTTP and will fail silently when a HTTP/407 is encountered. Similarly, Windows’ CAPI component’s Certificate Revocation Checks will fail because the svchost.exe process doesn’t have the required proxy credentials.

If you need to sell your software into an enterprise that uses proxies, or just want to make your software robust against even uncommon network configurations, be sure to test manual proxy authentication scenarios!

-Eric

Brain Dump: Shims, Detours, and other “magic”

2012-07-31T02:05:00Z

Since IE6, Internet Explorer has implemented major architectural changes without accompanying breaking changes to its binary extension model. While new extension features have been introduced (e.g. Search Providers, Web Slices, and Accelerators), they are all based on markup rather than code and have been relatively straightforward to keep working from version to version.

In contrast, Internet Explorer’s binary extension models: ActiveX, Browser Helper Objects (BHOs), Toolbars, etc, are all architected such that 3rd-party COM code runs within the Internet Explorer process. In many cases, extensions originally designed for IE6 (and earlier) continue to run without modification even in IE9 and IE10 on the Desktop. That’s despite the fact that virtually everything else around these extensions has changed: tabbed browsing and Protected Mode were introduced for IE7, Loosely-Coupled IE was added in IE8, Hang Resistance was introduced in IE9, and IE10 introduced Enhanced Protected Mode and other major changes throughout Windows. Each of these architectural shifts would break the majority of the binary extensions if not for a corresponding set of investments in compatibility features undertaken in each release of the browser.

Windows Vista’s introduction of the Integrity Level system was accompanied by the UAC Virtualization system, designed to help accommodate applications that expected to be running with Administrative privileges. If a 32-bit executable’s manifest lacks a requestedExecutionLevel element (e.g. iexplore.exe’s embedded manifest doesn’t have one), then UAC Virtualization will be applied for file and registry operations. Browser extensions running in Internet Explorer benefit from this virtualization, enabling legacy add-ons that expect to be able to read or write to protected locations to continue working. Virtualization works by redirecting write operations from read-only areas to a per-user “virtualized” location. For instance, attempting to write a file to the Desktop from Low Integrity would ordinarily fail, but virtualization permits the operation to succeed by writing the file to a hidden folder elsewhere in the file system. (IE’s Low Integrity virtualization uses a shim to redirect writes to %USERPROFILE%\AppData\Local\Microsoft\Windows\Temporary Internet Files\Virtualized\, while UAC virtualization writes to %USERPROFILE%\AppData\Local\VirtualStore).

However, virtualization alone isn’t enough to ensure compatibility. For instance, when tabbed browsing was introduced in IE7 and Hang Resistance was introduced in IE9, the behavior of windows and dialogs needed to be updated to be compatible with these features. For instance, when an extension in a background tab attempts to show a prompt, this prompt must be suppressed until that tab is activated (otherwise, a confusing experience would result). To accommodate that behavior, a system of shims and detours is used.

These two technologies are similar:

Shims work by rewriting a module’s import address table at runtime to point to a different target function
MSR’s Detours work by rewriting the start of one or more target functions at runtime to point to a wrapper function

These technologies allow Internet Explorer to intercept calls to important functions (e.g. CreateProcess, CoCreateInstance, CreateWindow, etc) and modify the behavior of those calls to improve compatibility with the restrictions and desired behaviors of the tab/content process in which HTML and add-ons run. For instance, the CreateProcess and CoCreateInstance APIs are wrapped such that the Protected Mode Elevation Policies can be applied. Similarly, CreateWindow is designed to accommodate the creation of new windows by background tabs, and to properly parent those windows to the correct window handle even though the window hierarchy was changed due to the hang resistance feature.

In IE10, we’ve moved most functionality away from Detours to Shims for enhanced compatibility and because we’re shipping to a new platform (Windows RT) to which we otherwise would have had to port the IE version of Detours. In most cases, this was a seamless change, but we recently ran into one ancient toolbar that was impacted by the change.

The toolbar in question was a simple one that offered a standard search box, a few notification icons, and a short set of menus that would launch dialog boxes to configure the toolbar and show information about it. Our compatibility testing team noticed that in IE10, the dialog boxes from the toolbar would never come up. Debugging native code extensions without source or symbols is never fun, but I decided to take a look anyway. I ran the installer and verified that the dialog boxes didn’t come up. Knowing nothing about the technology (e.g. maybe the dialogs were written in HTML), I took a quick look at the installation folder. I got an idea of how old the code was when I saw that the installation folder contained unicows.dll, an ancient library designed to help enable compatibility with pre-Unicode versions of Windows (e.g. 95/98).

I next ran through the repro with IE10 running under the debugger and found a nested function deep inside a call to CreateWindow() was returning Access Denied. I then ran the same repro in IE9 under the debugger and found that CreateWindow succeeded, but observed that in IE9, there were detoured compatibility wrappers in the stack trace, but those wrappers were not present in the scenario in IE10.

I spent several hours pondering this question and aimlessly touring around in the debugger. I was whining about this scenario to a colleague, complaining about code so ancient that it was shipping with unicows.dll, when I realized that I’d never used this library myself, and in fact I’d never seen a toolbar use it before. When trying to explain what it did to the colleague, I decided that I’d probably stop hand-waving and pulled up unicows up on Wikipedia. And bam, there it was, plain as day:

By adding the UNICOWS.LIB to the link command-line [ ... ] the linker will resolve referenced symbols with the one provided by UNICOWS.LIB instead. When a wide-character function is called for the first time at runtime, the function stub in UNICOWS.LIB first receives control and [ ... ] if the OS natively supports the W version (i.e. Windows NT/2000/XP/2003), then the function stub updates the in-memory import table so that future calls will directly invoke the native W version without any more overhead.

…and there’s the problem!

When IE first loads a toolbar, the shims run against the module and wrap all calls to CreateWindow with a call to the compatibility wrapper function. But when IE loaded this toolbar, it didn’t find any calls to CreateWindow, because those calls had been pointed at a function inside unicows.dll instead of at the original function in user32.dll. As a result, the compatibility shim wasn’t applied, and the function call failed.

Now, this wouldn’t have happened if unicows did its import-table fixup the “normal” way, using the GetProcAddress function. That's because the compatibility shims are applied to GetProcAddress as well, and the fixup would have been applied properly at the time that unicows did the update of the import table. However, for reasons that I thought were lost to the mists of time (see below), the implementers of unicows instead copied the source code of GetProcAddress from user32 into their own DLL, so the shims had no way to recognize it. While we could add a new shim to handle unicows.dll, the obscurity and low priority of this scenario mean that we instead decided to outreach to the vendor and request that they update their build process to remove the long-defunct support for Windows ‘9x.

-Eric

Update: Over on his blog, Michael Kaplan provided a history of why unicows.dll works the way it does.

PS: This MSDN article is a great resource that explains the PE file format and how linking and delay loading features work.

Content-Length and Transfer-Encoding Validation in the IE10 Download Manager

2012-07-16T19:12:00Z

Back in March of 2011, I mentioned that we had encountered some sites and servers that were not sending proper Content-Length headers for their HTTP responses. As a result, we disabled our attempt to verify Content-Length for IE9.

Unfortunately, by April, we’d found that this accommodation had led to some confusing error experiences. Incomplete executable files were not recognized by SmartScreen’s Application Reputation feature, and other signed filetypes would show “xxxx was reported as unsafe” because WinVerifyTrust would report that the incomplete file’s signature was corrupt. This problem was very commonly reported for large files (e.g. 50mb installers) by users in locations with spotty network access (e.g. where such connections are often interrupted).

With IE10, we’ve reenabled the Content-Length / Transfer-Encoding checks in IE’s Download Manager. If the Download Manager encounters a transfer that does include the number of bytes specified by the Content-Length header, or the transfer fails to include the proper 0-sized chunk (when using Transfer-Encoding: chunked), the following message will be shown:

If the user clicks Retry, IE will attempt to resume (or restart the download). In many cases of network interruption, this feature helps ensure that the user is able to download the complete file. As a compatibility accommodation, if the retried transfer again does not provide the expected number of bytes, Internet Explorer will permit the download to be treated as “finished” anyway, so that users are not blocked from interacting with buggy servers.

For instance, one buggy pattern we've seen is a server which delivers the HTTP response body as a single chunk, then calls HttpResponse.Close() instead of the proper HttpApplication.CompleteRequest.

// Add Excel as content type and attachment
Response.ContentType = "application/vnd.ms-excel";
Response.AddHeader("Content-Disposition", "attachment; filename=" + binTarget);

  mStream.Position = 0;
  mStream.WriteTo(Response.OutputStream);
  Response.Flush();

// BAD PATTERN: DO NOT USE.
// See http://blogs.msdn.com/b/aspnetue/archive/2010/05/25/response-end-response-close-and-how-customer-feedback-helps-us-improve-msdn-documentation.aspx
Response.Close();

Calling Close() like this omits the final chunk, and would cause the server's output to fail in the Download Manager if not for the compatibility accommodation.

You can test how browser’s handle incorrect transfer sizes using these two Meddler scripts:

-Eric

Brain Dump: International Text

2012-07-13T17:00:00Z

Handling of non-ASCII text is a common source of compatibility and interoperability problems. This post covers a variety of tidbits related to this topic, and it will be expanded (and likely corrected) over time.

RFC2616 defining HTTP/1.1 suggests that non-ISO-8859-1 text in HTTP headers must be encoded according to the rules of RFC2047, an approach that was not commonly implemented by many web clients. Many clients will instead send or accept raw UTF-8 or bytes encoded using the current system’s ANSI codepage instead. Character-set mismatches often result in interoperability problems.

Internet Explorer’s handling of non-ASCII text is partially controlled by these checkboxes in the Advanced tab:

Always show encoded addresses is disabled by default will force IE to show the raw Punycode in the address bar at all times when viewing an IDN site, even if that site’s IDN URL is following the non-spoofability rules.

Send IDN server names is enabled by default and will force IE to encode hostnames in URLs following the rules of RFC3491 and RFC3492. The user will be shown the URL in the address bar in Unicode form if and only if the URL is deemed non-spoofable. Please see this IEBlog post on the rules of IDN Non-spoofability.

Send IDN server names for Intranet addresses is disabled by default for compatibility with legacy Windows networks that were using UTF-8 to support non-ASCII hostnames. Other browsers, to the best of my knowledge, do not have special handling for Intranet sites, and I believe that current versions of Active Directory and the Windows DNS server support punycoded hostname registration and lookup.

Send UTF-8 URLs is checked by default, but doesn’t behave as broadly as its name implies. This option controls whether certain URL components and headers are sent and interpreted using UTF-8 or the system’s ANSI codepage, but it does not apply to the entire URL.

Show Notification bar for encoded addresses checked by default, informs the user that they are seeing punycoded text in the address bar only because the non-spoofability rules have determined that the current site’s address follows the rules for IDN non-spoofability except that the address uses characters outside of the current user’s configured Accept-Languages. The notification bar allows the user to adjust the configured Accept-Languages using the Internet Control Panel.

Use UTF-8 for mailto links is unchecked by default, but is checked when installing current versions of Outlook. You can learn a lot more about this option in this IEBlog post. The option has been removed for Windows 8 / Internet Explorer 10, and mailto links are always passed to the client application using %-encoded UTF-8.

Submission of text in HTML forms in Internet Explorer is a fascinating and complex topic. The design of form encoding in IE8 and earlier was to submit forms using the encoding of the submitting page by default. If the FORM element on the page declared the Accept-Charset attribute equal to UTF-8 (which is the only supported value) and if the form results contained data that could not be encoding in the page's encoding, then the form results would be sent as UTF-8. In IE9 standards-mode and later, IE will always encode form results as UTF-8 if the accept-charset="UTF-8" attribute is present.

If your web form contains an INPUT TYPE=HIDDEN element with the name _charset_ this field will be automatically filled with the name of the character set used to encode the form when it is submitted. This helps permit your server to decode the form using the proper encoding.

In contrast, it’s not always possible to reliably reconstruct querystrings at the server (no, that was not a typo!), because IE does not pass any state information to the server which would indicate what encoding was used.

URLs in IE may use up to three (!!) different encodings at once: punycode in the hostname, %-escaped UTF-8 for the path, and raw codepaged-ANSI for the query and fragment components. This is clearly a mess, but fixing it to match the IRI specification incurs compatibility costs. (Trust me, we’ve tried!)

Internet Explorer’s XMLHTTPRequest object will not automatically encode your URIs for you (e.g. %-escaping UTF8 characters). If you want to send such characters to the server following the rules of IRI, you should encode them before passing them to the open() method, using the encodeURIComponent JavaScript API.

If you’re downloading files to IE9+ or other modern browsers, you should use RFC5987 encoding for the Content-Disposition header. If you need to support old versions of IE, the story is more complicated. This IEInternals post explores that topic.

In WordPad (and most RichEdit controls in Windows) you can simply type a four-digit hexadecimal number, (e.g. 30C4) and then hit ALT+X to convert that sequence to the corresponding Unicode character (i.e. ツ). Similarly, you can paste a Unicode character into WordPad and hit ALT+X to convert it back to its Unicode value.

In Windows, encoding of non-ASCII characters in File-scheme URIs (e.g. file://server/path/file.txt) is different than in other schemes. %-encoded octets in a FILE uri are always interpreted using the system’s ANSI codepage, not UTF-8. Learn more about this and File URIs in general here.

-Eric