IEInternals

A look at Internet Explorer from the inside out. @EricLaw left Microsoft in 2012, but was named an IE MVP in '13 & an IE userAgent (http://useragents.ie) in '14

Brain Dump: Shims, Detours, and other “magic”

Brain Dump: Shims, Detours, and other “magic”

  • Comments 8

Note: The “brain dump” series is akin to what the support.microsoft.com team calls “Fast Publish” articles—namely, things that are published quickly, without the usual level of polish, triple-checking, etc. I expect that these posts will contain errors, but I also expect them to be mostly correct. I’m writing these up this way now because they’ve been in my “Important things to write about” queue for ~5 years. Alas, these topics are so broad and intricate that a proper treatment would take far more time than I have available at the moment.

Since IE6, Internet Explorer has implemented major architectural changes without accompanying breaking changes to its binary extension model. While new extension features have been introduced (e.g. Search Providers, Web Slices, and Accelerators), they are all based on markup rather than code and have been relatively straightforward to keep working from version to version.

In contrast, Internet Explorer’s binary extension models: ActiveX, Browser Helper Objects (BHOs), Toolbars, etc, are all architected such that 3rd-party COM code runs within the Internet Explorer process. In many cases, extensions originally designed for IE6 (and earlier) continue to run without modification even in IE9 and IE10 on the Desktop. That’s despite the fact that virtually everything else around these extensions has changed: tabbed browsing and Protected Mode were introduced for IE7, Loosely-Coupled IE was added in IE8, Hang Resistance was introduced in IE9, and IE10 introduced Enhanced Protected Mode and other major changes throughout Windows. Each of these architectural shifts would break the majority of the binary extensions if not for a corresponding set of investments in compatibility features undertaken in each release of the browser.

Windows Vista’s introduction of the Integrity Level system was accompanied by the UAC Virtualization system, designed to help accommodate applications that expected to be running with Administrative privileges. If a 32-bit executable’s manifest lacks a requestedExecutionLevel element (e.g. iexplore.exe’s embedded manifest doesn’t have one), then UAC Virtualization will be applied for file and registry operations. Browser extensions running in Internet Explorer benefit from this virtualization, enabling legacy add-ons that expect to be able to read or write to protected locations to continue working. Virtualization works by redirecting write operations from read-only areas to a per-user “virtualized” location. For instance, attempting to write a file to the Desktop from Low Integrity would ordinarily fail, but virtualization permits the operation to succeed by writing the file to a hidden folder elsewhere in the file system. (IE’s Low Integrity virtualization uses a shim to redirect writes to %USERPROFILE%\AppData\Local\Microsoft\Windows\Temporary Internet Files\Virtualized\, while UAC virtualization writes to %USERPROFILE%\AppData\Local\VirtualStore).

However, virtualization alone isn’t enough to ensure compatibility. For instance, when tabbed browsing was introduced in IE7 and Hang Resistance was introduced in IE9, the behavior of windows and dialogs needed to be updated to be compatible with these features. For instance, when an extension in a background tab attempts to show a prompt, this prompt must be suppressed until that tab is activated (otherwise, a confusing experience would result). To accommodate that behavior, a system of shims and detours is used.

These two technologies are similar:

  • Shims work by rewriting a module’s import address table at runtime to point to a different target function
  • MSR’s Detours work by rewriting the start of one or more target functions at runtime to point to a wrapper function

These technologies allow Internet Explorer to intercept calls to important functions (e.g. CreateProcess, CoCreateInstance, CreateWindow, etc) and modify the behavior of those calls to improve compatibility with the restrictions and desired behaviors of the tab/content process in which HTML and add-ons run. For instance, the CreateProcess and CoCreateInstance APIs are wrapped such that the Protected Mode Elevation Policies can be applied. Similarly, CreateWindow is designed to accommodate the creation of new windows by background tabs, and to properly parent those windows to the correct window handle even though the window hierarchy was changed due to the hang resistance feature.

In IE10, we’ve moved most functionality away from Detours to Shims for enhanced compatibility and because we’re shipping to a new platform (Windows RT) to which we otherwise would have had to port the IE version of Detours. In most cases, this was a seamless change, but we recently ran into one ancient toolbar that was impacted by the change.

The toolbar in question was a simple one that offered a standard search box, a few notification icons, and a short set of menus that would launch dialog boxes to configure the toolbar and show information about it. Our compatibility testing team noticed that in IE10, the dialog boxes from the toolbar would never come up. Debugging native code extensions without source or symbols is never fun, but I decided to take a look anyway. I ran the installer and verified that the dialog boxes didn’t come up. Knowing nothing about the technology (e.g. maybe the dialogs were written in HTML), I took a quick look at the installation folder. I got an idea of how old the code was when I saw that the installation folder contained unicows.dll, an ancient library designed to help enable compatibility with pre-Unicode versions of Windows (e.g. 95/98).

I next ran through the repro with IE10 running under the debugger and found a nested function deep inside a call to CreateWindow() was returning Access Denied. I then ran the same repro in IE9 under the debugger and found that CreateWindow succeeded, but observed that in IE9, there were detoured compatibility wrappers in the stack trace, but those wrappers were not present in the scenario in IE10.

I spent several hours pondering this question and aimlessly touring around in the debugger. I was whining about this scenario to a colleague, complaining about code so ancient that it was shipping with unicows.dll, when I realized that I’d never used this library myself, and in fact I’d never seen a toolbar use it before. When trying to explain what it did to the colleague, I decided that I’d probably stop hand-waving and pulled up unicows up on Wikipedia. And bam, there it was, plain as day:

By adding the UNICOWS.LIB to the link command-line [ ... ] the linker will resolve referenced symbols with the one provided by UNICOWS.LIB instead. When a wide-character function is called for the first time at runtime, the function stub in UNICOWS.LIB first receives control and [ ... ] if the OS natively supports the W version (i.e. Windows NT/2000/XP/2003), then the function stub updates the in-memory import table so that future calls will directly invoke the native W version without any more overhead.

…and there’s the problem!

When IE first loads a toolbar, the shims run against the module and wrap all calls to CreateWindow with a call to the compatibility wrapper function. But when IE loaded this toolbar, it didn’t find any calls to CreateWindow, because those calls had been pointed at a function inside unicows.dll instead of at the original function in user32.dll. As a result, the compatibility shim wasn’t applied, and the function call failed.

Now, this wouldn’t have happened if unicows did its import-table fixup the “normal” way, using the GetProcAddress function. That's because the compatibility shims are applied to GetProcAddress as well, and the fixup would have been applied properly at the time that unicows did the update of the import table. However, for reasons that I thought were lost to the mists of time (see below), the implementers of unicows instead copied the source code of GetProcAddress from user32 into their own DLL, so the shims had no way to recognize it. While we could add a new shim to handle unicows.dll, the obscurity and low priority of this scenario mean that we instead decided to outreach to the vendor and request that they update their build process to remove the long-defunct support for Windows ‘9x.

-Eric

Update: Over on his blog, Michael Kaplan provided a history of why unicows.dll works the way it does. 

PS: This MSDN article is a great resource that explains the PE file format and how linking and delay loading features work.

  • any chance of naming this Addon and publisher or adding it to the Addons compatibility lists so that support ppl will know to check for updates for it if they encounter it on a users machine?

  • @Rob: Sorry, it's probably best if I follow RaymondC's pattern of not naming names. The symptoms are pretty clear: Expected dialogs do not display (no crashes, just no display) and the install folder contains unicows.dll.

  • It would be nice if IE had better plugin compatibility inbetween versions. Some now-unupdated plugins provide crucial functionality (such as: user scripts, url dragging, user selectable per-file adblock), but they are getting more and more broken with new versions of IE. Yeah, I'm thinking of IE7Pro.

  • @Arieta: Thanks for the tee up.

    You've provided a great example of why it's a very bad idea to attempt to use unsupported mechanisms to extend IE. While we go to great lengths to prevent breakages of supported APIs, there's no way for us to keep extensions working when they perform unsupported hackery inside the IE process. IE7Pro, for instance, uses a variety of tricks to modify the IE UI in unsupported ways, and these break when internal data structures or architectures change underneath them. Add-ons that use unsupported mechanisms to extend IE are also subject to blocking by the IE Add-on Advisor feature, as they violate the IE Extensibility Guidelines.

  • I guess there wouldn't be a reason for that if IE provided all those features from get go. URL and Picture dragging has been in every other browser for years, for example.

  • @Arieta: I'm not sure what you're asking about. You've been able to drag URLs and images in IE since IE3 or earlier.

  • 1. Drag a link onto a new tab -> IE can do this. 2. Drag a picture onto a new tab -> they can only be dragged into the URL, where they open in the CURRENT tab. 3. Drag a plaintext url onto a new tab where the page is opened: not supported in any form.

    Also, userscripts, and user styles are increasingly important features as well among power users. IE7pro and TrixIE are the only plugins that can do that, but they are both old, unupdated and buggy. Internet Explorer can currently load global alternate style sheets since many, many years, so I imagine it would not be that difficult to extend that function to work on a per-site basis.

    I know I could go on and on about these features, and it's not the correct place to do so, but please understand that small perks like these can be a deal breaker for many users.

  • I'm not sure I understand why you'd ever want to do #2, if you already have the picture, why would you want the picture in a new tab? For #3, I simply right-click the selection and choose "Linkify and open", which is a trivial menu extension (http://www.bayden.com/ietoys/).

    For user-scripts, I use blogs.msdn.com/.../my-favorite-ie-add-on-mouse-gestures-by-ralph-hare.aspx

Page 1 of 1 (8 items)
Leave a Comment
  • Please add 5 and 4 and type the answer here:
  • Post