IEInternals

A look at Internet Explorer from the inside out. @EricLaw left Microsoft in 2012, but was named an IE MVP in '13 & an IE userAgent (http://useragents.ie) in '14

Understanding Web Proxy Configuration

Understanding Web Proxy Configuration

  • Comments 23

Over the last decade, I’ve come to learn a lot about web proxies, having chosen to implement my web debugger as a proxy. In today’s post, I’ll provide an overview of proxy-related information, including information on changes in Internet Explorer 11 / Windows 8.1.

The “System Proxy”

Unlike on other systems like Mac OSX, Windows doesn’t really have the concept of a “system” proxy. Most applications respect the WinINET proxy setting, but a few do not.

  • Firefox respects the WinINET setting only when configured to “Use System Proxy Settings” in its Network configuration.
  • WinHTTP-based applications may or may not respect the WinINET setting (see below).
  • Applications built on the .NET Framework typically will adopt the system proxy only when they start (and do not detect changes at runtime). Also, the proxy settings can be overridden by the app.exe.config or machine.config files.

Historically, most applications were built on WinINET and thus respected its proxy settings directly, but today more and more applications are built on WinHTTP (or a descendent architecture like BITS) and System.NET and thus may require additional configuration.

Settings are Per-User

WinINET proxy settings are typically per-user rather than per-machine. This means that individual (even non-admin) users can set their own proxy settings without impacting the proxy settings of other user-accounts. WinINET can be configured to apply proxy changes on a machine-wide basis by creating a registry DWORD named HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\ProxySettingsPerUser with value 0. Subsequently, changes to the proxy settings can only be made by elevated applications running with Administrative permissions.

Configuring the Proxy

The WinINET proxy setting can be configured in Internet Explorer by clicking Tools > Internet Options > Connections. On the connections tab, select a Connection and click the Settings button. For most users, the proxy is configured by pressing the LAN Settings button at the bottom of the dialog:

image

Settings Precedence: Part 1: While the proxy settings for the Dialup/VPN/RAS connection and the LAN settings button can be configured independently, internally WinINET will use the settings from any active/connected dialup/VPN/RAS Connection. Otherwise, the proxy setting specified in LAN settings is used. Many users incorrectly expect WinINET to first determine which network interface is used to establish a given connection before selecting a proxy; it doesn’t work like this.

Proxy Configurations

When you configure your proxy settings, you’re presented with the following dialog box:

image

Settings Precedence: Part 2: The settings in this dialog box are presented in the order of their precedence. First, the Automatically detect settings option is consulted, next the automatic configuration script option is evaluated. If neither of those options is enabled or neither specifies a proxy to use, only then are the fixed proxy server settings consulted.

Automatically Detect Settings

When Automatically Detect Settings is enabled, Internet Explorer performs a process called Web Proxy Auto-Detection (WPAD). This process consists of two phases:

  1. Issuing a DHCP INFORM query, asking DHCP to provide the URL of a proxy script
  2. Searching for a server in the current domain prefixed with WPAD (e.g. wpad.corp.contoso.com)

These operations may or may not be performed in parallel, depending on the network stack. Some browsers (e.g. Firefox) only undertake DNS detection and skip DHCP detection. For performance reasons, some clients may cache the result of the detection on a per-network basis ("SmartWPAD"), skipping the WPAD process in future sessions when connected to networks known not to be using WPAD.

If the client is able to automatically detect a proxy script's URL, it will then attempt to download and use that proxy configuration script.

Automatic Configuration Script

The user may also directly specify the URL of a proxy configuration script using the second checkbox in the dialog. The URL field below points directly at the target script (e.g. http://proxy.contoso.com/proxy.pac).

Proxy configuration scripts, whether discovered via WPAD, or manually specified by the user, are JavaScript files that expose at least one function FindProxyForURL(url, host). This function is called by the browser each time that it needs to decide where to send a given request. It returns a string which is either:

  1. "DIRECT" indicating that the request should bypass the proxy
  2. "PROXY PrimaryProxy:8080; BackupProxy:81" indicating that the request should be forwarded to the proxy PrimaryProxy on port 8080, unless that server is unreachable in which case the request should be sent to BackupProxy on port 81.

WinINET also offers support for another function FindProxyForURLEx, an extension which supports IPv6-aware proxy scripts.

NOTE: Proxy Scripts Impact the Zone

Warning: One sometimes surprising aspect of proxy scripts is that they impact the Internet Explorer Security Zone determination. To summarize my long MSDN article and blog post, by default, if a proxy script is in use and returns DIRECT, the target site will be mapped to the Local Intranet Zone.

 

NOTE: File://-based Proxy Scripts Deprecated

In IE11, the WinINET team has made some changes to attempt to improve interoperability of proxy scripts.

image

Internet Explorer 10 and other WinINET-based clients would use the specified proxy script. However, WindowsUpdate and any other application built on a WinHTTP-based network stack would silently ignore the specified proxy script because WinHTTP has never supported the use of the file:// protocol as the origin of a proxy configuration script. Note: In November 2012, the .NET Framework was changed to use WinHTTP for PAC processing, which means that, by default, .NET Applications will not support file://-based proxy scripts any longer either.

In Internet Explorer 11, the WinINET team has disabled WinINET’s support for file:// based scripts to promote interoperability across network stacks. Corporations are advised to instead host their proxy configuration scripts on a HTTP or HTTPS server. As a temporary workaround, this block can be removed by setting the following registry key:

Key:   HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\
Value: EnableLegacyAutoProxyFeatures
Type:  REG_DWORD
Data:  1

Keep in mind that this should only be a temporary measure, as this block was introduced for good reasons, and removing the block won’t magically fix your WinHTTP-based applications.

 

NOTE: File://-based Proxy Scripts Require “Unhealthy” Syntax

While you shouldn’t use a file://-based proxy script, if you do so, IE requires that you use the legacy “unhealthy syntax” for file URIs in the proxy configuration dialog.

As you can see, the “Healthy” syntax is incorrectly parsed as a relative URL.

image

The legacy “unhealthy” syntax works properly:

image

But, to reiterate, don’t use file:// to host your proxy script.

NOTE: IE11 Deprecates Non-Silent Auth when downloading Proxy Script

When downloading a proxy configuration script, it’s possible that the server will demand credentials from the client application. A problem arises if the server requests credentials using HTTP BASIC or HTTP DIGEST authentication, as these authentication methods require that the user respond to a credential prompt dialog box. (In contrast, NTLM and Negotiate authenticate silently).

Many applications cannot handle showing prompts as a part of this workflow and will fail silently. To promote interoperability, IE11 also blocks the use of Proxy Configuration scripts that require authentication. As a temporary workaround, this block can be removed by setting the following registry key:

Key:   HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\
Value: EnableLegacyAutoProxyFeatures
Type:  REG_DWORD
Data:  1

Keep in mind that this should only be a temporary measure, as this block was introduced for good reasons, and removing the block won’t magically fix applications which cannot show UI prompts.

Generally speaking, you should attempt to migrate away from BASIC/DIGEST authentication both for access to the proxy script and for access to use the proxy itself. As I’ve noted previously, Manual Proxy-Authentication breaks many applications.

 

Fixed Proxy Configuration

Beyond automatic proxy detection, the user may manually configure “fixed” proxies. By default, the simple UI is shown:

image

…but clicking the Advanced button shows more options:

 

image

You can either specify a different proxy server/port for each protocol, or you may use the default option which applies the same proxy server to all protocols.

The Exceptions box allows you to specify what hostnames are configured to bypass the proxy. Beyond hostnames, it supports two special tokens:

  • <local> – Matches “plain” hostnames (with no embedded dots)
  • <-loopback> – Unmatches localhost, 127.0.0.1 and other “loopback” addresses.
    • WinINET defaults to automatically bypassing the proxy for loopback addresses; this token disables that behavior.
    • This token impacts WinINET and current versions of WinHTTP; the .NET Framework simply ignores it.

Note that Internet Explorer only supports SOCKSv4.

 

Setting the Proxy Programmatically 

Fiddler and some other applications attempt to programmatically adjust the WinINET proxy setting. Rather than attempting to “poke” the registry directly, the proper way to update the proxy setting is to use the InternetSetOption API.

Using the API is straightforward, whether you’re using C/C++ or .NET.

IE10+ Regression on System Shutdown

As a part of Internet Explorer 10, proxy configuration was centralized in an existing system service (still labeled “WinHTTP Web Proxy Auto-Discovery Service”) so that proxy configuration code no longer needs to run in every process using WinINET (e.g. each IE tab is in its own process).

Unfortunately, as a part of this update, a regression was inadvertently introduced. Code that attempts to change the WinINET proxy setting during a system shutdown (like Fiddler’s shutdown code, for instance) will find that the APIs return success, but upon restart you will see that the proxy settings change was discarded. No fix for this issue (also present in IE11) is yet available. A possible workaround for this problem would be to create an out-of-proc COM server and run a method (DllHostKeepAlive) that doesn’t return until the client calls a shutdown method (CancelDllHostKeepAlive). This would open a DLLHost that can be used to adjust the proxy settings during shutdown.

CONNECT Bug: https://connect.microsoft.com/IE/feedback/details/838086/internet-explorer-10-11-wininet-api-drops-proxy-change-events-during-system-shutdown

WinHTTP Considerations

Thanks to Eric Loewenthal, owner of WinHTTP's proxy code, for providing the expertise for this section

The WinHTTP stack was designed for use in services and other contexts where WinINET would be inappropriate (e.g. because WinINET may show UI or not scale well for unattended use). WinHTTP offers a limited subset of proxy support-- generally speaking, the calling code is expected to set the proxy configuration settings directly on the session or request objects. The user may attempt to configure the "default" proxy for WinHTTP using the proxycfg.exe (XP) or netsh.exe (Vista+) tools, but these tools only support fixed proxy settings (not autodetection or PAC script URLs) and WinHTTP-based applications may or may not use those settings.

The configuration tools were primarily intended or WinHTTP apps that are running as System (without a user to impersonate). 

Well-written WinHTTP apps will:

  1. If not running in a user's process, impersonate a user
  2. Call WinHttpGetIEProxyConfigForCurrentUser
  3. Plug the returned settings into WinHttpGetProxyForUrl

Sample code for this technique can be downloaded from MSDN. 

Note: WinHTTP will not process a proxy script if the Content-Type response header is not "application/x-ns-proxy-autoconfig" or the URL's file extension is not one of .js, .pac, or .dat. Also, remember that WinHTTP will not use proxy script files from file:// URLs, on any version of Windows.

 

There’s still quite a bit more to say about proxies. I’ll update this post as time permits and questions arise.

thanks!

-Eric

  • How do I inspect a proxy script obtained via WPAD?

  • Proxy scripts are simply text files that can be examined in any text editor. If you're looking for an actual PAC-script debugger, check out chentiangemalc.wordpress.com/.../c-proxy-pacdbg-pac-debugging.

  • Hey Eric,

    Great article - we came across this as we are facing a problem due to this change and we have a valid reason for wanting to have a file based proxy.

    Our product uses a bunch of URLs that go to our data center and integrates with a bunch of URLs that belong to the clients of our product. Our URLs have our proxy rules and client URLs have their own rules -- we make both work by downloading both scripts and merging them to a combined script file on the desktop. We then call InternetSetOption and point it to the local file... It seems this is no longer working

  • I am behind a proxy server providing free internet via my town hall. Since upgrading to windows 8.1 and explorer 11 I receive error 8024401C when I try to connect to windows update. I believe this is a proxy server problem is there anything you can suggest to help? thanks

    EricLaw - What are your proxy settings? Did you download and run the WindowsUpdate Troubleshooter?

  • It looks like IE10 and above doesn't look at dialup/VPN/RAS proxy setting any more, it seems LAN setting always override that. The testing shows this behaviour. Could you confirm if this is expected?

    [EricLaw] The WinINET team indicates that this issue was addressed by http://www.microsoft.com/en-us/download/details.aspx?id=40532

  • I am using an Automatic Configuration Script for my proxy settings that is working fine for IE, but I have a .NET application that attempts to make an SSL connection to a website.  The application doesn't see any Proxy settings and tries to connect to the website directly.  If I populate the Fixed Proxy Configuration, the application uses the proxy and connects without issue.  What are the implications with using BOTH the PAC script and the Fixed Proxy settings?

    [EricLaw]: Settings "flow" from the top of the box downward. So, if the PAC script is inaccessible, then WinINET/IE will fallback to using the fixed proxy settings instead of going direct.

  • I am using Dansguardian for content filtering and one of the things that I have been implementing over the last month is filtering groups so that pupils, staff, technicians get different filtering.

    Because I have schools with Windows 2003, 2008, and 2012 servers and because we have iPads, Nexus tablets etc I ruled out using either Kerberos or NTLM as they will no doubt work differently on different equipment or may not work at all.

    Dansguardian does not support group authentication, only user authentication so moving away from Digest Authentication as you suggest, just is not practical as it’s the simplest way to provide user authentication for Ipads, tablets et.  So I went with Digest Authentication using Squid.  I bypassed authentication for things like Windows Updates.

    But on Windows 7 and IE10, I encountered two problems.  I could not programmatically add the credentials for IE to the credential manager and the second issue is even when I added the username and password and clicked save settings for IE, every single time I opened up IE or did something using some program that used IE Proxy settings, I would get a authentication popup box with the username and password all filled out, but I would still have to click on ok.  This popup box came up every single time I rebooted the computer and windows 7 gadgets wanted to connect to the computer etc.

    It’s only since updating windows updates this week and updating to IE11, that both of these problems have now been fixed.  I can now programmatically add the credentials to credential manager and I no longer get any popup authentication boxes anymore!

    [EricLaw] That's an interesting surprise, and it suggests that WinINET has been updated to use the credential manager for proxy authentication without a prompt. That feature request had been made previously but I wasn't aware that it was supported.

    I have two questions about your article.  I believe that the reason that these problems have now been fixed is because IE11 and obviously some updates in windows 7 has made it so that WinInet is no longer being used.

    [EricLaw] As with all Internet Explorer updates, IE11 updates the system's version of WinINET.

  • Hi Eric, great article!  I'm having issues with IE11 on WIndows 8.1 with our TMG 2010 wpad.  It works fine on our Windows 7 SOE and with Chrome on 8.1, but IE11 doesn't seem to even look for it.  I've posted at social.technet.microsoft.com/.../windows-81-with-ie-1102-wpad-issue   - any ideas appreciated : )

    Thanks,

    Simon

  • Regarding this IE 10+ regression on system shutdown, do you know if this is applicable to any of the other events such as sleep, restart or logout?

    [EricLaw] It's unlikely that sleep is impacted. Restart and Logout might be; you could simply try it and then you'd know for sure.

  • Thank you so much for your article. About the change in WinINET in IE11, I have one question. If I need access at our http external site for read the .pac (now are using file://) how work in my internal LAN? My http connection are via Proxy defined in the .pac

    I'm using the .pac file for set our internal corporate squid proxy or don't use proxy (by example in my house)

    [EricLaw]: The common configuration for this is to set the PAC file URL to a HTTP:// URL that is only available within your corporate network (e.g. http://proxy/pac.js). That way, the URL is inaccessible from outside your network, so that when you take your device home, the script is not used. Alternatively, set up your internal network to use WPAD and don't manually configure a PAC URL on the client directly at all.

  • Hello Eric,

    I work on any enterprise which uses a third partner remote access solution, this third partner solution only support internet proxy hosted locally at %appdata% folder (using file://C:\...)

    While the windows computers are connected remotely, the internet browsing through I.E. works fine, but every time the user try to run a Click Once application, the Click Once fails to read the script pac and try to connects directly.

    If we manually change the I.E. proxy settings for the same script.pac hosted at any webserver using Http://...) the Click Once application works fine.

    This happens with I.E 8 and later, do you know if this behavior is expected?

  • Hi, @Diego! You say that IE "works fine" but keep in mind that, as this post describes, that will no longer be true when IE11 is installed.

    There was a security fix for .NET in November 2012 that changed System.NET to rely on WinHTTP for WPAD support (which in turn means that file://-sourced WPAD scripts will fail) unless the LegacyWPADSupport key is set. See http://technet.microsoft.com/en-us/security/bulletin/MS12-074:

    What does the update do? 
    The update addresses the vulnerability by updating Microsoft.Jscript within System.Net and restricting how the .NET Framework executes JavaScript contained within a WPAD configuration file. The update also changes the way WPAD is handled by using WinHTTP instead of Microsoft.Jscript. This change, while unlikely, may cause some compatibility issues for environments that have non-standard proxy auto-configuration environments. For more information about this change, see the More Information section in Microsoft Knowledge Base Article 2745030.

    That KB specifically notes:

    When you set the LegacyWPADSupport DWORD value to 1, the new proxy detection logic that is installed by this update is disabled. This value should be set only if you must support one or more of the following scenarios in which WPAD is used for proxy detection in the network:

    • Lookup of proxies for FTP uniform resource identifiers (URIs) is needed (Windows XP).
    • Lookup of proxies for URIs that have a scheme other than HTTP, HTTPS, or FTP is needed (Windows XP, Windows Vista, Windows 7, or Windows 8).
    • The proxy auto-configuration (.pac) file is located on an FTP server or a file share instead of an HTTP server (Windows XP, Windows Vista, Windows 7, or Windows 8).

     

     

     

  • Hi Eric, Yes, IE 11 needs the workaround you posted, but we are running IE 10 yet. The .NET KB solved my problem for now.Do you know if MS will remove the I.E. and .NET workaround on later patches? What should I follow to keep updated about this?

    -Thanks, Diego

    [EricLaw] Unfortunately, there's no way to know whether the "escape hatch" registry key will be kept around indefinitely; sometimes they're removed in a later version and sometimes they linger on for a decade or more. It would probably be wise to push the vendor to explore updating their code to be compatible with future products.

  • Eric, Great article and very helpful.  Wondering if you could help me out with something related to the PAC file.

    We are use a corporate proxy server and configured IE to point to a URL and Port (e.g. Use automatic configuration script.  Address = http://ur:port/).  Our Network Engineer is troubleshooting a problem and asked if I knew where the Proxy PAC file is stored in Windows. My initial assumption was that the PAC file is downloaded to the Temp I-Net files cache, but I have not been able to locate the file.

    I have done quite a bit of research and there are several references (Wikis, Proxy and PAC build articles) stating the PAC file is downloaded and process by the browser (e.g. support.microsoft.com/.../271361), but I cannot find any documentation stating where the PAC file is actually downloaded to.

    Do you know if IE (or any other browsers for that matter) downloads the PAC file to the local machine?  If so, do you know is it located? OS: Win7 64bit / IE9 or WinXP / IE8

    [EricLaw] Your best bet is to watch file writes using SysInternals' Process Monitor, because the behavior will vary by OS and by the browser version installed. On Windows 8.1 with IE11, the "WinHTTP Web Proxy Auto-Discovery Service" running inside a SVCHost.exe instance will download and cache the PAC file to a rudimentary cache: the file is written to a randomly-named *.cache file inside the c:\Windows\ServiceProfiles\LocalService\winhttp folder. In IE8 on Windows XP, I see the PAC file getting cached to %USERPROFILE%\Local Settings\Temporary Internet Files\Content.IE5\random\proxy[2].pac

  • Hi Eric, Good article!!! I am working on a VPN product. In some use cases, proxy comes between client PC and VPN server. In this case, once tunnel is established, VPN client creates a PAC file in such a way that only access to VPN server should go through proxy. Access to all other servers should go through directly.

    This is to ensure that all access to back end servers are going through tunnel adapter, not physical adapter.New PAC file is kept in the disk itself which is deprecated in IE 11 (as you said).

    As a solution to this problem, can we give PAC path like http://127.0.0.1/VPNProxy.pac ?

    Here our component will do the web server functionality. Thanks John

    [EricLaw] Yes, this should be a fine approach, and what Fiddler itself does when it finds that an upstream proxy was specified using a file:// based URL. There are a few caveats however-- if you're running an actual server, you need to be sure that the client doesn't already have a different webserver running on that default port, and you need to ensure that the machine's firewall, if enabled, doesn't interfere with the delivery of that script.

Page 1 of 2 (23 items) 12
Leave a Comment
  • Please add 4 and 1 and type the answer here:
  • Post