Windows Core Networking : WinINet

Why didn’t my cached content revalidate to the web server? (IE8 edition)

AriPernick — Mon, 06 Jul 2009 23:33:50 GMT

For the super eagle eyed developer or Web Server admin, you might have noticed that cache entry validation happens slightly less frequently in IE8. I’m talking about the check to the Web Server (via If-Modified-Since) to see if the cached version of content needs updating or is still current. The short version of why this happens is the way that IE uses WinINet sessions changed to accommodate Loosely Coupled IE and session’s start time is an important factor in evaluating if WinINet needs to issue an If-Modified-Since request.

Digging deeper, we can look at a couple of the factors in the logic IE/WinINet is using to decide to issue an If-Modified-Since request. First is the the Cache Setting: “Check for newer versions of stored pages” in the “Temporary Internet Files and History Settings” as described in KB article 263070 and pictured with this blog entry. These are settings that apply to cached content missing the Expires headers. “Every time I visit the webpage” and “Never” are pretty straightforward. “Every time I start Internet Explorer” and “Automatically” have logic dependant on the the session’s start time. Another factor to note that we have different behavior when using the Back and Forward browser buttons compared to what we do in hyperlink navigation. The Back and Forward behavior also depends on the session’s start time when content is expired or is missing the Expires header.

How does Loosely Coupled IE in IE8 change the session start time? Previously any IE window got a single session in WinINet by the fact it only uses a single process and sharing of the session handle across tabs, with Loosely Coupled IE we had to do work to let multiple processes share session state. This means that new IE8 windows share the same session compared to new IE7 windows which get a new session. Eric Lawrence on the IEBlog talks a bit more about this session sharing behavior. The effect on WinINet is that IE8 sessions last longer on average then they used to and that means potentially less often revalidation of content from the Web Server.

-- Ari Pernick

HTTP Connection Management

AriPernick — Fri, 08 May 2009 19:29:00 GMT

HTTP is a request/response protocol. You request some resource like the HTML of a webpage and the response comes back with the HTML attached. As each request is sent on a connection, the complete response must be read from the connection before the next response can be read. (There is an optimization where you make more requests on a connection before receiving the full response to the previous requests known as pipelining. However it doesn’t change the need to receive all the data from the previous requests first.)

So if an application is asking for another request to a server while the current one is in progress, what’s a HTTP Client API to do? Well, it can either wait till the connection is available or start a new connection. This decision is mostly controlled by a configurable maximum number of simultaneous connections to a server.

With that in mind, how should the following customer question get answered?

My client application’s performance is getting impacted by WinInet opening new SSL connections to my server rather than reusing the existing connection. How do I fix this?

Ask Perf explains how winInet is used

wndpteam — Tue, 21 Aug 2007 20:37:21 GMT

Ask Perf, the blog of the Enterprise Platforms Windows Server Performance Team, is spending some time explaining a bit of how WinInet/WinHTTP and their surrounding components work with each other. Go check it out!

-- Ari

WinInet ETW Logs: Part 1 - Reading Logs

wndpteam — Mon, 05 Feb 2007 19:00:00 GMT

Hello, my name is Marcus Frazier. I am a developer on the WinInet team and I am here to talk about reading and understanding WinInet ETW logs. If you need to know how to capture these logs, check out Jonathan’s post. Keep in mind that WinInet ETW tracing is only available on Vista.

Events

ETW logs are event based. When an event occurs an entry is placed in the log. These events are based on protocol behavior and decision points.

WinInet currently has seven event categories:

1) Handle Events – Events dealing with the creation and destruction of HINTERNET handles.

2) HTTP Events – Events dealing with the processing of HTTP requests and responses.

3) Connection Events – Events dealing with underlying network operations (TCP, DNS).

4) Authentication Events – Events dealing with authentication.

5) HTTPS Events – Events dealing with the HTTPS protocol.

6) Autoproxy Events – Events dealing with Autoproxy.

7) Cookie Events – Events dealing with cookies.

WinInet currently uses three event levels. Each event can be one of these levels:

1) Error – These events are raised when an error condition is reached.

2) Information – These events provide additional information regarding protocol handling.

3) Verbose – These events provide extra information that can help in diagnosing certain situations but are not required for all cases.

XML Logs

Let’s look at a single event entry in a log highlighting relevant information. This log was converted from ETL to XML format using the tracerpt as described in Jonathan’s post.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <EventID>103</EventID> <Level>4</Level> <TimeCreated SystemTime="2006-08-21T13:00:35.614957600Z" /> <Execution ProcessID="2480" ThreadID="3396" ProcessorID="1” KernelTime="30" UserTime="0" /> </System> <RenderingInfo Culture="en-US"> <Level>Information</Level> <Keywords> <Keyword>Flagged on all WinINet events dealing with creation or destruction of INTERNET handles</Keyword> </Keywords> <Message>Handle 0xCC0010 created by InternetConnect: SessionHandle=0xCC0004, Server=www.live.com:80, HTTP, Flags=0x0 </Message> <Channel>Microsoft-Windows-WinINet/Analytic</Channel> </RenderingInfo>
</Event>

When reading a log in XML format, there is a lot of information about every event. The first thing to look for is the open <event> tag. All the information from this tag to the </event> tag is about a single event. When quickly scanning through these logs I find the easiest thing to look for is the <message> tag within each event block. This is the English version of what has actually happened. Some other useful information is the SystemTime in the <TimeCreated> tag, which tells you when this event happened, and the <Execution> tag which shows the ProcessID and ThreadID.

Text Logs

In addition to XML format, ETW logs can also be converted to an easier-to-read text format using the following command (Vista only):

wevtutil qe <etl file path> /lf:True /f:Text > <output path>

An example of the full sequence of commands would look like this:

logman start "test1" -p "Microsoft-Windows-WinInet" -o "test1.etl" -ets <Run scenario> logman stop "test1" -ets wevtutil qe c:\logs\test1.etl /lf:True /f:Text > c:\logs\test1.txt
notepad test1.txt

Entries in the text formatted log appear in a concise format:

Event[21]: Log Name: N/A Source: Microsoft-Windows-WinINet Date: 2007-01-22T12:43:55.735 Event ID: 301 Task: N/A Level: Information Opcode: Info Keyword: Flagged on all WinINet events dealing with network operations (TCP, DNS) User: N/A User Name: N/A Computer: mfrazier2-test Description: TCP connection to server www.live.com established: ConnectionHandle=0xcc0014, SocketHandle=0x944, LocalPort=49779

These text files can be read with notepad. The “Description” is the same text as contained in the <message> tag from the XML log and describes what has happened.

Diagnosing problems with an ETW log can prove to be difficult at times. ETW logs will not always produce an error level event when a problem occurs. It takes some practice to recognize an “incorrect” sequence of events when there is no explicit error level event in the log. It is helpful to understand the normal flow of the protocol, and to understand what a “correct” series of events looks like. We suggest capturing the log of a working scenario and keeping it for reference.

The best way to get better at reading these logs is to see the difference between good and bad logs. Subsequent blog entries will give examples of diagnosing common problems using ETW logs. These examples will help you diagnose real ETW logs. The next post will deal with a simple case: No Internet Connectivity.

- Marcus

A Tale of 20 Cookies

wndpteam — Mon, 28 Aug 2006 15:00:00 GMT

As more applications move to leveraging the web, either through desktop-integration or complete migration to a web model, maintaining user state on the web becomes critical. For many web sites and applications this means the use of in-memory and persistent cookies.

Netscape originally defined HTTP cookies in a preliminary document and later the IETF standardized cookies in RFC’s 2109 and 2965. Interestingly, the WinINet cookie implementation is still (mostly) modeled after the early Netscape document, which specifies that browsers should:

Maintain at least 300 cookies total,
Support cookies up to 4KB in size each,
Support no fewer than 20 cookies per unique host or domain

User-agents, such as mobile phones, have more relaxed recommendations. Internet Explorer 7.0 and WinINet support up to 20 cookies of 5KB each per domain with no set limits on the total number of cookies overall on a system. Even with these totals, some web applications may need more.

A follow-on question might be to ask what happens when a web application sends more than 20 cookies for a domain (21 cookies perhaps). The answer is that we maintain the 20 most recent cookies. Recent is based on the order in which we received the cookies. That means if your web server sends 21 cookies named Cookie1, Cookie2 thru Cookie21 to WinINet (via Internet Explorer or otherwise), your web server would only receive Cookie2, Cookie3 thru Cookie21 on the next request from WinINet. We simply maintain the 20 most recent and quietly discard the other cookies for that host/domain.

Next, what happens when a web application exceeds the other maximum and sends a cookie with a size over 5120 bytes (>5KB)? Instead of truncating the values of oversized cookies, we simply discard these cookies. We chose to discard instead of truncate in order to avoid any type of cookie data corruption. I think you would agree that a cookie with a value of $10000 is very different from a truncated one with a value of $100. In addition, it is easier for the server application to check for the existence of a cookie and react if it is missing versus detecting if the value is correct.

There are ways to work-around our current 20x cookie limit. In my opinion, the best work-around is to leverage a durable back-end store for user information and use cookies for user tracking and lightweight values. This keeps the request/response more streamlined.

Another work-around, in cases where you absolutely need more data sent back-and-forth within the 20-cookie limit, is to pack more data into fewer cookies. This means that instead of:

Set-Cookie: FirstName=Billy
Set-Cookie: LastName=Anders

You can combine into something like:

Set-Cookie: FullName=Billy~Anders

Then, in your web application, you split the FullName cookie. Here is a rough (rough meaning no error handling) ASP.NET example in C#:

string[] Names = Request.Cookies[“FullName”].Split(‘~’);
if (Names.Length == 2) {
string FirstName = Names[0];
string LastName = Names[1];
}

Admittedly, this is not as straight-forward as the standard ASP.NET approach of:

string FirstName = Request.Cookies["FirstName"];
string LastName = Request.Cookies["LastName"];

However, this may prove helpful if you ever need to get around the 20x limit that we currently have.

Update: The sample above is to demonstrate the concept of packing more data into fewer cookies for any web development platform. ASP.NET natively supports subkeys which already allow you to store multiple values within cookies.

string FirstName = Request.Cookies["Name"]["First"];
string LastName = Request.Cookies["Name"]["Last"];

The current limits seem reasonable, but we would like to hear from you on whether that is case or not. Are you writing web applications that would like to use the browser’s cookie store beyond the 20x and 5KB limits that we have? Do you need 25, 50, 100, 250 cookies for your application?

- Billy Anders

Content-Encoding != Content-Type

wndpteam — Mon, 21 Aug 2006 18:00:00 GMT

RFC 2616 for HTTP 1.1 specifies how web servers must indicate encoding transformations using the Content-Encoding header. Although on the surface, Content-Encoding (e.g., gzip, deflate, compress) and Content-Type (e.g., x-application/x-gzip) sound similar, they are, in fact, two distinct pieces of information. Whereas servers use Content-Type to specify the data type of the entity body, which can be useful for client applications that want to open the content with the appropriate application, Content-Encoding is used solely to specify any additional encoding done by the server before the content was transmitted to the client. Although the HTTP RFC outlines these rules pretty clearly, some web sites respond with "gzip" as the Content-Encoding even though the server has not gzipped the content.

Our testing has shown this problem to be limited to some sites that serve Unix/Linux style "tarball" files. Tarballs are gzip compressed archives files. By setting the Content-Encoding header to "gzip" on a tarball, the server is specifying that it has additionally gzipped the gzipped file. This, of course, is unlikely but not impossible or non-compliant.

Therein lies the problem. A server responding with content-encoding, such as "gzip," is specifying the necessary mechanism that the client needs in order to decompress the content. If the server did not actually encode the content as specified, then the client's decompression would fail.

Here is a potentially over-simplified example:

Windows Vista Networking Rocks!
Jvaqbjf Ivfgn Argjbexvat Ebpxf!

If I mistakenly claim that string a) has been encoded using the simple ROT-13 obfuscation scheme when in actuality it has not, then the decoded message b) will be very different than the intended message.

Since the AI engine for WinINet isn't yet ready for production (joke), we try and work-around these non-compliant server responses but that isn't the right long-term approach. The fix and the ask, is for web server, extension and application authors to test their servers to see if they exhibit this behavior and if so fix their implementations before we remove our client-side hacks.

To test your server for compliance, issue a simple HTTP 1.1 request, including the "Accept-Encoding: gzip" for a .gz file and inspect the headers. If you see Content-Encoding: x-gzip or gzip, then the server is either gzip-encoding the already gzipped file or it is misstating that the content has been encoded by the server before transmission and therefore perpetuating client HTTP stacks, such as WinINet, having to absorb and hide this bad server behavior.

-Billy Anders

A WinINet Chunked-Encoding Story

wndpteam — Mon, 14 Aug 2006 18:05:00 GMT

A couple of months back, Nick Bradbury of Homesite and FeedDemon fame posted a blog entitled, "Microsoft, Please fix this WinINet bug!" where he mentioned some users of FeedDemon 2.0 were experiencing a significant CPU spike when downloading RSS feeds. Nick discovered that the spike occurred when downloading feeds from particular sites via WinINet InternetReadFile. The problem turned out to be a combination of issues; first, a handful of servers have non-compliant chunked-encoding transfer implementations and secondly, WinINet was poorly handling these situations.

At a basic level, chunked-encoding allows servers to begin sending responses without first having to determine the overall size of the content. When the server knows the size of the content beforehand, as the case with static content, this is less of a big deal. Things become more difficult and time-consuming when the server is responding with dynamic data, such as from a database or a live multimedia event.

With chunked-encoding, the web server responds with chunked in the Transfer-Encoding header instead of using the Content-Length header. Clients receiving chunked know that each chunk is only a piece of the overall content response. Each chunk is preceded by the size of the chunk (in hexadecimal) before being sent by the server:

Transfer-Encoding: chunked
5DC
<…1500 bytes of chunked data>
57D
<…1405 bytes of chunked data>

There is no set limit to the number of chunks that can be returned from the server. Clients simply process the chunks as they arrive depending on the type of content and the scenario. A live multimedia stream may begin playback as soon as enough buffered content has arrived, while a downloaded ISO file may write the bytes directly to a local file until the last chunk is received. The server indicates the end of content by returning a chunk indicator of 0 (zero):

Server: Microsoft-IIS/6.0
Date: Wed, 02 Jul 2006 08:15:00 GMT
Transfer-Encoding: chunked
Content-Type: text/xml
5DC
<…1500 bytes of chunked data>
57D
<…1405 bytes of chunked data>
0

This final zero-length chunk is where the problematic servers were breaking from RFC 2616, and where WinINet was failing not so gracefully.

The problematic servers were not ending their chunked responses with the zero-length chunk and instead were simply terminating the TCP connection. The WinINet routine that processed these chunks was overlooking the possibility of an abrupt termination and was steadfastly waiting for the illusive zero-length chunk. Of course, this issue has been fixed in WinINet and is now publicly available with IE 7 Beta 3 builds and newer. We have also engaged with the owners of the known sites, so that they could update their servers to be compliant. I recommend testing your web servers so that you are certain that your site is properly ending the chunked transfer.

As always, I encourage everyone to continue to report potential bugs with Windows Core Networking technologies. The easiest and most direct way to submit bugs is via the Microsoft Networking Connect site at http://connect.microsoft.com/networking. Issues submitted via Microsoft Connect make it directly to our use in minutes. Nick Bradbury, thank you for bringing this issue to our attention.

- Billy Anders

A bit about WinInet’s Index.dat – Q&A

wndpteam — Tue, 08 Aug 2006 02:59:00 GMT

In my previous post I tried to explain a bit about what the index.dat files are and what has changed in IE7/Windows Vista timeframe. The post got a couple questions that I'll attempt to answer here.

1) Mike: The real problem behind index.dat is that whether or not the indexes inside are still relevant or not, it keeps named urls forever. … As a user, I want to be able to turn on an hypothetical "auto-delete" of everything either anytime the web browser is restarted, or windows is restarted, or even on a schedule basis.

In IE7 and Windows Vista, deleted entries are actually zeroed out instead of just marked free till overwritten. There shouldn't be any residual URL names lying around if the entry was deleted.

As for "Auto-Delete" we have very limited support to do this right now. I believe there is an option to delete the content cache on IE exit, but nothing that allows cleaning up all the stuff. I suggest that you vote for this suggestion and this suggestion and if they don't cover what you are looking for you can file your own on the IE feedback site and/or the WNDP Feedback site.

2) Jorrit: Why is the directory still called IE5?

The cache directory is called Content.IE5 because we haven't changed the file format since IE5. The same is true for a lot of the registry settings. We have been tempted to change the file format a couple times, but there is always the cost of having to upgrade and potentially downgrade (on uninstall) and the benefits of changing the file format hasn't been worth the cost. So why not update the name but leave the file format alone? It is a smaller cost (yes, there is a cost, we would have to move the files on upgrade and downgrade) but there is no apparent benefit and the point of the AppData directory is to not be user facing.

3) Nicholas: Also, I remember there being issues with the index.dat becoming corrupted or full. This is what leads to the "right click-> view source -> nothing happens." bug, right?

Full is correct. Another manifestation of the same issue was the "Save Image" wanting to save a .bmp file. There are a number of features of IE and other programs that expect a file behind the cache entry, when the file isn't there they have to try something else. In the image case they give you the in memory bitmap version. With view source, they just give up. The root cause is that there are only so many entries that the index.dat file format can hold, and when this limited resource ran out of space, we weren't firing off the scavenger code to try to clear up some space. This has been fixed in IE7 and Windows Vista and maybe even on IE6, but you'll have to ask the IE team about that (lazy Ari…).

-- Ari Pernick

A bit about WinInet's Index.dat

wndpteam — Sat, 05 Aug 2006 01:55:00 GMT

Since a recent digg article and its underlying Wikipedia entry seems a little confused about index.dat, I’d like to give some more detail about what it is and what we have changed with it in IE7/Vista’s version of WinInet. As Jeffdav explained a while back, the index.dat file is a store for web related things; the URL content cache, cookies, RSS feeds, and visited links. Each of these collections, called a container, has their own index.dat file that lives in the user profile.

First, let’s talk a bit about these containers a bit more:

On most machines the biggest and most important container is the URL content cache index.dat. It lives (on vista) at \Users\<user>\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.IE5\index.dat. Content such as pages and images that we fetch from the web and that are cacheable get placed into this cache until they expire. The rules for if it is cacheable and when the entries expire from the cache are complex enough to warrant its own blog posting, but the common reasons that content doesn’t go in the cache is due to the server telling us not to via response headers, or the user telling us to not save any SSL resources to disk via the “Do not save encrypted pages to disk” option in Internet Options->Security->Advanced. Each cache entry has the URL and a file name to allow us to quickly find previously retrieved URLs and serve that content out of the content container. If a user just deletes all the files in the directory, the index.dat file will still contain all the URLs and paths until we realize that the cache entry is missing the file, and should be deleted from the index.dat.

The visited container is a listing of the URLs that you click on when web browsing, which is how IE can do URL auto completion and mark the links that you have visited a different color. This container is located on my vista box in \Users\<user>\AppData\Local\Microsoft\Windows\History\History.IE5\index.dat. Visited only needs to know about each URL once, since you have either visited the site or you haven’t.

The history containers are a set of containers for the different date ranges that IE displays, like today, yesterday, last week, etc. These containers are in \Users\<user>\AppData\Local\Microsoft\Windows\History\History.IE5\MShist01<date><date>\index.dat. Again top level links that you visit are stored in these containers. When the date shifts, IE does the bookkeeping often through merging these buckets.

The cookie container maps the cookie URLs to individual cookie files. It is stored in \Users\<user>\AppData\Roaming\Microsoft\Windows\Cookies\index.dat. The index.dat contains the associated URL, path to the cookie data and other cookie metadata information. You might notice that unlike the other containers this container is under a path called Roaming. This has to do with a domain feature that copy around your preferences from machine to machine on a domain. Cookies are one of those types of settings.

You might have also seen that starting in Vista almost all the containers have a Low\ directory with another index.dat. That is because these are specially marked directories that IE in protected mode can access. We completely partition off IE between the protected mode and normal modes. By design, normal only accesses the normal cache, cookies, etc. and by design and OS protection, protected mode only accesses the Low\ versions. The “how” of this partitioning is talked about on MSDN.

It’s important to note that pretty much all modern web browsers has to store these types of data stores. Firefox (1.5.0.6 at least) uses different types of file formats for each of its index.dat equivalents but they are there. The equivalent of the cache container index.dat is in Users\<user>\AppData\Local\Mozilla\Firefox\Profiles\59kuzm1n.default\Cache with the _CACHE_* files. The other containers are in the Roaming version of the directory over in Users\<user>\AppData\Local\Mozilla\Firefox\Profiles\59kuzm1n.default\. The history and visited are probably combined into one container; history.dat and the cookies container is cookies.txt.

There is one thing pretty special about WinInet and hence the index.dat files; they are OS components that many applications use, including explorer. That means that they were highly optimized for sharing data between processes. Each application’s copy of WinInet opens up the file for sharing read and write, but not for delete. As long as any program is using WinInet, the index.dat file can’t be deleted. If you could delete it, the applications actively using the file would probably crash or start corrupting data in memory. This also means that many applications leave their own footprints in the different containers. For Example: when Windows Music Player downloads an mp3 from the web from an URL, that file can end up in WinInet’s content cache.

So what’s new in IE7? Well the first thing is that IE made the interface for clearing up these files much simpler with “Cover My Tracks”. Under this idea WinInet made a bunch of improvements. The first improvement was in entry deletion. Those of you who remember the FAT file system on DOS might find the concepts behind this problem familiar. In DOS when you delete a file, the file is still around and special tools can undelete them unless some new files have already written over the old files. The way we use to delete entries in the index.dat file was pretty similar, the old URL data was marked free, but was still there, at least until it was overwritten by a new entry. In IE7 we now zero out the entry. Another problem was that some applications (cough Outlook Express cough) would write temporary files, like attachments, into the cache file directory to allow other applications to open them. If the index.dat file didn’t know about the file, we wouldn’t clean it up. Now when you use the “Delete Files…” button we delete everything in the directory regardless of if it’s in index.dat or not. There is one more feature in this area that I should mention even though it is not new. When we attempt to delete an entry from the cache, but can’t delete the actual storage file, we will still remove the entry from index.dat and stick the file on a list of things to periodically try to clean up.

Any Questions?

-- Ari Pernick

WinINet and WinHTTP IPv6 Support in Web Proxy Auto-Discovery (WPAD) scripts enabled in Windows Vista

wndpteam — Wed, 19 Jul 2006 01:06:00 GMT

Hi, my name is Jonathan Silvera and I am the WinINet and WinHTTP Program Manager. Today I would like to talk to you about changes we have made to add IPv6 support in the WinINet and WinHTTP WPAD helper functions.

The explosion of the Internet in the late 1990’s has caused an unexpected scarcity of IPv4 addresses, with the pool depleting on a daily basis. IPv6 provides a solution to this problem and although it is currently not widely deployed, its use will definitely become more prevalent in the future. WPAD is a protocol that allows web clients to automatically detect what the correct proxy configuration should be for their outgoing traffic. This is very useful for corporate deployments because it allows IT administrators to setup complex scripts that can route traffic for all clients to specific proxies based on the target server the clients are attempting to connect to. WinINet and WinHTTP support WPAD helper functions as defined by the Navigator Proxy Auto-Config (PAC) File Format specification, which has become a defacto standard. Unfortunately this specification was written in 1996 and does not define what the function behaviors should be when a WPAD script is deployed in an IPv6 capable network.

Here at Microsoft we are aware that IPv6 is the wave of the future and as part of a Windows wide push, we have required that all of our components support dual stack (IPv4 and IPv6) and IPv6 only networks. The first customer to approach us asking for IPv6 support in WPAD was our internal IT department after encountering issues with the IPv4 script dependency of their current deployment. When designing a solution to enable this scenario for Windows Vista, our team recognized the importance of providing a backwards compatible solution for IT administrators in order to avoid breaking the large number of existing corporate deployments with IPv4 script dependencies.

In order to meet our customer’s needs and support IPv6 without affecting existing deployment, we have added 6 new helper class functions as an extension to the Navigator Proxy Auto-Config (PAC) File Format specification and also added a new IPv6 capable function called FindProxyForURLEx that administrators can implement in the WPAD script.

New Proxy Helper API Definitons:

In order to take advantage of the IPv6 enabled functions, IT administrators must define within their WPAD script a function called FindProxyForURLEx (url, host) which will replace the legacy FindProxyForUrl (url, host) function. Only from the new FindProxyForURLEx function will administrators be able to execute the new functions.

We also attempted to simplify work for developers, by aligning our functions to return different types of IP addresses in the same order of preference as other networking components, specifically IPv6 addresses followed by IPv4 addresses (i.e. current behavior for Winsock’s getaddrinfo(..) function).

The following tables explain the differences between the new WPAD helper functions and the legacy WPAD helper functions. The new functions are marked with a *

Functions	Input	Output	Reason for Change
dnsResolve	Host	IPv4 address	Ex function will return a list of IPv6/IPv4. Necessary since IPv6 or IPv4 addresses can have multiple unicast addresses for a single interface
dnsResolveEx*	Host	List of IPv6/IPv4 addresses

Functions	Input	Output	Reason for change
isResolvable	IPv4 host	TRUE / FALSE	The Ex function will return TRUE if a host can resolve to an IPv6 or IPv4 address. The legacy function only returns TRUE if the host resolves to an IPv4 address
isResolvableEx*	IPv6/IPv4 host	TRUE / FALSE

Functions	Input	Output	Reason for change
myIPAddress	none	IPv4 address	Ex function will return a list of IPv6/IPv4. Necessary since IPv6 or IPv4 addresses can have multiple unicast addresses for a single interface
myIPAddressEx*	none	List of IPv6/IPv4 addresses

Functions	Input	Output	Reason for change
isInNet	Host, Dot separated IP address pattern, IP address Mask	TRUE / FALSE	Provide an IP version agnostic way to find if an IP address is in a given subnet. Also, the mask notation in IPv4 is deprecated.
isInNetEx*	IP Address IP Prefix	TRUE / FALSE

Functions	Input	Output	Reason for change
sortIPAddressList*	List of IPv6/IPv4 addresses	Sorted List of IPv6/IPv4 addresseslist of IPv6/IPv4 addresses	There is no counterpart legacy function because legacy functions only returned a single IPv4 address, therefore there was no need to sort

Functions	Input	Output	Reason for change
getClientVersion*	None	WPAD engine version number	Currently this function returns version 1.0. We added this function to allow IT administrators to update their WPAD to work with different versions of the WPAD engine without causing breaks to their existent deployment.

Again, see the extension spec for more details.

We are truly proud of our work extending the Navigator Proxy Auto-Config (PAC) File Format specification to work with IPv6 addresses while ensuring backwards compatibility, as we believe these changes will prove useful current and future IPv6 users. These changes are available for WinINet/WinHTTP in Windows Vista starting in Beta 2, for WinINet on XP and Windows 2003 through IE7 starting in Beta 2 and will be available in a future release of the .Net Framework’s system.net. We would love to hear feedback from the community regarding these changes and if you are planning to deploy a WPAD script in an IPv6 network please let us know about your experience.

Thanks
-Jonathan Silvera

Calling WinInet Asynchronously from managed code

wndpteam — Tue, 18 Jul 2006 23:29:00 GMT

Hi, I’m Reymarx Gereda, SDET at the Networking Development Platform in the Windows team. Recently I saw a question in one of the MSDN forums about calling WinInet API from managed functions; this is something we did for testing the component, and here I want to share something we found when calling some of the WinInet APIs in an asynchronous fashion. I’ll talk about specifically of the use of InternetReadFile in this mode. I assume you understand the asynchronous behavior of WinInet and a basic knowledge of interacting with native components using PInvoke.

When calling functions in WinInet from managed code using PInvoke, you can map the buffer arguments to managed byte arrays (byte[]) and pass them to the functions without having to pin the memory used by those buffers. For example, for InternetReadFile, the native signature is as follows:

BOOL InternetReadFile(
  HINTERNET hFile,
  LPVOID lpBuffer,
  DWORD dwNumberOfBytesToRead,
  LPDWORD lpdwNumberOfBytesRead
);

And the PInvoke declaration could look like this:

[DllImport("WinInet", SetLastError = true, ExactSpelling = true)]
internal extern static bool InternetReadFile(
    IntPtr hFile, 
    byte[] buffer, 
    Int32 numberOfBytesToRead, 
    ref Int32 numberOfBytesRead
);

This signature works really well in the synchronous case, since the marshaler takes care of pinning the memory used by the buffer for the duration of the function call.

For the asynchronous case, when the function returns ERROR_IO_PENDING, WinInet requires access, at the completion of the call, to the buffer passed in the function call; and it could happen that the memory location for buffer used in the function call, has changed due to the work of the garbage collector. In this case, you’re responsible for pinning the memory used by the buffer until the asynchronous call completes. In the case of InternetReadFile, WinInet needs to update the information inside the buffer and numberOfBytesRead parameters, hence the PInvoke declarations needs to be changed to:

[DllImport("WinInet", SetLastError = true, ExactSpelling = true)]
internal extern static bool InternetReadFile(
    IntPtr hFile, 
    IntPtr buffer, 
    Int32 numberOfBytesToRead, 
    IntPtr numberOfBytesRead
);

The change on the parameters type to IntPtr requires an additional overhead of pinning the memory used by this two variables and keeping them until the asynchronous call to the native API completes. To pin the memory, use the methods provided by the Marshal class; it also provides methods for dumping the content of the memory in a byte array, string or instance of a class/structure.

To use the Marshall class with the buffer passed to InternetReadFile, you can use something similar to this:

Remarks: To help illustrate the point, this code omits exception handling

...
//At this point a request has been sent to the server 
//and we’re going to read the response’s entity body
Int32 buffsize = 1024;
Int32 bytes2Read = 1024;

//Pointers to use with the WinInet call
IntPtr bytesRead = Marshal.AllocHGlobal(sizeof(Int32));
IntPtr buff = Marshal.AllocHGlobal(buffsize);

//Variables to copy the data returned from WinInet
Int32 totalBytesRead = 0;
Int32 bytesNum = 0;
byte[] tempByteArray = new byte[buffsize];

//We’ll allocate the response in a MemoryStream object
MemoryStream memStream = new MemoryStream();

InternetReadFile(this.Handle, buff, bytes2Read, bytesRead);
//Verify if the call completed asynchronously 
// Marshal.GetLastWin32Error()==ERROR_IO_PENDING (997)
//And wait for the completion of the call
... //Perform some other operations while the call completes

//At the completion of the call 
//Extract the information from the buffer
bytesNum = Marshal.ReadInt32(bytesRead);
totalBytesRead += bytesNum;
Marshal.Copy(buff, tempByteArray, 0, bytesNum);
memStream.Write(tempByteArray, 0, bytesNum);

//Repeat the call to InternetReadFile until bytesNum==0

//At the end of the process, release the memory allocated
Marshal.FreeHGlobal(bytesRead);
Marshal.FreeHGlobal(buff);

Using this approach you’ll ensure the appropriate behavior of the code, otherwise you could end with access violations or incomplete data inside the buffer.

For more information about the PInvoke wrappers, look at http://www.PInvoke.net.

-- Reymarx Gereda

Wanted: Developer feedback for our "Next Generation" client HTTP stack

wndpteam — Tue, 06 Jun 2006 22:48:00 GMT

As we begin the planning phase for our "Next Generation" client HTTP stack, we would love to hear from developers using our existing APIs (WinINet and WinHTTP). Please help us understand what your experience has been so far and what you would like to see in any future releases by filling out this survey.

Thanks

-Jonathan Silvera

WNDP Connect Site gets an upgrade!

wndpteam — Tue, 06 Jun 2006 19:32:00 GMT

Last year we setup a small site on connect.microsoft.com in order to let our blog readers, developers and users file bugs, make suggestions and get some conntent like whitepapers and samples early. The downside to the site was that you couldn't easily deep link and it required a Windows Live (aka Passport) login. Well the folks at connect.microsoft.com have launched a new version of connect that enables access to much of the site without a login including readonly access to the bugs filled there much like the popular MSDN Product Feedback Center.

Check it out: http://connect.microsoft.com/WNDP

-- Ari Pernick

TLS enabled by default in IE 7.0 and WinINet

wndpteam — Wed, 12 Apr 2006 22:00:00 GMT

Transport Layer Security (“TLS”) and Secure Sockets Layer (“SSL”) are transport-layer protocols for server and client authentication, data encryption, and data integrity of application layer protocols such as HTTP, SMTP, and FTP. In order to provide a more secure experience, Microsoft is enabling TLS and extensions by default in Internet Explorer 7.0 and WinINet. Starting with Windows Vista, Schannel - the Security Support Provider (SSP), which implements TLS 1.0, will send “Extensions” during SSL/TLS protocol negotiation. The negotiation of the best common protocol occurs automatically between hosts. However, if needed, users can toggle TLS and Extensions in the “Advanced” tab of the “Internet Options” dialog. We do not supply a mechanism for toggling Extensions independently of TLS.

The reason that we have enabled TLS Extensions (aka SSLv3.1) in Windows Vista by default is that TLS provides a number of enhancements beyond SSLv3. Some of the benefits of TLS include:

TLS is an IETF standard.
TLS provides more secure hashes through its use of the HMAC algorithm instead of the SSL MAC algorithm.
TLS adds a number of new alert messages.
TLS allows the use of an intermediary authority.
TLS support “extensions”, an RFC-specified mechanism for extending the protocol with additional functionality.
TLS supports AES and Elliptic Curve Cryptography-based (ECC) cipher suites

Although the SSLv3 and TLSv1 protocols differ only slightly, they are not interchangeable. Client and server must agree on which protocol, version, and extensions to use during the protocol negotiation phase. This negotiation happens transparently between the client and server – with TLS backing-off to SSLv3 when both parties do not claim support for TLS. Over the years, the IETF has expanded TLS beyond its original RFC 2246 to include support for Extensions and additional cipher suites such as AES, Kerberos and ECC. The extensions to TLS 1.0, defined in RFC 3546, provide additional behaviors beyond what the original drafters of RFC 2246 envisioned, such as:

IETF has added cipher suites that rely on algorithms beyond the original RC2, RC4, IDEA, DES and 3DES to the TLS protocol.
IETF has added support for virtual hosting of HTTPS websites on a single IP address/port with the “server_name” extension. With “server_name”, the client sends the hostname during its “extended” Client Hello to the server. This allows the TLS server to select the appropriate server certificate for the client. This is similar in capabilities brought on by the host-header requirement of HTTP/1.1. (server_name extension)
IETF has added support for client certificate URLs. Client Certificate URLs allow the client to tell the server to “go” to a particular URL for the client’s certificate, which helps in network-constrained environments, e.g. mobile phone. (client_certificate_url extension)
IETF allows for MAC truncation which helps in bandwidth constrained environments by allowing the use of an 80-bit (10 byte) truncated HMAC. (truncated_hmac extension)
RFC 3546 also added support for the certificate status extension (e.g. via OCSP) which allows the server to send revocation state of the server certificate to the client.

Although the vast majority of modern web servers behave correctly when receiving a handshake containing TLS 1.0 extensions, some web servers do not respond in the RFC required manner when support does not exist for extensions. RFCs for TLS require that servers simply ignore extensions that appear in the “client hello” message that they do not support, which effectively disables those extensions for the session. During our testing using tools like Microsoft Network Monitor (“netmon”), we learned that servers that do not understand TLS + extensions may respond incorrectly in one of three ways:

The server replies with a message that specifies that there was an invalid handshake;
The server unexpectedly resets the TCP connection; or
The server ignores the request and does not respond, which keeps the connection open past the WinINet 30 second connection timeout

TLS requires that a server ignore unsupported or unfamiliar extension requests. The client will know that the server does not support the particular extensions when it receives the “server hello” without the extensions it requested in the response. With an invalid handshake response (#1), we back-off and try SSLv3. Helping the problematic servers in cases of a TCP RST or timeout is not possible because these responses (or lack of) are not clear indicators of a problem with TLS extensions.

We want to be sure that help the TLS ecosystem and encourage web server creators and site owners to take advantage of the benefits offered by TLS extensions. Masking of these problems does little to help with adoption, in addition:

Interpreting a TCP reset from the server as a faulty TLS extension server is unacceptable because TCP resets can occur for other, non-TLS related, reasons such as an overloaded web server, proxy server, or firewall;
Interpreting a lack of response as an issue with TLS extensions is also not the answer. Network congestion anywhere along the route could be the real issue.

We decided that the best solution is to combine smart engineering with proactive communications. This means that IE7 and WinINet will simply scale-back and attempt to use SSLv3 when enabled in Internet Options whenever we receive an invalid handshake message from a server. For proactive communications, the Internet Explorer team is working with the known web servers and site owners to ensure that their site properly handles the TLS 1.0 extensions.

- Billy Anders

WinINet ETW Tracing Support in Windows Vista

wndpteam — Wed, 29 Mar 2006 06:29:00 GMT

In order to enhance debugging of wininet.dll for application developers, Microsoft has added ETW tracing support for this component. The feature will be available beginning with Windows Vista Beta 2 builds.

To use the feature, wininet.dll users should open a command prompt window with elevated privileges:

1. Click the Start button, point to All Programs, and then click Accessories.

2. Right-click Command Prompt, and then click Run Elevated or Run as administrator (depending on what Vista build you are running).

Initialize WinINet tracing:

logman start "wininettrace” -p “microsoft-windows-wininet” –o “wininettrace.etl” –ets

This command will initialize the ETW framework for WinINet tracing, using wininettrace.etl as the output file for trace data.

Log Events:

Run IE or an application that exercises the WinINet APIs

Stop tracing:

logman stop “wininettrace” -ets

Create a human readable (xml) log file:

tracerpt “wininettrace.etl” –y –o “wininetracelog.xml” –of xml

This command will generate an XML log file called wininetracelog.xml that contains the WinINet events logged.

--Jonathan Silvera