Windows Core Networking : Http.sys

Event Tracing in Http.sys: Part 3 - Typical Request

wndpteam — Thu, 01 Feb 2007 19:00:00 GMT

Now that you are somewhat familiar with a single ETW event, let’s illustrate what a typical HTTP request looks like. Here, I’ve made a simple HTTP request to a web server, IIS7 in this case.

I’ve taken the liberty of pulling out all important data from the XML file. You may notice that I've placed all the Data in the same cell for each event, this is simply to save space in this web format.

Event Name	Data
ConnConnect	ConnectionObj=0x840C9008 LocalAddr=[::1]:80 RemoteAddr=[::1]:50438
ConnIdAssgn	RequestId=0xFB00000080000004 ConnectionId=0xFB00000060000003 ConnectionObj=0x840C9008
RecvReq	RequestId=0xFB00000080000004 ConnectionId=0xFB00000060000003 RemoteAddr=[::1]:50438
Parse	RequestObj=0x840D8A38 HttpVerb=4 Url=http://localhost:80/
Deliver	RequestObj=0x840D8A38 RequestID=0xFB00000080000004 RequestQueueName=DefaultAppPool Url=http://localhost:80/
FastResp	RequestId=0xFB00000080000004 ConnectionId=0xFB00000060000003 StatusCode=200 Verb=GET EntityChunkCount=0 CachePolicy=0
FastSend	RequestId=0xFB00000080000004 HttpStatus=200
ConnCleanup	ConnectionObj=0x840C9008

The first part of almost any Http.sys event trace are a series of events that shows we’ve received a request:

ConnConnect - The first Http.sys event one will see is ConnConnect. This signals that a new connection has been initiated by a client. We log the memory address of the ConnectionObj here, the Local Address and port that received the request as well as the Remote Address, where the HTTP request originated. Here you can see that I’ve sent a request from localhost:Port 50438 to localhost:Port 80.
ConnIdAssgn – Http.sys will next assign a ConnectionId and a RequestId to the new incoming connection/request. We assign both these identifiers because a connection could have multiple HTTP requests associated with it. Notice that the ConnectionObj is the same for the first two events indicating that we are dealing with one connection.
RecvReq – This event asserts that’s we received the request and are about to parse it.

The rest of the trace can be interpreted as follows.

Parse – After Http.sys parses a request, we log the Parse event. With it comes the memory address of the HTTP_REQUEST object, the requested URL and the HTTP Verb. Here we are logging the enum value of the verb from our HTTP_VERB enum located in our public header file http.h.
Deliver – When the user mode application calls HttpReceiveHttpRequest() we hand them the parsed request (in the for of an HTTP_REQUEST structure) and log this event. Along with the RequestObject and RequestId, we also log the Url and the name of the RequestQueue where we’ve delivered the request. For IIS, their default queue is named “DefaultAppPool”.
FastResp – The FastResp event means the server application, in this case it’s IIS, has passed the given data to Http.sys for delivery (with HttpSendHttpResponse()). Here the application is responding to the request with requestId=0xFB00000080000004 on Connection with Id 0xFB00000060000003 with the Verb being GET. IIS is responding with status code 200, is not caching the send and is sending no entity chunks.
FastSend – Http.sys has queued the response, status 200, for the given requestId for sending
ConnCleanup – Cleanup has begun for the given request object. This is a result of a TCP Reset or mutual TCP Fins.

In the next ETW posts, we will discuss diagnosing typical Http.sys problems using Event Tracing.

-Jeff Balsley

Event Tracing in HTTP.sys: Part 2 - Anatomy of an Event

wndpteam — Thu, 25 Jan 2007 19:00:00 GMT

Continuing the series on Http.sys ETW Tracing, we will dissect an event as displayed in the default XML format. To review creating an ETW trace, see Capturing a Trace

Pictured is an example of a typical event in a trace. Note how the event is wrapped in an <Event> element. This particular event happens to be our Parse event:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
   <System>
      <Provider Name="Microsoft-Windows-HttpService" Guid="{dd5ef90a-6398-47a4-ad34-4dcecdef795f}" />
      <EventID>2</EventID>
      <Version>0</Version>
      <Level>4</Level>
      <Task>1</Task>
      <Opcode>12</Opcode>
      <Keywords>0x8000000000000002</Keywords>
      <TimeCreated SystemTime="2006-09-06T16:59:22.251258100Z" />
      <Correlation ActivityID="{80000002-0001-fb00-0000-000000000000}" />
      <Execution ProcessID="4" ThreadID="1276" ProcessorID="1" KernelTime="15" UserTime="0" />
      <Channel>Microsoft-Windows-HttpService/Trace</Channel>
      <Computer />
   </System>
   <EventData>
      <Data Name="RequestObj">0xFFFFFA8001BB5320</Data>
      <Data Name="HttpVerb">4</Data>
      <Data Name="Url">http://localhost:80/RequestQueueTests/</Data>
   </EventData>
   <RenderingInfo Culture="en-US">
      <Level>Information </Level>
      <Opcode>Parse </Opcode>
      <Keywords>
         <Keyword>Flagged on all HTTP events dealing with request processing </Keyword>
      </Keywords>
      <Task>HTTP Request Trace Task </Task>
      <Message>Parsed request (request pointer 0xFFFFFA8001BB5320, method 4) with URI http://localhost:80/. </Message>
      <Channel>HTTP Service Channel </Channel>
      <Provider>Microsoft-Windows-HttpService </Provider>
   </RenderingInfo>
</Event>

The <event> node has three sub-nodes, <System>, <EventData>, and <RenderingInfo>. Each contains specific data. The <System> node holds information such as the name of the traced component and the event time stamp. The <EventData> node holds information specific to the event. The Http.sys dev decides what information should be logged for each of our events. The <RenderingInfo> node holds much of the same information as the <System> node, but in a more readable form. Also, this node holds the <Message> sub-node, which describes the event. Below is complete description of the elements contained under each sub-node.

The System node

Provider: This contains the name and the Guid of the component responsible for the event.
EventID: Each event has a unique EventID associated with it. In the case of Http.sys, we have 65 unique events.
Version: Beyond scope of this post.
Level: This is the verbosity level of the event. Each event has a verbosity level, LogAlways, Critical, Error, Warning, Informational, and Verbose numbered from 0 to 5 respectively. I generally trace with the highest verbosity level to collect the maximum amount of data.
Task: Beyond scope of this post.
Opcode: A numeric value representing the name of the particular event, short for “Operational Code”.
Keywords: Keywords are tags attached to events in order to group various events based on their use.
TimeCreated: This is the timestamp of the event. It includes the data in YYYY-MM-DD format and the time in HH:MM:SS.SSSSSSSSS format.
Correlation: This includes the ActivityID for the request.
Execution: Beyond scope of this post.
Channel: Again, this is the name of the trace provider.
Computer: Beyond scope of this post.

The EventData node

Data: Each request has it’s own unique data. Typically, a data node includes the name of the data (show as the value of the Name attribute) and the data itself.

The RenderingInfo node

Level: Like the Level element under the <System> node, this is the verbosity of the event. However, instead of the numeric level, this gives the name of the level, such as Error or Informational.
Opcode: This is the same Opcode as above, only not rendered in it’s numeric form. Here it is displayed via the friendly name of the event. Hereafter, I will refer to the opcode as the “EventName”.
Keywords: Keywords are used to group events into groups.
Task: NA.
Message: This is a friendly, more readable message describing the event.
Channel: Beyond scope of this post.
Provider: This is the name of the provider.

I know this amount of information appears overwhelming, especially considering a typical trace can have hundreds or even thousands of events and be several megabytes in size. Consequently, when using ETW, I “cherry-pick” a few bits of information from the XML file:

The event name, or opcode element, under the <RenderingInfo> node and all the Data elements under the <EvenData> node.

EventName

Data1

Data2

Data3

Data4 ...

When sorted in a table format it’s easy to follow one event to another. We’ll go over the best ways to do this (using a script, xslt, etc...) in a separate post.

In the next ETW post, I’ll go over an ETW trace for a typical HTTP transaction with multiple events.

-Jeff Balsley

Event Tracing in HTTP.sys: Part 1 - Capturing a Trace

wndpteam — Thu, 18 Jan 2007 19:00:00 GMT

Hi, I'm Jeff Balsley a test developer in the HTTP.sys team in Windows Networking. In this series I will be showing you how to use and interpret ETW tracing in HTTP.sys.

Http.sys

Http.sys is the kernel-mode HTTP listener that first debuted in Windows 2003 Server. We are often closely associated with IIS6 as they are our most visible customer. The second version of our APIs will debut with Vista. Along with new features and extended functionality, we’ve also vastly improved our event tracing using the new Crimson APIs. (An intro to Crimson can viewed on the .NET Show.)

What is ETW?

Crimson is the next generation of the Windows Event Log shipping in Windows Vista and Longhorn Server. It is a high-performance, low overhead and highly scalable tracing facility provided by the Windows Operating System. Or, more simply, Event Tracing is similar to placing a series of “printf” statements throughout your code to aid in debugging and troubleshooting. ETW can be enabled or disabled dynamically and uses an efficient event buffering system for increased performance.

Why is it useful for Http.sys?

When Http.sys first appeared in Windows Server 2003, it lacked basic diagnosing and analyzing tools. While it did include some event traces, they were often unhelpful in identifying common customer issues. Often, customers had to rely on a packet sniffer and/or a kernel debugger to probe their Http.sys issues. For Vista and Longhorn Server, it was a goal of the Http.sys dev and test team to include ETW events that would aid customers in analyzing their Http problems. To do so, we analyzed customer issues and complaints occurring after the release of Windows 2003 server and included events that would easily reveal the problem at hand.

Creating an ETW trace

Creating an Http.sys ETW trace is similar to creating one for wininet. To start tracing events in Http.sys use the logman.exe tool. Logman.exe ships with every Windows installation, and can be used right out of the box. To enable ETW, type:

C:> logman start httptrace -p Microsoft-Windows-HttpService 0xFFFF -o httptrace.etl -ets

Here httptrace is the name of our trace. This will be used to identify this particular event trace in further commands. Logman needs to know which components of Windows to trace. Since we are only interested in tracing Http.sys, we pass the provider name, "Microsoft-Windows-HttpService". The 0xFFFF signifies we want full tracing, or all trace events. Our event trace will be written to a raw binary file, httptrace.etl. Please note that you must be an elevated Admin to use the logman command.

That's all that's needed to turn tracing on. Recycling services is not required as ETW is dynamic. Now, any activity in http.sys will be logged in our binary trace file, httptrace.etl.

To stop tracing we use the logman tool again:

C:> logman stop httptrace -ets

This will flush the event cache to your .etl output file, httptrace.etl in this case. By itself, the .etl file is of little use. We can create a human readable file in either XML or .csv format by using the tracerpt command (admin privs not necessary here):

For .csv output:

C:> tracerpt.exe httptrace.etl –of CSV -o httptrace.csv

For XML output:

C:> tracerpt.exe httptrace.etl -of XML -o httptrace.xml

To maximize the amount of useful information in each trace I would advise the XML format. However, the XML trace file can grow rather large quickly. For these traces, it may be best to stick with a simple .csv format.

In the next ETW Post I will look at multiple ways to consume and view this binary tracing data.

-- Jeff Balsley

URLACL Setting Day

wndpteam — Tue, 24 Oct 2006 01:54:00 GMT

Looking around the web today, I noticed that Keith Brown has a sample of using HTTP_SERVICE_CONFIG_URLACL_SET from managed code. Kenney Wolf, a while back, found the Windows Vista way of configuring it via netsh.

-- Ari Pernick

Content-Encoding != Content-Type

wndpteam — Mon, 21 Aug 2006 18:00:00 GMT

RFC 2616 for HTTP 1.1 specifies how web servers must indicate encoding transformations using the Content-Encoding header. Although on the surface, Content-Encoding (e.g., gzip, deflate, compress) and Content-Type (e.g., x-application/x-gzip) sound similar, they are, in fact, two distinct pieces of information. Whereas servers use Content-Type to specify the data type of the entity body, which can be useful for client applications that want to open the content with the appropriate application, Content-Encoding is used solely to specify any additional encoding done by the server before the content was transmitted to the client. Although the HTTP RFC outlines these rules pretty clearly, some web sites respond with "gzip" as the Content-Encoding even though the server has not gzipped the content.

Our testing has shown this problem to be limited to some sites that serve Unix/Linux style "tarball" files. Tarballs are gzip compressed archives files. By setting the Content-Encoding header to "gzip" on a tarball, the server is specifying that it has additionally gzipped the gzipped file. This, of course, is unlikely but not impossible or non-compliant.

Therein lies the problem. A server responding with content-encoding, such as "gzip," is specifying the necessary mechanism that the client needs in order to decompress the content. If the server did not actually encode the content as specified, then the client's decompression would fail.

Here is a potentially over-simplified example:

Windows Vista Networking Rocks!
Jvaqbjf Ivfgn Argjbexvat Ebpxf!

If I mistakenly claim that string a) has been encoded using the simple ROT-13 obfuscation scheme when in actuality it has not, then the decoded message b) will be very different than the intended message.

Since the AI engine for WinINet isn't yet ready for production (joke), we try and work-around these non-compliant server responses but that isn't the right long-term approach. The fix and the ask, is for web server, extension and application authors to test their servers to see if they exhibit this behavior and if so fix their implementations before we remove our client-side hacks.

To test your server for compliance, issue a simple HTTP 1.1 request, including the "Accept-Encoding: gzip" for a .gz file and inspect the headers. If you see Content-Encoding: x-gzip or gzip, then the server is either gzip-encoding the already gzipped file or it is misstating that the content has been encoded by the server before transmission and therefore perpetuating client HTTP stacks, such as WinINet, having to absorb and hide this bad server behavior.

-Billy Anders

Buffering in HTTP.SYS

wndpteam — Tue, 15 Aug 2006 18:00:00 GMT

My name is Chun Ye. I am a Software Design Engineer in the Microsoft Windows Networking Transports & Connectivity group. I'm here to describe the scenarios under which an application using HTTPAPI.DLL should set the HTTP_SEND_RESPONSE_FLAG_BUFFER_DATA flag. This flag applies to both HttpSendHttpResponse and HttpSendResponseEntityBody APIs.

This is the section in http.h that describes this flag:

//
// HTTP_SEND_RESPONSE_FLAG_BUFFER_DATA - Specifies that a caller wants the
// response to complete as soon as possible at the cost by buffering partial
// or the entire response.
//

#define HTTP_SEND_RESPONSE_FLAG_DISCONNECT 0x00000001
#define HTTP_SEND_RESPONSE_FLAG_MORE_DATA 0x00000002
#define HTTP_SEND_RESPONSE_FLAG_BUFFER_DATA 0x00000004

First of all, some background information related to the introduction of the buffer-data flag. Before Windows 2003, IIS 5.0 is implemented using winsock, which enables buffering by default. However, when IIS 6.0 in Windows 2003 moves to HTTP.SYS, users have encountered performances issues when sending a response or entity bodies. There are two underlying causes for this performance issue.

The first cause is related to the TCP Delayed ACK. The TCP stack in Microsoft Windows' implementation turns on the Delayed ACK algorithm by default. What this means is that the TCP receiver sends the ACK after every second segment or until a delayed ACK timer expires (the Windows TCP stack sets the timer value to 200 milliseconds). When an application sending a response or entity bodies results in odd number of TCP segments, the Delayed ACK is introduced, and the application's IO is blocked until the ACK is received.

Another issue that an application can encounter send performance issues is when the underlying network has long latencies (Round Trip Times). In this case, the ACKs take long time to come back therefore the IO is blocked for a long time until all ACKs are received.

An application using only a single send can be hit by both issues mentioned above. The result is under network utilization because the application is not keeping the network pipe full and it is serializing every send by RTT units of time.

To solve these problems, starting Windows 2003 SP1, HTTP.SYS introduced the HTTP_SEND_RESPONSE_BUFFER_DATA flag for the send APIs. HTTP.SYS applies the winsock-like buffering when the flag is set. By buffering, HTTP.SYS copies the user data into its own internal buffer and sends it to the transport layer. The original IO from the user mode is completed immediately once the data reaches the transport layer but without waiting for the all the ACKs. By buffering the data in the kernel, we are trading CPU/memory for network utilization. This allows a "one send at a time" application to queue sends in parallel.

Here is the recommendation for the three types of applications in terms of their IO models:

An application that does synchronous IO. It is recommended that such applications always set the HTTP_SEND_RESPONSE_BUFFER_DATA flag for the send IOs.
An application that does asynchronous IO but keeps maximum only one outstanding IO for send on each connection. Here it is recommended that the application should set the HTTP_SEND_RESPONSE_BUFFER_DATA flag for all the intermediate send IOs (when the HTTP_SEND_RESPONSE_MORE_FLAG flag is also set).
An application that utilizes multiple outstanding send IOs on each connection. For instance, an application sends out the next entity body without waiting for the previous send completion. In this case, there is no need to set the HTTP_SEND_RESPONSE_BUFFER_DATA flag since such an application is not blocked in any way. New and truly asynchronous application should try to use this IO model.

While setting the buffering flag solves the performance issue for most of the applications that fall under category 1 and 2 above, there are certain negative sides that the user should be aware of:

Setting the flag increases CPU and memory usage. Buffering requires extra memory to store the user response or entity bodies. So this means extra CPU cycles and memory usage. Compared to an truly asynchronous application (category 3 above), an application that sets the buffering flag can expect some decrement in terms of clients being serviced. A trade-off has to be made between application simplicity and system scalability.
When the buffering flag is passed, the send API will be completed as soon as the data is copied and passed to the transport layer. Thus, the completion status of the send API is not indicative of the completion status of the actual send.

- Chun Ye

Http.sys URL Registrations and Service SIDs

wndpteam — Wed, 12 Jul 2006 03:56:00 GMT

A common question we get when configuring HTTP.sys to listen on a URL (though HttpAddUrl, HttpAddUrltoUrlGroup, HttpListenerPrefixCollection or even CREATE ENDPOINT) is how you claim and protect your name space.

The first issue that people need to understand is the precedence rules for URLs. If you are really worried about having the whole namespace to yourself, then you may want to use the strong wildcard http://+:80/bar style URL so that your registration overrides hostname and IP address based reservations. A nice discussion of how this works is described on MSDN. On my laptop running a post Beta 2 build of Vista, 4 of the 7 reservations I have use the strong wildcard, including BITS and the Windows Media Player Network Service.

The next issue is understanding that HTTP.sys uses the Windows security model, which means that we use the process security token and compare it against the registered ACL that you probably set up during install time (while you had admin credentials). In an ideal world we would all have secure application Ids and just ACL the namespace resource to the application ID. On Windows XP and Windows 2003, the best level of isolation you get is the user account or groups your process is running under. If all users on the system needs to be able to run the application then you have to have wide open ACLs on the reservation, which means you have very little protection from any random application. The best you can do right now is to create your namespace with something that other people aren’t likely to step on. This is the same isolation model that applications writing to a user’s “Application Data” or even “Program Files” folders have today. If appropriate, you probably should use your company and/or app name as part of the URL namespace to help avoid collisions.

The same type of ACL problem exists for services, which often run under a couple of well known accounts, so in practice there isn’t much isolation except away from user processes. In Windows Vista things gets better because of a Service Hardening feature called Service SIDs. Since the service manager knows what service is running in any given process, it can maintain a list of per Service SIDs that show up when http.sys does its security checks. They built this functionality into windows security itself, so the Service SID concept works on anything that uses ACLs like files and registry keys. You can quickly recognize a Service SID because it starts with “S-1-5-80-“. Looking again at the default urlacl registrations on my laptop (“netsh http show urlacl”), I can see that two registrations are using Service SIDs in their namespace ACLS: BITS and Windows Media Player Network Service.

Service SIDs are generated automatically from the Service Name (this is the short name not the longer more friendly Display Name) in such a way that it is the same on every windows machine. When I run “sc showsid BITS” (which is easier then calling LookupAccountSid), I get back the exact same Service SID that was in HTTP’s URLACL list. A really astute observer might notice that the output of netsh actually lists user names that you have never seen before: NT SERVICE\BITS and NT SERVICE\WMPNetworkSvc. The Service SID alone isn’t reversible, but the system has another feature that caches the service SIDs and gives them friendly names for when you call the SID lookup API LookupAccountName. For an implementing service the only thing you need to call (with proper rights) is ChangeServiceConfig2 with SERVICE_CONFIG_SERVICE_SID_INFO structure set to SERVICE_SID_TYPE_UNRESTRICTED (or “Sc sidtype XXX unrestricted”) and an administrator could actually use the command line tools to alter the service and namespace URLACLs to lock down on vista a service that doesn’t know better.

Of course we still have to deal with the process as the security boundary which leads to an interesting “attack”. If a service MISBEHAVING is running in the same process as out ANGEL service, then the process will have the SIDS for both services and MISBEHAVING can register ANGEL’s URLs, but this only works while ANGEL is started and in the same process ANGEL’s service SID goes away when ANGEL stops. Of course the administrator had to have installed MISBEHAVING to begin with.

What is especially neat about this feature is that it wasn’t developed for HTTP.sys and we didn’t change any code to support it. It was free :). With Service SIDs I’ve had a glimpse at how an application id could work in Windows and I hope in some future release it gets added, finishing off this class of namespace security issues for good.

-- Ari Pernick (arip)

WOW64 on HTTP.sys

wndpteam — Tue, 11 Jul 2006 01:56:00 GMT

Back when Windows Server 2003 SP1 shipped, HTTP.sys hit an important milestone, WOW64 support.
What is WOW64 support and why is it important?
I’m glad you asked.

WOW64 lets 32 bit applications run on top of 64 bit windows. WOW64 support for HTTP.sys is an important piece of letting 32 bit IIS worker processes run on top of 64 bit windows. You want this because of all the existing applications that run on top of and plug in to IIS, like ISAPIs are still distributed as 32 bit binaries. You want the 64 bit kernel because big sites are getting squeezed from two directions; the number of connections and memory address space use in user mode. The number of connections that windows and HTTP.sys can support is dependent on non-paged pool memory (kernel memory that can’t be paged out to disk), since every connection in TCP/IP and HTTP.sys require a non paged pool structure. Some people try to trade off between these two limits via the /3gb PAE switch which constrains kernel mode addressing to 1 GB (down from 2 GB) which causes there to be less room for non-paged pool and hence constrains the number of simultaneous connections. 64-bit addressing gives us a lot of room to address non-paged pool in the kernel and hence can greatly expanded the connection limit. On the user mode side each process now gets much of the full 4 GB address space for 32 bit processes running under WOW64. The people who run Microsoft.com have a nice article about the trade off and how much more memory they get in both spaces when they moved to 64 bit.

So what was the work needed to make this happen (and why didn’t we ship support for this in win2k3 original)?

For us it is nontrivial work that didn’t fit in the win2k3 ship schedule. HTTP.sys is a driver that communicates with user mode via an IOCTL interface and it turns out that the vast majority of HTTPAPI.dll is just mapping the API to an IOCTL (Part of testing HTTP.sys we have to test the IOCTL interface as well as the API). This interface DeviceIOControl, much like IDispatch::Invoke, turns all possible function calls in to a single call. It takes a handle, the IOCTL code, an input buffer, an output buffer, the number of bytes returned and an OVERLAPPED structure. Depending on what you stick in the buffer the IO manager takes care of almost all the 64/32 bit book-keeping, like the change in HANDLE size and work with the top level buffers. However HTTP.sys structures tend to be a bit more complex. Looking at HTTP_RESPONSE we see at least two pointers who can’t point to other structures. Since the IO manager isn’t psychic we have to do the translation of those pointers our selves. A lot of the HTTP.sys IOCTL code has bits like this (much like the guidelines under IOCTL Support on WHDC 64-bit checklist):


#if defined(_WIN64)
if (IoIs32bitProcess(Irp)) {
GET_OUTPUT_BUFFER(Irp,
    STRUCTURE32,
    &OutputBufferLength);
}
else
#endif //_WIN64
{
GET_OUTPUT_BUFFER (Irp,
    STRUCTURE,
    &OutputBufferLength);
}

It turns out we were real gluttons for punishment because our structures tend to have pointers to other structures who also contain pointers. Each buffer and pointer has to translated both in the api call as well in the structures that we return back to user mode like HTTP_REQUEST.. In the send path we capture all the 32 bit information up front and in one place, but in the receive path we fill out the user data buffer in a number of places and we do the translation as we go, avoiding a copy step later to translate from the native 64 bit structure to a 32 bit one.

When we tested this feature we found a number of alignment issues that the devs didn’t expect. The alignment issues were caused by 32 bit aligned structures not being 64 bit aligned and it can be tough figuring out all the places this could occur. In the cases where this happens we had to write our own byte by byte copy routine since the system one expects byte alignment.

I remember that we got some flak from David Cutler for being one of the few components that didn’t have WOW64 support at Win2k3 ship.

-- Ari Pernick

WNDP Connect Site gets an upgrade!

wndpteam — Tue, 06 Jun 2006 19:32:00 GMT

Last year we setup a small site on connect.microsoft.com in order to let our blog readers, developers and users file bugs, make suggestions and get some conntent like whitepapers and samples early. The downside to the site was that you couldn't easily deep link and it required a Windows Live (aka Passport) login. Well the folks at connect.microsoft.com have launched a new version of connect that enables access to much of the site without a login including readonly access to the bugs filled there much like the popular MSDN Product Feedback Center.

Check it out: http://connect.microsoft.com/WNDP

-- Ari Pernick

Network Programming with Winsock Kernel (WSK)

wndpteam — Thu, 04 May 2006 16:00:00 GMT

Winsock Kernel (WSK) is the latest network programming interface introduced by the WNDP team in Windows Vista. As evident by its name, WSK can be used by kernel-mode drivers for sending and receiving data over the network. But less evident to many developers, WSK is not an interface for performing network “filtering”. Hence, to clarify a common misconception up front, if all you want is to perform some form of network traffic filtering or interception, then you are strongly advised to look at the Windows Filtering Platform (WFP) interface first. WFP is the one-stop shop for network filtering in Windows Vista.

As we have first presented in last year’s Driver DevCon and WinHEC 2005, our main goal with WSK is to provide a programming interface which is easier-to-use and has higher performance relative to its predecessor Transport Driver Interface (TDI) for kernel-mode network applications. Since last year, we have considerably improved the WSK documentation in the Windows Driver Kit (WDK), and have also added a WSK sample driver to the WDK. A preview version of the pre-Beta2 WSK documents and sample is available as described in a previous blog on this site by Mike Flasko.

In WinHEC 2006, the session on WSK will be geared more towards the guidelines and best practices to follow in WSK to achieve optimal performance and stability. For those of you planning to attend the WSK session in WinHEC 2006, we encourage you to get familiar with the available WSK documents and the sample prior to the session to get the most out of the presentation. We have identified a number of areas in WSK over the past year based on feedback from both external and internal WSK clients that require more explanation and guidance. These include how to start and stop using WSK (WSK registration and deregistration), IRP handling, building and processing WSK_BUFs, when to use socket callbacks, how to achieve optimal throughput when sending stream data, how transport address security works, and how to use a single IPv6 socket for both IPv6 and IPv4 traffic. We will discuss all of these areas in depth in the WSK session in WinHEC 2006.

An important question I would like to address here is when to use WSK. Let’s look at this starting with a Winsock2 application in user-mode. If your Winsock2 application is working fine in the user-mode land, then you have no reason to consider a WSK-based implementation for that application. An important misconception here is to assume that a kernel-mode implementation will automatically provide much better performance than a user-mode implementation, which is not true. Remember also that kernel-mode programming has much stricter requirements in order to ensure a high degree of system stability and robustness; if your kernel-mode code is not rock solid, then moving your application into the kernel will cause more grief than good. Lastly, user-mode Winsock2 interface is a richer higher-level interface while, even though we have made it relatively simple and easy-to-use, WSK is still quite a low-level interface which sometimes requires an intimate familiarity of protocol-specific behavior from its clients to avoid pitfalls. To give a few concrete examples, Winsock2 does have dedicated routines for complex tasks like transmitting a file or connecting by name to remote peers whereas WSK doesn’t. Also as another less obvious example, Winsock2 performs buffering in the send direction, which allows even simple applications using blocking send requests to achieve decent throughput whereas WSK does not perform any “socket-level” buffering, hence requires the application to know about and account for things like Nagling, delayed-ack, bandwidth-delay product, etc to achieve decent throughput.

My personal guidance on implementing anything in kernel is to ask this question first: Is there a significant performance, stability, robustness, or security benefit in a kernel-mode implementation for a given functionality that can not otherwise be achieved by a user-mode implementation? This guidance may sound a bit abstract without a concrete example. So, let’s take a look at a real example. The HTTP.sys component in Windows Vista implements a kernel-mode HTTP stack by using the WSK interface. Prior to HTTP.sys, HTTP applications like Internet Information Services (IIS) used to use Winsock2 directly. The move to the HTTP.sys model was driven by several important factors. First, multiple HTTP applications running on the same system often needed to share a single TCP port (e.g. 80). Implementation of this sharing by keeping a clean and secure isolation between multiple HTTP application instances was not straight-forward via Winsock2 in user-mode. HTTP.sys has brought a robust solution to this problem by taking on the responsibility of managing multiple HTTP connections in kernel over a single WSK socket. This allowed applications like IIS to support multiple 3^rd party plug-ins running in user-mode, in isolation, and with least-privilege in order to achieve better system stability and security. Second, HTTP.sys has a kernel-mode cache implementation that allows it to satisfy incoming HTTP requests directly in the kernel without making a user-mode transition. A kernel-mode cache for HTTP.sys makes sense due to the static nature of HTTP content. The full benefit of this cache is made possible by receiving and sending data over a WSK socket in kernel directly. This boosts the overall performance. However, note that this performance factor alone can not be the sole reason for implementing an HTTP stack in kernel without the former factor.

I hope what was stated so far above gives you a pretty good idea about when to use WSK. As for how to use WSK in the best possible way, we will address that topic in the WSK session in WinHEC 2006. We hope to see all of you there!

-Osman N. Ertugay

System .NET features and future

wndpteam — Tue, 06 Sep 2005 19:43:00 GMT

As many of you may already know, the release of Visual Studio .NET 2005 is right around the corner… hooray! As we put the final touches on “Whidbey” for the November 7th launch we are starting a parallel effort focused on planning for our next major release of Visual Studio, codenamed “Orcas”.

As we switch hats from ”Whidbey” release to “Orcas” planning, the System.Net team would like to hear directly from networking application developers on what they would like to see from System.Net in the future.

Some questions that we have been asking ourselves that you may also want to consider …

What features would you like to see enhanced in the System.Net namespace?
What functionality would you like to see from the Visual Studio .Net IDE as it relates to creating rich networking applications?
Do you find yourself always writing the same lines of code to accomplish a networking-related task that we should consider simplifying? An example of this is the work that we did with the WebClient class that allows applications to easily send/receive data to an URI with just a few lines of code.
Is there anything that you believe we have missed the mark on with System .Net?
Have you went down the path designing and/or developing an application with System .Net that you later could not finish due to a limitation with the classes?
Are there features in other platforms that you would to see in System.Net?
How would you rate the overall experience of creating network applications in .Net?
Are you using the HttpListener classes? And if so, what type of functionality would you like to see added?
Are there any HTTP-based protocols that you would like see added or expanded-on in the System.Net classes?

Beyond the features and functionality of the classes, we also would like to learn about network applications, past, present and future that are being built by .NET developers! So please let us know about the applications that you have created or are in the process of creating by replying to this post in the .NET Framework and Communication forum, as a comment to this post on our WNDP team blog, or simply by sending the System.NET team an email.

The System.Net classes are about enabling developers to write robust, secure networking applications for the client and the server. And as we begin planning for the future of the networking classes we want to be sure that we hear about what you would like to see. We look forward to hearing from you.

Thanks!

Billy Anders
Group Program Manager
Windows Networking Developer Platform
billy.anders@microsoft.com
http://blogs.msdn.com/wndp/

This has been cross-posted on the WNDP blog and the System .Net forum.

Kernel Mode SSL in HTTPAPI 2.0

wndpteam — Sat, 30 Jul 2005 01:50:00 GMT

Win2k3 SP1 introduced Kernel mode SSL which had great perf benefits over Win2k3 but the implementation was limited and required a restart of HTTP.sys to change the server certificate configuration. Hence it was not turned on by default. In Vista Beta 1 these limitations have been fixed with dynamic updates to configuration settings and support for client certificates. Previously unparsed requests and formatted responses were pushed to a user mode service for decryption and encryption. This overhead is saved by doing the encryption and decryption in kernel mode and eliminating the extra service. (One less service on Vista! Wohoo!). With the new API it is possible to bind certificates to IPV6 addresses. In SP1 this had to be done with wild card IPV4 addresses.

-- Narasimhan Venkataramaiah (narave)

David Wang discusses http.sys's kernel response cache.

wndpteam — Fri, 15 Jul 2005 23:43:00 GMT

Check out David Wang's recent post about http.sys's response cache. He describes the configurable registry keys and how the scavanger works.

Run ASMX without IIS

wndpteam — Thu, 23 Jun 2005 11:02:00 GMT

Back in late 2004, Aaron Skonnard wrote an interesting article that details how a developer can create a lightweight, special-purpose web server without IIS. In his sample, Aaron chooses to host ASMX web services, but one could easily create other types of specialty web servers. Interestingly enough along with hosting the ASP.NET Runtime the other enabling technologies that allow for this functionality are none other than Microsoft's kernel-mode HTTP listener, Http.sys and the HttpListener classes of System.Net. Both of which are WNDP API's.

Run ASMX Without IIS

http://msdn.microsoft.com/msdnmag/issues/04/12/ServiceStation/

Now you may ask "why would I want to create my own web server"? And to that question we would have to agree that for the vast majority you would not want to for a production system. But the article and sample does demonstrate that it is indeed possible to create some of the basic web serving functionality with very little code and effort.