August, 2009

• Some History of the Named Pipe, Part 1 (Unix pipes)
• Some History of the Named Pipe, Part 2 (Unix fifo)
• Some History of the Named Pipe, Part 3 (Windows named pipe)
• Some History of the Named Pipe, Part 4 (WCF named pipe)

In earlier parts we looked at the anonymous pipe and the fifo, which were predecessors similar in spirit to the Windows named pipe. A Windows named pipe is an inter-process communication mechanism that can provide either a one-way or duplex flow of data, organized as either a stream of bytes or a series of messages.

Creating a named pipe on Windows has significantly grown in complexity from what we’ve seen before as the number of options and capabilities of the named pipe system has increased. Here’s the named pipe creation function for comparison although I’m not going to go into detail on all of the parameters.

HANDLE WINAPI CreateNamedPipe(
  __in      LPCTSTR lpName,
  __in      DWORD dwOpenMode,
  __in      DWORD dwPipeMode,
  __in      DWORD nMaxInstances,
  __in      DWORD nOutBufferSize,
  __in      DWORD nInBufferSize,
  __in      DWORD nDefaultTimeOut,
  __in_opt  LPSECURITY_ATTRIBUTES lpSecurityAttributes
);

The name parameter places the pipe somewhere in the file system, similar to the Unix fifo. On Windows though, the creation of pipes is limited to a special namespace, \\hostname\pipe\, which keeps pipes separate from other types of files and devices. The hostname portion of the pipe address allows pipe connections to span machines (next time we’ll see how to prevent remote access to a pipe, which is something that WCF does). For the local host, a dot is used instead of the hostname.

While a Windows named pipe can behave somewhat like a fifo, it also can behave somewhat like a socket. The pipe mode controls whether data is transferred through the pipe as a stream of bytes or as a sequence of messages. Instancing allows multiple pipe connections to be serviced using different threads where the pipe connections share the same pipe name. For example, you can construct a multi-threaded pipe server that talks to many clients, each of which connected to the same named pipe server address.

In summary, some of the things that make Windows named pipes different than the other named pipes we looked at are:

• Duplex communication
• Message-oriented reads and writes
• Special pipe namespace
• Remote machine connections
• Multiple instances at the same server address

WCF uses only a portion of these capabilities and hides nearly all of the details of named pipe communication. Next time we’ll look at both how WCF uses Windows named pipes and how WCF makes a named pipe look even more like a socket.

Over on MSDN, Michele Bustamante will be doing a biweekly series called WCF Essentials on best practices and how to use WCF. The first two introductory articles are available now.

Getting Started with WCF (an introduction to the WCF Essentials series): This section will serve as a summary of the steps you will take to design, implement, host and consume WCF services. The purpose is not to explain the details behind each design choice or feature- but to make you aware that these are things you should consider. Later sections of this whitepaper will provide recommendations to narrow the choices you face.

WCF Templates and Tools: This section summarizes the available project templates, when to use each; and discusses the limitations and benefits of the core WCF tools.

• Some History of the Named Pipe, Part 1 (Unix pipes)
• Some History of the Named Pipe, Part 2 (Unix fifo)
• Some History of the Named Pipe, Part 3 (Windows named pipe)
• Some History of the Named Pipe, Part 4 (WCF named pipe)

The anonymous pipes that I talked about last time satisfy the basic requirements of an inter-process communication mechanism but have the considerable drawback of only working between two processes that share a parent-child relationship. The need for inter-process communication between two unrelated processes is reflected in another mechanism called a fifo (also sometimes called a named pipe).

A fifo is in many ways similar to a pipe on the surface but the two are different entities. Like a pipe, a fifo supports unidirectional communication from a writer to a reader. Unlike a pipe though, a fifo shares several characteristics with file-based devices, such as a name and an entry in the file system. A pipe is opened through the act of creating it while a fifo is first created and then opened for reading and writing by other processes.

The similarity between a fifo and a file device is actually substantial. In fact, the standard Unix system call for creating a fifo is int mknod (char *pathname, int mode, int device), which is the same system call for creating other kinds of devices. Creating most other devices requires special privileges though while ordinary users can create a fifo. A fifo is distinguished by the use of a particular flag bit in the mode, allowing the two uses to exist side by side. The pathname is what gives the name to the named pipe.

Once created, the fifo can be opened like an ordinary file, and after that can be read, written, or closed just like any pipe.

The Windows named pipe isn’t precisely the same as a fifo though, and those differences are what I’ll talk about next time.

• Some History of the Named Pipe, Part 1 (Unix pipes)
• Some History of the Named Pipe, Part 2 (Unix fifo)
• Some History of the Named Pipe, Part 3 (Windows named pipe)
• Some History of the Named Pipe, Part 4 (WCF named pipe)

Let’s start with the more ancient history and work over a few steps towards what pipes look like today. Pipes started out as inter-process communication mechanisms that provide a one-way flow of data.

The standard Unix pipe is created by the pipe system call int pipe (int*). The pointer parameter is really an array to which the pipe system call will write a pair of file descriptors for reading and writing from the pipe. Thus, you’re more likely to see the pipe system call written these days as int pipe(int[2]).

These pipes are anonymous pipes as their only representation is a file descriptor, which is a non-significant integer value that’s not portable between processes. The pipe will already be opened upon creation so the interesting operations to do with it are to read, write, or close the pipe.

This might lead to the question of how a pipe actually gets used for inter-process communication if the pipe has no name and there’s no way to open it from another process. The only way a file descriptor gets copied around is when a process is forked to create a child process. After calling fork(), both the parent and child process initially have the same file descriptors, effectively replicating the endpoints of the pipe elsewhere. One of the processes can then use the read file descriptor while the other process uses the write file descriptor. For example, you might build bidirectional communication between a client and server with the following approach:

1. Create two pipes
2. Call fork()
3. Child closes the read side of one of the pipes and the write side of the other pipe
4. Parent closes the opposite side of each pipes
5. The two processes can then exchange one-way messages using the sides of the pipe that are still open

These anonymous pipes are the fundamental building block of the pipeline for commands. The basic composition for running commands is to sequentially pipe data from one command to another. The command processor takes care of the step of creating the pipes and wiring them up to be the input and output of various commands in the pipeline.

Anonymous pipes are all well and good but it’s a big hindrance that they only work between two related processes. This spurred the creation of another inter-process communication mechanism that we’ll talk about next time.

Why are the members of a data contract expected in a particular order in order to be read?

The order of members is frequently used to build an expectation of what element might be appearing next. Knowing the possible order of data instead of potentially having to accept data in any order tends to make deserialization both faster and cheaper. In particular, you can frequently use an expectation of order to reduce or eliminate the buffering of data as compared to having to reorder the data after the fact. This is one of the many reasons for being able to specify the order property of a data member to be able to control the expected order.

This list is a bit different in places from what you’ve seen in the beta 1 release and there’s always a chance that things might change again.

The third part of the list covers some of the new features for service configuration. I haven’t mentioned some of the larger changes for simplified configuration and standard endpoints because those are already a part of the official beta 1 list.

• Short type name support: many of the configuration extensions require registering a CLR type for the extension object. We generally recommend using fully qualified type names, but this is not always practical during development. Several of the extension elements for WCF configuration now recognize and attempt to resolve short type names.
• Location tag for service files: there is a standard ASP.NET configuration feature that allows a location element to describe configuration specific to a part of the application. With ASMX it was possible to use that location element with a path to the ASMX file. We’ve similarly enabled the use of location elements with SVC files.
• Channel factories with custom configuration: we’ve added new channel factory subclasses that support passing in your own instance of a configuration object. This allows you to build configuration objects from a source other than the application configuration without having to construct the proxy objects entirely by hand.

I’ve defined my own XML namespaces for the members of a data contract and now when I generate the client proxies I get some ugly CLR namespaces for the types. How do I fix this?

The mapping from an XML or WSDL namespace to a CLR namespace is a mechanical transformation of breaking apart the authority and path of the XML namespace, doing minimal fix-ups to avoid illegal CLR type names, and spitting out a type in an equivalently delimited CLR namespace. The generated CLR namespaces might be neither pretty nor support CLS compliance needed for maximum portability.

You can fix those generated CLR namespaces though by overriding the mapping for an XML namespace. The svcutil program has a /namespace option that takes as parameters the target XML namespace and the CLR namespace that you would like to use instead. You can also use * for the target namespace to set the CLR namespace for all target namespaces without an explicit mapping. For example, you could say:

svcutil /namespace:*,Example.Contracts

If you’re using Visual Studio instead of svcutil, then you instead need to set the namespace mappings in the generated svcmap file in your solution. Inside the svcmap file is a NamespaceMappings collection to which you can add NamespaceMapping elements with attributes for the TargetNamespace and ClrNamespace. You aren’t able to use * for the target namespace though in an svcmap file because Visual Studio already has a wildcard definition and you’re not allowed to have conflicting definitions.

The broader Silverlight team is running a one day event here at Microsoft on September 17 to talk about Silverlight 3. It’s possible to sign up either for in-person attendance or for a live meeting presentation.

Here’s what’s on the agenda for the day:

• Keynote by Scott Guthrie
• Key Silverlight Scenarios by Tim Heuer
• Expression 3 Overview by Adam Kinney
• Sketch Flow by Janete Perez
• Toolkit & Controls by Justin Angel and Shawn Oster
• RIA Services by Brad Abrams
• Building Silverlight UIs with XAML Power toys by Karl Shifflett

If you’re interested, the sign up links are below for both ways of attending the event.

• Silverlight 3 Firestarter In-Person
• Silverlight 3 Firestarter Live Meeting

What would you most like to see at PDC for WCF?

1. New features in WCF for .Net 4
2. How to use WCF better in applications today
3. Internal details of WCF
4. Using WCF with workflow services

Include as much commentary with your selection as you’d like. I generally don’t publish comments for survey questions but many people are auto-approved to show up. If you’d prefer your response to definitely not be visible to others, you can send it in the contact form.

Moonlight, an open source implementation of Silverlight for Linux systems, has released their feature complete beta corresponding to Silverlight 2 (and a small bit of Silverlight 3 too).

You can get the Moonlight 2 beta as either binaries or source code.

• Prebuilt Moonlight 2 Beta 1 binaries for Firefox
• Moonlight 2 Beta 1 source code
• Release notes

This list is a bit different in places from what you’ve seen in the beta 1 release and there’s always a chance that things might change again.

The second part of the list covers some of the new features and changes to existing features to improve service performance. These performance features are tailored to specific scenarios. Enabling these features when unnecessary might even slow your application down. You should measure whether your application actually gets some benefit before taking advantage of these.

• Increased default values for service throttles: almost everyone has to change the throttle settings today so we’ve made the default throttle settings a better match for the capacity of modern machines. The MaxConcurrentCalls and MaxConcurrentSessions are now scaled by the number of processors and the base value of MaxConcurrentSessions has additionally increased from 10 to 100. You’re still able to set your own values for service throttles through a ServiceThrottlingBehavior if you don’t like the defaults.
• Asynchronous service replies: in most cases it’s faster for the service to reply synchronously than asynchronously (by contrast, the opposite is true for the service to receive requests). However, if you have a service that frequently completes most of its outstanding calls at the same time, then you might see better performance by sending the replies asynchronously. We’ve added an asynchronous send behavior that can be applied to an endpoint to enable this option. This might change the order in which replies are sent so you have to be sure that the application is able to tolerate this before enabling the feature.
• Multiple asynchronous receives: although a service might receive messages from many channels and may be processing many messages at once, the handoff between a channel and the service is typically done one at a time. This is in most cases faster than trying to handoff multiple messages at once as the handoff is made simpler and each one is supposed to be extremely inexpensive. However, it is possible to install service extensions that block until completion during this handoff. We’ve introduced an endpoint behavior that allows you to increase the number of messages that a service will try to retrieve from a channel at once. This changes the timing for handing off messages so you have to be sure that the application is able to tolerate this before enabling the feature.

Aaron Skonnard also has a list going on MSDN of new features for WCF in .Net 4. Our lists are going to look a bit different because I’m covering more, smaller features and Aaron’s list is targeted at beta 1. Some of the topics in the MSDN list though are:

• Simplified configuration
• Standard endpoints
• IIS hosting without an SVC file
• Discovery
• Routing service (previously included with Dublin)
• REST caching and help page
• Workflow services
• Non-destructive queue receive
• Simple byte stream encoding
• ETW tracing

Microsoft Research has a Network Monitor 3.3 plugin called TCP Analyzer that helps identify performance problems from a captured TCP session. TCP Analyzer takes a Network Monitor trace and performs visualization and analysis of the TCP connection. The analysis tries to identify what factor is most limiting to the transfer rate of the TCP connection, such as the rate of dropped packets, the latency in adjusting the transmission windows, or the available connection bandwidth.

• TCP Analyzer (32-bit)
• TCP Analyzer (64-bit)
• Network Monitor 3.3

As we get closer to the release of .Net 4 it becomes easier to talk about exactly what new features we’re planning to provide. This list is a bit different in places from what you’ve seen in the beta 1 release and there’s always a chance that things might change again.

The first part of the list comes from the area of WCF channels.

• Non-destructive queued receive: the current support for queuing has a choice between transacted messaging and purely best effort messaging. It’s hard to reason about systems that are best effort while it’s hard to achieve high volume and scale while using transactions. We’ve introduced a new queued receive pattern that is amenable to having multiple consumers work off of a single queue while retaining the ability to do transactional processing in your application.
• Extended protection of secure channels: extended protection is a new mechanism in Windows for binding one security system, such as an SSL stream, to another security system, such as Kerberos or HTTP authentication. By associating the two security systems together and ensuring that the association is preserved over time, it becomes harder for an attacker to get a foothold even when they’re able to intercept and modify your network traffic.
• Simple byte stream encoding: when talking to something other than a web service you often need to be able to directly influence the stream of bytes that is being exchanged with the network transport. The byte stream encoder is a low overhead way of enabling your application to work with messages that are essentially binary objects.
• Automatic HTTP client decompression: compression is an approach for reducing the amount of network traffic in exchange for spending more CPU time on the client and server. Many web servers have support for compression modules but the client and server first need to agree to exchange data in a compressed format. We’ve made that easier when using HTTP by allowing the client to automatically negotiate using gzip or deflate compressed streams and then automatically decompress them.

A month ago I asked for feedback on whether you’d be interested in occasionally seeing some longer articles with the tradeoff that the articles would come out less frequently. I got about 30 responses, with almost everyone interested in having longer articles and almost no one interested in changing away from a daily update schedule.

One compromise is to build the longer articles over time by seeding them from the daily updates. I’ve tried that out in a sample article on other technologies of interest to WCF developers. I’ve gotten started with the ASP.NET MVC and Managed Services Engine technologies that I talked about recent, but you might imagine seeing over time additional entries such as the Managed Extensibility Framework and the Windows API Code Pack. The goal of the summary page isn’t to be comprehensive but to tie together many related posts to make the content easier to find.

You might also imagine seeing similar types of summaries for topics such as WCF/Silverlight releases, conferences, or technical topics like IIS troubleshooting.

Let me know if you love or hate the first edition.

Going along with Windows 7, the final version of the Windows API Code Pack was released last week. The code pack is a way to provide access to Windows features that lack a managed interface in the framework today, primarily for things added in Windows Vista and Windows 7. For example, the code pack adds managed wrappers around native Windows APIs allowing you to integrate with newer shell, dialog, power, and DirectX features.

The API that WCF programmers are most likely to be interested in is the Network List Manager interface that I talked about when it was added to the code pack beta. The network list manager service tracks information about all of the network connections the machine has seen and knows whether they’re currently connected or disconnected. This allows you to enumerate all of the computer’s connections and query properties such as the IP address, network adapter, protocol versions, gateway settings, and whether the connection joins the machine to a Windows domain.

• Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 (Web setup)
• Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 (Full install)
• Release notes

Most people will want to use the most recent SDK since it supports the last several framework and operating system versions, but if you're looking for the older releases, here they are:

• Windows SDK for Windows Server 2008 and .NET Framework 3.5
• Microsoft® Windows® Software Development Kit Update for Windows Vista^TM

The 2009 Professional Developers Conference will be held this year from Tuesday November 17th to Thursday November 19th, located again at the Los Angeles convention center. A pre-conference workshop day happens ahead of time on Monday. Workshops are a more in-depth look at particular technologies while the main conference is built around breakout sessions and keynote presentations.

The scheduled workshops are:

• Getting the most out of Silverlight 3 by Richard Griffin and Ian Griffiths
• Patterns of Parallel Programming by Richard Ciapala, Ade Miller, Herb Sutter, and Stephen Toub
• Developing Quality Software using Visual Studio Team System 2010 by Todd Girvin and Chris Tullier
• Architecting and Developing for Windows Azure by Chris Auld
• Developing Microsoft BI Applications- The How and The Why by Andrew Brust and Don Demsak
• Microsoft Technology Roadmap by Michele Leroux Bustamante
• Software in the Energy Economy by Juval Lowy

The schedule for the rest of the sessions is completely fictional. If you're signing up now it's because you know you want to attend regardless of who's saying what. A reasonably accurate list of sessions generally starts shaping up closer to the event.

It’s been a year since I last talked about the Managed Services Engine, which is a prebuilt solution for virtualizing web services through metadata inspection and message routing. Since then they’ve released two more CTPs, the most recent one being the May 2009 CTP.

• Managed Services Engine May 2009 CTP

You’ll find guides covering installation, the technical details of the services engine, and securing virtualized services at the same place.

Aaron Skonnard has an article in MSDN Magazine covering virtualization of cloud and REST services with the Managed Services Engine for those interested in a walkthrough. A whitepaper that greatly extends that article is also available.

The latest information about the Managed Services Engine can be found in the list of supporting technologies.

MVC 2 resources:

• MVC 2 reference documentation
• Get the source code for ASP.NET MVC 2 Preview 1
• The MVC Roadmap talks about upcoming features and timelines for shipping together with Visual Studio 2010
• Phil Haack and Scott Hanselman talk about MVC 2 on Channel 9

MVC 1 resources:

• Aaron Skonnard has an article on writing RESTful Services With ASP.NET MVC
• Stephen Walther's ASP.NET MVC Framework Unleashed book is now available, although the sample chapters haven't been taken down yet

The MVC forums cover both MVC 1 and 2.

The latest information about ASP.NET MVC can be found in the list of supporting technologies.

The second version of ASP.NET MVC is already underway (it’s on the same type of frequent release schedule as Silverlight). MVC is a model-view-controller framework on top of the existing ASP.NET runtime that separates display and application logic as well as makes test-driven development of ASP.NET applications easier. MVC could be used with a variety of types of web applications but is frequently associated with REST applications.

The MVC 2 Preview 1 runs on Visual Studio 2008 with .Net 3.5 SP1 and you can have both MVC 1 and MVC 2 installed at once.

• ASP.NET MVC 2 Preview 1
• ASP.NET MVC 2 Preview 1 Release Notes

If you’re not looking for a preview release, you can get the original ASP.NET MVC 1.

The latest information about ASP.NET MVC can be found in the list of supporting technologies.