Whenever I've talked about tracing I've always used the System.Diagnostics trace listeners in the example. However, there's a second tracing system that is much more powerful but much less talked about called ETW (Event Tracing for Windows).
ETW is much less talked about because of the way it combines extreme ununsability with a copious lack of documentation. If you can master ETW though, then it can become an extremely valuable tool. ETW is highly performant, can be made very granular, and allows programmatic control for starting and stopping traces. This type of tracing is extremely useful for advanced debugging and performance monitoring tasks.
The history of ETW derives from a need for very fast and general-purpose tracing facilities in the operating system. ETW used kernel level buffering and logging to provide a tracing mechanism for user applications and kernel drivers starting with Windows 2000. Because of this history, ETW developed several useful features such as dynamic control of traces (to avoid having to restart applications or reboot the machine to change tracing options) and very low overhead asynchronous logging. ETW has gotten an overhaul in Windows Vista to define a unified provider model and in Orcas to add some basic managed APIs.
You probably have a large number of ETW providers already on your system. You can run "logman query providers" to see the list.
Defining an ETW provider and listener for use with WCF is fairly complex. There is an ETW tracing sample for WCF though that takes you through defining and registering a provider, defining events, defining and registering a listener, and receiving trace events.
Next time: Common Problems Composing Security with Streaming
I have a schema file that describes a choice between multiple layout formats for a type. How do I build a proxy that matches this schema?
The reason that this is probably not working is because DataContractSerializer for the most part does not support ambiguous, optional, or multiply-described sections of types. Using svcutil and specifying the serializer to be XmlSerializer is more likely to generate the expected proxy type.
Here's a rough guide to what DataContractSerializer does not support for complex type content. If you're using any of these features in your schema, then it's likely that you'll need to use XmlSerializer instead.
Next time: Using ETW Tracing
Last night a release candidate for the Silverlight 2 developer runtime and tools went live. This release candidate lets developers update their applications before the end user runtime is released in a few weeks. The release candidate is feature complete. There are some breaking changes from the last beta release, most of which are the removal of scattered methods and constructors to clean up the classes that were ported from the desktop framework. Here are the changes that may impact WCF developers.
The MIME type of Silverlight applications should now be the final release version application/x-silverlight-2.
Cross domain hosts of Silverlight must set the right MIME type for XAP (application/x-silverlight-app).
Some exceptions that were previously thrown in HttpWebRequest.BeginGetResponse() are now thrown in HttpWebRequest.EndGetResponse(). This may cause a similar shift in timing when using the asynchronous WCF APIs to send HTTP messages.
Many of the XML APIs will now throw NullReferenceException where they used to throw ArgumentNullException. This may cause a similar change with messages and serialization when dealing with bad data.
Cross domain policy files no longer need to explicitly allow the Content-Type request header. The Content-Type header is always settable on cross domain POST requests.
There are some more detailed changes for the HTTP polling duplex channel.
Yesterday came with the announcement that Windows 7 would be making a public appearance at PDC. The PDC sessions list has been updated with over twenty new sessions covering all aspects of Windows 7. A Windows 7 alpha build will be among the various bits handed out to attendees at the conference. We'll have quite a bit to say about how the advances we're making in Silverlight and the .NET platform relate to upcoming operating system releases.
A few weeks ago Ythos posted an interesting case study for migrating a healthcare application to Orcas. The case study covers a lot of ground including the use of WCF, WPF, and LINQ in the updated application. I like how the case study spends less time talking about what the application does and more time talking about the thought process for understanding how to use the technology. You get a lot more insight from reading how someone thinks than just reading a summary of the solutions they came up with.
I've talked a bit in the past about the tradeoffs of using IIS to host your service applications. In order to use IIS as a host, you must give up some level of control about how your application functions. This is similarly true with any other host you might imagine, even if we use host in a broader sense than just WCF services, such as with SQL Server. By selecting a host, you exchange control over certain aspects in your application in return for getting certain services for free. For example, IIS gives you a configuration model, tooling, health management, process activation, and other services. A service hosted inside of a Windows process might receive a different set of benefits.
In order to make a good selection of a host, you have to understand these tradeoffs and be able to apply them to your application scenario to make a hosting decision. As an example, consider the aspect of service lifetime. When you host a service application in IIS, you give up some control over when the service is stopped, started, and executed. Your service will be activated by message arrivals. Your service may be stopped at any time based on the needs of the host, such as due to a lack of available memory, a lack of recent activity, or because of receiving a certain number of requests. These service shutdowns may or may not be predictable to you or give you an opportunity to react to them.
A service that stops and starts at times necessitates a different strategy of design than a service that runs all of the time. A process that is long-running but intermittent needs to have some way to persist progress so that starting and stopping does not lose important work. A process that is continuously long-running might have different concerns, such as remaining extremely stable and carefully cleaning up resources. Both strategies can lead to the same goal and there's no reason to believe that there has to be a significant difference in performance or latency between the two. However, the two strategies are very different and very much aligned to different hosting models.
When you commit to using a particular host with your application, you should take some time to understand how your application can best be adapted to the host's expectations. Similarly, when you are designing an application before committing to using a particular host, you should understand what the important constraints are for each of the hosts that you are considering. This obviously requires some planning to have a candidate list of hosts and understand their tradeoffs. The return on that investment comes once the host is selected and you find that your application needs few modifications to work well with the host.
Next time: No Choice for Data Contracts
Peter Marcu is looking for people that are responsible for the deployment or installation programs of a .NET application to answer a few questions. The questions are mainly about how you deal with the big dependency of a .NET application, which is the .NET framework. I figured that since every WCF application is a .NET application, you might have some experiences to share with Peter.
Every time you open a connection to another machine you need to have a port both at the local machine and the remote machine for exchanging data. The port at the remote machine is typically well known in the sense that the port number is a fixed number or published through some mechanism that doesn't change very frequently. This allows the service that is listening on the port to have a well known address. The port at the local machine can be any port number and often you don't care what that number is. Every time you make a new connection the port number can change.
These short lived port allocations are typically called ephemeral ports and are allocated using a wildcard request. Your request is just for any free ephemeral port rather than for a specific port number. On older Windows systems the default range allowed for ephemeral ports was 1025 to 5000 giving you a little bit less than 4000 free ports. In rare cases there may be no free ephemeral ports available, which will cause the connection open to fail or timeout. This would be unlikely on a client machine but is more likely to take place on a middle-tier machine that is creating connections for every client request that is processed.
On Windows Vista and Server 2008 the default range of ephemeral ports is now 49152 to 65535 giving you a little bit more than 16000 free ports. It would now be very unlikely for all of the ephemeral ports to be in use. You would probably want to start caching and reusing connections faster than the operating system recycled ports once you have that many ports in use. However, you can change this ephemeral port range if you need more ports or if the range conflicts with a port number used by your application.
The basic command for changing the ephemeral port range looks like this.
netsh int ipv4 set dynamicport tcp start=49152 num=16384
You can use ipv6 instead of ipv4 and udp instead of tcp depending on the protocols used by your application. A similar command shows what your current ephemeral port range is.
netsh int ipv4 show dynamicport tcp
Next time: Hosted Service Shutdown
Fifteen years ago, a change to the way IP addresses were assigned was the first major step taken to combat the problem "the Internet is rapidly running out usable IP addresses". You hear that problem still talked about today as one of the reasons to adopt IPv6, which has been very slowly increasing in usage over the past nine years. The change I'm talking about though is the adoption of classless routing between domains. The reason that classless routing was adopted relatively quickly is that it solved not only the problem of running out of usable addresses but also a more urgent problem at the time of routers running out of memory.
The original allocation of IP addresses used three fixed-size classes to split up the 32-bit address space. There was a range of addresses that were class A addresses (having 24 bits of address space), a range of addresses that were class B addresses (having 16 bits of address space), and a range of addresses that were class C addresses (having 8 bits of address space).
The problem with this class approach was both that address assignments were typically wasteful as well as having no way of performing routing besides knowing about every single active class. To avoid assigning an entire class B address to an organization with only 800 machines, you had to make active routing entries for several class C addresses. It was a tradeoff between being so wasteful with addresses that you would run out or advertising so many routing entries that routers would fail.
The solution to this problem used a technique called variable-length masking to both subdivide and aggregate the existing classes. Variable-length masking allows a route to be advertised for any number of contiguous significant bits rather than just 8, 16, or 24. An organization might be assigned a 10-bit address space instead of a 16-bit address space to use with 800 machines. However, while avoiding waste with less hassle than using multiple class C addresses, you would still have an explosion of routes if every organization's allocation had to be advertised.
Combined with variable-length masking was the assignment of large contiguous blocks to service providers. Service providers could subdivide their blocks into smaller contiguous regions for subscribers without having to advertise the individual small blocks. These smaller blocks could then be divided and redivided as necessary as long as the allocations were always kept contiguous. This allowed routing across the Internet to work without having to know about a whole bunch of routes. You only needed to know about one level of detail within your block as well as the biggest blocks in the world, both of which could have reasonable limits in quantity.
Today's post wraps up the series on cross-domain policy files with some side stories that you probably don't have to deal with when using WCF in Silverlight.
I've been giving all of the examples using the defined Silverlight policy format. However, if the host you’re trying to connect to doesn't have a policy defined for Silverlight, then the client will fall back and check to see if the host has the equivalent cross-domain policy file used by Adobe Flash.
The Adobe Flash policy format is slightly simpler and is located in a file called crossdomain.xml instead of clientaccesspolicy.xml. Both files are expected to be located at the root of the domain. If the host returns a response for the Silverlight policy file, then the Adobe Flash policy file will not be checked.
Just like I gave you the DTD for the Silverlight policy format, here's the DTD for the Adobe Flash policy format.
<!ELEMENT cross-domain-policy (site-control?,allow-access-from*,allow-http-request-headers-from*)><!ELEMENT site-control EMPTY><!ATTLIST site-control permitted-cross-domain-policies (all|by-content-type|by-ftp-filename|master-only|none) #REQUIRED><!ELEMENT allow-access-from EMPTY><!ATTLIST allow-access-from domain CDATA #REQUIRED><!ATTLIST allow-access-from to-ports CDATA #IMPLIED><!ATTLIST allow-access-from secure (true|false) "true"><!ELEMENT allow-http-request-headers-from EMPTY><!ATTLIST allow-http-request-headers-from domain CDATA #REQUIRED><!ATTLIST allow-http-request-headers-from headers CDATA #REQUIRED><!ATTLIST allow-http-request-headers-from secure (true|false) "true">
The other part of policy that I didn't talk about was the policy used for socket resources. Currently there is no TCP channel for WCF in Silverlight so including a policy for socket connections wouldn't be of much use for web services. However, here are the differences between HTTP and TCP resources.
Instead of a resource path, you have a socket-resource port and protocol. The socket-resource port is the range of ports that the domain is allowed to make connections to. Additionally, there is a restriction that the allowed port range has to be within the range of ports 4502 to 4534. The socket-resource protocol is equivalent to the scheme, except in this case the only value supported currently is "tcp". If we added TCP support to WCF in Silverlight, then we'd probably define our own "net.tcp" scheme similar to what we do on the desktop.
Next time: Ephemeral Port Limits
Last time I talked a bit about the need for a cross-domain access policy in the Silverlight security policy system. Today I'll go into more detail about that example cross-domain policy file that I showed you.
<?xml version="1.0" encoding="utf-8"?><access-policy> <cross-domain-access> <allow-from http-request-headers="*"> <domain uri="*”/> </allow-from> <grant-to> <resource path="/" include-subpaths="true"/> </grant-to> </cross-domain-access></access-policy>
You might see four spots where you can plug in your own values to define policy.
The http-request-headers attribute is a comma separated list of HTTP headers that are allowed to be passed by the domain. The default is that no HTTP headers are allowed. Using WCF with Silverlight requires allowing HTTP headers in order to work correctly.
The domain uri is the allowed domain that is being granted access by this policy. The scheme used in the domain uri is significant.
The resource path is the allowed resource that you are granting access to the domain uri by this policy. The resource path is always relative to the root of the domain.
The include-subpaths attribute allows you to specify whether the resource path should be treated as a prefix or as an exact match. The default is that the match has to be exact.
The official definition of this format is provided by this DTD.
<!ELEMENT access-policy (cross-domain-access)><!ELEMENT cross-domain-access (policy+)><!ELEMENT policy (allow-from)><!ELEMENT policy (grant-to)><!ELEMENT allow-from (domain+)><!ATTLIST allow-from http-request-headers CDATA><!ELEMENT domain EMPTY ><!ATTLIST domain uri CDATA #REQUIRED><!ELEMENT grant-to (resource+)><!ELEMENT grant-to (socket-resource+)><!ELEMENT grant-to EMPTY><!ATTLIST resource path CDATA #REQUIRED><!ATTLIST resource include-subpaths (true|false) "false"><!ATTLIST socket-resource port CDATA #REQUIRED protocol #REQUIRED>
You'll notice that because I've only been talking about HTTP resources, there are some parts specific to sockets in the DTD that don't appear in the example.
Tim Heuer has a translation of the cross-domain policy file DTD to a schema and directions for using the schema with Visual Studio.
Next time: Cross Domain Policy Extras
Here are a few articles about Silverlight cross-domain communication that got bumped for announcements a few months back and I forgot to reinsert them into the normal posting order. Hopefully they are still timely enough to be useful.
Silverlight applications are run under a security policy system to prevent some common networking threats that can occur with untrusted or low-trusted application code. This type of security policy system is a standard part of executing applications from a web browser, where you may be downloading application code from a variety of sources without a lot of consideration beforehand of whose code you're executing.
One aspect of the security policy system is that applications executing in the web browser are limited to creating connections back to the site of origin. The site of origin refers to the host and port pair from which the application code was downloaded. However, it's common for a web service client application to want to interact with network resources at many different locations. Permission to access a network resource that is not the site of origin is defined by a cross-domain policy file.
The cross-domain policy file is a file called clientaccesspolicy.xml located at the root of the requested target domain. This policy file defines an access policy in terms of sites of origin that requests are allowed from and the permissions granted to those locations.
Here's a quick example of an extremely permissive cross-domain policy file to demonstrate the syntax.
I'll go into more detail about cross-domain policy files next time.
Next time: Defining Cross Domain Policy
An index of all of the different standard stores for certificates used to authenticate with a service. There are two store locations- one for the stores for the local machine and one for the stores for the current user. There's no way to directly reference the location of the stores for some other particular user besides the current user.
Here are the eight standard stores.
Next time: Cross Domain Policy
Channel 9 has put up a new video that mostly plugs the Advanced Windows Debugging book by Mario Hewardt and Daniel Pravat. I recommend the book for someone that needs to debug difficult issues dealing with native resources. For most people working with WCF this is never really a concern, but custom transport authors will occasionally find themselves dealing with these types of issues. This is particularly true when tracking down resource leaks and synchronization issues in native code that are hard to diagnose due to the heavy use of asynchronous execution in WCF.
How do I specify that the client proxy should not have a setter for a particular data member?
It doesn't make sense for the service to be able to dictate what the client can do with a piece of data. Once data has been put on the wire, you can't stop the other side that is receiving the message from doing whatever they want with that data.
Schema and metadata don't have any notion of access modifiers, but let's say that they did. You might say in the metadata that a certain piece of data has restricted access. First, there's nothing requiring me to use the same metadata and tools that you are, so I may not even know about your access modifiers. The representation that gets written to the wire is the truth about how your service works, not the metadata. Second, even if I am using the same metadata and tools as you are, I can always change the client after it's created because the generated code is something that I own rather than you as the service author.
If the client author and the service author are the same person, then there are much more effective ways of providing a preferred programming model than trying to describe those preferences through some aspect of the service description. Instead, just change the client library directly to do what you want and give people that.
Next time: Certificate Stores
The September update to the PDC schedule is available with 50 new sessions being published.
My session has now been unhidden with this update. Someone else wrote the title and abstract for me. I'll update the session details with more specific information when I get the chance.
Windows Communication Foundation: Tips and Tricks for Performance and Scale
Presenter: Nicholas Allen
Join us for lunch to discuss the different kinds of performance and scale requirements that are a crucial part of any distributed systems development life cycle. Learn great tips from Nicholas Allen on WCF throughput and responsiveness optimization throughout the lifecycle of a distributed system. Hear about WCF scalability improvements in the next version of the Microsoft .NET Framework.
Tags:
Advanced, WCF
Related Sessions:
WCF 4.0: Building WCF Services with WF in .NET 4.0
As Raymond Chen has said: Information was believed to be accurate at press time. Speakers subject to change. Talk may be changed or cancelled for reasons outside my control, or possibly even reasons within my control.
Don and Doug have started dropping hints about what they're going to talk about on the topic of Oslo for PDC. If you haven't read the other articles, the essence of Oslo is that it is a system for modeling applications that works throughout the entire lifecycle of development, deployment, and management. A modeling system encompasses many things but the key deliverables divide into a language for describing models, tools that build models, tools that use models, and a repository for storing models. You'll start seeing all of these types of components as betas of Oslo become available.
What Oslo doesn't change is what your services can actually do. There's nothing forcing you to start modeling your applications to make use of features in the Microsoft frameworks. Modeling is a tool that is supposed to enhance the productivity of developers and IT professionals rather than be the single way to do things. You're going to hear a lot about Oslo because it's new, but we're still planning to make the next versions of Windows and the .Net framework awesome. In the two years since Indigo has shipped, I've spent a few weeks working on Oslo and the vast majority of the time making WCF and WF better.
To go along with yesterday's post on targeting a particular version of the framework for proxy generation, FxCop 1.36 supports some new features for platform targeting. The portability section contains a targeted framework version rule that allows you to validate that all API calls used by your program are available in a particular version of the framework. The target framework version can either be the compilation target version in your project or a version that you specify manually as an option.
Orcas introduced a new DateTimeOffset class that is easier to use for representing absolute times than the original DateTime class. However, if you run svcutil on a contract that contains a DateTimeOffset, you'll get an ugly generated structure because DateTimeOffset isn’t recognized as a natively supported type by the system. A new class is generated by svcutil to match the schema for DateTimeOffset in the metadata.
namespace System{ using System.Runtime.Serialization; [DebuggerStepThroughAttribute()] [GeneratedCodeAttribute("System.Runtime.Serialization", "3.0.0.0")] [DataContractAttribute(Name="DateTimeOffset", Namespace="http://schemas.datacontract.org/2004/07/System")] public partial struct DateTimeOffset : IExtensibleDataObject { private ExtensionDataObject extensionDataField; private DateTime DateTimeField; private short OffsetMinutesField; public ExtensionDataObject ExtensionData { get { return this.extensionDataField; } set { this.extensionDataField = value; } } [DataMemberAttribute(IsRequired=true)] public DateTime DateTime { get { return this.DateTimeField; } set { this.DateTimeField = value; } } [DataMemberAttribute(IsRequired=true)] public short OffsetMinutes { get { return this.OffsetMinutesField; } set { this.OffsetMinutesField = value; } } }}
There's a new option on svcutil, /targetClientVersion:Version35, that can be used to indicate that code generation should use new features in Orcas.
As far as I know, there are three places where this option makes a difference.
Next time: Read Only Data Members
Encryption is counterproductive for compression if the two features aren't used together correctly. Generally, you want to compress first and then encrypt. This is the order that naturally happens when you compress at the encoding level and encrypt at the transport level. You tend to get disadvantageous results if you encrypt first and then compress. This order can happen when you encrypt early on, such as when you use message security with transport compression, or if you attempt to apply compression from outside the system after encryption has already taken place.
If you think about how a typical lossless compression method works, it is exploiting repeatability or non-randomness in the uncompressed content. Completely random content will tend to compress very poorly, possibly even growing in size, because there is no statistical redundancy to eliminate. On the other hand, completely predictable content will tend to compress very well. Text and many kinds of binary content that are not already compressed tend to at least be somewhat predictable.
Most secure encryption mechanisms will transform content such that the result is very close to being random. Statistical tendencies in the resulting encryption output could be a way to attack the encryption mechanism, possibly revealing the original content or even the secrets used for encryption. Therefore, the encrypted output tends to be more random than the original content. If the original content was fairly predictable, then this can cause a significant decline in compression effectiveness. This makes encrypted content a poor candidate for compression.
Next time: Getting Better Time Formats
When I create a channel to a service, how do I know when the service is ready to process the data for that channel?
A channel doesn't really know what the service is doing. The service might be actively processing the data being sent over the channel. Or, the service might not.
There is a constant tension in the system between components that want to push data and components that want to pull data.
Components that push data actively work as long as there is data available until a back pressure builds up in the system that resists their ability to push. This back pressure is typically the result of some queue or buffer that has filled up and is no longer able to accept the things being pushed into it. Components that push include several transport and protocol channels, as well as the service dispatcher that pushes messages to the service implementation. Components that pull data remain quiescent even if data is available until there is someone that actually wants to consume some data. Many transport and protocol channels pull messages rather than push.
A single implementation may feature both push and pull modes. For example, the TCP channel has a small portion that works in a push mode for connection establishment and transferring some initial data while the application portion of TCP works in a pull mode. Depending on the visibility of these push and pull modes, you might be able to tell from the behavior of a particular protocol what the service is doing.
For example, if you're using just TCP, then the initial transmissions needed to open a connection can only be completed once a little bit of pulling has been done on the receiving application side, thus telling you that the service is doing some active processing. On the other hand, if you're using a ReliableMessaging channel or a OneWay channel, then those protocols have a visible running state where they are in push mode. You can't actually tell whether the service is working until you fill up some of the buffers in the protocol stack and start getting push back in the form of rejected messages. That means the service is not working as fast as you're sending data. A queued channel would be an extreme example of push mode. A message queue allows you to push large amounts of data when the service is not even running.
Therefore, to know in the general case whether the service is ready to process for you, you need to be able to ask that question to the service rather than to the channel.
Next time: Composing Compression and Encryption