Network Class Library Team (System.Net)

New Picture Hunt Silverlight Socket Sample

2011-04-27T23:42:00Z

Introducing “Picture Hunt”, the newest Silverlight sample app. This multiplayer game demonstrates the power of the System.Net Socket API inside a browser. No more long-polls, or trying to turn HTML requests into something that’s almost a socket; you can use sockets directly! The sample is also a “good network citizen”. It shows the network status nicely, without any irritating pop-ups; it automatically tries to reconnect, and it will automatically switch networks as needed.

Check out the Powerpoint deck that goes into detail about the demo and complete source code for the whole demo, including the game server, pictures, and the Silverlight Policy server.

The sample is on Silverlight TV! An earlier version of the game (without the fancy animations) is on Silverlight TV on Channel 9 on MSDN. The discussion includes when to use sockets (answer: when you need to send and receive any time, not just when you’re polling, or when you need to match an existing protocol, or when you need to reduce your server resources), plus tips on how to get started, and common issues with network programming.

Enjoy!

Silverlight 4 Socket Policy Changes

2010-04-15T21:57:00Z

After installing the Silverlight plugin, a Silverlight application is able to run on your computer without your explicit consent whenever you browse to a webpage which links to it. Because there is no explicit user action required to run, Silverlight applications are executed within what is known as a security sandbox – a locked-down environment which prevents a malicious application from doing anything to attack your computer or your local network. For the Socket and UDP multicast client classes, this sandboxing takes the form of a required policy check which the application must pass before being able to communicate with network resources.

While these policy checks prevent a malicious application from attacking your computer or other devices on your local network, they also pose a barrier to deploying legitimate Silverlight applications. In order to make it easier to write and deploy Silverlight applications which communicate using Sockets and the UDP multicast clients, a number of changes have been made to the policy systems for Silverlight 4.

Elevated Trust

Silverlight 4 introduces the notion of Elevated Trust applications. Using this feature, an out-of-browser Silverlight application can indicate to the user that it wants to run in Elevated Trust while it is being installed for out-of-browser use. If the user consents, the application is run within an expanded sandbox which allows it to do things that a normal Silverlight application is explicitly denied the ability to do for security reasons.

In Silverlight 4, Elevated Trust applications have been given permission to use the Socket and UDP multicast client classes without policy checks. The Socket class may be used to create a TCP connection to any port on any host without the need for a cross-domain policy file. Similarly, the UDP multicast clients may join any multicast group on any port greater than or equal to 1024 without the need for a policy responder to authorize it. No extra API calls are needed in order to take advantage of this work; the existing APIs light up with the extra functionality when an app is running in Elevated Trust. In-browser and non-elevated out-of-browser applications still have the same policy check requirements as before, and will continue to throw a SocketException with the SocketErrorCode property set to SocketError.AccessDenied if the policy check fails.

Socket Policy via HTTP

When running in the browser sandbox, the policy check for the Socket class requires downloading a policy file from the intended target that controls which Silverlight applications are allowed to connect to it. In previous versions of Silverlight, this download was conducted using a custom protocol running on TCP port 943. While this protocol can be implemented in a very lightweight and easily-secured fashion on the server side, it requires running a dedicated service and opening port 943 in any firewalls between the client and server.

In Silverlight 4, applications using the Socket class may opt in to retrieving the policy file via the HTTP protocol (on TCP port 80) instead of the custom TCP protocol. This allows servers which are already running HTTP services to authorize Socket connections from Silverlight apps without having to deploy a new service on the machine or open up additional firewall surface. To toggle this behavior, set the new SocketAsyncEventArgs.SocketClientAccessPolicyProtocol property on the SocketAsyncEventArgs passed to Socket.ConnectAsync to SocketClientAccessPolicyProtocol.Http.

When this option is set, the Socket class will attempt to download the policy file using an HTTP GET request for http://[target ip address]/clientaccesspolicy.xml. It will NOT attempt to download the policy via the custom TCP protocol if the policy check fails. The path of the Uri (/clientaccesspolicy.xml) is the same as that used by the HttpWebRequest class for its policy check. The two policies may be combined into a single file; the Socket class will safely ignore any <resource> elements, and the HttpWebRequest class will safely ignore any <socket-resource> elements.

Unlike the HttpWebRequest class, the Socket class always specifies the target IP address as the host when retrieving the policy file, even if the connection is made to a DnsEndpoint. This prevents confusion in the case of a single machine hosting websites from multiple domains. As a result, the server must be configured to respond to requests made to its IP address in addition to requests made to a specific hostname.

System.Uri FAQ

2010-02-23T23:04:00Z

NCL includes classes for networking related technology such as the base type System.Uri. URI’s are used extensively to identify network resources, especially on the Internet. We’ve found that different customers frequently ask us the same questions about System.Uri and its capabilities, so we would like to share some of those questions with you here. For space reasons we will address some of the harder questions in an additional post about System.Uri Customization.

1. System.Uri vs. Strings

Q. If a URI is just a simple string of characters, why do you need the System.Uri type at all?

A. A URI is a string of characters, but it is not necessarily simple. For example, take the following URI:

HTTPS://basicUser:password@www.xn--fek.COM:443/path/../realpath/./ル/my-App.Aspx?QueryParam1=value1;param2=value2#document_Fragment

There are a number of public standards that outline the syntax for URIs. System.Uri’s role is to validate the given string is in the appropriate syntax, as well as to provide easy access to the subcomponents.

Microsoft took this seriously enough to add a series of Code Analysis rules in Visual Studio to discourage developers from storing URIs as Strings. See: http://msdn.microsoft.com/en-us/library/ms182174.aspx

2. Standards and History

Q. What public standards does System.Uri support? Has this changed in newer versions of the .NET framework?

A. When the System.Uri class was first introduced in .NET 1.0, it implemented the public standards described in RFCs 1736, 1738, and 2396, along with other functionality necessary for a base type in the .NET Framework. Since then, these standards have been updated or modified by new standards introduced in RFCs 3490, 3986, 3987, and others. In .NET 3.5 System.Uri was expanded to add support for RFCs 3490 and 3987 as described in question 10 below. In order to maintain backwards compatibility very few changes have ever been made to migrate System.Uri from RFC 2396 to 3986.

3. URI Components

Q. Which System.Uri properties give me which components of my URI?

A. Here are some of the common properties, along with the (canonicalized) data they represent from the previous sample URI. See RFCs 2396 and 3986 for more details on the component definitions. Note: IRI and IDN are enabled for these samples (see question 10).

Property/Method	Result
OriginalString	HTTPS://basicUser:password@www.xn--fek.COM:443/path/../realpath/./ル/my-App.Aspx?QueryParam1=value1;param2=value2#document_Fragment
Scheme	https
UserInfo	basicUser:password
Host	www.ル.com
DnsSafeHost	www.xn--fek.COM
Port	443
Authority*	www.ル.com (+ ‘:’ + Port, for non-default ports)
AbsolutePath	/realpath/%E3%83%AB/my-App.Aspx
Query	?QueryParam1=value1;param2=value2
Fragment	#document_Fragment
AbsoluteUri	https://basicUser:password@www.ル.com/realpath/%E3%83%AB/my-App.Aspx?QueryParam1=value1;param2=value2#document_Fragment
ToString()	https://basicUser:password@www.ル.com/realpath/ル/my-App.Aspx?QueryParam1=value1;p aram2=value2#document_Fragment

* The RFC’s indicate that Authority can optionally include UserInfo. To enhance security and limit the exposure of user credentials, System.Uri does not include UserInfo in the Authority.

4. Canonicalization

Q. Why don’t I get back exactly what I put in?

A. According to the RFCs, some URI components may be accepted in not-quite-standard formats as long as they are converted to standard formats. This process is called canonicalization or normalization. Examples of such are that the scheme and host should be lower case, default port values should be dropped, and dot segments in the path should be compressed. Additionally, System.Uri expands IPv6 addresses to their longest form. All of these processes are required to facilitate equivalency checking when accessing resources and verifying security access.

5. Uri Manipulation

Q. Why can’t I change any of System.Uri’s properties?

A. This is because System.Uri is an immutable base type, the same as System.String. You couldn’t trust System.Uri as a base type if anybody could change its contents underneath you. To conveniently remove, change out, or assemble components of a URI use the System.UriBuilder class, or System.UriTemplate. Also see the relative Uri APIs; Uri.MakeRelativeUri, and a System.Uri constructor.

6. Supported Schemes

Q. What schemes does System.Uri support?

A. System.Uri has built in parsing rules for Http, Https, File, Ftp, Gopher, MailTo, NetPipe, NetTcp, News, Nntp, Uuid, as well as generic parsing rules applied to unrecognized schemes. Here are some samples for each of these schemes.

http://user:password@host:80/path/file.txt?Query#fragment

https://user:password@host:443/path/file.txt?Query#fragment

ftp://user:password@host:21/path/file.txt#fragment

file://host/path/file.txt?Query#fragment

c:\path\file.txt (implicit DOS file)

\\host\share\path\file.txt (implicit UNC file)

gopher://user:password@host:70/path/file.txt#fragment

news:///path/file.txt#fragment

nntp://user:password@host:119/path/file.txt#fragment

telnet://user:password@host:23/path/file.txt#fragment

ldap://user:password@host:389/path/file.txt?Query#fragment

mailto:user:password@host:25/path/file.txt?Query#fragment

net.pipe://host/path/file.txt?Query#fragment

net.tcp://host:808/path/file.txt?Query#fragment

uuid:///path/file.txt#fragment

7. UriFormatExceptions & Validating User Input

Q. Do I have to try/catch UriFormatExceptions every time I create a new System.Uri?

A. Not always. In .NET 3.5 we added the System.Uri.TryCreate methods so you can check for problems with your data input without all the hassle and performance problems of try/catch blocks and exception handling.

Uri result = null;

if (!Uri.TryCreate(userUriString, UriKind.Absolute, out result))

{

// Fail gracefully, ask the user to try again.

return false;

}

The down side here is that when a URI fails to parse, these methods do not return an error message explaining what was wrong with the input. If you need to display the error message then you should still use the try/catch pattern.

8. MailTo Multiple E-mail Addresses

Q. Why doesn’t System.Uri’s MailTo scheme support multiple e-mail addresses?

A. System.Uri supports RFC 1738 which only allows for a single e-mail address in the Authority of a MailTo URI.

An alternative way to represent multiple e-mail addresses with one System.Uri is by using query parameters as discussed in RFC 2368. The following example works well with Microsoft Outlook:

mailto:jim@contoso.com?cc=test@contoso.com;billy@contoso.com&bcc=sally@contoso.com&subject=High%20there&to=boby@contoso.com

The only caveat with this format is that Outlook will only populate the To field with the primary e-mail address (jim@contoso.com); Any To field data in the query (boby@contoso.com ) is ignored. Other e-mail application’s behaviors may vary.

Alternatively, if you need to parse out multiple e-mail addresses from a data string, you can use the System.Net.Mail.MailAddressCollection class.

If you have access to a mail server, you may also consider adding all of the addresses to a mailing list. Then you only need a single address (mailto:MyGroup@constoso.com), and the lists membership can be updated without changing this address.

9. Implicit Dos/Unc File Paths

Q. Why can’t I put # in my file:/// URI path? It’s a valid file name character!

A. In a standard URI, the # symbol identifies the start of the Fragment portion. Because this may conflict with local file paths, System.Uri treats # differently for implicit Dos and Unc file paths.

	Explicit	Implicit
Dos	file:///c:/path/file.txt#Fragment	c:\path\file#name.txt
Unc	file://server/share/path/file.txt#Fragment	\\server\share\path\file#name.txt

As shown in the table, explicit file URIs are allowed to have fragments, so the # symbol is not considered part of the file path for the LocalPath or AbsolutePath properties. Implicit file URIs cannot have fragments, so the # symbol remains part of the path.

10. Unicode, IRI, and IDN

Q. Why does System.Uri turn my Unicode into xn--fek or %E3%83%AB?

A. Many computer programs and protocols do not yet understand non-Latin/ASCII character sets, so System.Uri has implemented RFC defined logic for encoding and decoding these non-Latin Unicode characters. This support was significantly expanded in .NET 3.5 to include the IDN and IRI RFC standards described below. For backwards compatibility reasons these new behaviors can only be enabled through configuration file settings. See the System.Uri documentation for details.

RFC 3490 outlines International Domain Names (IDNs). This standard covers the encoding of non-Latin Unicode characters in the Host portion of a URI. This encoding is designed for backwards compatibility with existing DNS standards and implementations. The Uri.DnsSafeHost property was extended to show this new format.

RFC 3987 introduces Internationalized Resource Identifiers (IRIs). This standard discusses how to encode non-Latin Unicode characters outside of the Host portion of a URI. Use the Uri.AbsoluteUri property rather than Uri.ToString() to get this encoding.

Conclusion

As you can see System.Uri is a powerful tool that helps you handle a large variety of complex URI syntaxes. In a following post we will address how you can expand System.Uri’s support for additional syntaxes through customization.

UDP Multicast in Silverlight 4

2009-11-18T22:40:00Z

In Silverlight 4, we have added multicast support. If you’re not familiar with multicast, here’s a quick scenario to explain what it is and why it might be useful to you.

Scenario Overview

Suppose your company provides market data and you need to distribute the same commodity and value records to 50,000 client workstations. It would be quite a load on the server and network infrastructure to do this in the traditional client/server TCP model where all 50,000 clients connect to the server to receive the data feed in round-robin fashion.

Wouldn’t it be great if the server could just send the data once and let the network infrastructure pass it on to all subscribers in an efficient manner? That would take the load off the server since it would only be sending out the data once instead of 50,000 times. Conveniently, that’s precisely how multicast works, and this makes it a useful technique for video and content distribution.

User Datagram Protocol (UDP)

User Datagram Protocol (UDP) supports multicast and is the most common low-level protocol for IP multicast. Because it’s so low-level, there are some key things you should be aware of:

UDP by nature is not “reliable” like TCP. This means messages are not guaranteed to be received in order or at all. There are, however, forms of reliable multicast, such as Pragmatic General Multicast (PGM) which can be implemented on top of UDP.
UDP has no built in IP address spoofing protection. Just because a message indicates it came from a particular IP address, you can’t be sure it did, so your application protocol needs to implement this protection if you need to be sure of the origin of your messages.

Multicast Concepts

Multicast Groups and Addresses

People are most familiar with unicast addresses, which are used to direct packets to a single host. Alternatively, a special multicast address can be used to deliver a packet to a group of destinations; specifically, all nodes which have explicitly joined the multicast group as identified by the multicast address. The 224.0.0.0/4 IPv4 and ff00::/8 IPv6 address ranges have been reserved for identifying multicast groups.

Joining a Multicast Group

When a computer or device wants to join a multicast group, it performs a special exchange with its router using the Internet Group Management Protocol (IGMP) or the Multicast Listener Discovery (MLD) protocol. This lets the router know the computer is interested in receiving traffic from the multicast group. To leverage multicast, it must be supported by the router and not otherwise blocked.

Multicast is often deployed within corporate networks for efficient transmission of video. Be aware that it is common for corporate firewalls to block multicast traffic at the edge so that it cannot traverse the boundary of the corporate network. Many home ISPs enforce similar restrictions.

It’s also worth mentioning that there are multiple versions of IGMP and MLD, but the specifics of negotiation are up to the client OS stack and the first-hop router; it’s not something your application needs to worry about.

Finally, in addition to joining a multicast group, a complementary operation to leave the group is also available.

Multicast Sources

The node or nodes sending data to the group are known as multicast sources. There are two common forms of IP multicast differentiated by the number of sources in the group:

Single Source Multicast / Source Specific Multicast (SSM) – A single, well-known source identified by a unicast IP address will send to the group (one-to-many).
Any Source Multicast (ASM) – Any member of the group may behave as a multicast source and transmit data to the group (many-to-many).

In the case of any source multicast, additional operations are provided to block and unblock specific sources so that messages can be filtered by source address. One of the benefits of SSM is that with a single source known ahead of time, the multicast tree can be built more efficiently when nodes join the group.

New APIs

In correlation to the multicast forms above, two new classes have been introduced to the System.Net.Sockets namespace, UdpSingleSourceMulticastClient and UdpAnySourceMulticastClient. If you’re familiar with Sockets or UdpClient in the .NET Framework, the available operations should look very familiar to you. Here are some brief, simplified samples to demonstrate the basics.

Construction

// Construct a client for the multicast group 224.0.0.1 port 3000
// Note that only ports >= 1024 are supported for Silverlight
var client = new UdpAnySourceMulticastClient(IPAddress.Parse(“224.0.0.1”), 3000);

Join Group

// Join the multicast group
client.BeginJoinGroup((result) => EndJoinGroup(result), null);

Message Receive

// Receive a message
client.BeginReceiveFromGroup(buffer, offset, count,
(result) => { messageLength = EndReceiveFromGroup(result, sourceAddress); }, null);

Leave Group and Dispose

// Leave the group and cleanup connections
client.Dispose();

In a real application, the async calls would be chained together. For a more detailed example showing this, see SilverChat on MSDN Code Gallery.

Browser Security

Because multicast in corporate environments is often blocked at the edge, Silverlight should not be able to circumvent that trust model by allowing code from an untrusted domain to run unchecked within the corporate network where it could potentially receive and forward privileged data. To prevent this, Silverlight requires that a Silverlight Multicast Policy Responder join the multicast group to authorize Silverlight clients. If multicast has been blocked at the corporate edge, then only a service behind the corporate firewall will be able to authorize Silverlight clients, thus maintaining the original network boundary.

You are free to deploy the responder in any way you choose, whether on the same or separate machine as your multicast source.

For details of the policy protocol used and to review a sample implementation, please see the Silverlight Multicast Policy Responder sample also on MSDN Code Gallery.

Conclusion

This article covered how multicast can be used to reduce server and network infrastructure load, the new APIs added in Silverlight 4 to support these features, the special requirements to maintain browser security, and finally, some important points to consider for your application protocols built on UDP. Please let us know if you have questions about these new features. In addition to blog comments, you may also direct inquiries to the forum.

New Performance Counters for HttpWebRequest

2009-08-08T06:56:00Z

Performance counters! You can never get enough! And starting with the .NET version 4.0 Beta 2, we in the System.Net team have added six new counters to the old set. But before we begin, some ground rules:

Performance counters have to be enabled in your application’s .config file. I’ve put a sample of my config file at the end of this post.
The new counters need the .NET version 4 Beta 2 runtime.
The counters are now called “.NET CLR Networking 4.0.0.0” (the version number is new). You only get the new performance counters when you run using the .NET version 4 CLR. If your program runs using an older CLR, it will use the older performance counters.
The counters are now initialized on a separate thread. This means that for the first small amount of time, activity isn’t recorded by the performance monitor.

Without any further delay, let’s take a look at the new counters.

The first two performance counters let you know that your application is running smoothly.

1. HttpWebRequests Created/Sec
2. HttpWebRequests Average Lifetime

The ‘Created/Sec’ is just what it says: it’s updated whenever an HttpWebRequest is created. The performance counter takes care of adjusting the raw numbers into a “per sample interval” – just beware that the label says “per second” but it’s really “per performance counter sample time interval”. It turns out that the default sample interval is one second, and the label is much shorter when we just call it “/Sec”.

You can see the effects of this adjustment for yourself – make a program that creates a fixed number of HttpWebRequests per second. When the performance monitor is set to “Sample every 1 seconds”, it gives that amount. Set it to “Sample every 2 seconds” and the number doubles.

But watch out! System.Net will make HttpWebRequest objects for you – in particular, the System.Net automatic proxy detection on Windows XP will create an HttpWebRequest internally to resolve your proxy. System.Net will also make extra objects for tunneling through proxies. It’s even possible for FTP requests to silently make HttpWebRequests. Exactly when we make these extra objects can change from one release to the next.

The average lifetime is a little sneakier. It’s the lifetime of the entire HTTP transaction, starting when the HttpWebRequest is created all the way until the resulting HttpWebResponse is closed. If you ever see this counter stuck at zero even though you’re making requests and getting replies, it’s probably because you haven’t closed your HttpWebResponse! The ‘average’ is the average for the last performance sample interval, so it’s just for the time in the “Sample every ___ seconds” option in the Performance Monitor properties box.

There are two performance counters to look for Queuing issues. The .NET framework doesn’t let you (by default) send lots of requests to a server all at once. Instead, only up to ‘ConnectionLimit’ requests can be active at once (by default, this is two); after that we put the request onto a queue. Thanks to pipelining, a request can be ‘waiting’ without being ‘queued’. This happens when you make a request, and there’s a perfectly good connection to the server ready to be used, but it’s busy with another request. The new request will be pipelined with an earlier request and not queued (but it still has to wait for the earlier request to end).

The connection limit is trickier than you might realize because HttpWebRequests that are bound for different places but which share the same proxy share the same ServicePoint and therefore share the same set of connections.

The Queue performance counters are:
3. HttpWebRequests Queued/Sec
4. HttpWebRequests Average Queue Time

The Queued/Sec counter is updated whenever an item is put onto the queue; the Average Queue Time is updated whenever an item is removed from the queue. This means that it’s possible to see items being added to the queue like crazy even though the average queue time is zero – it could just mean that the server is slow and nothing is coming off of the queue.

Lastly, there are two counters to tell if your requests aren’t completing correctly.
5. HttpWebRequests Aborted/Sec
6. HttpWebRequests Failed/Sec

A request is Aborted when the ‘Abort()’ method is called. The Abort() method is automatically called by System.Net in some cases, all having to do with failures – when the request times out, for example, or sometimes when the request handler code triggers an exception. Your application code can also call Abort() on a request.

The most common reason that a request is Failed is that the final server response is a “could not fulfill your request.” For example, a server reply code of 404 Not Found is a failure. Technically speaking, the failed counter is updated when a request triggers an exception (other than exceptions thrown in your application code). This is a good definition of failed because all of the failed server codes will throw a WebException. Server codes that are not final – for example, a 401 Unauthorized return code will usually result in the System.Net code automatically retrying with proper credentials. In this case, the 401 Unauthorized is not final and does not trigger an exception.

That’s a lot: how about a diagram?
Certainly! Here’s a diagram that our developer created to help understand what all of the timings are. Each of the seven green circles represents one of the six performance counters (there are two ‘5’ items because 5 is the average lifetime, and there are two code paths that will affect that counter).

As promised, here’s a sample of my config file:

The performanceCounters value is documented on MSDN at http://msdn.microsoft.com/en-us/library/ms229151.aspx.

<?xml version="1.0" encoding="utf-8" ?>

<system.net>

</settings>

</system.net>

</configuration>

I also had to add in a “requiredRuntime” to my .config file: <startup><requiredRuntime version="v4.0" /></startup>. Otherwise, my application – which I compiled with an older compiler – wouldn’t use the 4.0 CLR.

What's new in System.Net.Mail

2009-08-06T23:10:00Z

What’s new in System.Net.Mail

We’ve made a number of enhancements to our SMTP support for .NET 4.0, mostly in the area of Unicode support and increased standards compliance, which is an important aspect in ensuring that legitimate emails do not get flagged as spam, as well as a few other useful features. In this post I’ll go over these enhancements and what they mean to you as a developer.

Decreased likelihood of being accidentally flagged as spam

Many spammers don’t bother to follow the SMTP protocol correctly in sending their spam, so standards violations tend to be a major factor in calculating spam scores. In .NET 4.0, we’ve made some significant improvements in how we encode headers and fold long lines of text in order to better comply with the SMTP protocol to reduce the likelihood that your legitimate emails will accidentally be caught by an overzealous spam filter. You don’t need to change a thing in your code to take advantage of this since it’s all internal to System.Net.Mail.

Increased Unicode support

You can now include Unicode in your custom headers thanks to a new property to the MailMessage class called HeadersEncoding, which allows you to specify what Unicode encoding that custom headers are using so that they can be sent correctly (the default is UTF-8). HeadersEncoding only affects custom headers that you add; the subject, body, and email address display names can have different encodings that are controlled by other properties in the MailMessage and MailAddress classes.

Clarification on setting header values

While MailMessage allows you direct access to the headers collection some headers being malformed could cause the email message to become corrupted. The following is a list of headers that you should not set via the headers collection; any values you set for these headers will be silently discarded or overwritten when the message is sent (use the appropriate properties of MailMessage to control these headers):

Bcc, Cc, Content-ID, Content-Location, Content-Transfer-Encoding, Content-Type, Date, From, Importance, MIME-Version, Priority, Reply-To, Sender, Subject, To, X-Priority

Note: if you do not specify an X-Sender header, we will create one for you however we will not overwrite an X-Sender that you set.

Multiple Reply-To addresses with the new property ReplyToList

We’ve added another new property to MailMessage called ReplyToList that allows you to add multiple Reply-To addresses to an email. The ReplyTo property has been obsoleted.

Content Disposition time zones

We’ve received some feedback on how ContentDisposition does not allow you to specify time zones in the character formats dictated by RFC 822, such as “GMT” or “PST.” This functionality was actually obsolete by RFC 2822, which replaces RFC 822, and dictates that an email time should always contain the offset for a timezone (e.g. “-0800” or “+0000”). However, we will now accept RFC 822 time zone formats and convert them to their semantically equivalent offsets. Whenever we are unable to determine the correct semantically equivalent offset, it is converted to “-0000” meaning that the time zone is unknown. This affects strings passed to the constructor or Parameters collection of ContentDisposition.

Mail Address formats

We’ve improved our ability to parse email address strings in the MailAddress class. Display names that contain Unicode cases and quoted local parts in the address will now be accepted correctly in most circumstances. If an address does contain an invalid character, the exception message will inform you of what character was invalid. Remember, you should format your email addresses as “displayname” <local@domain>.

Authentication

Some legacy clients do not advertise their AUTH mechanisms correctly. Normally, in the EHLO reply you would expect to see something like AUTH LOGIN NTLM advertising that this server supports LOGIN and NTLM authentication. Some legacy servers will instead reply with AUTH=LOGIN NTLM which is a violation of the RFC, however we have added support to SmtpClient for these legacy servers so authentication will work correctly should you encounter one.

We hope that you find this new and improved functionality useful for your applications and, as always, feel free to drop us a line if you have any feedback on these features or anything else you would like to see!

End-to-end connectivity with NAT traversal

2009-07-27T21:35:00Z

Like street numbers for a house, the Internet was originally designed so that all network devices could be directly addressed. Every connected device was given at least one unique identifier, or IP address, which could be used to route network packets to and from the device. For a while this worked well and devices had end-to-end connectivity.

IPv4 addresses can be used to route to about 4 billion (2³²) devices, but the rapid growth of the Internet quickly exhausted that available real estate. In the late 1980’s, several methods were developed to conserve this rapidly dwindling address space. Network isolation became a key strategy in this effort, and it had a beneficial side-effect – increased security. If an attacker was unable to address a device to directly establish connectivity, then attacking it was more difficult.

Home Routers with NAT

A common means for achieving this network isolation in the home is through an IPv4 router which supports Network Address Translation (NAT). These devices do some useful things for you:

They pick up a single, public IPv4 address from your Internet provider
They hand out IPv4 addresses to devices on your local network, most often through DHCP, drawing from a pool of the IPv4 address space reserved for private networks
They allow all of the devices on your network to share the single, public IPv4 address received from your Internet provider and handle the translation automatically from private to public address in conjunction with port information

The following diagram illustrates this kind of deployment for two private networks both connected to the Internet through a NAT.

With these features also come limitations. Note how both networks share the same private address space, both routers have the same 192.168.1.1 private IP address, and two different computers each have the same 192.168.1.20 private IP address. Devices on each private network can communicate within their own network, but how can they communicate with each other over the Internet?

Internet packets are routed using public addresses, and in the scenario above, each network only has one public IPv4 address. That means all packets go to the home router and it figures out whether to send them on to a device on the home network.

Here are 3 common strategies for achieving this routing between public and private networks:

Automatic translation can occur when a home device makes an outgoing network connection. This allows the NAT to match the destination address and port to the originating local address and port, then translate between them as packets are sent and received.
Manual configuration can be performed by an administrator where a given home router port can be redirected to a private device address and port. This allows communication from the public Internet to be directed to a private device by using the public IPv4 address of the home router in combination with an administrator defined port.
Automatic configuration can be performed by devices or applications behind the home router using UPnP. This allows a device or application to map ports on the router automatically so an administrator doesn’t have to. Most home routers support UPnP today. An unfortunate problem with this approach is that UPnP requires a specific port to be chosen at the time of registration, and this can cause collisions if two callers attempt to register the same port. UPnPv2 and NAT-PMP address this shortcoming.

NAT Traversal & Security Considerations

Because of the security boundary offered by NAT configurations, a key tenant for any traversal technology is to continue to maintain that security boundary for existing applications and services. The technologies discussed in this article adhere to that tenant. For a detailed discussion of this topic, you may refer to this informational on Security Concerns With IP Tunneling.

Impact to Applications

Generally these strategies result in a decent connectivity story, though complicated, and this carries over into application development.

In NAT situations, devices don’t really have end-to-end addressability by default, and the port begins to play an overdeveloped role in routing to your home devices to compensate for private addresses not being publicly reachable.

If your application only makes outgoing connections, the NAT solution will generally handle this transparently for you. The complications occur in applications that not only make but also receive connections – again, very common with peer-to-peer networks and video games.

If your application listens on a particular port for incoming connections and relies strictly on IPv4, you might ordinarily need to resort to one of the following techniques:

Add port mapping support to your application so it can automatically detect when it is behind a home router and configure the router to direct traffic to the application instance and port
Instruct your users to do this manually (which requires far more networking knowledge than is desirable)

A key point and drawback to keep in mind is that there is a 1:1 mapping between the Internet facing port on the router and the private IP and port pair of your local device.

If you want to run a web server from behind your NAT, you might allocate port 80 on the NAT and have it direct traffic to port 80 on Computer 1. If you wanted Computer 2 on the same network to also host a website available on the Internet, a different port on the NAT would need to be allocated, such as port 81 since port 80 is already in use and mapped to Computer 1.

Visitors would then need to use the router’s public IP address combined with a distinct port to reach the appropriate web server. This is not ideal since web visitors may not be accustomed to having to specify a specific, non-default port.

Contention for well-known ports demonstrates just one of the problems that may be encountered when relying on port mapping instead of an IP address with enough specificity to more fully handle addressing.

With that in mind, we’ve made many of these complexities transparent for you in the .NET Framework 4.0.

If you’re using TcpListener or UdpClient, just pass into the constructor IPAddress.IPv6Any, then call AllowNatTraversal with a value of true.

var listener = new TcpListener(IPAddress.IPv6Any, 8000);
listener.AllowNatTraversal(true);
listener.Start();

You’ll notice we have been discussing IPv4, but the example above mentions IPv6. Have no fear, this will work over intermediate IPv4 networks, it just relies on the origin or destination endpoints supporting IPv6 and a couple key technologies which I’ll cover in more detail below. If your application must run on a PC which only has IPv4 installed, a condition that is becoming more rare, then unfortunately this new solution won’t help you right now.

Another point worth mentioning is that applications which wish to listen on all IP addresses today have to set up two listeners, one for IPv4 using IPAddress.Any and one using IPAddress.IPv6Any. We’re investigating ways to take advantage of “dual mode” sockets so you won’t need to worry about tying your application to a specific IP version in the future.

IPv6 Restores Addressability

What if all your network devices had a public IP address which could be used by other devices to directly communicate with them? This would eliminate the need to use NAT for addressing and once again achieve end-to-end connectivity. With IPv6, this is possible.

IPv6 has a significantly larger address space (2¹²⁸ or about 3.4×10³⁸), and with an address space this size, it is once again possible for every device connected to the Internet to be given a unique address.

But, there’s a problem. Much of the world is still using IPv4, so exactly how can your application take advantage of IPv6 addresses today?

Fortunately, a number of transparent solutions have been devised and implemented on platforms and devices so that your application doesn’t need to worry with the specifics so long as it listens on all available IP addresses.

6to4 Tunneling

Although applications generally don’t need to worry about the underlying mechanism used to allocate an IPv6 address, for context, one such solution is 6to4 tunneling.

This solution works great for devices that already have a public IPv4 address. A special range of IPv6 maps to the IPv4 address space, and so with a 6to4 tunnel gateway deployed at the edge, IPv6 connectivity can be automatically enabled.

Windows supports 6to4, so if your computer already has a public IPv4 address, or you take your laptop to a hotspot that assigns it one, it probably also has a public IPv6 address through the 6to4 pseudo adapter.

Of course, in the NAT scenario we’ve been discussing, only the home router has a public IPv4 address. So, to take advantage of 6to4, your router would need to be IPv6 compatible, it would need to be able to assign IPv6 addresses to the local network, and and it would need to have 6to4 tunneling built into it.

Presently, most home routers don’t support this, though it is something a Windows Server can be configured to do as can standard Windows versions supporting Internet Connection Sharing (ICS).

So if a home router holds the one and only public IPv4 address, and 6to4 isn’t an option, how can computers behind the NAT use IPv6?

Teredo Tunneling

Like 6to4, Teredo is another IPv6 transition technology which brings IPv6 connectivity to IPv4 networks. It is described by RFC 4380, is further extended by MS-TERE, and has even been implemented by the open source community as Miredo.

There are a number of technical articles on exactly how Teredo encapsulates IPv6 over IPv4/UDP, which most NATs can forward, so if you’re interested in those details you can find out more from the links at the end of the article.

What’s important to know here is that Windows OS versions starting with Windows XP SP2 and Windows Server 2003 provide a virtual Teredo adapter which can give you a public IPv6 address in the range 2001:0::/32. This address can be used to listen for incoming connections from the Internet and can be provided to IPv6 enabled clients that wish to connect to your listening service.

Teredo and related transition technologies free your application from worrying about how to address a computer behind a home router or NAT since typically all you need to do is connect to it using its IPv6 addresses.

New for .NET 4.0

We’re making some additions in the .NET Framework 4.0 starting with Beta 2 to make these great technologies easier for you to use. The .NET Framework client APIs already support address-based NAT traversal, so the main updates are for listeners.

Listeners

The TcpListener and UdpClient changes were previously mentioned. Use these new methods to toggle whether your application explicitly wants to allow or restrict NAT traversal support.

public class TcpListener
{
public void AllowNatTraversal(bool);
}

public class UdpClient
{
public void AllowNatTraversal(bool);
}

Sockets

If you’re using sockets directly, the interface is a little closer to what Winsock exposes via the IPV6_PROTECTION_LEVEL socket option. We have exposed this on the Socket class as a new method called SetIPProtectionLevel.

public class Socket
{
public void SetIPProtectionLevel
(IPProtectionLevel);
}

public enum IPProtectionLevel
{
    Unspecified = –1,    // platform default
    Unrestricted = 10,   // global with NAT traversal
    EdgeRestricted = 20, // global without NAT traversal
    Restricted = 30,     // site local
}

Set the IPProtectionLevel to Unrestricted prior to Bind to allow clients to connect to your listener deployed behind a NAT. This is what the System.Net listener implementations do when you invoke the previously mentioned AllowNatTraversal method.

EdgeRestricted allows clients to only connect to your listener on IP addresses that aren’t used for NAT traversal (like Teredo).

Restricted only allows intranet connectivity (site and local link).

The actual default setting is Unspecified and left to the underlying platform to determine.

Starting with Windows Vista, this is equivalent to Unrestricted when the Windows Firewall is enabled and an appropriate rule is configured per the instructions below and EdgeRestricted when it is disabled.

This honors that security point mentioned at the beginning of the article. Many IPv4 networks rely on NAT as a limited form of protection. The default setting protects applications from unintentionally exposing themselves to the Internet in NAT scenarios. Instead, applications must explicitly opt-in to NAT traversal using this socket option or by configuring a Windows Firewall rule.

Configuration

To enable you to easily turn these features on for your existing applications, you can also control this through app.config.

<system.net>
<settings>
    <!-- default is platform defined (Unspecified) –>
    <socket ipProtectionLevel="Unrestricted |
      EdgeRestricted | Restricted | Unspecified"/>
</settings>
</system.net>

This setting will affect all listening sockets in an AppDomain.

Address Discovery

It’s not recommended to implement behavior that relies on direct knowledge of IP addresses since this would typically be handled through name resolution, but in some cases, like building peer to peer applications or your own discovery service, it can be useful.

To get the list of addresses on a host, you could do it the traditional way using System.Net.NetworkInformation to enumerate NetworkInterfaces and their addresses, but we’re adding some new methods which make this simpler and also “wake up” Teredo if it hasn’t been used recently.

public class IPGlobalProperties
{
public UnicastIPAddressInformationCollection
    GetUnicastAddresses();

public IAsyncResult BeginGetUnicastAddresses
    (AsyncCallback, object);

public UnicastIPAddressInformationCollection
    EndGetUnicastAddresses(IAsyncResult);
}

This allows you to enumerate addresses as follows.

var addressInfoCollection =
IPGlobalProperties.GetIPGlobalProperties()
.GetUnicastAddresses();

foreach(var addressInfo in addressInfoCollection)
{
Console.WriteLine("Address: {0}",
addressInfo.Address);
}

Once you have acquired this list of addresses, you can give the address list to your clients out of band and they can use Socket.Connect, TcpClient.Connect, or even WebRequest.Create to establish a connection to your service.

For a savvy way to publish your addresses so others can discover them, check out our Peer Name Resolution Protocol (PNRP). Another great discovery mechanism is the Collaboration API. The traditional mechanism is of course to use DNS and the System.Net APIs which accept a host name.

Finally, we’re also adding a convenient property to the IPAddress class so you can tell if an address you’re dealing with is an IPv6 Teredo address.

public class IPAddress
{
public bool IsIPv6Teredo { get; }
}

This property is primarily intended to be used for debugging and test scenarios since you will typically want to listen on all available IP addresses so your application can automatically take advantage of new platform enhancements.

Right now, client support works transparently the whole way up the transport stack, and for listeners we support these new features with Sockets, TcpListener, and UdpClient. We hope to extend this to HttpListener in the future.

Windows Communication Foundation

WCF supports Teredo today for TCP channels when using the NetTcpSection.TeredoEnabled and TcpTransportElement.TeredoEnabled properties.

Firewall Considerations

By default, the Windows Firewall blocks incoming connections. For your listener to be accessible, you will want to create a firewall rule. This is true of any listening application, not just ones that wish to take advantage of NAT traversal. You can do this programmatically, and since adding firewall rules requires UAC elevation, application installation is the best time to do this. Even though the rule can be added at install time, it can be configured to activate only while the application is running.

The firewall can be controlled using COM interop. Add a project reference to FirewallApi.dll. You can then add a rule with the following code.

Guid netFwRuleUuid = new Guid("{2C5BC43E-3369-4C33-AB0C-BE9469677AF4}");
INetFwRule rule = (INetFwRule)Activator.CreateInstance(Type.GetTypeFromCLSID(netFwRuleUuid));

rule.Action = NET_FW_ACTION_.NET_FW_ACTION_ALLOW;
rule.ApplicationName = @"C:\Program Files\My Application\MyApplication.exe";
rule.Description = "My Rule Description";
rule.Direction = NET_FW_RULE_DIRECTION_.NET_FW_RULE_DIR_IN;
rule.EdgeTraversal = true;
rule.Enabled = true;
rule.Grouping = "My Rule Group";
rule.Name = "My Rule Name";
rule.Protocol = (int)ProtocolType.Tcp;

Guid netFwPolicy2Uuid = new Guid("{E2B3C97F-6AE1-41AC-817A-F6F92166D7DD}");
INetFwPolicy2 policy = (INetFwPolicy2)Activator.CreateInstance(Type.GetTypeFromCLSID(netFwPolicy2Uuid));
policy.Rules.Add(rule);

Note that to enable NAT traversal, the EdgeTraversal flag must be set to true since a firewall rule with a setting of false (the default) is used to prevent NAT traversal even when an application sets the IPProtectionLevel for its sockets to Unrestricted.

Starting with Windows 7, although you can still use the approach above, there is more control over the default NAT traversal behavior using the InetFwRule2 interface and NET_FW_EDGE_TRAVERSAL_TYPE.

Third party firewalls may require custom configuration or may prompt the user for permission on first application launch.

Parting Thoughts

With IPv6 and its related transition technologies becoming ubiquitous, now is a great time to take advantage of these capabilities in your applications, and the .NET Framework 4.0 makes it easy to get started.

Additional References

Microsoft IPv6 Website
IPv6 Transition Technologies
TechNet Teredo Overview
TechNet Using IPv6 and Teredo
MSDN Teredo Site
Firewall requirements for coexisting with Teredo
Security Concerns With IP Tunneling

Special thanks to Dave Thaler for his insights and expert feedback.

~ Aaron Oneal | NCL Program Manager

New NCL Features in .NET 4.0 Beta 2

2009-07-20T18:59:00Z

We’re introducing some new features starting with .NET 4.0 Beta 2 that you may find useful. Additional information will be available on MSDN and in subsequent articles. If you have any questions or comments, let us know!

Sockets

DnsEndPoint

This feature was first introduced in Silverlight 2 and it allows you to connect to a listening socket by DNS name, simplifying the connect experience. Now you can take advantage of this in .NET as well.

Before

IPAddress[] IPs = Dns.GetHostAddresses("www.contoso.com");

foreach(IPAddress ip in IPs) {
try {
socket.Connect(new IPEndPoint(ip, port));
break;
}
catch(SocketException) { continue; }

}

After

socket.Connect(new DnsEndPoint("www.contoso.com", port));

IP Version Neutrality

Starting with Windows Vista, the TCP/IP stack has features which allow you to write sockets applications that can transparently handle multiple versions of the Internet Protocol. This works by creating an IPv6 socket and setting the IPv6Only socket option to false. Once set, the IPv6 socket is also able to accept IPv4 traffic. This frees your application from the traditional model of having to create multiple sockets for each IP version.

Socket socket = new Socket(AddressFamily.InterNetworkV6, SocketType.Stream, ProtocolType.Tcp);

socket.SetSocketOption(SocketOptionLevel.IPv6, SocketOptionName.IPv6Only, 0);

socket.Bind(new IPEndPoint(IPAddress.IPv6Any, 8000));
socket.Listen(5);
Socket client = sock.Accept();

SSL Encryption Policy

Traditionally SSL is used both for encryption and for authentication / message integrity. In some cases, however, encryption is not required or desired. This could be for interoperability with other systems or performance reasons. Therefore, you can now change the default encryption policy for SSL. In some circles, this is also referred to as using a “null cipher”.

namespace System.Net.Security
{
public enum EncryptionPolicy
{
    RequireEncryption, // always require encryption
    AllowNoEncryption, // prefer full encryption; allow none if server agrees
    NoEncryption       // require no encryption; fail if server requires it
}
}

This setting can be configured in the following places.

SslStream
ServicePointManager for HttpWebRequest, FtpWebRequest, SmtpClient
Machine.config or app.config

SmtpClient

SSL Configuration

It is now possible to enable explicit SSL in SmtpClient via application config in addition to the runtime option that was already available.

Header Encoding

Prior versions of .NET restricted headers to ASCII encoding. A new MailMessage.HeadersEncoding property will allow for an alternate encoding to be specified for the headers added to the MailMessage.Headers collection.

Multiple ReplyTo Addresses

The MailMessage.ReplyTo property only allows a single address to be added. We’re adding a new collection property called MailMessage.ReplyToList to support multiple addresses.

HttpWebRequest

64-bit Range Header Support

When a client program makes an HTTP request, that request includes a number of headers which determine how the server will respond. One of those headers is the Range header as described in RFC 2616 (obsoletes 2068). The Range header on a request allows a client to request that it only wants to receive some part of the specified range of bytes in an HTTP entity. Servers are not required to support Range header requests.

An example of a range header in the HTTP request is:

Range: bytes=1024-2047

Some of the uses of the range header include:

1. Manually maintaining “QOS” (quality of service) for streaming media. For example, a sophisticated movie viewer program might download the video and audio of a movie separately, downloading the “next” pieces only when needed

2. Manually handling file caching for interrupted downloads

The existing framework includes a set of public HttpWebRequest.AddRange methods as specified on MSDN at http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.addrange.aspx.

Additional overloads supporting Int64 ranges have been added.

Date Header Support

A new HttpWebRequest.Date property has been added to support setting of the HTTP Date header.

Host Header Support

A new HttpWebRequest.Host property has been added to allow configuration of the HTTP Host header. This header can be used during testing and with load balancers, for example, to ensure the TCP connection is established to a particular IP address but that the client can still address the intended web site at the specified HTTP server endpoint. For example:

var request = WebRequest.Create("http://127.0.0.1/") as HttpWebRequest;
request.Host = "contoso.com";
var response = request.GetResponse();

WebRequest Performance Counters

Troubleshooting a server application can be difficult, and one key tool for this is the Windows Performance Monitor. With the addition of new performance counters for WebRequest, this is now easier. The diagram below (click for a larger view) shows the new counters and the points in the lifetime of an HttpWebRequest where they are updated.

NAT Traversal

The term NAT traversal refers to the ability for client devices to address and communicate with listening devices behind a NAT. This turns out to be an incredibly useful thing to do for games, peer-to-peer, and a variety of other applications. A subsequent article will go into full detail about what is available, but here’s a quick preview of one aspect of this feature.

If you’re using TcpListener or UdpClient, just pass into the constructor IPAddress.IPv6Any, then call AllowNatTraversal with a value of true to support NAT traversal in your application.

var listener = new TcpListener(IPAddress.IPv6Any, 8000);
listener.AllowNatTraversal(true);
listener.Start();

Escaped Character Support in Uri

With REST becoming a more popular way of creating web services, and with the key role Uri plays in resource identification, we are providing a way to relax some of the canonicalization performed by our current http(s) scheme parser. If you have encountered issues with special characters such as / or \ being unescaped, you now have the following configuration options at your disposal.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section name="uri" type="System.Configuration.UriSection, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
</configSections>

<system.net>
    <settings>
      <httpListener unescapeRequestUrl="false" />
    </settings>
</system.net>
</configuration>

This will allow you to turn off this form of escaping in Uri and also in the RequestUrl received from HttpListener. The latter setting will also impact application hosted WCF services using HTTP bindings since those are built on HttpListener.

Bug Fixes

No release would be complete without bug fixes, and we’re addressing plenty of those! The NCL team has fixed over 150 bugs as of this milestone. Our focus during triage was on:

Stability (fix hangs, crashes and improve long-running/data center scenarios such as fix memory leaks)
Performance
RFC compliance with URI, FTP, HTTP, SMTP
Internationalization (better handling of Unicode characters, etc.) in URI, FTP, HTTP, SMTP
IPv6 connectivity
Customer reported bugs through our Connect, MSDN Forums, and other channels

Please try out the new release when available and let us know what you think!

~ NCL Team

Why does Silverlight have a restricted port range for Sockets?

2009-06-23T22:52:00Z

Silverlight restricts the ports of outgoing TCP socket connections to the range 4502 – 4534. Connecting to a different port requires the use of a server-side proxy or port redirector.

One of the most common questions we hear from customers about this is, “Why do you restrict the port range in Silverlight? It doesn’t add any extra security.”

Actually, it does. The short explanation is, it gives network administrators control over their infrastructure by providing a convenient way to distinguish and route Silverlight traffic. For the long answer, read on.

Desktop trust model

When you run an application on your desktop you typically:

Have intentionally downloaded and/or installed the application
Have intentionally executed the application

In the case of a managed corporate environment you:

Have been granted permission by an IT administrator to install and/or execute applications on your PC

In short, by being able to install and explicitly execute an application, you are asserting your trust of that application to not go rummaging around your file system or corporate network, for instance.

Web trust model

The web trust model is different. A web browser is a trusted desktop application, so per above, there are expectations it will not do anything malicious. Since web content can come from anywhere, there are no security guarantees about the intentions of the content provider.

Moreover, the explicitness of application install and execution is not present by design. That is, just by navigating to a website, a number of Silverlight applications could be started – an advertisement playing in the corner of the page, a hidden application with no UI, etc.

None of these Silverlight applications should be able to break out of the “sandbox” trust model without your knowledge, and nothing short of application signing, a domain trust model, prompting, etc. could establish that trust.

We’ve worked hard to keep the experience as unobtrusive as possible by generally avoiding prompting. But even when necessary, such trust models are fragile in nature because there is such a tendency to just click OK when you’re on a trusted site, even if the content came from elsewhere.

User vs. IT admin security decision

The other consideration is whether the decision to trust a website or Silverlight application rests with the web browser user or with the IT administrator. An insecure client on the network can be an entry point to other normally secured systems.

Here is one well-known FTP attack for why a trust decision like this is necessary and why it needs to rest with the IT admin.

The FTP protocol has a PORT command which is typically used to establish connections using “active” FTP. It can also be used to initiate server-to-server transfers.

In a malicious case, the command can be exploited to perform port scanning, or in the case of some active packet filtering devices, to actually open ports in the firewall.

With a desktop FTP client application that the user or IT admin has installed, the behavior of that client is trusted to be benign and these commands are issued at the request of the user. The active packet filter is doing what it was configured to do by opening the necessary ports to allow the connection.

Now imagine if Silverlight were allowed to send those same commands. You visit a website, a hidden application sets up a TCP connection back to its server of origin, and then it sends the PORT command which promptly opens a hole in your firewall. The website then uses this open port to establish a connection back to a victim machine on the internal network. The user never intended this action and therefore the trust model is broken. Moreover, the entire network has been placed at risk since the connection could be to a different computer than the user’s who indicated trust of the application.

Now, such attacks can typically be mitigated through additional configuration and patching, but this class of attack tends to re-surface with various protocols because of the liberties active filters take.

Similar exploits exist for HTTP. You’ll notice that Silverlight, Flash, XmlHttpRequest, etc. block a number of request headers. Now imagine if we allowed TCP connections over port 80 and a malicious application could craft their own HTTP request, effectively bypassing our HTTP implementation which has these checks.

For a detailed look at one such threat, please see this article about CERT’s VU#435052. Security researcher Dan Kaminsky has published a presentation and paper on the subject.

In a corporate environment, IT administrators need to be able to secure their networks against such attacks, so there must be a way for them to retain control of these security decisions and to also be able to distinguish Silverlight traffic from trusted application traffic that might be using similar protocols.

Port ranges and transparent proxy abuse

So how do port ranges help? Well, it’s unlikely your active packet filter scanning FTP port 21 is going to notice if someone attempts to send a maliciously crafted command over port 4502. Moreover, with a clearly identified port range of Silverlight-only traffic, it’s easy to configure such filtering devices to handle that traffic differently and with an appropriate level of trust.

Conclusion

For the time being, we’ve settled on the port range restrictions as the best compromise to maintain security and protect our customers. We understand this makes interoperability and connectivity challenging for some deployments, but we are planning to address that feedback with future work in this area.

I hope this helps to clear up some of the questions out there about this, and if you have more, please let us know. Thanks!

Custom HTTP Authentication Schemes

2009-05-05T21:06:00Z

Introduction

My name is Chris Ross and I work as a developer for Microsoft’s .NET Framework networking components. As part of the Network Class Library (NCL) team I get lots of networking questions from other developers. This post resulted from my research into a question about using custom HTTP authentication schemes. The custom scheme in question was Google’s GoogleLogin scheme, so I’ll use it as an example to show how System.Net.WebRequest and System.Net.WebClient can work with custom schemes. Fist though, I should explain how standard HTTP authentication works.

Standard HTTP Authentication

It has become increasingly important to secure online information from unauthorized users. But once secured, how do clients figure out how to login? Well, the HTTP protocol utilizes a back and forth negotiation pattern to let the client know that something is secured and what type of credentials they must submit to gain access.

A common HTTP authentication pattern goes as follows:

WebRequest.Credentials = new NetworkCredential(username, password);
WebRequest sent (without Credentials)	To	Secured web page
Error 401 – List of login schemes supported (Basic, Digest…)	From	Secured web page
WebRequest +Credentials (formatted according to the selected login scheme) sent	To	Secured web page
200 – Success – Here is your secured content	From	Secured web page

The best part is that .NET handles this back and forth for you under the hood, so all you end up seeing is the content you asked for. The .NET Framework (WebClient & WebRequest) has built in support for Basic, Digest, NTLM, Kerberos, and Negotiate authentication schemes.

Here is an example on how you would use standard authentication schemes with WebRequest:

public static void NormalHTTPAuthExample(String username, String password)

{

NetworkCredential creds = new NetworkCredential(username, password);

WebRequest request = WebRequest.Create("http://www.contoso.com/");

request.Credentials = creds;

// Send the request and process the response

WebResponse response = request.GetResponse();

StreamReader responseStreamReader =

new StreamReader(response.GetResponseStream());

String result = responseStreamReader.ReadToEnd();

responseStreamReader.Close();

Console.WriteLine(result);

}

WebRequest Authentication Practices

WebRequest takes great care with the credentials you provided. Many of these precautions are to help prevent your username and password from being exposed to unauthorized servers. Two features related to our example are CredentialCache and PreAuthenticate.

WebRequest by default facilitates automatic request redirection from site to site. However if a server you have logged into decides to redirect you elsewhere, WebRequest will remove its Credentials property before automatically redirecting to prevent your username and password from being exposed. If you are aware that redirects will take place, you can use the CredentialCache class to manage what servers are allowed access to your credentials. A CredentialCache pairs credentials with the URIs and authentication schemes they’re allowed to be used with. When you have created and configured a CredentialCache object, you then assign it to the WebRequest.Credentials property. You can see this in the next code sample. A CredentialCache will not be removed from the Credentials property when redirecting because WebRequest knows where you will allow your credentials sent. You may also reuse your cache by assigning it to subsequent requests. All of this also applies to WebClient.Credentials, as it uses WebRequest for its underlying operations.

A second WebRequest property that can be useful with HTTP authentication is PreAuthenticate. Without it, WebRequest will not include your credentials on a request until it has first received a challenge from the server. Setting this property is optional but will help boost performance on secured sites as follows. After you’ve successfully authenticated once, this flag causes your credentials to be included in the Authorization header automatically on subsequent requests and redirects that match the same URI path at the folder level. This way the server does not have to issue a challenge for authentication on each page. WebClient does not expose this property.

Here is a code sample that shows both of these properties:

public static void RedirectHTTPAuthExample(String username, String password)

{

NetworkCredential creds = new NetworkCredential(username, password);

CredentialCache credCache = new CredentialCache();

// Cached credentials can only be used when requested by

// specific URLs and authorization schemes

credCache.Add(new Uri("http://www.contoso.com/"), "Basic", creds);

WebRequest request = WebRequest.Create("http://www.contoso.com/");

// Must be a cache, basic credentials are cleared on redirect

request.Credentials = credCache;

request.PreAuthenticate = true; // More efficient, but optional

WebResponse response = request.GetResponse();

StreamReader responseStreamReader =

new StreamReader(response.GetResponseStream());

String result = responseStreamReader.ReadToEnd();

responseStreamReader.Close();

Console.WriteLine(result);

}

Custom HTTP Authentication Schemes

What if the sever you’re requesting information from does not utilize one of the standard authentication schemes? Take for an example Google’s custom GoogleLogin authentication scheme. I’m no expert on Google’s APIs, but its differences from standard HTTP authentication schemes make for a good example. GoogleLogin requires first making a request to a completely different login server for an Auth token, then adding that token to every subsequent request’s Authorization header. You can still do this with .NET, but because GoogleLogin is a custom scheme you might have to handle some of the back and forth yourself, such as turning off automatic redirects so you can be sure to include the Auth token on each request.

For example, you would first make a request to the login server for your user token. Then you’d add that token to a second request to the Calendar server. The Calendar server responds with a redirect to your specific calendar at a session specific url. You then issue one final request (again including the Auth token) to that session specific url for your calendar data.

WebRequest (with user name and password) sent	To	Login server
Auth token	From	Login server
WebRequest + Auth token	To	Secured Calendar server
302 – Redirect to session specific URL	From	Secured Calendar server
WebRequest + Auth token sent to new URL	To	Your Secured Calendar
200 – Success – Here is your secured content	From	Your Secured Calendar

While this approach can work well enough to use once, I find it unwieldy to make three different requests where only one would normally be required, especially if you needed to do it often.

Luckily .NET provides a better answer to this sort of problem: Custom Authentication Modules. You can add your own modules to the list of supported authentication schemes for WebClient & WebRequest, and let .NET juggle the authentication, redirection, proxies, and other complications for you. Below is an example of how to do this for Google’s custom GoogleLogin scheme.

How to use a custom module with WebRequest/WebClient

How do we use WebRequest and a custom authentication module to log in to Google services? Here is a sample:

public static void GoogleAuthManagerExample(String username, String password)

{

AuthenticationManager.Register(new GoogleLoginClient());

// Now 'GoogleLogin' is a recognized authentication scheme

GoogleCredentials creds = // user@gmail.com

new GoogleCredentials(username, password, "HOSTED_OR_GOOGLE");

CredentialCache credCache = new CredentialCache();

// Cached credentials can only be used when requested by

// specific URLs and authorization schemes

credCache.Add(new Uri(

"http://www.google.com/calendar/feeds/default/private/"),

"GoogleLogin", creds);

WebRequest request = WebRequest.Create(

"http://www.google.com/calendar/feeds/default/private/full?q=null+item");

// Must be a cache, basic credentials are cleared on redirect

request.Credentials = credCache;

request.PreAuthenticate = true; // More efficient, but optional

WebResponse response = request.GetResponse();

StreamReader responseStreamReader =

new StreamReader(response.GetResponseStream());

String result = responseStreamReader.ReadToEnd();

responseStreamReader.Close();

Console.WriteLine(result);

// Erase cached auth token unless you'll use it again soon.

creds.PrevAuth = null;

}

The only meaningful difference between this code block and the previous authentication example I showed is the registration of the GoogleLoginClient. The GoogleLoginClient is a custom authentication module that WebRequest can refer to for handling the special symantics of Google’s API. As I described before, the alternative is to include much of the folowing Google API code inline, as well as managing all of your own redirection. I definitely prefer the custom modules, so lets look how to implement this custom authentication module.

How to write a custom authentication module for GoogleLogin

The first minor element is a storage place for Google’s Auth token and login parameters. Caching the token is ok so long as you’re careful where you put it and you know when you need to clear it.

public class GoogleCredentials : NetworkCredential {

private Authorization prevAuth = null;

public Authorization PrevAuth { // Cached login token

get { return prevAuth; }

set { prevAuth = value; }

}

private String accountType;

public String AccountType {

get { return accountType; }

// Validate "GOOGLE","HOSTED", or "HOSTED_OR_GOOGLE"

set { accountType = value; }

}

public GoogleCredentials(String user, String pswd,

String accountType)

: base(user, pswd) {

this.AccountType = accountType;

}

Then we need to implement the IAuthenticationModule interface. When we are challenged for GoogleLogin authorization this will do the actual request to Google’s login server and fetch us an Auth token.

public class GoogleLoginClient : IAuthenticationModule {

internal const string AuthType = "GoogleLogin"; // Scheme identifier

internal static string AuthServer

= "https://www.google.com/accounts/ClientLogin";

public String ServiceString = "cl"; // Calendar

public String Source = "MSTest"; // Our program name

public Authorization Authenticate(string challenge,

WebRequest webRequest, ICredentials credentials) {

// Careful, if your challenge contains more than one

// authorization scheme, this one might not be first

// in the list. Also ignore parameter names and quoted

// parameter values. See RFC 2617 Section 1.2

// ie: Basic, Digest nonce=122352354,

// realm="www.GoogleLoginDirections.com/help",

// GoogleLogin realm="http://login.google.com/",Ntlm

if (!challenge.Contains(AuthType)

/* && ContainsNotInQuotes(challenege,AuthType)

* && MoreValidation(challenege,AuthType) */) {

return null;

}

return Login(webRequest, credentials);

}

public Authorization PreAuthenticate(WebRequest webRequest,

ICredentials credentials) {

return Login(webRequest, credentials);

}

public bool CanPreAuthenticate {

// Some schemes do not support PreAuthentication

get { return true; }

}

public string AuthenticationType {

get { return AuthType; }

}

private Authorization Login(WebRequest webRequest,

ICredentials credentials) {

// Do we have credentials for this site?

NetworkCredential NC = credentials.GetCredential(

webRequest.RequestUri, AuthType);

GoogleCredentials gcreds = NC as GoogleCredentials;

if (gcreds == null)

return null; // none or wrong type of credentials

if (gcreds.PrevAuth != null)

return gcreds.PrevAuth; // Cached from last login

ICredentialPolicy policy =

AuthenticationManager.CredentialPolicy;

if (policy != null && !policy.ShouldSendCredential(

webRequest.RequestUri, webRequest, NC, this))

return null;

WebRequest client = WebRequest.Create(AuthServer);

client.ContentType = "application/x-www-form-urlencoded";

client.Method = "POST";

// Custom authentication string:

// http://code.google.com/apis/accounts/docs/AuthForInstalledApps.html

String requestParams = "accountType=" + gcreds.AccountType

+ "&Email=" + gcreds.UserName + "&Passwd=" + gcreds.Password

+ "&service=" + ServiceString + "&source=" + Source;

byte[] bytes = Encoding.UTF8.GetBytes(requestParams);

// Google's API says that the custom authentication string

// goes in the body of this request. This is unusual.

Stream requestStream = client.GetRequestStream();

requestStream.Write(bytes, 0, bytes.Length);

requestStream.Close();

// The Auth token comes in the response body. Also unusual.

WebResponse response = client.GetResponse();

StreamReader responseStreamReader =

new StreamReader(response.GetResponseStream());

String result = responseStreamReader.ReadToEnd();

responseStreamReader.Close();

String authToken = ""; // Parse out the Auth token

String[] tokens = result.Split(new String[] { "\n" },

StringSplitOptions.None);

foreach (String token in tokens) {

if (token.StartsWith("Auth=",

StringComparison.OrdinalIgnoreCase)) {

authToken = token;

break;

}

if (authToken == "")

throw new WebException("GoogleLogin authentication failed");

// Assemble the Authorization header and cache it

gcreds.PrevAuth = new Authorization(AuthType + " " + authToken);

return gcreds.PrevAuth;

}

And you’re done. Now any time your application is challenged for GoogleLogin authentication, WebRequest can just refer to this new module automatically.

Conclusions

HTTP authentication can take many forms, and .NET includes support for several standard schemes. When these are not sufficient, it is easy to add custom schemes for many other services with minimal changes to your existing code.

Notes & references

- WebClient: http://msdn.microsoft.com/en-us/library/system.net.webclient.aspx

- WebRequest: http://msdn.microsoft.com/en-us/library/system.net.webrequest.aspx

- IAthenticationModule: http://msdn.microsoft.com/en-us/library/system.net.iauthenticationmodule.aspx

- Register a new custom module in the App.Config file without modifying existing code: http://msdn.microsoft.com/en-us/library/y9b82x09.aspx

- CredentialCache: http://msdn.microsoft.com/en-us/library/system.net.credentialcache.aspx

- GoogleLogin API: http://code.google.com/apis/accounts/docs/AuthForInstalledApps.html

- RFC 2617 – HTTP Basic & Digest authentication - http://www.ietf.org/rfc/rfc2617.txt

- RFC 2616 – HTTP 1.1 - http://www.ietf.org/rfc/rfc2616.txt

- VB example: http://support.microsoft.com/default.aspx/kb/331501

~Chris Ross

System.Net

Network Class Library Team (System.Net)

New Picture Hunt Silverlight Socket Sample

Silverlight 4 Socket Policy Changes

System.Uri FAQ

1. System.Uri vs. Strings

2. Standards and History

3. URI Components

4. Canonicalization

5. Uri Manipulation

6. Supported Schemes

7. UriFormatExceptions & Validating User Input

8. MailTo Multiple E-mail Addresses

9. Implicit Dos/Unc File Paths

10. Unicode, IRI, and IDN

Conclusion

UDP Multicast in Silverlight 4

Scenario Overview

User Datagram Protocol (UDP)

Multicast Concepts

New APIs

Browser Security

Conclusion

New Performance Counters for HttpWebRequest

What's new in System.Net.Mail

End-to-end connectivity with NAT traversal

Home Routers with NAT

NAT Traversal & Security Considerations

Impact to Applications

IPv6 Restores Addressability

6to4 Tunneling

Teredo Tunneling

New for .NET 4.0

Listeners

Sockets

Configuration

Address Discovery

Windows Communication Foundation

Firewall Considerations

Parting Thoughts

Additional References

New NCL Features in .NET 4.0 Beta 2

Sockets

DnsEndPoint

IP Version Neutrality

SSL Encryption Policy

SmtpClient

SSL Configuration

Header Encoding

Multiple ReplyTo Addresses

HttpWebRequest

64-bit Range Header Support

Date Header Support

Host Header Support

WebRequest Performance Counters

NAT Traversal

Escaped Character Support in Uri

Bug Fixes

Why does Silverlight have a restricted port range for Sockets?

Desktop trust model

Web trust model

User vs. IT admin security decision

Port ranges and transparent proxy abuse

Other solutions

Conclusion

Custom HTTP Authentication Schemes

Introduction

Standard HTTP Authentication

WebRequest Authentication Practices

Custom HTTP Authentication Schemes

How to use a custom module with WebRequest/WebClient

How to write a custom authentication module for GoogleLogin

Conclusions

Notes & references