Feroze Daud's WebLog : System.Net

Enabling Tracing for System.Net

Feroze Daud — Thu, 12 May 2005 22:09:00 GMT

In whidbey, System.Net has a cool retail tracing implementation. It writes most calls made to public API's to a TraceListener. You can setup your own tracelisteners, or use the config file to setup a default listener.

This facility is very useful in debugging problems with applications.

Here is the example config file which enables Verbose tracing.

<?xml version="1.0" encoding="UTF-8" ?>

<configuration>
<system.diagnostics>
<trace autoflush="true" />

<sources>
<source name="System.Net">

  <listeners>
   <add name="System.Net"/>
  </listeners>
</source>
<source name="System.Net.Sockets">
  <listeners>
   <add name="System.Net"/>
  </listeners>
</source>
<source name="System.Net.Cache">
  <listeners>
   <add name="System.Net"/>
  </listeners>
</source>
</sources>
        <sharedListeners>
                <add
                  name="System.Net"
                  type="System.Diagnostics.TextWriterTraceListener"
                  initializeData="System.Net.trace.log"
                />
        </sharedListeners>
<switches>
                <add name="System.Net" value="Verbose" />
  <add name="System.Net.Sockets" value="Verbose" />
  <add name="System.Net.Cache" value="Verbose" />
</switches>
</system.diagnostics>
</configuration>

NOTE: If you are doing logging inside of the ASP.NET process, make sure to give the ASP.NET process identity WRITE permissions to the directory where you want the log to be written.

HttpWebRequest and Host Headers

Feroze Daud — Thu, 31 Mar 2005 23:46:00 GMT

A user posted on the newsgroups asking about how to set Host headers in HttpWebRequest.

HttpWebRequest does not allow the host header to be set by the user. It is set automatically from the URI supplied to HttpWebRequest constructor.

However, it turns out that there is a roundabout way to accomplish this. This is shown in the following reply to the original post:

http://groups-beta.google.com/group/microsoft.public.dotnet.framework/browse_thread/thread/17c5369520221c8/4e768c733ad2f451?q=HttpWebRequest+and+Host+Header&rnum=1#4e768c733ad2f451

This solution works, but there is a subtle difference in the request sent on the wire. To illustrate that, look at the following program:

using System;
using System.Net;

public class EP {
public static void Main()
{
  test("http://www.company.com/test.aspx");
  test("http://www.contoso.com/test.aspx");

}

public static void test(string uri)
{
WebRequest req = WebRequest.Create(uri);
req.Proxy = new WebProxy("http://myserver", false);

WebResponse resp = req.GetResponse();

resp.Close();
}
}

To get this program to work, you need to do the following:

Have a server named "myserver" on which you get IIS/ASP.NET running. If you dont want to run ASP.NET that is fine as well.
On the server, go into the IIS console, and create two websites. Set the host header on the first website to be www.company.com and the other to be www.contoso.com
Compile and run the above program.

If you look at the request sent on the wire, you will notice that this program sends the following request to the server "myserver"

GET http://www.contoso.com/test.aspx HTTP/1.1
Host: http://www.contoso.com

Whereas the actual request you want sent is:

GET /test.aspx HTTP/1.1
Host: www.contoso.com

NOTE: Even though the request URI sent on the wire is an absolute URI, and is different from what should actually be sent if the client had control of the host header, it will still work on the server, because as per RFC/2616, a server is supposed to honor all requests which use an absolute URI instead of a relative URI to identify the resource.

Forcing reuse of connections when doing Windows Authentication

Feroze Daud — Tue, 23 Nov 2004 03:40:00 GMT

When doing authentication, HttpWebRequest manages connections differently, depending on the authentication mode being used. If it uses Windows Integrated Authentication, it will close the connection after every request completes. The reasons for this are complex, and I wont go into them at this point.

However, there are cases where you dont want this to happen. For example, imagine that you have a 3-tier architecture, where the client is talking to an asp.net application, which in turn is hitting a backend webserver. The asp.net webapplication (henceforth called the MiddleTier) is using HttpWebRequest to talk to the back-end, and the back-end is secured by Windows Integrated Authentication.

If the middle-tier web application ends up getting a lot of requests, it will inturn issue a lot of requests to the back-end. The default behavior of webrequest is to create a new connection for each request, and over time, the middle-tier might end up running out of wildcard TCP ports, causing the HttpWebRequest to fail with a "Unable to connect to the remote server" exception.

To mitigate this, you want to use a property on the webrequest called UnsafeAuthenticatedConnectionSharing . Setting this property will cause HttpWebRequest to reuse authenticated connections (making sure that it honors ServicePoint.ConnectionLimit).

Security Note: You dont want to use this property lightly. It has security consequences.

Async Programming Model - Reloaded

Feroze Daud — Fri, 08 Oct 2004 01:25:00 GMT

This topic is about the asynchronous programming model, and how it is used when doing Networking.

Introduction to Asynchronous Programming

The DotNetFramework supports an asynchronous programming model that is uniform across all classes & features in the framework. What is cool about this model is that there is a uniform way of doing asynchronous operations across all the classes in the Framework.

The following links give an overview of Asynchronous programming in the .NETFramework.

http://msdn.microsoft.com/library/en-us/cpguide/html/cpovrasynchronousprogrammingoverview.asp?frame=true

http://msdn.microsoft.com/library/en-us/cpguide/html/cpconasynchronousfileio.asp?frame=true

Async programming in HttpWebRequest/HttpWebResponse

Let us proceed on to asynchronous programming in System.Net.HttpWebRequest class. The following code snippet demonstrates how to download data from a webserver using asynchronous programming pattern.

class Ep {

static ManualResetEvent doneevent = New ManualResetEvent(False);

static Byte [] Buffer = New Byte[1024];

static MemoryStream DownloadedBytes = New MemoryStream();

static void Main() {

HttpWebRequest Req = (HttpWebRequest)WebRequest.Create("http://www.Contoso.Com/");

Req.BeginGetResponse(new AsyncCallback(Respcallback), Req);

doneevent.Waitone();

}

static void Respcallback(Iasyncresult Ar) {

HttpWebRequest Req = (HttpWebRequest) Ar.AsyncState;

HttpWebResponse Resp = Null;

try {

Resp = Req.EndGetResponse(Ar);

Stream Respstream = Resp.GetResponseStream();

Respstream.BeginRead(Buffer, 0, Buffer.Length, new AsyncCallback(Readcallback), Respstream);

} catch(Exception E) {

Console.Writeline(E);

RespStream.Close();

doneevent.Set();

}

static void ReadCallback(IasyncResult Ar) {

Stream Rs = (Stream)Ar.AsyncState;

int Read = -1;

try {

Read = Rs.EndRead(Ar);

} catch(Exception E) {

Console.WriteLine(E);

rs.Close();

doneevent.Set();

return;

}

if(Read > 0) {

// Write To Memorystream

DownloadedBytes.Write(Buffer,0,Read);

try {

rs.BeginRead(Buffer,0,Buffer.Length, new AsyncCallback(Readcallback), Rs);

} catch(Exception E) {

Console.WriteLine(E);

doneevent.Set();

}

} else

if( Read == 0) {

// No More Data To Read.

// Close The Response Stream To Free Up The Connection

rs.Close();

doneevent.Set();

}

The above code is the standard asynchronous pattern as applied to the HttpWebRequest/Response classes.

However this code could be improved. The most important thing to note here is that we are always calling an asynchronous Read method on the Response Stream returned by the response object. Sometimes, this might be overkill, because the data that we are trying to read is already present in the buffers of the networking stack on the machine. When data is already present to be read, the Read operation will complete very quickly. So, we can avoid some overhead with issuing asynchronous calls if we are smarter about when we issue them. If we know that data is already present to be read, we do a synchronous Read, otherwise we do an asynchronous Read.

Well, you might ask: How do I know that there is already data available to be read? The answer is to look at the members of the IAsyncResult structure which is passed to the callback delegate. This interface contains a property called “CompletedSynchronously”, which indicated if the async operation completed synchronously or not. We can utilize this property to decide if we should do a synchronous Read or an asynchronous Read.

Here is the modified code for the ReadCallback, with this change.

static void ReadCallback(IAsyncResult Ar) {

Stream Rs = (Stream)Ar.AsyncState;

int Read = -1;

try {

Read = Rs.EndRead(Ar);

} catch(Exception E) {

Console.WriteLine(E);

rs.Close();

doneevent.Set();

return;

}

if(Read > 0) {

// Write To Memorystream

DownloadedBytes.Write(Buffer,0,Read);

try {

if(rs.CompletedSynchronously) {

Read = rs.Read(Buffer,0,Buffer.Length);

while(Read > 0) {

DownloadedBytes.Write(Buffer,0,Read);

Read = rs.Read(Buffer,0,Buffer.Length);

}

rs.Close();

} else {

rs.BeginRead(Buffer,0,Buffer.Length, new AsyncCallback(ReadCallback), rs);

}

} catch(Exception E) {

Console.WriteLine(E);

rs.Close();

doneevent.Set();

}

} else

if( Read == 0) {

// No More Data To Read.

// Close The Response Stream To Free Up The Connection

rs.Close();

doneevent.Set();

}

With this minor rewrite, we have now accounted for the Read completing synchronously. By doing this, we are preventing the underlying framework from taking an extra thread ( it needs to take an extra thread because the thread on which the async callback is scheduled to be called is different from the thread on which the BeginRead was issued).

HttpWebRequest and PreAuthentication

Feroze Daud — Mon, 12 Jul 2004 17:52:00 GMT

The topic of this post is PreAuthentication.

HTTP Webservers support multiple authentication options. On Windows, IIS supports Basic, NTLM, Digest, Negotiate (a.k.a Integrated Windows Authentication). Of these, Basic & Kerberos support PreAuthentication.

"PreAuthentication" is sometimes misunderstood to mean "Send credentials on the very first request". That is not the way it works. For Preauthenticate to work, the following things have to happen:

1) You make a webrequest to a server. You might (or might not) have specified credentials on the first request.

2) The server want to authenticate the request you are making. So it replies with a 401.

3) The client does the authentication handshake with the server, and sends your credential.

4) If "Basic" was chosen as the authentication scheme by the client, then Pre-Authentication is enabled for this request. So, every subsequent request from now on to the same Uri-Prefix will result in the credentials being sent preemptively.

The main thing to note here is: It requires atleast one request before the client can do pre-authentication. Credentials are not automatically sent on the first request if PreAuthenticate is true.

If you want to force credentials to be sent on the first request (and thus avoid the first auth handshake), you can set credentials manually on the request:

WebRequest req = WebRequest.Create("http://....");

string cre = String.Format("{0}:{1}", username, password);

byte [] bytes = Encoding.ASCII.GetBytes(cre);

string base64 = Convert.ToBase64String(bytes);

req.Headers.Add("Authorization", "basic " + base64);

WebResponse resp = req.GetResponse();

This will set the authorization header on the request, and send it on the first request itself. If the credentials are correct, the first handshake will not happen.

However, remember that you have to use this code to set the authorization header on every request. This is because we bypassed the authentication handshake, and HttpWebRequest has no knowledge that the server requires authentication. So it wont do pre-authentication by itself anymore.

The recommended way to get preauthentication is to set the credentials on the request, set PreAuthenticate=true, and send the request. Once the first authentication handshake happens, and credentials are accepted by the server, preauthentication will happen behind the scenes for every request to the same Uri-Prefix.

Downloading content from the web using different encodings

Feroze Daud — Wed, 31 Mar 2004 02:30:00 GMT

The other day, somebody asked me: How do I download a webpage, or other content from a webserver, where the content is stored using a specific encoding ? They want to do this using for eg: System.Net.HttpWebRequest

Why is this necessary ?

Well, for starters, webservers around the world store their content in various encodings. For eg, webadmins in Japan server their pages using the Shift-JIS encoding to account for the japanese characters in their pages.

If you just attach a StreamReader to the stream given by HttpWebResponse.GetResponseStream(), then you will most likely get bad characters in your data. Or, your stream might be truncated in the middle. This is because StreamReader uses a default encoding (UTF8) which might not match the encoding of the bytes you are reading into the StreamReader.

So, lets get down to coding.

There are two places where a server can indicate the encoding of the entity in the response. The first is the response header. The second is the entity body itself, if the entity is an HTML page (this is indicated by “content-type: text/html“ response header).

The response headers you need to look at are:

“Content-Type: foo/bar; charset=<charset encoding>“

If the Content-Type header exists, and the value for this header contains a charset=<value>, then the <value> portion gives the encoding of the response entity.

If this header is not present, or if a “charset=” token is not present in the header value, then you need to look at the header of the HTML page (if the entity contains HTML). There will be some meta tags in the begining of the entity which indicate the charset of the entity:

What you need to do is to read the entity as ASCII into a string. Then, you extract the encoding information from the header of the entity. Once you know the encoding info, you can reprocess the raw entity using the correct encoding. of course, you should make sure to store the raw entity in a MemoryStream or other buffer, so that you can use it when you want to read the entity using its actual encoding.

Here is the code which demonstrates this:

private static String DecodeData(WebResponse w) {

      //
      // first see if content length header has charset = calue
      //
      String charset = null;
                  String ctype = w.Headers["content-type"];
      if(ctype != null) {
          int ind = ctype.IndexOf("charset=");
          if(ind != -1) {
              charset = ctype.Substring(ind + 8);
              Console.WriteLine("CT: charset=" + charset);
          }
      }

                // save data to a memorystream
                MemoryStream rawdata = new MemoryStream();
                byte [] buffer = new byte[1024];
                Stream rs = w.GetResponseStream();
                int read = rs.Read(buffer,0,buffer.Length);
                while(read > 0) {
                    rawdata.Write(buffer,0,read);
                    read = rs.Read(buffer,0,buffer.Length);
                }

rs.Close();

      //
      // if ContentType is null, or did not contain charset, we search in body
      //
      if(charset == null) {
          MemoryStream ms = rawdata;
          ms.Seek(0,SeekOrigin.Begin);

          StreamReader srr = new StreamReader(ms,Encoding.ASCII);
          String meta = srr.ReadToEnd();

          if(meta != null) {
                    int start_ind = meta.IndexOf("charset=");
                    int end_ind = -1;
                    if(start_ind != -1) {
                     end_ind = meta.IndexOf("\"", start_ind);
                     if(end_ind != -1) {
                         int start = start_ind + 8;
                         charset = meta.Substring(start, end_ind - start + 1);
                         charset = charset.TrimEnd(new Char[] { '>','"' });
                      Console.WriteLine("META: charset=" + charset);
                     }
                 }
          }
      }

      Encoding e = null;
      if(charset == null) {
          e = Encoding.ASCII; //default encoding
      } else {
          try {
              e = Encoding.GetEncoding(charset);
          } catch(Exception ee) {
              Console.WriteLine("Exception: GetEncoding: " + charset);
              Console.WriteLine(ee.ToString());
              e = Encoding.ASCII;
          }
      }

rawdata.Seek(0,SeekOrigin.Begin);

StreamReader sr = new StreamReader(rawdata, e);

String s = sr.ReadToEnd();

return s.ToLower();
}

A tale of threads

Feroze Daud — Mon, 02 Feb 2004 03:20:00 GMT

Today's lesson is about thread interaction between Asp.Net and the HttpWebRequest object of the System.Net namespace.

The CLR provides a threadpool. This facility provides threads for normal work ( worker threads ), I/O work ( I/O threads) and Timer Threads. Normally, the limit of threads for managed processes is set as follows:

WorkerThreads: 25 per CPU

I/O Threads: 1000

Asp.Net uses worker threads to dispatch incoming requests to page handlers (ASPX/ASMX etc). If you write an ASPX page, your request will run on a worker thread.

It turns out that System.Net also uses the threadpool. Worker Threads are used to kick of connect's to servers, and I/O threads are used to service I/O completions for Sockets.

Normally, people create middle tier applications by creating a WebRequest inside of their Page_Load(), and then doing a GetResponse(). Normally, a synchronous function (like GetResponse()) would complete on the same thread on which it was invoked. However, GetResponse() is actually implemented internally as follows:

void GetResponse() {

IAsyncResult ar = BeginGetResponse(null,null);

ar.WaitOne();

return EndGetResponse(ar);

}

It so happens that while you are waiting for the operation to complete, another thread is being spawned iternally to service the request, by kicking off the connect to the server, parsing the response etc.

If you have a page which is under high load, then you can run into potential deadlock issues. The user's code is running on a worker thread. When the user does a GetResponse(), another threadpool worker thread is used internally to kick off the operation. Imagine that there are 25 threadpool worker threads which are executing asp.net code (eg: page handlers etc). One of these threads is executing a page which is trying to do the webrequest.GetResponse().

System.Net will do a ThreadPool.QueueUserWorkItem() to complete the GetResponse(). However, since the CLR threadpool is exhausted ( uniprocessor limit is 25), CLR will not create a new thread to service the request. If no thread becomes available, then the GetResponse() will never complete. At the same time, the worker thread that asp.net is running the user code on, will not complete either. Now multiply this situation across all threads, and you have a classic deadlock.

This problem is mitigated somewhat in the V1 & V1.1 versions of the framework. There, System.Net will query the threadpool for available threads. If it determines that there are no free threads available to service the request, it will throw an exception, so that the users code can handle it.

Note that this problem can also happen in standalone applications.

So, the next time you get an exception from GetResponse() or GetRequestStream(), saying that “There are no available free threads to service this operation“ it is by design :-)

HttpWebRequest and Connections

Feroze Daud — Thu, 22 Jan 2004 04:32:00 GMT

The .NET Framework has a HttpWebRequest object in the System.Net namespace. This object allows you to talk to HTTP Servers.

Usually, people forget that the underlying connection created by the request is not freed up,unless you call Close() on the response. So, you hit the connection limit, and no more webrequests will go through. This is illustrated in the following example:

for(int i=0; i < 3; i++) {
HttpWebRequest r = WebRequest.Create(“http://www.microsoft.com“) as HttpWebRequest;
HttpWebResponse w = r.GetResponse() as HttpWebResponse;
}

The above code is buggy, because the third request is going to hang in GetResponse().

If you close the response, by calling Response.Close() , then the underlying connection gets freed up, and the request will succeed.