Http Client Protocol Issues : HttpWebRequest

How to get a dump for a System.Net.WebException using DebugDiag (Debug Diagnostics)

jpsanders — Tue, 29 Sep 2009 17:13:46 GMT

This information is available in the help file as well. For more options please refer to the Help Documentation include with DebugDiag.

Install the latest Debug Diagnostics from http://download.microsoft.com (search for Debug Diagnostics once on that site)

Configure a crash rule for all w3wp.exe processes (if this dialog does not appear simply press the ‘Add Rule…’ button).

Select ‘A specific process’

Make sure you DO NOT check ‘This process instance only’ and type in the process we wish to monitor (in this case asp.net is running in w3wp.exe).

In the ‘Advanced Configuration Dialog’ select the 'Exceptions...’ button at the bottom of this dialog:

Choose ‘Add Exception…’. Clicking this button will bring up the Configure Exception dialog. Use this to configure actions for a specific type of first chance exception.

In the Configure Exception Dialog, Choose 'CLR (.NET) Exception'

For the 'Net Exception Type' field: (This will only be available when choosing the E0434F4D CLR (.NET) Exception code) You must specify the exact name of the exception in this field. This field is case sensitive. In our case we want System.Net.WebException

In the ‘Action Type’ pull down specify ‘Full User Dump’ and you will be able to capture the state when we have the 401 (or other) WebException in the w3wp.exe process.

The ‘Action Limit’ by default is set to 1. This means only one dump will be created. If you wish to catch more than one dump, increase this number (usually 3 is sufficient). Then hit OK and proceed by hitting the ‘Save & Close’ button (note you will see the configured CLR exception now).

Finally hit the ‘Next >’ button twice and then ‘Finish’. The rule will be activated.

You do not need to keep the UI up for DebugDiag. If you do keep it up you will see the ‘Userdump Count’ increase with each exception you encounter (up to the limit you set previously) and the location of the dump is listed as well.

Finally to analyze the dump, you can use DebugDiag to analyze the script. You can use the special script I developed here to get a good picture of what the problem may be.

Please drop me a note and let me know if this Post helped you!

Using HttpWebRequest with Credential Manager

jpsanders — Thu, 03 Sep 2009 21:39:00 GMT

A little know fact is that the .NET framework will use the stored credentials in the Credential Manager when accessing a network resource if the credentials exist for that particular resource (host). I intend to clear up how this functionality works for the HttpWebRequests (you could extend this to WebService calls as well).

In Internet Explorer, you can access a website that challenges you for credentials by filling in a username and password when the dialog is presented and asks you for authentication. If you check ‘Remember my password’ these credentials will be stored in protected storage (Credential Manager). You can see the stored credentials by running this command: Control Keymgr.dll. These credentials are stored in protected storage in each user’s My folder. You can store these credentials by using control Keymgr.dll, by using the interface available in Internet Explorer or by using C++/C and the Credentials API’s (Such as CredWrite). If you do not check ‘Remember my password’ then these credentials are not stored in Credential Manager and .NET will not be able to use these credentials.

The .NET framework can use the currently logged on user’s credentials to authenticate but if you have a web site that requires different credentials you can use store these credentials in Credential Manager and .NET will use these credentials instead of the currently logged on user credentials. To illustrate this I put together a real simple sample.

The target web server in my sample is jpsandershv2003. I only enabled Windows Integrated Authentication in IIS on that box. I then created a User ID JPSGuest on that IIS machine. Finally I wrote a real simple .NET console application to create an HttpWebRequest and get the response.

HttpWebRequest aReq;

HttpWebResponse aResp;

aReq = WebRequest.Create("http://jpsandershv2003/") as HttpWebRequest;

aReq.UseDefaultCredentials = true;

aResp = aReq.GetResponse() as HttpWebResponse;

aResp.Close();

When you run this code it will first look in the credential manager first for the credentials for attaching to jpsandershv2003 and if found use those, otherwise the Currently Logged on User credentials will be used. In my example since I was logged on as jpsanders to my machine I expect that the credentials used should be my domain credentials jpsanders. I turned on logging and ensured the IIS log was logging the username and sure enough, jpsanders was authenticated in IIS.

My next goal was to see if I could change this default behavior and make it log on using the jpsandershv2003\JPSGuest user credentials. Using the command control keymgr.dll I was able to add the credential as a Windows logon credential:

Now when I ran the console application I saw success in the IIS log! I was logging in now as JPSGuest.

Taking a fiddler (http://fiddlertool.com) trace of Internet Explorer also confirmed I was using the stored credentials and these were NTLM credentials.

Domain: jpsandershv2003
User: JPSGuest
Host: JPSANDERS3

Can I store credentials and have it use Kerberos? In theory yes, simply type domain based credentials in. However I could not test this because I only have one logon to my domain and you cannot save your currently logged on credentials in the Credential Manager.

What about basic and digest? NO, this would be a huge security hole. Can you tell me why (it should be obvious)?

Note that the credentials used are independent of the PORT used. So if I change the server to bind HTTP traffic on port 8089 then change my code to access http://jpsandershv2003:8089 the code will still access the web site using the stored credentials.

To revert back to the interactive user, simply delete the credentials stored for jpsandershv2003.

Further observations

What is the second radio button for (A Web site or program credentials)? This is for programs that are writing against the credentials API directly. Some examples of this are Terminal Server (RDP) and Windows Live credentials. Here are some great links about the APIs:
http://msdn.microsoft.com/en-us/library/aa374789(VS.85).aspx
http://msdn.microsoft.com/en-us/library/aa480470.aspx

NOTE: Because the credentials are stored in the context of the logged on user, this technique will not work for non interactive user accounts such as the service accounts and ASP.NET applications.

Please send me a quick note if you found this Blog helpful!

Web Service call results in Exception: A socket operation was attempted to an unreachable host

jpsanders — Wed, 03 Jun 2009 20:31:00 GMT

There can be many reasons for this error. In some cases this error can be confusing. I have seen a few instances where an ASP.Net webservice call resulted in this error. More perplexing is the fact that Internet Explorer on the same machine can reach the WebService fine. If a firewall was blocking the traffic, most likely the Internet Explorer request to the same port would also be blocked. Also to add to the confusion is when adding the usesystemdefault=true (default setting) did the proxy not get used. The reason that usesystemdefault did not get the proxy is that IE stores the hard coded proxy information in the registry of the logged on user. When ASP.Net runs, it is not running in the context of the currently logged on user (normally) and so it cannot read the same information that IE is reading to connect to the proxy.

Some other errors around this exeption include: WebException: Unable to connect to the remote server, SocketException (0x2751): A socket operation was attempted to an unreachable host, Socket operation encountered a dead network

Typical call stacks look something like this.
Exception Details: System.Net.Sockets.SocketException: A socket operation was attempted to an unreachable host
[SocketException (0x2751): A socket operation was attempted to an unreachable host <<ipaddr:port>>]
System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) +1073657
System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP) +33
System.Net.ServicePoint.ConnectSocketInternal

WebException: Unable to connect to the remote server
System.Net.HttpWebRequest.GetRequestStream() +1534317
System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke

Each time, these issues were related to proxy settings. IE was able to get to the WebService but .NET could not. The reason is, the Proxy information was manually configured for IE.

A simple change to the Web.Config fixes the issue:
You will need to change "http://192.168.1.10:3128" to be the address of your proxy server. You should be able to determine the proxy from your Internet Explorer Settings.
<configuration>
<system.net>
    <defaultProxy>
      <proxy
        proxyaddress="http://192.168.1.10:3128"
        bypassonlocal="true"
      />
    </defaultProxy>
</system.net>
</configuration>

Here is an article on how proxy detection works in the 2.0 framework:
http://msdn.microsoft.com/en-us/magazine/cc300743.aspx

You may also now need to provide credentials to get through the proxy. If the call does not go through and the exception indicates an HTTP 407 error, you can add the credentials to your config as well. Here are articles on the <system.net> schema and proxy setting: http://msdn.microsoft.com/en-us/library/kd3cf2ex(VS.80).aspx, http://msdn.microsoft.com/en-us/library/sa91de1e(VS.80).aspx

If the proxy settings are not the issue, ProcMon may help you identify any permissions related issues on the server. You can download this tool from http://www.microsoft.com/sysinternals

Leave me a message if you were able to solve a problem with this article!

Understanding MaxServicePointIdleTime and DefaultConnectionLimit

jpsanders — Wed, 20 May 2009 22:39:00 GMT

To understand these settings you need to understand how the HttpWebRequest class relates to the ServicePointManager and ServicePoint classes. When you make a request with the HttpWebRequest class, the ServicePointManager provides a ServicePoint object to the HttpWebRequest object to handle the connection to a web resource (web server). This ServicePoint object manages all the requests for a particular URI Host and protocol. For example: There will be one ServicePoint object for all the requests in my sample because they are all going to the server JSAN1 and using the HTTP protocol. The ServicePoint object has some settings to control how it services the requests. You can set the default values for these settings on the ServicePointManager before any ServicePoint objects are created. Once the ServicePoint object is created these default values will not affect existing ServicePoint objects, only newly created ones.

The ServicePoint model allows optimization, control and performance enhancements for the request. When you make an HTTP protocol request, ultimately that request and response will take place over a TCP Socket. Creating and initializing this TCP Socket is a very expensive operation in terms of computer resources and time so if you can re-use this connection for the next request, you have saved the time of creating and initializing a new socket. The HTTP 1.1 protocol provisions for this optimization by keeping the socket connection alive for subsequent requests by the client to the server.
Reference - How ServicePointManager works: http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.aspx (see remarks)
Reference - ServicePoint: http://msdn.microsoft.com/en-us/library/system.net.servicepoint.aspx

The ServicePoint object contains a queue of connections to the server identified by the URI of the server passed in. By default the number of simultaneous connections to a server is controlled by the ServicePoint.ConnectionLimit property. This property is set on this object when the ServicePointManager creates the ServicePoint object for your connections to a host. Connections to the server are queued and will be served by the connections to the server as they become freed from use. The default number of connections is set to 2. Ref: http://msdn.microsoft.com/en-us/library/system.net.servicepoint.connectionlimit.aspx

Obviously if the number of connections your application has requested to a resource is greater than the connection limit, requests are queued and may timeout before they are taken from the queue and processed on an active connection. An example may help you understand this. This example creates 64 Worker threads and executes HTTP requests on them. Create a simple HTTP page on a server in your network (not on the same machine you run this code on) and run this code, passing the URI to the page on the command line. You will see some timeout exceptions generated:

Complete code listing for sample (Copy Code):

using System;

using System.Net;

using System.Threading;

using System.Text;

using System.IO;

// ClientGetAsync issues the async request.

class ClientGetAsync

{

// This class is used in the worker thread to pass information in

public class threadStateInfo

{

public string strUrl; // the URL we will be making a request on

public ManualResetEvent evtReset; // used to signal this worker thread is finished

public int iReqNumber; // used to indicate which request this is

}

// this does the work of making the http request from a worker thread

static void ThreadProc(Object stateInfo)

{

threadStateInfo tsInfo = stateInfo as threadStateInfo;

try

{

HttpWebRequest aReq = WebRequest.Create(tsInfo.strUrl) as HttpWebRequest;

aReq.Timeout = 4000; // I want the request to only take 4 seconds, otherwise there is some problem

Console.WriteLine("Begin Request {0}", tsInfo.iReqNumber);

HttpWebResponse aResp = aReq.GetResponse() as HttpWebResponse;

System.Threading.Thread.Sleep(500); //simulate a half second delay for the server to process the request

aResp.Close(); // if you do not do this close, you will timeout for sure! The socket will not be freed.

Console.WriteLine("End Request {0}", tsInfo.iReqNumber);

}

catch (WebException theEx)

{

Console.WriteLine("Exception for Request {0}: {1}",tsInfo.iReqNumber,theEx.Message);

}

//signal the main thread this request is done

tsInfo.evtReset.Set();

}

public static void Main(string[] args)

{

if (args.Length < 1)

{

showusage();

return;

}

// The number of worker threads to make simultaneous requests (do not go over 64)

int numberRequests = 64;

// Used to keep the Main thread alive until all requests are done

ManualResetEvent[] manualEvents = new ManualResetEvent[numberRequests];

// Get the URI from the command line.

string httpSite = args[0];

// 2 is Default Connection Limit

ServicePointManager.DefaultConnectionLimit = 2;

// Get the current settings for the Thread Pool

int minWorker, minIOC;

ThreadPool.GetMinThreads(out minWorker, out minIOC);

// without setting the thread pool up, there was enough of a delay to cause timeouts!

ThreadPool.SetMinThreads(numberRequests,minIOC);

for (int i = 0; i < numberRequests; i++)

{

//Create a class to pass info to the thread proc

threadStateInfo theInfo = new threadStateInfo();

manualEvents[i] = new ManualResetEvent(false); // thread done event

theInfo.evtReset = manualEvents[i];

theInfo.iReqNumber = i; //just to track what request this is

theInfo.strUrl = httpSite; // the URL to open

ThreadPool.QueueUserWorkItem(ThreadProc, theInfo); //Let the thread pool do the work

}

// Wait until the ManualResetEvent is set so that the application

// does not exit until after the callback is called.

// can wait on a maximum of 64 handles here.

WaitHandle.WaitAll(manualEvents);

Console.WriteLine("done!");

}

public static void showusage() {

Console.WriteLine("Attempts to GET a URL");

Console.WriteLine("\r\nUsage:");

Console.WriteLine(" ClientGetAsync URL");

Console.WriteLine(" Example:");

Console.WriteLine(" ClientGetAsync http://www.contoso.com/");

}

Please read the comments in the code. There are some interesting points here. The first is that if you do not set the ThreadPool MinThreads property to the anticipated number of threads you will use, the delay of creating these threads on the fly is enough to fail all of the requests with a timeout exception. Next, note the Close() call on the response. No matter what we do to the code, if you do not do the Close(), those sockets will remain open and a new socket will be created for each request (you will see that in a network trace). This will also prevent any of the connections in the ServicePoint object from being freed, so no additional requests can be served. Another thing that is sort of cool about this code is that it simulates the ASP.NET environment as far as the thread pool idea goes. If your sample code works OK in this test app, chances are you will be able successfully run similar code in ASP.NET with the same ThreadPool and Connection settings. Reference Asp.Net tuning: http://msdn.microsoft.com/en-us/library/ms998583.aspx

Next play with this line of code:

// 2 is Default Connection Limit

ServicePointManager.DefaultConnectionLimit = 2;

Set it to 12 and see how the number of successful requests increases. Now set it to 4. You will see a different number of requests succeed and fail depending on the sleep value simulated in the request (System.Threading.Thread.Sleep(500);), the response time of your target server to provide the content, network traffic and the performance of your machine. Similar factors will affect your real world application. Also, I set the timeout for the web request to 4 seconds because I expect the result to come back very quickly. This may or may not be realistic in your situation. Only testing and careful design will allow you to determine what best works for your application.

The question often comes up “why can’t I set the DefaultConnectionLimit to a HUGE number”? You can set this number in the hundreds if you want. This will take more computer resources however. More threads need to be created, more sockets open and the target server will be taxed more. It is all a balance of resources and expected performance. These factors need to be considered during design and testing.

The value MaxServicePointIdleTime comes in to play when you consider performance as well. This value indicates how long a socket will be kept open after a request and response, before the client gracefully closes the socket down. Again, the design and performance of your system will determine what a proper value would be for you situation. If you close unused sockets down too fast, then the socket will never get reused by another request (remember I said the connection creation time is very expensive). If the value is too long, then sockets (read resources, threads cpu time and memory) may be used unnecessarily at the source and target computer. Finding the correct balance is important. In addition, if the MaxServicePointIdleTime is not less than the server Keep-Alive timeout, there is a chance that your code will start using a connection at the same time the server is sending a message to close the socket resulting in a WebException.

Example App Design

Let’s work through an example together. Let’s say this application is designed to handle 40 users every 4 seconds. Let’s also assume it will take .5 seconds to process a web request in your code. If this was a single threaded application, 40 users processed in 4 seconds means 4/40 or .1 seconds per user would have to be the performance of your application. Given that the request to the backend server is .5 seconds (normally) there is no way that your single threaded application can handle this load and meet the spec. The very best you could hope for is 20 seconds! So you decide to create a multi threaded solution based on my example and try to apply the principles in this article. Change the number of requests to 40 in the sample code to simulate all the requests coming in. If this were a single threaded app, then the best we could do is 4 seconds/.5 processing time = 8 requests. So in theory if we could do 6 simultaneous requests over the 4 second time period at 8 requests per thread, we would get abound 6*8 or 48 requests done in 4 seconds. So set the DefaultConnectionLimit to 6 and let’s test.

Well, my results are: about 10 of the 40 connections timed out. Why is that, and what in my calculation is wrong? First, the sleep value I have in my code to simulate the time it takes to get a request back from the server is added to the actual time it took for the request to come back, so the socket was not freed in exactly .5 seconds, it was a little longer. Also, it takes some time for those 40 worker threads to start processing, they are background worker threads an run at a lower priority than the main thread, so the main thread (adding the work) is going to get a higher priority and is running in a tight loop and so the worker threads will be delayed a little before they start working. Finally there is the overhead of the connections initially spinning up! To see if the time spent to get the connections up is part of the delay, I added this code to see if once the 6 connections are created and maintained by the ServicePoint object, would the performance be as I expected after the WaitHandle.WaitAll(manualEvents); code in the sample code:

for (int i = 0; i < numberRequests; i++)

{

//Create a class to pass info to the thread proc

threadStateInfo theInfo = new threadStateInfo();

manualEvents[i].Reset();//Unsignalled state
theInfo.evtReset = manualEvents[i];

theInfo.iReqNumber = i+40; //just to track what request this is

theInfo.strUrl = httpSite; // the URL to open

ThreadPool.QueueUserWorkItem(ThreadProc, theInfo); //Let the thread pool do the work

}

WaitHandle.WaitAll(manualEvents);

Sure enough! On my system, once the 6 connections are created in the first loop, each of the second set of requests succeeds.

Next let’s play with the MaxServicePointIdleTime to see how this affects the performance. Let’s say in our case we expect that the 40 user in 4 seconds rate occurs in bursts every 6 seconds. In other words, 40 users come in, there is 4 seconds processing time, 2 second delay and another 40 users come in. Simulate this in code with this sleep statement just before the new loop you added in the previous step:

System.Threading.Thread.Sleep(2000);

On my machine that seemed to perform well. But let’s see how the MaxServicePointIdleTime affects this. Set the idle time to .5 seconds at the top of the code:
// 2 is Default Connection Limit
ServicePointManager.DefaultConnectionLimit = 6;
ServicePointManager.MaxServicePointIdleTime = 500;

I was surprised to see that this still succeeded. I started Netmon to see what was happening with the network sockets, and indeed with the timeout, more sockets were being used. I picked a couple of the conversations and filtered by the port and indeed saw that the connection was being reset by the client after .5 seconds. Apparently in MY case, the overhead of the ServicePoint object initializing internal structures was greater than the cost of setting up the socket. Realize in my case the target server was doing nothing else and was on the same switch that the source computer is on so the TCP overhead for creating the connection was indeed negligible.

All this said, pick a MaxServicePointIdleTime that is less than the Keep-Alive timeout of the server and small enough so that you do not have ports open when it is not necessary. In my scenario I expected 40 users in a burst after 2 seconds of activity so a value of 2-6 seconds seems like a good choice. I set it to 2500 milliseconds.

Next I wanted to get a good setting for the DefaultConnectionLimit that would allow the requests to succeed all of the time. Simple experimentation showed that 10 seemed to always succeed. Confident that I had a good bunch of settings, I now had to plan for the fact that sometimes the network, local resources or target server may create a timeout condition. One way to address this would be to increase the HttpWebRequest.Timeout value. Another may be to increase the number of connections available. Finally you could simply retry the request when it fails.

To simulate the system under a heavy load and test retrying the request, I increased the sleep value in my Worker ThreadProc to sleep for a second. I then added a retry immediately after the catch:

static void ThreadProc(Object stateInfo)

{

threadStateInfo tsInfo = stateInfo as threadStateInfo;

bool success = false;

try

{

HttpWebRequest aReq = WebRequest.Create(tsInfo.strUrl) as HttpWebRequest;

aReq.Timeout = 4000; // I want the request to only take 4 seconds, otherwise there is some problem

Console.WriteLine("Begin Request {0}", tsInfo.iReqNumber);

HttpWebResponse aResp = aReq.GetResponse() as HttpWebResponse;

System.Threading.Thread.Sleep(1000); //simulate a half second delay for the server to process the request

aResp.Close(); // if you do not do this close, you will timeout for sure! The socket will not be freed.

Console.WriteLine("End Request {0}", tsInfo.iReqNumber);

success = true;

}

catch (WebException theEx)

{

Console.WriteLine("Exception for Request {0}: {1}",tsInfo.iReqNumber,theEx.Message);

}

try

{

if (success == false)

{

Console.WriteLine("Retry Request {0}:", tsInfo.iReqNumber);

HttpWebRequest aReq = WebRequest.Create(tsInfo.strUrl) as HttpWebRequest;

aReq.Timeout = 4000; // I want the request to only take 4 seconds, otherwise there is some problem

Console.WriteLine("Begin Retry Request {0}", tsInfo.iReqNumber);

HttpWebResponse aResp = aReq.GetResponse() as HttpWebResponse;

System.Threading.Thread.Sleep(1000); //simulate a half second delay for the server to process the request

aResp.Close(); // if you do not do this close, you will timeout for sure! The socket will not be freed.

Console.WriteLine("End Retry Request {0}", tsInfo.iReqNumber);

success = true;

}

catch (WebException theEx)

{

Console.WriteLine("2nd Exception for Request {0}: {1}", tsInfo.iReqNumber, theEx.Message);

}

//signal the main thread this request is done

tsInfo.evtReset.Set();

}

This allowed all of the requests to succeed! Retrying the after the exception has the benefit that we do not use additional resources unnecessarily and in the event there is a temporary problem with the system, the requests will still go through.

This allowed all of the requests to succeed! Retrying the after the exception has the benefit that we do not use additional resources unnecessarily and in the event there is a temporary problem with the system, the requests will still go through.

Summary

This is only a simple example but should give you some hands on experience with the two settings; MaxServicePointIdleTime and DefaultConnectionLimit. If you work through this document and sample code on your own, you will have a good idea of how these values can be changed to help you create an application that is efficient and that will work for you. NOTHING can replace careful design, initial specification and analysis. You will need to load test and see how your system behaves in the normal case and when unexpected events affect the normal processing. You should be able to extend this knowledge to ASP.NET and Windows applications that use the HttpWebRequest class and successfully design or troubleshoot them. Network Monitor will be a great tool for you to see what is actually happening over the network as well. I only showed once how I used the tool to see the connection activity over the network, but I use this tool daily to troubleshoot HTTP issues.

Let me know if you find this a useful resource!

HttpWebRequest exception: "Unable to connect to the remote server" when using .INS config files

jpsanders — Tue, 24 Mar 2009 17:27:00 GMT

Problem

Taking a System.Net Trace (see My Favorite System.Net Tracing File) you will see that the HttpWebRequest is trying to use a .INS file.

System.Net Verbose: 0 : [0371] WebRequest::Create(http://myserver/ie/update/windows/IEConfig.INS)

It is successfully retrieved but when the request is made you see something similar to this in the .NET log:

System.Net.Sockets Error: 0 : [0371] Exception in the Socket#4094998::Connect - No connection could be made because the target machine actively refused it 192.168.0.202:443

and

System.Net Error: 0 : [0371] Exception in the HttpWebRequest#33574999:: - Unable to connect to the remote server

Resolution

System.Net by design does not process .INS files. This results in the HTTP Request being made direct and the Proxy is denying this action.

You need choose one of these methods to resolve the issue:

1. Change your environment to use .PAC files. For example, set this in IE or use the AutoConfigUrl key.

2. Change your .NET code to hard code the proxy information

3. Add proxy configuration information to your <<application>>.exe.config, web.config or machine.config files.

-Jeff

My Favorite System.Net trace configuration file dumps Process ID and Date Time Information

jpsanders — Tue, 24 Mar 2009 15:28:00 GMT

Often you need to correlate a .NET trace with a Netmon trace. To do that you need the date and time of the call from the System.Net trace. This will do it!

NOTE: This section is configures where the log file is and where it should be written: initializeData="System.Net.trace.log"
Specify the path and file name and make sure the process running has permissions to write that file in the location you specify!

Copy Code:

HttpWebRequest WebExcepton: The remote server returned an error: (407) Proxy Authentication Required.

jpsanders — Tue, 24 Mar 2009 14:57:00 GMT

Problem

System.Net Tracing (see My Favorite System.Net Tracing File) revealed that we are getting the 407 error and the .NET framework is not retrying the request with Credentials.

System.Net Error: 0 : [4811] Exception in the HttpWebRequest#33574938:: - The remote server returned an error: (407) Proxy Authentication Required.

We can see in a Fiddler2 trace that Internet Explorer does submit the 407 credentials of the currently logged on user, but the HttpWebRequest does not even try. We also see that for HTTP traffic there is no 407 in either request. Only HTTPS traffic was requiring credentials for the proxy.

Resolution

By default the Proxy object does not supply the credentials of the Currently Logged on User. You can fix this by:

1. Supply the credentials of the Currently Logged on User to the Proxy object similar to this:

Copy Code:

// Begin code change by jeff

// Obtain the 'Proxy' of the Default browser.

IWebProxy theProxy = aReq.Proxy;

// Print the Proxy Url to the console.

if (theProxy != null)

{

// Use the default credentials of the logged on user.

theProxy.Credentials = CredentialCache.DefaultCredentials;

}

// End code change by jeff

HttpWebResponse aResp = aReq.GetResponse() as HttpWebResponse;

2. Add this property to the <<application>>.exe.config or machine.config file:

Copy Code:

<system.net>
<defaultProxy useDefaultCredentials="true"> </defaultProxy>
</system.net>

Anaylyze HttpWebRequest Hangs with a VB DebugDiag script`

jpsanders — Mon, 23 Feb 2009 22:39:00 GMT

Here is a VB Version of the JScript DebugDiag script for analyzing WebRequest hangs using DebugDiag http://blogs.msdn.com/jpsanders/archive/2008/10/24/analyzing-httpwebrequest-connection-saturation-in-net.aspx. It has several improvements over the previous version. First you will notice many similarities to the previous script. Some minor changes were added to translate from Jscript to VBscript. One big change is the .loadby command. This will load the sos.dll from the same path the mscorwks is loaded. You would normally run this script on the machine that you took the dump from. This would ensure that the sos version matches the mscorwks version in the dump. Another change is to seach not only for GetRequest but for SubmitRequest as well. Finally I am dumping out more information like when there request is fine and the ConnectionGroupName.

Problem! All available connections used up.
HttpRequest URI:http://10.10.10.10/Service/Service.asmx
ServicePoint - ConnectionLimit:12 CurrentConnections:14
ConnectionGroupName: MyConnectionPool

I thought it would be equally important to dump when there were no issues:

No Hang Problems found
HttpRequest URI:http://Mydev/output.asmx/test?&Name=test2
ServicePoint - ConnectionLimit:10 CurrentConnections:1
ConnectionGroupName:

Finally, fomatting the URI in a more readable format was fairly simple. I used the -nofields switch when dumping the string and parsed out the String: text.

function getUriString(aObjAddr)
Dim aRetStr
Dim aCmdOutput
Dim aFoundIdx
aCmdOutput = g_Debugger.Execute("!do -nofields poi(poi(" & aObjAddr & "+" & g_HttpObjectOffsets.Item("_Uri") & ")+" & g_UriObjectOffsets.Item("m_String") & ")" )

aFoundIdx = InStr(1,aCmdOutput,"String:",1)

if (aFoundIdx <> 0) then
aRetStr=trim( Mid(aCmdOutput, aFoundIdx + Len("String:")))
end if
getUriString = aRetStr

end function

As with ANY sample code, I leave it to you to debug and test! I also left some "TODO" sections for you to fill in yourself. I hope you find it useful. An industrious person could embed this into the existing DebugDiag scripts to included it in the HangAnalysis portion of the scripts and really have a powerful tool for troubleshooting WebRequest issue.

Full script below (Copy Code):

<%@ Language = vbscript %><%@ Category = Crash/Hang Analyzers %><%@ Description = Template System.Net analysis script to build from...%><%@ TopLevelScript = True %><%

Option Explicit
Dim g_Progress
Dim g_Debugger

Dim g_HttpObjectOffsets
Dim g_SvcPointObjectOffsets
Dim g_UriObjectOffsets
Dim g_StringObjectOffsets

Dim g_MaxConnectionErrorThreads ' array of thread id's that have exceed max connection.
Dim g_MaxConnectionWarningThreads ' array of thread id's that have 50% max connection.

main()

' this kicks off everything
sub main()

Dim a_DataFile
Dim i
Dim a_NumberOfDumps

Manager.Write "<b>System.Net sample analysis script</b><BR>"

Manager.Progress.SetOverallRange 0,100
Manager.Progress.OverallPosition = 10
Manager.Progress.OverallStatus="Initializing"
a_NumberOfDumps = Manager.DataFiles.Count

' for each data file specified...

for i=0 to a_NumberOfDumps-1

' get the dump
a_DataFile = Manager.DataFiles(i)
Manager.Progress.OverallStatus="Starting analysis of: " & a_DataFile

' get an instance of a debugger
set g_Debugger = Manager.GetDebugger(a_DataFile)

Manager.Write "<h2>Analysis of: " & a_DataFile & "</h2>"

if g_Debugger.IsCrashDump then
Manager.ReportInformation "DebugDiag determined that this dump file (" & a_DataFile & ") is a <font color='Red'><b> crash </b></font>" &_
"dump and this script does not analyze crashes (only hangs). Please run analysis using a crash Script."

end if

' try to load the managed debugging extension
if(LoadSOS)then

' initialize some object information for performance
initGlobals

' walk all threads but report only specific managed ones
walkManagedThreads
reportErrors

else
' todo
end if

' done with the debugger, so close it
Manager.CloseDebugger a_DataFile
Manager.Progress.OverallPosition = 100/a_NumberOfDumps * (i+1)

next
end sub

sub initGlobals()

set g_HttpObjectOffsets = CreateObject("Scripting.Dictionary")
ManagedFieldOffsets g_HttpObjectOffsets, Array("_ServicePoint", "_Uri", "_HttpRequestHeaders", "_Proxy","_ConnectionGroupName"),"system.dll!System.Net.HttpWebRequest"

set g_SvcPointObjectOffsets = CreateObject("Scripting.Dictionary")
ManagedFieldOffsets g_SvcPointObjectOffsets, Array("m_ConnectionLimit", "m_CurrentConnections"),"system.dll!System.Net.ServicePoint"

set g_UriObjectOffsets = CreateObject("Scripting.Dictionary")
ManagedFieldOffsets g_UriObjectOffsets, Array("m_String"), "system.dll!System.Uri"

set g_StringObjectOffsets = CreateObject("Scripting.Dictionary")
ManagedFieldOffsets g_StringObjectOffsets, Array( "m_firstChar"), "mscorlib.dll!System.String"

g_MaxConnectionErrorThreads = ""
g_MaxConnectionWarningThreads = ""

end sub

' this starts the analysis...

sub walkManagedThreads()

Dim aCmdOutput
Dim aStackOutput
Dim aThreadInfo
Dim aDebugThread
Dim aNumThreads
Dim aAddr

Dim i
Dim aFoundIdx
Dim bFoundFirstThread
bFoundFirstThread =false

set aThreadInfo = g_Debugger.ThreadInfo
aNumThreads = aThreadInfo.Count

Manager.Progress.SetCurrentRange 0,aNumThreads
Manager.Progress.CurrentStatus="Finding threads waiting on 'System.Net.HttpWebRequest.GetResponse()' or 'System.Net.HttpWebRequest.SubmitRequest()'"

for i=0 to aNumThreads-1

Manager.Progress.CurrentPosition = i

set aDebugThread = aThreadInfo(i)

if Not( aDebugThread is nothing ) then

aStackOutput = g_Debugger.Execute("~" & aDebugThread.ThreadID & "e!CLRStack")
aFoundIdx = InStr(aStackOutput,"System.Net.HttpWebRequest.GetResponse")

if (aFoundIdx = 0)then
aFoundIdx = InStr(aStackOutput,"System.Net.HttpWebRequest.SubmitRequest")
end if

' highlight the call???
'is the tread processing GetResponse?

if (aFoundIdx <> 0) then
if Not(bFoundFirstThread) then

bFoundFirstThread=true
Manager.Write "<br><h2>Managed Threads Waiting on 'System.Net.HttpWebRequest.GetResponse()'</h2><br>"

end if

Manager.Write "<A NAME='ThreadAnchor" & aDebugThread.ThreadID & "'>"

' dump managed objects on the stack
aCmdOutput = g_Debugger.Execute("~" & aDebugThread.ThreadID & "e!dso")

' Collect the HttpWebRequest objects on the stack.
aFoundIdx = InStr(aCmdOutput,"System.Net.HttpWebRequest")

while aFoundIdx <> 0

' if we found an object of interest on the stack,
' sav the address fo the object and the thread ID.

aAddr = Mid(aCmdOutput,aFoundIdx-9, 8)

checkObj aAddr,aDebugThread.ThreadID

'build error/warning strings here instead.
'display total number of managed threads?

' look for another one...
aFoundIdx = 0 'aCmdOutput.indexOf("System.Net.HttpWebRequest", aFoundIdx + 25)

wend

Manager.Write regexReplace(aStackOutput, "\n", "<br>" )
end if
end if

end sub

function regexReplace(aSrcStr, aPattern, aReplacement )

dim aRetStr
dim aRegEx
set aRegEx = New RegExp
aRegEx.Pattern=aPattern
aRegEx.Global=True

aRetStr = aRegEx.Replace( aSrcStr,aReplacement)
regexReplace = aRetStr

end function

sub reportErrors()

Dim i
Dim threadList
if Len(g_MaxConnectionErrorThreads) then

Manager.ReportError "The number connections exceeds the number of available connections for these threads:" & g_MaxConnectionErrorThreads,"Increase the number of connections available per this article: <a TARGET=_blank href='http:'msdn.microsoft.com/en-us/library/7af54za5.aspx'>Managing Connections</a>"

end if

end sub

'''''''''''/ Managed code helpers '''''''''''''''''/

' use .loadby to find where mscorwks is loaded and then load SOS from the same place
function LoadSOS()

Dim abSuccess
Dim begPos
Dim endPos
Dim CmdOutput
Dim Module

abSuccess = false
CmdOutput = g_Debugger.Execute(".loadby sos mscorwks")
'todo, check CmdOutput to ensure SOS loaded!

LoadSOS = true

end function

' given an Array, populate a dictionary object with offets for these array items to be used later.
sub ManagedFieldOffsets( dictObj ,aArray, aObject)

Dim aStr
Dim i
Dim aCmdOutput
' get the Class information
aCmdOutput = g_Debugger.Execute("!DumpClass " + getEEClass(aObject))

for each aStr in aArray
dictObj.Add aStr, findOffset(aCmdOutput,aStr)
next

End Sub

' simple helper functions

function getManagedLongVal( aObjAddr, aOffset )
getManagedLongVal = g_Debugger.Execute("dt long " & aObjAddr & "+" & aOffset)
end function

function getManagedStringVal( aObjAddr, aOffset )

Dim retStr
Dim aCmdOutput
Dim aFoundIdx
aCmdOutput = g_Debugger.Execute("!do -nofields poi(" & aObjAddr & "+" & aOffset & ")")

aFoundIdx = InStr(aCmdOutput,"String:")

if (aFoundIdx <> 0) then
retStr=trim( Mid(aCmdOutput, aFoundIdx + Len("String:")))
end if

getManagedStringVal = retStr

end function

''''''''' Helper Function - findOffset
' given a string, extract the field offset specified
' this is really an excercise it string parsing
function findOffset(aCmdOutput, aField)

Dim retStr
retStr=""

Dim aFoundIdx
Dim aBOL
Dim sArray

' find the field
aFoundIdx = InStr(1,aCmdOutput,aField,1)

if aFoundIdx <> 0 then

' look backwards from there for the offset.
' find the previous line
aBOL = InStrRev(aCmdOutput,Chr(10),aFoundIdx,1)+1

'need to strip all but one space from the string and make an array of the column values
retStr=normalizeWhitespace(Mid(aCmdOutput,aBOL, aFoundIdx-aBOL))
sArray = Split(retStr)
retStr=""
' 7 columns total, offset is third column (0 based array)
retStr = sArray(2)

else
'TODO Manager.Write "Did not find Field: " & aField & "<<<<<<<<<<<<<<"
end if

findOffset= retStr

end function

''''''''' Helper Function - getEEClass
' given a string, extract the EEClass address
' theClass name is direct input to !Name2EE must be MODULENAME!CLASSNAM
function getEEClass( theClassName )

Dim retStr
retStr=""
Dim aCmdOutput
Dim aFoundIdx
Dim aEOL

aCmdOutput = g_Debugger.Execute("!Name2EE " & theClassName)

aFoundIdx = InStr(1,aCmdOutput,"EEClass:",1)

if (aFoundIdx <> 0) then
aEOL = InStr(aFoundIdx,aCmdOutput,"Name:",1)
retStr = Mid(aCmdOutput, aFoundIdx + Len("EEClass:") , aEOL - (aFoundIdx + Len("EEClass:")))
end if

getEEClass= retStr

end function

' utility functions

' utility function removing extra whitespace and replacing it with a single space
' for example " some text here" will be converted to " some text here"

function normalizeWhitespace(stringToTrim)

dim aRegEx
set aRegEx = New RegExp
aRegEx.Pattern="\s+"
aRegEx.Global=True
normalizeWhitespace = aRegEx.Replace( stringToTrim," ")

end function

' calculate and save offsets for the Object fields we need
'''''''''''''''' Non-generic managed object helpers....

' Given an address of an object get the service point object address
function getSvcPointFromReq(aHttpObjAddr)

Dim aRetStr
aRetStr=""
Dim aCmdOutput

aCmdOutput = g_Debugger.Execute("dd " & aHttpObjAddr & "+" & g_HttpObjectOffsets.Item("_ServicePoint") & " L1")

aRetStr = Mid(aCmdOutput, Len(aCmdOutput) - 8,8)
getSvcPointFromReq = Trim(aRetStr)

end function

aFoundIdx = InStr(1,aCmdOutput,"String:",1)
if (aFoundIdx <> 0) then
aRetStr=trim( Mid(aCmdOutput, aFoundIdx + Len("String:")))
end if

getUriString = aRetStr

end function

function getConnectionGroupName(aObjAddr)

getConnectionGroupName=g_Debugger.Execute("du poi(poi(" & aObjAddr & "+" & g_HttpObjectOffsets.Item("_ConnectionGroupName") & ")+" & g_UriObjectOffsets.Item("m_String") & ")+" & g_StringObjectOffsets.Item("m_firstChar")).substr(9)

end function

' obj is an address we can do !do on to get information from
sub checkObj(aObjAddr, threadID)

'get service point and see if it is OK.
Dim bSvcPointErrorOrWarning
bSvcPointErrorOrWarning = false

Dim maxConn
Dim currConn
Dim connGroup
Dim aCmdOutput
Dim i
Dim aFoundIdx

aCmdOutput = getSvcPointFromReq(aObjAddr)

if( (aCmdOutput<>"00000000") And (aCmdOutput<>"????????")) then

maxConn = getManagedLongVal( aCmdOutput, g_SvcPointObjectOffsets.Item("m_ConnectionLimit"))
currConn = getManagedLongVal( aCmdOutput, g_SvcPointObjectOffsets.Item("m_CurrentConnections"))
connGroup = getManagedStringVal( aObjAddr, g_HttpObjectOffsets.Item("_ConnectionGroupName"))

if(currConn>=maxConn) then

' report out info and error
g_MaxConnectionErrorThreads = g_MaxConnectionErrorThreads & "<a href='#ThreadAnchor" & threadID & "'>" & threadID & "</a>, "

Manager.Write "<br> <font color='red'><b>Problem! All available connections used up.</b><br> HttpRequest URI:" & getUriString(aObjAddr) & "<br>ServicePoint - ConnectionLimit:" & maxConn & " CurrentConnections:" & currConn &"<br>ConnectionGroupName: " & connGroup & "</font><br><br>"

else

if(((currConn)*2)>=maxConn) then

' report out info and error
g_MaxConnectionWarningThreads = g_MaxConnectionWarningThreads & "<a href='#ThreadAnchor" & threadID + "'>" & threadID & "</a>, "

Manager.Write "<br> <font color='orange'><b>Warning, half of the availabe connections are being used</b><br> HttpRequest URI:" & getUriString(aObjAddr) & "<br>ServicePoint - ConnectionLimit:" & maxConn & " CurrentConnections:" & currConn &"<br>ConnectionGroupName: " & connGroup & "</font><br><br>"

else

' no problems found
Manager.Write "<br> <font color='green'> <b>No Hang Problems found</b><br>HttpRequest URI:" & getUriString(aObjAddr) & "<br>ServicePoint - ConnectionLimit:" & maxConn & " CurrentConnections:" & currConn &"<br>ConnectionGroupName: " & connGroup & "</font><br><br>"

end if
end if
end if

End Sub
%>

You receive one or more error messages when you try to make an HTTP request in an application that is built on the .NET Framework 2.0

jpsanders — Wed, 07 Jan 2009 23:46:00 GMT

Please send me a message or leave a comment to let me know if this Post helped you!

When you try to make an HTTP request in an application that is built on the .NET Framework, you may receive one or more of the following error messages or Exceptions.

One of the first things you should do is see what the framework is doing. Get a System.Net trace: http://blogs.msdn.com/jpsanders/archive/2009/03/24/my-favorite-system-net-trace-configuration-file-dumps-process-id-and-date-time-information.aspx

Then you can get a description of the errors here (note this is an enumeration which is the status member of the WebException):
http://msdn.microsoft.com/en-us/library/system.net.webexceptionstatus(VS.80).aspx

Many of these errors are the same in the 1.1 .NET framework:
http://support.microsoft.com/kb/915599

Some of these errors can be trapped and immediately retried such as:

System.Net.WebExceptionStatus.ConnectFailure:
System.Net.WebExceptionStatus.KeepAliveFailure:
System.Net.WebExceptionStatus.RequestCanceled:
System.Net.WebExceptionStatus.ConnectionClosed:
System.Net.WebExceptionStatus.SendFailure:
System.Net.WebExceptionStatus.PipelineFailure:
System.Net.WebExceptionStatus.UnknownError:

Errors:

The underlying connection was closed: The remote name could not be resolved

This is a DNS error. For some reason the server name cannot be resolved. Do you have permissions to the HOST file? Is this a service such as ASP.NET? See this article for help: The underlying connection was closed: The remote name could not be resolved. Also look at this KB: http://support.microsoft.com/kb/330221

The underlying connection was closed: Unable to connect to the remote server

This is a connection issue. The DNS resolved the server name but something is preventing the client from connecting to the server. Lookup the error in this article http://support.microsoft.com/kb/915599.

The underlying connection was closed: An unexpected error occurred on a receive

This problem occurs when the server or another network device unexpectedly closes an existing Transmission Control Protocol (TCP) connection. Lookup the error in this article http://support.microsoft.com/kb/915599.

The underlying connection was closed: An unexpected error occurred on a send

This problem occurs when the client computer cannot send an HTTP request. The client computer cannot send the HTTP request because the connection has been closed or is unavailable. Lookup the error in this article http://support.microsoft.com/kb/915599. Antivirus software can interfere with the send as well. If the stack from the exception includes something similar to this:

System.IO.IOException: Authentication failed because the remote party has closed the transport stream.

It is possible that the server is an older server does not understand TLS and so you need to change this as specified in the 915599 kb to something like this:

ServicePointManager.SecurityProtocol= SecurityProtocolType.Ssl3;

Before you make any HTTPS calls in your application.

The underlying connection was closed: A pipeline failure occurred

Pipelined requests are requests sent out on the same socket without waiting for a reply (http://search.live.com/search?q=http+pipelined+request). All HTTP 1.1 servers are required to support pipelining. In general if you see this error it is probable related to a network related failure. You could turn off pipelining and try to see what the underlying issue is but pipelining itself is rarely the cause of this error message. You should retry the request if you get this error but if you get this exception multiple times, you should troubleshoot the root cause.

The underlying connection was closed: The request was canceled

The underlying connection was closed: The connection was closed unexpectedly

This occurs with the server cancels the request with a TCP RST (reset). The server may do this because of the thread Execution Timeout Setting being too low and it is tearing down the thread that is processing the request (Thread Abort Exceptions may be logged): http://msdn.microsoft.com/en-us/library/e1f13641.aspx

Also check and ensure the Keep-Alive timeouts on the server, load balancer and client (.NET) are set so that the client is set to less than the load balancer, which in turn is set less than the server. See this article as well:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel

For a service application the whole certificate chain needs to be in the LOCAL_MACHINE store. This means the Root in the Trusted Root Certificate and all the Intermediate Certificates in the Trusted Intermedeate store. For applications that run in the context of the logged on user, If Internet Explorer does not have any Certificate problems, then the .NET application should work fine. Often people will find that Internet Explorer is fine and only the ASP.NET or Service application fails. This is because the root and intermediate certificates are installed correctly in the user store but not the local machine store.

You can dump out what the failure is by using this delegate: http://msdn.microsoft.com/en-us/library/system.net.security.remotecertificatevalidationcallback.aspx. Finally you could use process monitor and verify that there are no access denied errors when trying to access the certificate store which would prevent reading the certificates.

The request was aborted: Could not create SSL/TLS secure channel

Lookup the error in this article http://support.microsoft.com/kb/915599 Resolution J. It also may be that you are not supplying a client certificate. Most likely this is an issue with TLS or SSL3 being used and the server not understanding it.

Error: The server committed a protocol violation

The Exception will tell what the HTTP protocol violation is:

"The server committed a protocol violation. Section=ResponseHeader Detail=Header
name is invalid"

Fix the server to conform to the HTTP RFCs to avoid this error. .NET is a stickler that the HTTP Protocol be followed.

The underlying connection was closed: A connection that was expected to be kept alive was closed by the server

Specifically a Keep-Alive connection was torn down before it should have been. This is often caused because the MaxIdleTimeout setting in the client is greater than that of the server or intermediate network device (load balancer or proxy).

http://support.microsoft.com/?id=941633 and http://support.microsoft.com/kb/915599 Resolution E

The operation has timed out

This could be caused by a stale DNS entry for a proxy or if a request is not received before the socket times out.

An example of this Exception is when for example a server returns information but does not correctly set the content-length header. The call will wait for the rest of the data (which will never come) until it throws this message. It could be that the server takes a long time to respond (which is a different issue) and you could increase System.Net.ServicePointManager.DefaultConnectionLimit. This could also be because you are issuing a lot of request and have not increased the DefaultMaxConnections for your application and the timeout happens waiting for a connection thread to process your request. MaxConnections can be set in the application.config file or in code.

The proxy name could not be resolved

This is a DNS issue. Find out what proxy it is looking for (from the System.Net trace) and find why this does not resolve in your network.

The request was aborted: The request cache-only policy does not allow a network request and the response is not found in cache

The request was aborted: The request could not be satisfied using a cache-only policy

The request was not permitted by the cache policy. In general, this occurs when a request is not cacheable and the effective policy prohibits sending the request to the server. You might receive this status if a request method implies the presence of a request body, a request method requires direct interaction with the server, or a request contains a conditional header.

You can control if the request should use only the cache: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.defaultcachepolicy.aspx

If you do this and the file in not cached, then you will get this error.

Also see: http://msdn.microsoft.com/en-us/library/system.net.cache.httprequestcachepolicy(VS.80).aspx

Using .NET HttpWebRequest class with SiteMinder you get a 403 or 401 error

jpsanders — Thu, 11 Dec 2008 19:22:00 GMT

Download Fiddler from http://fiddlertool.com and run this for an Internet Explorer browser session that successfully connects.

Then hit the same site with your .NET 2.0 client application and make sure you set the credentials in your code as well: http://msdn.microsoft.com/en-us/library/system.net.networkcredential.aspx (Note you can set myCreds directly on the Credentials property of the HttpWebRequest object... no need for the Credential Cache object, but it will not hurt anything).

Compare the two traces in Fiddler in the Sessions Tab on the right side. If the credentials are successfully passed you should see the 401 challenge from the site answered with the same Basic Authentication header (a base 64 encoded jumble of characters). If they are the same then you are passing the credentials correctly so what could be the problem?

Using my post on http troubleshooting: http://blogs.msdn.com/jpsanders/archive/2008/06/25/troubleshooting-code-that-uses-the-http-protocol.aspx

to simplify the issue you will see no Cookies in the failed case. What is going on?

The .NET classes do not implement a cookie container by default and if you wish to use cookies, you need to provide a cookie container. You may want to make this container global to your app or simply local to the function (this is your design decision). Here is the documentation:

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx

Implement the container and use Fiddler once again if it does not work to ensure the successful and failed cases send the exact same data (hint they do not send the same headers and data if this fails still).

Analyzing HttpWebRequest Connection Saturation in .NET

jpsanders — Fri, 24 Oct 2008 23:15:00 GMT

WinDbg is a powerful tool and can use the SOS extension (that ships with the .NET framework) to look into and analyze Managed Code. Another powerful tool is Debug Diagnostics. It uses the same Debugger Engine that WinDbg uses and can run analysis scripts. You can use Debug Diagnostics and a custom script to analyze hangs and performance issues with the HttpWebRequest class.

I will walk you through some techniques that will allow you to customize your own Debug Diagnostics script to analyze .NET HttpWebRequest connection saturation.

The technique I will use is to:

1. Walk the threads in a Dump and look for Managed threads that have System.Net.HttpWebRequest.GetResponse on the stack. To me that means we are waiting on a response and may indicate an issue.

2. For each of the threads found above, I will get the ServicePoint object and see how many connections it allows vs. how many it currently has.

3. Make a suggestion on how to fix it!

DebugDiag (Debug Diagnostics) has a great help file and some starter scripts. I wanted to use jscript so the starter scripts did not do much for me since they were written in VBScript. I put all the metadata for the script on one line so that when I get script errors I can easily map this to the script I am editing in Visual Studio 2008. Save the script (with a .asp extension) to the Scripts sub-directory of where DebugDiag is installed and it will show up when you start DebugDiag.

Here is the starting script:

<%@ Language = javascript %><%@ Category = Crash/Hang Analyzers %><%@ Description = Template System.Net analysis script to build from...%><%@ TopLevelScript = True %><%

main();

// this kicks off everything

function main()

{

Manager.Write( "<b>System.Net sample analysis script</b><BR>");

}

Select a dump file and run this script against it. You see it will just print some text.

Now let’s get to work. We need to iterate over all the dump files the DebugDiag is pointing to (in the event there is more than one) and we want to iterate over all of the threads.

Here is the Main function:

// this kicks off everything

function main()

{

var a_DataFile;

var i;

var a_NumberOfDumps;

g_GlobalBookMark=0;

Manager.Write( "<b>System.Net sample analysis script</b><BR>");

// interesting that this does not work

//for (a_DataFile in Manager.DataFiles)

Manager.Progress.SetOverallRange(0,100);

Manager.Progress.OverallPosition = 10;

Manager.Progress.OverallStatus="Initializing";

a_NumberOfDumps = Manager.DataFiles.Count;

// for each data file specified...

for(i=0;i<a_NumberOfDumps;i++)

{

// get the dump

a_DataFile = Manager.DataFiles(i);

Manager.Progress.OverallStatus="Starting analysis of: " + a_DataFile;

// get an instance of a debugger

g_Debugger = Manager.GetDebugger(a_DataFile);

Manager.Write("<h2>Analysis of: " + a_DataFile + "</h2>");

// try to load the managed debugging extension

if(LoadSOS())

{

// initialize some object information for performance

initGlobals();

// walk all threads but report only specific managed ones

walkManagedThreads();

reportErrors();

}

// done with the debugger, so close it

Manager.CloseDebugger(a_DataFile);

Manager.Progress.OverallPosition = 100/a_NumberOfDumps * (i+1);

}

Now all that remains is to fill in the functions: LoadSOS(), initGlobals(), walkManagedThreads() and reportErrors();

// Find where mscorwks is loaded and then load SOS from the same place

function LoadSOS()

{

var abSuccess;

var begPos;

var endPos;

var CmdOutput;

abSuccess = false;

// Find where mscorwks module came from…

CmdOutput = g_Debugger.Execute("lmv m mscorwks");

begPos = CmdOutput.indexOf("Image path:");

if (begPos>0)

{

begPos=begPos+11; // skip over Image path:

endPos=CmdOutput.indexOf("mscorwks.dll");

Manager.Write ("<b>Loading SOS from: </b>" + CmdOutput.substr(begPos,endPos-begPos) + "<br>");

// load it!

CmdOutput = g_Debugger.Execute("!load " + CmdOutput.substr(begPos,endPos-begPos) + "sos.dll");

if (CmdOutput.length == 0)

{

abSuccess=true;

}

else

{

Manager.Write( "Cannot load Managed Debugging extensions. Error: " + CmdOutput + "<br>" );

}

return abSuccess;

}

function initGlobals()

{

g_HttpObjectOffsets = new ManagedFieldOffsets(new Array("_ServicePoint", "_Uri", "_HttpRequestHeaders", "_Proxy"),"system.dll!System.Net.HttpWebRequest");

g_SvcPointObjectOffsets = new ManagedFieldOffsets(new Array("m_ConnectionLimit", "m_CurrentConnections"),"system.dll!System.Net.ServicePoint");

g_UriObjectOffsets = new ManagedFieldOffsets(new Array("m_String"), "system.dll!System.Uri");

g_StringObjectOffsets = new ManagedFieldOffsets(new Array( "m_firstChar"), "mscorlib.dll!System.String");

g_MaxConnectionErrorThreads = "";

g_MaxConnectionWarningThreads = "";

}

The ManagedFieldOffsets function is pretty interesting. You could use several techniques in this script to get the information you want from a managed object. The SOS extension has a !do command that can dump the fields of an managed object and you can hunt through that for each object to get the value of a field you want, then !do that object etc… until you get to the value you want. However, I want this script to perform well, so I will use offsets from the object base address to get to the data I want. I could hard code these offsets, but if they change or I am using a different version of the CRL, I would have to change my script. So this handy function (actually used lick a class) is an expando based class that will store these offsets for me.

NOTE: If you want to learn about these commands, load a dump file with HttpWebRequest on the heap and play with each of the g_Debugger.Execute commands I am using!

/////////////////// Class - ManagedFieldOffsets

// this class uses expando properties and managed helpers to cache offsets

// in managed classes.

// usage: var obj = new ManagedFieldOffsets(new Array("_ServicePoint", "_Uri", "_HttpRequestHeaders", "_Proxy"),"system.dll!System.Net.HttpWebRequest");

// aArray is an array of fields you are intersted in. It will create properties

// of the same name and look up the offset value for this field

// the second parameter aObject is the location and name of the managed object.

// you then can use the obj as defined above like this obj._Proxy to get the offset values

function ManagedFieldOffsets( aArray, aObject )

{

var aStr;

var i;

var aCmdOutput;

// get the Class information

aCmdOutput = g_Debugger.Execute("!DumpClass " + getEEClass(aObject));

for(i in aArray)

{

this[aArray[i]] = findOffset(aCmdOutput,aArray[i]);

}

////////////////// Helper Function - getEEClass

// given a string, extract the EEClass address

// theClass name is direct input to !Name2EE must be MODULENAME!CLASSNAM

function getEEClass( theClassName )

{

var retStr="";

var aCmdOutput;

var aFoundIdx;

var aEOL;

aCmdOutput = g_Debugger.Execute("!Name2EE " + theClassName);

aFoundIdx = aCmdOutput.indexOf("EEClass:");

if (aFoundIdx != -1)

{

aEOL = aCmdOutput.indexOf("\n",aFoundIdx);

retStr=trim( aCmdOutput.substr( aFoundIdx + "EEClass:".length , aEOL - (aFoundIdx + "EEClass:".length)));

}

return retStr;

}

////////////////// Helper Function - findOffset

// given a string, extract the field offset specified

// this is really an excercise it string parsing

function findOffset(aCmdOutput, aField)

{

var retStr="";

var aFoundIdx;

var aBOL;

var sArray;

// find the field

aFoundIdx = aCmdOutput.indexOf(aField);

if (aFoundIdx != -1)

{

// look backwards from there for the offset.

// find the previous line

aBOL = aCmdOutput.lastIndexOf("\n",aFoundIdx)+1;

//need to strip all but one space from the string and make an array of the column values

sArray = normalizeWhitespace(aCmdOutput.substr(aBOL, aFoundIdx-aBOL)).split(" ");

// 7 columns total, offset is third column (0 based array)

retStr=sArray[2];

//Manager.Write("<br> 0:" + sArray[0] +"<br> 1:" + sArray[1] +"<br> 2:" + sArray[2] +"<br> 3:" + sArray[3] + "<br>");

}

return retStr;

}

Finally this function walks the threads and if it finds the thread is waiting on a request, dumps it out and inspects for connection saturation:

// this starts the analysis...

function walkManagedThreads()

{

var aCmdOutput;

var aStackOutput;

var aThreadInfo;

var aDebugThread;

var aNumThreads;

var aAddr;

var i;

var aFoundIdx;

var bFoundFirstThread =false;

aThreadInfo = g_Debugger.ThreadInfo;

aNumThreads = aThreadInfo.Count;

Manager.Progress.SetCurrentRange(0,aNumThreads);

Manager.Progress.CurrentStatus="Finding threads waiting on 'System.Net.HttpWebRequest.GetResponse()'";

for (i=0;i<aNumThreads;i++)

{

Manager.Progress.CurrentPosition = i;

aDebugThread = aThreadInfo.Item(i);

if (aDebugThread)

{

aStackOutput = g_Debugger.Execute("~" + aDebugThread.ThreadID + "e!CLRStack");

aFoundIdx = aStackOutput.indexOf("System.Net.HttpWebRequest.GetResponse");

// highlight the call???

//is the tread processing GetResponse?

if (aFoundIdx != -1)

{

if (!bFoundFirstThread)

{

bFoundFirstThread=true;

Manager.Write("<br><h2>Managed Threads Waiting on 'System.Net.HttpWebRequest.GetResponse()'</h2><br>");

}

Manager.Write( "<A NAME='ThreadAnchor" + aDebugThread.ThreadID + "'>" );

// dump managed objects on the stack

aCmdOutput = g_Debugger.Execute("~" + aDebugThread.ThreadID + "e!dso");

// Collect the HttpWebRequest objects on the stack.

aFoundIdx = aCmdOutput.indexOf("System.Net.HttpWebRequest");

while (aFoundIdx != -1)

{

// if we found an object of interest on the stack,

// sav the address fo the object and the thread ID.

aAddr = aCmdOutput.substr(aFoundIdx-9, 8);

checkObj(aAddr,aDebugThread.ThreadID);

//build error/warning strings here instead.

//display total number of managed threads?

// look for another one...

aFoundIdx = -1;

}

// print the stack pretty by replacing line feeds with breaks Manager.Write( aStackOutput.replace(/\n/g, "<br>" ) + "<br>");

}

I have included the completed script. It needs some error checking and polishing, but it will get the job done. It will report out any threads waiting on requests and dump out any HttpWebRequest objects that have ServicePoint objects with the CurrentConnections >= ConnectionLimit

Here is the complete code listing (save to the DebugDiag Scripts directory as WebRequestConnections.asp):

<%@ Language = javascript %><%@ Category = Crash/Hang Analyzers %><%@ Description = Template System.Net analysis script to build from...%><%@ TopLevelScript = True %><%

var g_Progress;

var g_Debugger;

var g_HttpObjectOffsets;

var g_SvcPointObjectOffsets;

var g_UriObjectOffsets;

var g_StringObjectOffsets;

var g_MaxConnectionErrorThreads; // array of thread id's that have exceed max connection.

var g_MaxConnectionWarningThreads; // array of thread id's that have 50% max connection.

main();

// this kicks off everything

function main()

{

var a_DataFile;

var i;

var a_NumberOfDumps;

Manager.Write( "<b>System.Net sample analysis script</b><BR>");

// interesting that this does not work

//for (a_DataFile in Manager.DataFiles)

Manager.Progress.SetOverallRange(0,100);

Manager.Progress.OverallPosition = 10;

Manager.Progress.OverallStatus="Initializing";

a_NumberOfDumps = Manager.DataFiles.Count;

// for each data file specified...

for(i=0;i<a_NumberOfDumps;i++)

{

// get the dump

a_DataFile = Manager.DataFiles(i);

Manager.Progress.OverallStatus="Starting analysis of: " + a_DataFile;

// get an instance of a debugger

g_Debugger = Manager.GetDebugger(a_DataFile);

Manager.Write("<h2>Analysis of: " + a_DataFile + "</h2>");

// try to load the managed debugging extension

if(LoadSOS())

{

// initialize some object information for performance

initGlobals();

// walk all threads but report only specific managed ones

walkManagedThreads();

reportErrors();

}

// done with the debugger, so close it

Manager.CloseDebugger(a_DataFile);

Manager.Progress.OverallPosition = 100/a_NumberOfDumps * (i+1);

}

function initGlobals()

{

g_HttpObjectOffsets = new ManagedFieldOffsets(new Array("_ServicePoint", "_Uri", "_HttpRequestHeaders", "_Proxy"),"system.dll!System.Net.HttpWebRequest");

g_SvcPointObjectOffsets = new ManagedFieldOffsets(new Array("m_ConnectionLimit", "m_CurrentConnections"),"system.dll!System.Net.ServicePoint");

g_UriObjectOffsets = new ManagedFieldOffsets(new Array("m_String"), "system.dll!System.Uri");

g_StringObjectOffsets = new ManagedFieldOffsets(new Array( "m_firstChar"), "mscorlib.dll!System.String");

g_MaxConnectionErrorThreads = "";

g_MaxConnectionWarningThreads = "";

}

// this starts the analysis...

function walkManagedThreads()

{

var aCmdOutput;

var aStackOutput;

var aThreadInfo;

var aDebugThread;

var aNumThreads;

var aAddr;

var i;

var aFoundIdx;

var bFoundFirstThread =false;

aThreadInfo = g_Debugger.ThreadInfo;

aNumThreads = aThreadInfo.Count;

Manager.Progress.SetCurrentRange(0,aNumThreads);

Manager.Progress.CurrentStatus="Finding threads waiting on 'System.Net.HttpWebRequest.GetResponse()'";

for (i=0;i<aNumThreads;i++)

{

Manager.Progress.CurrentPosition = i;

aDebugThread = aThreadInfo.Item(i);

if (aDebugThread)

{

aStackOutput = g_Debugger.Execute("~" + aDebugThread.ThreadID + "e!CLRStack");

aFoundIdx = aStackOutput.indexOf("System.Net.HttpWebRequest.GetResponse");

// highlight the call???

//is the tread processing GetResponse?

if (aFoundIdx != -1)

{

if (!bFoundFirstThread)

{

bFoundFirstThread=true;

Manager.Write("<br><h2>Managed Threads Waiting on 'System.Net.HttpWebRequest.GetResponse()'</h2><br>");

}

Manager.Write( "<A NAME='ThreadAnchor" + aDebugThread.ThreadID + "'>" );

// dump managed objects on the stack

aCmdOutput = g_Debugger.Execute("~" + aDebugThread.ThreadID + "e!dso");

// Collect the HttpWebRequest objects on the stack.

aFoundIdx = aCmdOutput.indexOf("System.Net.HttpWebRequest");

while (aFoundIdx != -1)

{

// if we found an object of interest on the stack,

// sav the address fo the object and the thread ID.

aAddr = aCmdOutput.substr(aFoundIdx-9, 8);

checkObj(aAddr,aDebugThread.ThreadID);

//build error/warning strings here instead.

//display total number of managed threads?

// look for another one...

aFoundIdx = -1; //aCmdOutput.indexOf("System.Net.HttpWebRequest", aFoundIdx + 25);

}

Manager.Write( aStackOutput.replace(/\n/g, "<br>" ) + "<br>");

// Manager.Write( "Thread ID: " + aDebugThread.ThreadID + " Waiting <br>");

}

function reportErrors()

{

var i;

var threadList;

if (g_MaxConnectionErrorThreads.length)

{

Manager.ReportError("The number connections exceeds the number of available connections for these threads:" + g_MaxConnectionErrorThreads.replace(/, +$/g, "") ,"Increase the number of connections available per this article: <a TARGET=_blank href='http://msdn.microsoft.com/en-us/library/7af54za5.aspx'>Managing Connections</a>");

}

/////////////////////// Managed code helpers ///////////////////////////////////

// Find where mscorwks is loaded and then load SOS from the same place

function LoadSOS()

{

var abSuccess;

var begPos;

var endPos;

var CmdOutput;

abSuccess = false;

CmdOutput = g_Debugger.Execute("lmv m mscorwks");

begPos = CmdOutput.indexOf("Image path:");

if (begPos>0)

{

begPos=begPos+11; // skip over Image path:

endPos=CmdOutput.indexOf("mscorwks.dll");

Manager.Write ("<b>Loading SOS from: </b>" + CmdOutput.substr(begPos,endPos-begPos) + "<br>");

CmdOutput = g_Debugger.Execute("!load " + CmdOutput.substr(begPos,endPos-begPos) + "sos.dll");

if (CmdOutput.length == 0)

{

abSuccess=true;

}

else

{

Manager.Write( "Cannot load Managed Debugging extensions. Error: " + CmdOutput + "<br>" );

}

return abSuccess;

}

/////////////////// Class - ManagedFieldOffsets

// this class uses expando properties and managed helpers to cache offsets

// in managed classes.

// usage: var obj = new ManagedFieldOffsets(new Array("_ServicePoint", "_Uri", "_HttpRequestHeaders", "_Proxy"),"system.dll!System.Net.HttpWebRequest");

// aArray is an array of fields you are intersted in. It will create properties

// of the same name and look up the offset value for this field

// the second parameter aObject is the location and name of the managed object.

// you then can use the obj as defined above like this obj._Proxy to get the offset values

function ManagedFieldOffsets( aArray, aObject )

{

var aStr;

var i;

var aCmdOutput;

// get the Class information

aCmdOutput = g_Debugger.Execute("!DumpClass " + getEEClass(aObject));

for(i in aArray)

{

this[aArray[i]] = findOffset(aCmdOutput,aArray[i]);

}

function getManagedLongVal( aObjAddr, aOffset )

{

return parseInt(g_Debugger.Execute("dt long " + aObjAddr + "+" + aOffset));

}

////////////////// Helper Function - findOffset

// given a string, extract the field offset specified

// this is really an excercise it string parsing

function findOffset(aCmdOutput, aField)

{

var retStr="";

var aFoundIdx;

var aBOL;

var sArray;

// find the field

aFoundIdx = aCmdOutput.indexOf(aField);

if (aFoundIdx != -1)

{

// look backwards from there for the offset.

// find the previous line

aBOL = aCmdOutput.lastIndexOf("\n",aFoundIdx)+1;

//need to strip all but one space from the string and make an array of the column values

sArray = normalizeWhitespace(aCmdOutput.substr(aBOL, aFoundIdx-aBOL)).split(" ");

// 7 columns total, offset is third column (0 based array)

retStr=sArray[2];

//Manager.Write("<br> 0:" + sArray[0] +"<br> 1:" + sArray[1] +"<br> 2:" + sArray[2] +"<br> 3:" + sArray[3] + "<br>");

}

return retStr;

}

////////////////// Helper Function - getEEClass

// given a string, extract the EEClass address

// theClass name is direct input to !Name2EE must be MODULENAME!CLASSNAM

function getEEClass( theClassName )

{

var retStr="";

var aCmdOutput;

var aFoundIdx;

var aEOL;

aCmdOutput = g_Debugger.Execute("!Name2EE " + theClassName);

aFoundIdx = aCmdOutput.indexOf("EEClass:");

if (aFoundIdx != -1)

{

aEOL = aCmdOutput.indexOf("\n",aFoundIdx);

retStr=trim( aCmdOutput.substr( aFoundIdx + "EEClass:".length , aEOL - (aFoundIdx + "EEClass:".length)));

}

return retStr;

}

// utility functions

// utility function removing leading and trailing white space

function trim(stringToTrim) {

return stringToTrim.replace(/^\s+|\s+$/g,"");

}

// utility function removing extra whitespace and replaceing it with a single space

// for example " some text here" will be converted to " some text here"

function normalizeWhitespace(stringToTrim) {

return stringToTrim.replace(/\s+/g,' ')

}

// calculate and save offsets for the Object fields we need

// this is how you would get the information in the debugger with the sos.dll extension

//0:000> !Name2EE system.dll!System.Net.HttpWebRequest

//Module: 7a726000 (System.dll)

//Token: 0x020003d0

//MethodTable: 7a7681d4

//EEClass: 7a768154 <<----- This is it!

//Name: System.Net.HttpWebRequest

//0:000> !DumpClass 7a768154

//Class Name: System.Net.HttpWebRequest

//mdToken: 020003d0 (C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)

//Parent Class: 7a768574

//Module: 7a726000

//Method Table: 7a7681d4

//Vtable Slots: 2c

//Total Method Slots: da

//Class Attributes: 102001

//NumInstanceFields: 45

//NumStaticFields: 7

// MT Field Offset Type VT Attr Value Name

//790fd0f0 400018a 4 System.Object 0 instance __identity

//7a7978b0 4001d3b 14 System.Int32 1 instance m_AuthenticationLevel

//...

//7a769fd8 4001f19 40 ....Net.ServicePoint 0 instance _ServicePoint

//...

//7a757bf0 4001f22 5c System.Uri 0 instance _Uri

//////////////////////////////// Non-generic managed object helpers....

// Given an address of an object get the service point object address

function getSvcPointFromReq(aHttpObjAddr)

{

var aRetStr = "";

var aCmdOutput;

aCmdOutput = g_Debugger.Execute("dd " + aHttpObjAddr + "+" + g_HttpObjectOffsets._ServicePoint + " L1");

aRetStr = aCmdOutput.substr(aCmdOutput.length - 9,8);

return aRetStr;

}

function getUriString(aObjAddr)

{

//strip the first 9 chars off (the address)

return g_Debugger.Execute("du poi(poi(" + aObjAddr + "+" + g_HttpObjectOffsets._Uri + ")+" +g_UriObjectOffsets.m_String + ")+" + g_StringObjectOffsets.m_firstChar).substr(9);

}

// obj is an address we can do !do on to get information from

function checkObj(aObjAddr, threadID)

{

//get service point and see if it is OK.

var bSvcPointErrorOrWarning = false;

var maxConn;

var currConn;

var aCmdOutput;

var i;

var aFoundIdx;

aCmdOutput = getSvcPointFromReq(aObjAddr);

if( (aCmdOutput!="00000000") && (aCmdOutput!="????????"))

{

maxConn = getManagedLongVal( aCmdOutput, g_SvcPointObjectOffsets.m_ConnectionLimit);

currConn = getManagedLongVal(aCmdOutput, g_SvcPointObjectOffsets.m_CurrentConnections);

if(currConn>=maxConn)

{

// report out info and error

g_MaxConnectionErrorThreads = g_MaxConnectionErrorThreads + "<a href='#ThreadAnchor" + threadID + "'>" + threadID +"</a>, ";

Manager.Write("<br> <font color='red'> HttpRequest URI:" + getUriString(aObjAddr) + " ServicePoint - ConnectionLimit:" + maxConn + " CurrentConnections:" + currConn +"</font><br>");

}

else

{

if(((currConn)*2)>=maxConn)

{

// report out info and error

g_MaxConnectionWarningThreads = g_MaxConnectionWarningThreads + "<a href='#ThreadAnchor" + threadID + "'>" + threadID +"</a>, ";

Manager.Write("<br> <font color='orange'> HttpRequest URI:" + getUriString(obj) + " ServicePoint - ConnectionLimit:" + maxConn + " CurrentConnections:" + currConn +"</font><br>");

}

Troubleshooting code that uses the HTTP protocol

jpsanders — Wed, 25 Jun 2008 21:02:00 GMT

Overview

You can solve issues encountered when using the HTTP protocol by applying the principles in this article. By simplifying the issue into a small reproducible problem you can effectively troubleshoot and resolve most problems encountered in code that uses the HTTP protocol. Crucial steps to troubleshooting code are: Gather Information, Simplify the Problem and Detailed Troubleshooting.

Technologies Covered

WinHttp, WinInet, Internet Explorer, XMLHTTP (MSXML), ServerXMLHTTP, System.Net classes such as WebClient, HttpWebRequest, SOAP classes, WebServices, System.Xml classes such as XmlUrlResolver, SoapHttpClientProtocol.

Error Messages Encountered (so search will find this document)

The operation timed out, 80072EE2, -2147954402, ERROR_WINHTTP_TIMEOUT, ERROR_WINHTTP_INVALID_SERVER_RESPONSE, The server name or address could not be resolved, 0x80072EE7, -2147012889, ERROR_WINHTTP_NAME_NOT_RESOLVED, 0x80072EFD

-2147012867, A connection with the server could not be established, -2146697208 (800c0008), The download of the specified resource has failed, Cannot connect to Remote server, Access is Denied, System error: -2146697191, 401, 500, 407, Server refused connection, Http protocol violation, Unauthorized, Cannot find server.

Step 1. Gather Information

You may solve the issue or get close to the root cause quickly by gathering information about the problem. Then you can continue to the Simplify the Problem section of this document to further isolate the issue and determine where the problem is occurring. Do this only after Gathering the information listed below.

1. What HTTP Verb are you using (EX: GET, POST, PUT)?

2. What is the destination URL?

3. Did this ever work? What changed if it did work previously (moved to production server, applied Service pack or Anti Virus update)?

4. What technology are you using to reach the URL that fails?

5. Can you reach the destination URL with another Technology such as Internet Explorer, other Browser, scripted Http Components, a simple console application?

6. Can you reach ANY URL from the computer (Perhaps DNS is not set, ports blocked, or you need a proxy setting)?

7. What is the failure (exact error message)?

8. What is the call stack of the failure?

9. Is this a Desktop or service based application (ASP and ASP.NET is a service)?

10. Are you running with the latest Service Packs and fixes etc…(Go to Windows Update)?

Step 2. Simplify the Problem

Armed with the answers from Step 1, reproduce the problem with the simplest code possible. In the process of simplifying the issue you may discover the issue is not in the technology you are using but a problem being obscured by your program’s logic or system configuration. For example, In one case I assisted in, numerous troubleshooting actions had been taken in the ASP pages to determine what was going wrong. When I was brought in to assist, I simply asked, “Can you hit the target URL in a Browser”? The answer (“no”) instantly revealed to all of us that the issue was the DNS server setting on that machine and not the code.

How to simplify the issue:

1. Duplicate the issue with a technology that is easier to troubleshoot such as the Web Browser or System.Net 2.0 which has the ability to log the HTTPS traffic.

2. If the problem occurs with HTTPS, see if you can reproduce with HTTP as well then troubleshoot HTTP traffic instead of HTTPS traffic (which is securely encoded). If you cannot duplicate the issue with HTTP then the problem may lie with your HTTPS configuration.

3. If this is a service based application, such as ASP or ASP.NET, move the code out of the Service and test. If it works, then the issue is not with the technology you are using. The issue is how you have your service configured or the context it is running in.

4. Try simple code from script or a command line application to see if that succeeds. Then use that same simple code in your application and test. If it fails in either case you have a much smaller problem to deal with.

5. Simplify the code by executing the simplest possible HTTP call in a command line script or console application. Start with using a HEAD or GET verb to a resource (perhaps an image or static file on the server) and add complexity until you get the failure.

Below are some code samples you can use to simplify the problem. Simplify! For example if your application is doing a POST verb, start with a GET verb to start your troubleshooting and then only if you cannot duplicate the issue using the GET verb, move to the POST verb.

VBScript samples:

Dim aRequest

Set aRequest = CreateObject ("Microsoft.XMLHTTP")

aRequest.Open "GET","http://www.microsoft.com/",False

aRequest.Send