When using a WebRequest, one important thing to keep in mind is how many connections are allowed to be made to the same server.  The maxConnection setting will affect how many connections you can concurrently have to a given server.  When you set up a proxy, System.Net sees the proxy as the server and so it will limit the number of concurrent connections to that instead of all the various servers.

For example, doing something like this:

HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(requestUri);
wr.Proxy = new WebProxy("http://myproxy");

will cause the connection to funnel through a proxy.  This means that from the IIS perspective, the “client” is connecting to the proxy as the server.  So the number of connections, as set by maxConnection, will be applied to that proxy, not the Uri that it is eventually going to.  This is based off the RFC: RFC 2616, the RFC for Hypertext Transfer Protocol -- HTTP/1.1:

"Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion."

So a few suggesting of what we could do:

- we could just increase maxConnection to 500 or 800 or whatever would be needed, knowing that the proxy can handle that much and pass the requests through.

- We could switch to using an NAT and route the requests through the proxy that way, using Big IP or Cisco Local Director or something like that.  Then we wouldn’t set the proxy on the HttpWebRequest and so it would not see them all as going to one machine.

If you have other suggested methods to solve this, or any thoughts around it, I’d like to hear them.