Wenlong Dong's Blog : Performance

Crack Throttling: Debugging WCF TimeoutException for a Middle-Tier Application

wdong — Sun, 06 Sep 2009 10:04:00 GMT

Recently I helped a customer with a few friends to debug a TimeoutException issue happened in their middle-tier which uses WCF proxies. The issue turned out to be simple but the experience was quite interesting and I would like to tell you how we proceeded with it.

Problem

The customer’s scenario is a multi-tier application as following:

ASPX clients [AC] ó ASPX invokes WCF clients [FE] ó WCF service (Basic Http with Binary) [BE] ó Backend services (over NamedPipe/HTTP) [BS]

The “ASPX clients” ([AC]) are web clients for a load testing. They send about 60 – 100 concurrent requests to the frontend ASP.NET web pages ([FE]) which are hosted on a front-end server machine. The web pages use WCF proxies to invoke the backend WCF service ([BE]) which is hosted on a back-end server machine. And then the WCF service invokes other backend services ([BS]). All is based on request/reply message exchange pattern.

After a while, the front-end ([FE]) starts to get TimeoutException for some requests as showing in the WCF trace log:

<ExceptionType>System.TimeoutException, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>

<Message>The HTTP request to 'http://co1wacwb201/Conversion.svc' has exceeded the allotted timeout of 00:00:20. The time allotted to this operation may have been a portion of a longer timeout.</Message>

at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)

at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.End()

at System.ServiceModel.Channels.RequestChannel.EndRequest(IAsyncResult result)

at System.ServiceModel.Dispatcher.RequestChannelBinder.EndRequest(IAsyncResult result)

at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishSend(IAsyncResult result, Boolean completedSynchronously)

The SendTimeout of the binding was set to be 20 seconds. However, based on the statistics, the backend service ([BS]) should not take 20 seconds. The number of concurrent requests is not very high. So why did this happen?

Debugging

ASP.NET/WCF Settings

By working with the customer, we first checked the different throttling settings for both ASP.NET and WCF to see whether everything is running as expected. The following is the checklist that we have used.

1) WCF Service Throttling

The first step is to make sure that the backend has big enough WCF service throttling settings. It turns out that the backend has used quite large settings:

· MaxConcurrentCalls: it’s set to 100.

· MaxConcurrentSessions: it’s set to 100.

· MaxConcurrentInstances: it’s set to 200.

These settings should be big enough for the scenario given that number of concurrent clients. Actually the binding is BasicHttpBinding. So MaxConcurrentSessions should not matter at all here.

2) InstanceContextMode/ConcurrencyMode

As per my early blog, the default InstanceContextMode is PerSession and the default for ConcurrencyMode is Single. However, PerSession is equivalent to PerCall for BasicHttpBinding. So there should be no contention here if everything is used as default. After checking the code snippet, it turns out that the settings are not default:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single,

ConcurrencyMode = ConcurrencyMode.Multiple)]

public sealed class MyService

This usage allows concurrent requests for the same service operation and it is also fine.

3) ASP.NET Threading Settings

Now the next thing that I thought was to check the ASP.NET threading settings as per the whitepaper. The server machine is dual-proc/4-core and thus 8-way box. So the following default should be enough:

<system.web>

</system.web>

In this case, ASP.NET would be able to execute 12 * 8 = 96 concurrent requests. They should be also big enough.

4) MaxConcurrentRequestsPerCPU

As per Thomas Marquardt, this setting has a default value of 12 and it’s available through the following registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0

“MaxConcurrentRequestsPerCPU”=dword:0000000C

By default, this registry does not exist, but the value is 12. It’s also settable from the ASP.NET application pools setting file %windir%\Microsoft.NET\{Framework|Framework64}\v2.0.50727\Aspnet.config:

<system.web>

</system.web>

This setting again is good enough for the scenario as the throttling setting is actually 96.

5) Client Settings

To this point, it does not look like any other server settings would matter here. So we need to check whether all client requests were pushed to the backend WCF service. So we asked the customer to try to increase the “maxconnection” setting as per my previous blog entry:

<system.net>

</connectionManagement>

</system.net>

The customer did that but the problem did not go away after this setting.

To this point, I have run out of the throttling settings that I can think of. So the next step is to gather more data for the system.

Trace Log

We asked the customer to enable Warning and Error level WCF tracing to see whether the WCF service has generated any exception and silently fails the client, though it is quite unlikely to happen in usual case. As expected, the log did not show anything interesting. Only the front-end log showed the TimeoutException and the callstack that I pasted above. So the log did not help here.

WinDBG Dump

Process dump can usually tell us a lot of information of the server process. We can examine the managed objects, states, threads, and call stacks. The caveat is that a dump only reflects a single data point of the process and it does not have overall progress. So we ask the customer to generate the process dump with WinDBG.

We examined the objects in the heap and searched for the corresponding WCF objects such as ThreadPool, FlowThrottle, InputQueue, etc by using the SOS debugging extension. We did not find anything wrong with those. We noticed that there were 9 I/O threads:

00000000`0961f160 000007fe`f8aaf0ac mscorwks!ThreadpoolMgr::CompletionPortThreadStart(void * lpArgs = 0x00000000`00000000)+0xee

00000000`0961f200 00000000`7737466d mscorwks!Thread::intermediateThreadProc(void * arg = 0x00000000`00000000)+0x78

There are totally 48 CLR worker threads:

00000000`0497fd70 00000000`7737466d mscorwks!Thread::intermediateThreadProc(void * arg = 0x00000000`00000000)+0x78

And there are about 41 other threads. With this data, we could not figure out the root cause of the timeout issue yet, especially when the process dump is only at a single time stamp. So at this moment, we moved on to look for other data first.

NetMon Capture

In many cases, Network Monitor capture data can provide very detailed information and can be very useful for analyzing network performance and reliability issues. We can analyze the TCP packages and frames and examine the conversations to find out how the data were transferred between the client and the server. The tricky part is that it took about an hour to reproduce the problem and the captured data size was pretty big (around 500MB). Searching for correlated requests between the client and the service was quite a challenge. This is another hard route as using WinDBG to debug the problem. So we skipped this step for this investigation for now.

Client Latency Distribution

We further gathered the distribution data for the client latency as the following graph. It shows that that system worked just fine in the first 92 seconds. All requests completed in sub-seconds. Then it started to show erratic behavior with longer latencies from 4 up to 12 seconds in the subsequent half minute. We need to understand better why this started to happen. So next we really needed to check the distribution for other data types on the front-end and the back-end machines.

Performance Counter Data

A more useful view of the system is to use the data for the ASP.NET and WCF performance counters. Performance counters are very light-weighted and it’s easily to gather the data without affecting the product performance much.

ASP.NET performance counters are turned on by default. We just need to enable WCF performance counters. In order to enable WCF ServiceOnly performance counters, you can add the following to the application configuration file (such as web.config in this case):

<system.serviceModel>

</system.serviceModel>

The following is a list of interesting performance counters that we used to investigate the issue:

Perf Counter Category	Perf Counter Name
ASP.NET	Request Execution Time
ASP.NET	Requests Queued
ASP.NET	Request Wait Time
ASP.NET Applications	Requests Executing
ASP.NET Applications	Pipeline Instance Count
ServiceModelService 3.0.0.0	Call Duration
ServiceModelService 3.0.0.0	Calls Outstanding

To find out more details, you can check the following two articles:

1) ASP.NET Performance Monitoring guide: http://msdn.microsoft.com/en-us/library/ms972959.aspx

2) WCF Performance Counters: http://msdn.microsoft.com/en-us/library/ms735098.aspx

Here is the data that we got from the customer:

This graph gives us very interesting data:

1) The CPU usage on both the front-end server FESERVER1 and the back-end server BESERVER002 is low. So there shouldn’t be traffic congestion/starvation problem.

2) There are no queued requests in the ASP.NET queues because “Requests Queued” counters are 0 on both machines.

3) Both the “Pipeline Instance Count” and the “Requests Executing” counters are pretty high on the front-end server FESERVER1. They are 91 and 69 in average respectively and they matched the number of concurrent clients.

4) However, the above counters are very low on the back-end server BESERVER002. They are 11 and 7 in average respectively. This is a very interesting data point.

From the above data, basically all of the requests are accumulated in the front-end server and they cannot be pushed all the way to the backend. The requests did not reach to the ASP.NET layer on the backend server. Something must be set incorrectly. What could have blocked the requests from being pushed to the backend server?

We checked the code usage of WCF client proxies in the front-end. The code did not use cached proxies. In this case, there should be no contention in the code. We have to check the configuration again.

Configuration Rechecking

Based on the analysis, we believed that that requests were blocked by the throttling in the front-end. We still have the doubt of the System.Net’s “maxconnection” setting.

My friend Mauro Ottaviani spotted an incorrect setting in the machine.config:

</section>

The highlighted line was incorrect placed in the section handler definition part of machine.config instead of under the <system.net> node. The system simply ignored the above <add> element and thus used the default values.

Resolution

After fixing this setting, the customer re-ran the scenario and the test passed without getting into the TimeoutException again. Here is the updated performance counter data:

As from this new graph, we noticed the following:

1) The number of concurrent clients has been increased. The max "Requests Executing" now is 457.

2) The system reached much higher concurrent requests on both the front-end and the backend servers, which is great. For example, the average “Requests Executing” is 42 now. This is a good symptom.

3) There was some high latency in the backend server. The max of “Calls Duration” for WCF service is 5.05 seconds. It is not far from the client timeout setting 20 seconds. For the front-end, it can reach 11 seconds. This may be a problem for the long running system. The backend service must have done some heavy-weighted work. We need to either increase the timeout or optimize the backend service logic more to make it faster.

4) The good news is that much higher CPU usage on both frontend and backend were observed. Apparently the system runs much more efficiently. Also higher throughput was observed from the system which was not recorded in the performance counters.

At this point, we have got the problem solved.

WCF 4: Higher Default Throttling Settings for WCF Services

wdong — Sun, 26 Jul 2009 09:36:00 GMT

It has been quite a while that I did not write anything. This is because I was fully focusing on WCF 4. Now that we have got most of the work done, it is good time to continue my writing. Starting from this blog entry, I will show you performance related changes in WCF 4. Please let me know what you think about WCF performance and what you would like to see in future releases.

When WCF was first released in .NET Framework 3.0, the default values for WCF Service throttling settings such as MaxConcurrentSessions and MaxConcurrentCalls were set very conservatively. For example, here are the default settings in 3.0:

MaxConcurrentSessions (default: 10)
MaxConcurrentCalls (default: 16)
MaxConcurrentInstances (default: 26)

These settings are not enough for many real production environments. With limited concurrency due to the low settings, the power of the production server is normally not well utilized. So people have to bump up these values for enterprise applications. However, it is hard for people to figure out how to set these values correctly. Most people would set very high numbers or even int.MaxValue to workaround the throttling which defeats the purpose of the throttling. The main purpose for the throttling settings can be classified into the following two aspects:

1) Controlled resource usage: With the throttling of concurrent execution, the usage of resources such as memory or threads can be limited to a reasonable level so that the system works well without hitting reliability issues.

2) Balanced performance load: Systems always work in a balanced way when the load is controlled. If there are too much concurrent execution happening, a lot of contention and bookkeeping would happen and thus it would hurt the performance of the system.

In WCF 4, we have revised the default values of these settings so that people don’t have to change the defaults in most cases. Here are the main changes:

· MaxConcurrentSessions: default is 100 * ProcessorCount

· MaxConcurrentCalls: default is 16 * ProcessorCount

· MaxConcurrentInstances: default is the total of the above two, which follows the same pattern as before.

Yes, we use the multiplier “ProcessorCount” for the settings. So on a 4-proc server, you would get the default of MaxConcurrentCalls as 16 * 4 = 64. The basically consideration is that, when you write a WCF service and you use the default settings, the service can be deployed to any system from low-end one-proc server to high-end such as 24-way server without having to change the settings. So we use the CPU count as the multiplier.

We also bumped up the value for MaxConcurrentSessions from 10 to 100. Based on customer’s feedback, this change fits the need for most applications.

Please note, these changes are for the default settings only. If you explicitly set these settings in either configuration or in code, the system would use the settings that you provided. No “ProcessCount” multiplier would be applied.

Scale WCF Application Better with Asynchronous Programming

wdong — Mon, 09 Feb 2009 09:53:00 GMT

WCF is a framework that is asynchronous inside out. Internally, asynchronous patterns are used to achieve non-blocking calls between different layers inside WCF. Externally WCF exposes the asynchronous patterns for both client and service programming.

Why Asynchronous Pattern?

From performance perspective, it’s not always ideal to use asynchronous programming if you don’t have to. Any asynchronous pattern involves at least some thread switches and thus causes delays to requests. So you should not abuse it if you don’t have to.

However, for blocking calls such as I/O operations or remote calls, you should always thinking about using asynchronous patterns. In those cases, you would be able to reduce number of threads used in the system while waiting for operations to complete. Here are a few cases where you would like to use asynchronous patterns:

1) A WCF service operation is waiting for a database operation to complete.

2) A WCF service operation calls into another WCF client proxy of a backend WCF service.

3) A UI thread waits for data from WCF client calls.

4) Any other case when the calling thread is blocked to wait for a slow operation but non-CPU bound operation to complete.

If you are using synchronous calls for the above cases, you would clearly see the threads are blocked from a dump of the process. This is wasting resources and you are preventing more requests from being processed if the system is capable of. This becomes more serious if you are holding hundreds or thousands of threads. So whenever you are seeing a thread blocked waiting for something, you would have to think about why that is the case? Why can’t the thread go away and come back with a different thread when there is real work to do?

Of course in reality, there are a small number of threads doing bookkeeping work, for example, they are doing event wait handling, finalizing, garbage collection, AppDomain unloading, ThreadPool gating, timer handling, etc. The rest of threads are worker threads or I/O threads that do the real work and they are what you need to pay attention to.

Basic Asynchronous Pattern with IAsyncResult

There are a few common asynchronous programming patterns with .NET Framework. Here is a rough list:

1) Begin/End pattern with IAsyncResult

2) Raw Callback model using a delegate as in C++ world

3) Implicit callback model with synchronous APIs

WCF advocates the first approach from the high level and uses it in most of its implementation. For performance critical parts, WCF uses 3) to avoid extra GC costs dealing with IAsyncResult objects. Here I just want to describe the basic idea of how to use 1).

The basic pattern of using IAsyncResult is to expose a pair of Begin/End operations from your type as following:

public class IAsyncWork

{

public IAsyncResult BeginDoWork(MyPropertyBag propertyBag, AsyncCallback callback, object state);

public WorkResult EndDoWork(IAsyncResult result);

}

When a caller invokes the methods of this interface, it supplies some custom arguments (here it’s “propertyBag”), a callback delegate, and the state object that will be consumed by the callback. The callback delegate wraps a callback method of the caller with the following type:

public delegate void AsyncCallback(IAsyncResult ar);

For best performance, here are two principles when you call/implement the above asynchronous pattern:

· Principle 1: Do not do heavy-weighted work inside the Begin method (which is BeginDoWork above).

The reason for this is that you should return the calling thread as soon as possible so that the caller can schedule other work. If it’s a UI thread, the application needs to use the thread to respond to user inputs. You should always put heavy operations in a different thread if possible.

· Principle 2: Avoid calling End method (which is EndDoWork above) on the same thread of the Begin method.

The End method is normally blocking. It waits for the operation to complete. If you implement the End method, you would see that it actually calls IAsyncResult.WaitHandle.WaitOne(). On the other hand, as a normal implementation (for example, the sample attached in this Blog entry), this WaitHandle is a delay allocated ManualResetEvent. As long as you don’t call it, it would be not allocated at all. For fast operations, this is pretty cheap. However, once End is called, you would have to allocate it. The right place to call End is from the callback of the operation. When the callback is invoked, it means that the blocking work is really completed. At this point, you can call End to get data retrieved without sacrificing performance.

In online WCF samples, you would find a typical implementation IAsyncResult. I also included one in the sample attached.

Service Asynchronous Pattern

WCF provides asynchronous service operation contract support. An asynchronous service operation can be defined as following:

[ServiceContract

public interface IAsyncOrderService

{

[OperationContract(AsyncPattern=true)]

IAsyncResult BeginGetOrders(int numOrders, AsyncCallback callback, object state);

Order[] EndGetOrders(IAsyncResult result);

}

As you can see, other than the ServiceContract/OperationContract attributes, the interface is exactly the same as standard asynchronous patterns. One thing to note is that it is not your code but WCF that calls into these methods. So the operation names have to follow the Begin/End convention.

Another minor thing to note is that, if you also have a synchronous operation for the service that matches the signature of the asynchronous operation as following, WCF would call this synchronous operation instead of the asynchronous operation:

Order[] GetOrders(int numOrder);

You may have control of which one to call in future releases.

Asynchronous Database Access

Now it’s time for you to think about your own service implementation. In order to leverage the asynchronous advantage of WCF, you need to implement your service to be purely asynchronous so that no threads are blocking.

In the sample that I attached to this blog, I used SQL database access with System.Data API as an example. For simplicity, I used SQL Express which is available for free. To get asynchronous support, you need to use the “Asynchronous Processing” property of the connection string as following:

const string ConnectionString = @"Server=LOCALHOST\SQLEXPRESS;Database=MyOrderDatabase;

Integrated Security=True;

Asynchronous Processing=True";

Once you have the asynchronous support from the SqlConnection, you can invoke the asynchronous API SqlCommand.BeginExecuteReader as following:

IAsyncResult sqlAsyncResult = this.command.BeginExecuteReader(sqlGetOrdersCallback, this);

With this, you would have a good asynchrony for your service side.

Client Asynchrony

On the client side, you can also asynchronous pattern to access the service. When you invoke the service operations, you would call into the asynchronous methods of the service contract. For example, the sample calls it in the following way:

IAsyncResult result = client.BeginGetOrders(driver.numOrders, onGetOrders, this);

It supplies a callback delegate that handles the response when the operation is completed.

One important thing to note is that WCF is a fully decoupled platform. This means that the client-side asynchrony has nothing to do with the service-side asynchrony. So an asynchronous client can basically invoke either a synchronous or an asynchronous service. The decoupling happened at the transport layer when data is serialized or deserialized as byte arrays and being transmitted through the transport.

If you are building N-tier WCF services, you would have WCF service operations invoking WCF client proxies for other backend services. In this case, you would need to make sure that the middle-tier (routing layer) has asynchronous service operation invoking asynchronous WCF proxy operations. In this way, your middle-tier won’t run out of threads when processing many slow operations.

Exception Handling

The logic of exception handling for asynchronous programming is more complicated than synchronous programming. Basically when an exception is thrown from a callback thread, you should not let it bubble up to the caller, which is almost the top of the callstack of a worker thread or an I/O thread. If you do so, your application would blow up as no other code would catch the exception and handle it.

To best handle an exception happened from the callback thread, you would need to record the exception as a state somewhere, for example, in the AsyncResult as showing in the sample. At some point, when the End method is invoked, this exception will be thrown on that thread and thus causes the exception to be thrown on the caller’s thread.

In the sample attached, we showed this once in SqlGetOrdersAsyncResult. In its callback “OnSqlGetOrdersCallback”, it catches the exception and passes to the AsyncResult.Complete method which then stores the exception there. When the SqlGetOrdersAsyncResult.End is invoked from WCF, this exception would be rethrown from AsyncResult.End. In this way, WCF would be able to catch the exception internally and convert it into a fault and send it to the client side.

Sample Setup

The attached sample implements a simple performance test that demonstrates the 3-tier model as showing above: WCF client, WCF service, and SQL database. The code is pretty self-explanatory and I copy the readme file here:

Prerequisite

1) It’s better to use Windows 2008 so that you can leverage the asynchronous WCF HttpModule/HttpHandler that were released in .NET 3.5 SP1. Check here.

2) Visual Studio 2008 or above installed.

3) SQL Express or SQL server installed

4) IIS, IIS Management tools, and WCF HTTP Activation components installed.

Steps to Run the Test

1) Open the solution file "WCFOrderService.sln" with VS 2008 and compile the whole solution with "Release" type.

2) Run "SQL\SetupSQL.bat" to create database "MyOrderDatabase".

3) Run "OrderCreator\bin\Release\OrderCreator.exe 1000" to insert 1000 distinct orders into database.

4) Open IIS Manager, create a virtual application that points to the full path of "OrderService".

5) Use IIS Manager to configure the default ApplicationPool to have identity "OrderPerfUser". This user was created in step 2) above.

6) Browse the OrderService.svc file in the browser. Make sure that you can see the WCF service correctly responds to the request from browser.

7) Run the client "OrderClient\bin\Release\OrderClient.exe"

8) Tune different parameters in the OrderService\web.config and OrderClient\bin\Release\OrderClient.exe.config

9) Use the tool "Tools\WcfAsyncWebUtil.exe" to configure WCF to use the async HttpModule and increase the ASP.NET registry "MaxConcurrentRequestsPerCpu". Run the test again to see the difference.

10) Open PerfMon to monitor requests such as "Calls Outstanding" counter.

Why Only Two Concurrent Requests for WCF Load Testing?

wdong — Sun, 08 Feb 2009 10:05:00 GMT

This is not a new topic in the world of traditional web service usage. But I was still asked by people who use WCF. So I think it would be valuable to post a blog entry here. One of our customers asked me the following question:

“We are using WCF in our application on Windows 2008 Server. We found the web role instance only have two concurrent threads to handle the incoming requests no matter how many requests we throw into it. We tried 50, 100, 200 clients sending requests to the WCF service at the same time. But web role only have two threads to handle it. We used the following settings in the web.config to set up the number of threads for WCF. But it seems not work. Why is that?”

<serviceThrottling maxConcurrentCalls="32" maxConcurrentInstances="100" maxConcurrentSessions="20"/>

When I looked at this question, my first response is that: the client did not really send enough requests to the server. Why is that? Here are the reasons:

1) If you use the synchronous WCF HttpModule/HttpHandler (installed by default), you would get the maximal number of concurrent requests (held by that number of ASP.NET worker threads) as 12 * [Number of CPU for the Server].

2) WCF throttling is specified above.

There is no other place that has a throttling of “2” on the server side. If you really want to know how many outstanding requests are processed by WCF, you can enable WCF performance counters and check “Calls Outstanding”. It will tell you that information.

Then the real problem should be on the load testing client. There is an apparent throttling on the client side that has default value 2: “maxconnection” from System.Net. This setting is according to RFC 2616, the RFC for Hypertext Transfer Protocol -- HTTP/1.1:

"Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion."

So for load testing purpose, you should increase this setting to an appropriate limit. You can find a lot of documents on how to set this setting. Let me simply re-iterate it here.

1) Using <connectionManagement> configuration

You can use the following configuration to tune this setting:

<add address="www.contoso.com"

maxconnection="16"/>

</connectionManagement>

You can use address=”*” to set the same setting for all remote addresses.

2) Setting it from code with ServicePointManager.DefaultConnectionLimit

You can also set the setting from code before you call into any web service:

System.Net.ServicePointManager.DefaultConnectionLimit = 16;

After this, you would see your client application generate more concurrent requests to the server.

Orcas SP1 Improvement: Asynchronous WCF HTTP Module/Handler for IIS7 for Better Server Scalability

wdong — Wed, 13 Aug 2008 10:19:00 GMT

Introduction

As mentioned in my last blog entry, for IIS-hosted WCF services, WCF holds the worker thread coming from ASP.NET until the whole request is completed to avoid a Denial of Service (DOS) attack. I also mentioned that on Windows 2008 Server, IIS7 has introduced the following registry setting to provide request throttling for all incoming requests:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0]

“MaxConcurrentRequestsPerCpu”=dword:0000000c

Based on this, in Orcas SP1 (.NET 3.5 SP1), WCF has implemented the asynchronous HTTP Module/Handler to allow better server scalability for high latency requests.

Asynchronous WCF HTTP Module/Handler

Besides the existing synchronous WCF HTTP Module/Handler types, WCF introduced the asynchronous versions. Together, WCF has the following four types implemented:

· Synchronous Module:

System.ServiceModel.Activation.HttpModule, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089

· Asynchronous Module (new):

System.ServiceModel.Activation.ServiceHttpModule, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089

· Synchronous Handler:

System.ServiceModel.Activation.HttpHandler, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089

· Asynchronous Handler (new):

System.ServiceModel.Activation.ServiceHttpHandlerFactory, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089

Asynchronous HTTP Module

The new type ServiceHttpModule implements the System.Web.IHttpModule and it registers the async event HttpApplication.AddOnPostAuthenticateRequestAsync with a pair of async handlers:

static public IAsyncResult BeginProcessRequest(object sender, EventArgs e, AsyncCallback cb, object extraData)

static public void EndProcessRequest(IAsyncResult ar)

When a request comes in, WCF does some validation and returns from BeginProcessRequest immediately after passing the request up to internal processing logic. In this way, the worker process is immediately released.

If the service operation is asynchronous and it is waiting for slow I/O operations, the whole stack does not hold any thread. The only performance concern would be the memory usage to hold objects used to process the request. This allows many concurrent requests to be served by WCF without using a lot of threads and thus greatly improves the scalability of the service. This is very helpful when completing each request takes significant amount of time and many clients are served. In some private testing, WCF can easily support more than 10K concurrent slow requests, which is not possible with the synchronous HttpModule.

Asynchronous HTTP Handler Factory

You might have already noticed that WCF implements the Handler Factory instead of the asynchronous Handler directly. This allows more flexibility in the future to support different types of HTTP Handlers. The ServiceHttpHandlerFactory creates a stateless asynchronous HTTP handler of the following type which implements System.Web.IHttpAsyncHandler:

System.ServiceModel.Activation.ServiceHttpHandler

It implements the pair of asynchronous request handlers:

public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback cb, object extraData)

public void EndProcessRequest(IAsyncResult result)

Default Installation – Synchronous Version

Though the asynchronous version of HTTP Module/Handler is added in this release, the default installation is still the synchronous version. Why is this?

First of all, you should not use asynchronous version of HTTP Module/Handler on Windows 2003 Server (with IIS6) without knowing the risk. If you have many clients, the service would be easily DOS attacked by too much memory usage due to huge amount of pending requests queued up in WCF transport layer without the throttling inside IIS/ASP.NET.

Because of the limitation on IIS6, WCF did not change the setup to switch to purely asynchronous due to the ramifications between IIS6 and IIS7 for this quick release. This means that in order to use the asynchronous version, you have to perform some manual registration.

Don’t worry, I have provided a private tool to do this registration for you.

WCF Module/Handler Registration Tool

You can find the simple tool called WcfAsyncWebUtil.exe attached in this blog. Here is the usage:

Usage: WcfAsyncWebUtil <options>

Options:

/is

Install WCF Synchronous HTTP module and handler into IIS7 configuration

in Integrated Mode.

/ia

Install WCF Asynchronous HTTP module and handler into IIS7 configuration

in Integrated Mode.

List WCF HTTP module and handler registered into IIS7 configuration in

Integrated Mode.

/t [ <throttle> ]

Configure the throttling registry MaxConcurrentRequestsPerCpu for the ASP.NET

integrated pipeline to the value <throttle>. Default is 100. This argument is

used together with /ia only

For example, if you want to register asynchronous HTTP Module/Handler with the ASP.NET throttling to be 1000, you can run the following command:

WcfAsyncWebUtil.exe /ia /t 1000

To list the installed WCF HTTP Module/Handler, you can run:

WcfAsyncWebUtil.exe /l

WCF Request Throttling and Server Scalability

wdong — Mon, 21 Apr 2008 10:09:00 GMT

Two Threads per Request

In .NET 3.0 and 3.5, there is a special behavior that you would observe for IIS-hosted WCF services. Whenever a request comes in, the system would use two threads to process the request:

· One thread is the CLR ThreadPool thread which is the worker thread that comes from ASP.NET.

· Another thread is an I/O thread that is managed by the WCF IOThreadScheduler (actually created by ThreadPool.UnsafeQueueNativeOverlapped).

When you have a high latency request, you would see the following callstack in the debugger before the request is completed (with .NET 3.5):

0ee6ee8c 5094dad5 mscorlib_ni!System.Threading.WaitHandle.WaitOne()+0xa

0ee6ee8c 50951e3b System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.ExecuteSynchronous(System.Web.HttpApplication, Boolean)+0x8d

0ee6eeb4 65fe626d System_ServiceModel_ni!System.ServiceModel.Activation.HttpModule.ProcessRequest(System.Object, System.EventArgs)+0x143

0214e4f4 65fe3fd1 System_Web_ni!System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x5d

0ee6ef08 65fe804f System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0x41

0ee6efb8 65fe4df2 System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x63b

060f05dc 66003a92 System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x56

0ee6f040 65fd8022 System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+0x352

0ee6f0c4 65fd7e07 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x1f2

0ee6f0f4 01782374 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x1b

0ee6f138 6a2abf3c webengine!MgdGetCurrentNotification+0x236

0ee6f19c 74a22ea0 webengine!MgdGetPreloadedSize+0x4d

0ee6f1b0 74a23696 iiscore+0x2ea0

0ee6f560 6a2ac222 webengine!MgdCanDisposeManagedContext+0xb5

0ee6f570 015f1311 webengine!MgdIndicateCompletion+0x22

System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x289

This is the worker thread that comes from ASP.NET. Another thread is still processing the request in WCF:

01fbf0ac 0e9001c7 App_Web_rsfhd6l1!HelloWorld.SimpleService.Hello(System.String)+0x14

01fbf0ac 50b8d90b System_ServiceModel_ni!DynamicClass.SyncInvokeHello(System.Object, System.Object[], System.Object[])+0x3f

01fbf0ac 50b6d245 System_ServiceModel_ni!System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(System.Object, System.Object[], System.Object[] ByRef)+0x1fb

01fbf108 509137ad System_ServiceModel_ni!System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0xd5

01fbf148 509136a6 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0xad

01fbf174 50913613 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0x76

062b75a8 50913459 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0x33

062b75a8 50912257 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0x59

062b75a8 50911f8f System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0x127

01fbf1f0 509115ff System_ServiceModel_ni!System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean)+0xdf

01fbf394 5090f8c9 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(System.ServiceModel.Channels.RequestContext, Boolean, System.ServiceModel.OperationContext)+0x1ff

01fbf3dc 5090f35e System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext, System.ServiceModel.OperationContext)+0x169

01fbf42c 5090f2f1 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult)+0x5e

01fbf42c 50232d68 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(System.IAsyncResult)+0x41

01fbf42c 50904501 SMDiagnostics_ni!System.ServiceModel.Diagnostics.Utility+AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult)+0x28

01fbf468 50992b36 System_ServiceModel_ni!System.ServiceModel.AsyncResult.Complete(Boolean)+0xb1

01fbf4d8 50992215 System_ServiceModel_ni!System.ServiceModel.Channels.InputQueue`1+AsyncQueueReader[[System.__Canon, mscorlib]].Set(Item<System.__Canon>)+0x46

01fbf4d8 50991ffb System_ServiceModel_ni!System.ServiceModel.Channels.InputQueue`1[[System.__Canon, mscorlib]].EnqueueAndDispatch(Item<System.__Canon>, Boolean)+0x1f5

00000000 5091d7e5 System_ServiceModel_ni!System.ServiceModel.Channels.InputQueue`1[[System.__Canon, mscorlib]].EnqueueAndDispatch(System.__Canon, System.ServiceModel.Channels.ItemDequeuedCallback, Boolean)+0x6b

00000001 50977b7e System_ServiceModel_ni!System.ServiceModel.Channels.SingletonChannelAcceptor`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Enqueue(System.__Canon, System.ServiceModel.Channels.ItemDequeuedCallback, Boolean)+0x75

01fbf570 5094f396 System_ServiceModel_ni!System.ServiceModel.Channels.HttpChannelListener.HttpContextReceived(System.ServiceModel.Channels.HttpRequestContext, System.ServiceModel.Channels.ItemDequeuedCallback)+0x1ce

01fbf5b8 5094e4cf System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(System.ServiceModel.Activation.HostedHttpRequestAsyncResult)+0xc6

0216803c 5094defd System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()+0xdb

01fbf60c 5094dea5 System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()+0x19

01fbf638 50903be6 System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(System.Object)+0x29

01fbf674 50903b26 System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke2()+0x36

01fbf688 50903ab5 System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke()+0x4a

01fbf6bc 5090390f System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.ProcessCallbacks()+0x185

01fbf6e8 5090388b System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.CompletionCallback(System.Object)+0x6f

01fbf720 50232e1f System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0xf

01fbf720 79405534 SMDiagnostics_ni!System.ServiceModel.Diagnostics.Utility+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x2f

01fbf744 79e7c74b mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x60

01fbf758 79e7c6cc mscorwks!LogHelp_TerminateOnAssert+0x3433

01fbf7d8 79f00eca mscorwks!LogHelp_TerminateOnAssert+0x33b4

01fbf7f8 79f00e75 mscorwks!GetCLRFunction+0x1de16

Why is this? Is this a scalability issue for WCF? I will explain this below.

Asynchronous Request Processing

One of the most significant pieces for WCF is that it is highly asynchronous from bottom to top for the whole channel stack and the programming model. The advantage is apparent: applications have high flexibility by issuing operations asynchronously, doing some other stuff in parallel, and coming back to handle the completed operations once they are completed. This is also scalable when the asynchronous operations are I/O bound so that you do not need to hold extra threads to wait for operation completions.

ASP.NET also supports asynchronous request processing mechanisms through IHttpModule and IHttpAsyncHandler. With these interfaces, ASP.NET can push many concurrent requests up to the layers which have such asynchronous implementations. In this way, the whole stack need a small number of threads to process a large number of concurrent requests given appropriate queuing for the requests.

However, this also exposes a memory problem when this is on IIS 6.0. When there is unlimited number of incoming requests, all of the requests would be pushed up by ASP.NET to upper layers and it would cause significant memory consumption. Thus it would cause the server to be unavailable to do its work correctly.

So in the release of .Net 3.0 and 3.5, WCF implemented synchronous versions of HTTP module and handler instead of asynchronous ones. See the following for more details.

ASP.NET Threading on IIS 6.0

When ASP.NET is hosted in IIS 6.0, requests are handed over to ASP.NET on IIS I/O threads. ASP.NET uses CLR ThreadPool worker threads to handle requests. The CLR ThreadPool automatically adjusts the number of threads based on the server workload. In order to prevent from using too many threads or using too much memory, ASP.NET has a limit on the number of threads concurrently executing requests. This is controlled by the httpRuntime/minFreeThreads and httpRuntime/minLocalRequestFreeThreads settings. If the limit is exceeded, the request is queued in the application-level queue, and executed later when the concurrency falls back down below the limit. Thomas Marquardt has written an excellent blog on this:

http://blogs.msdn.com/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx (TM0707)

In .NET 2.0, the magic config “processModel/autoConfig” in machine.config automatically configures the ASP.NET threading settings for most applications. For those which require special settings need to be configured differently. The following KB article was provided as the general guidance:

http://support.microsoft.com/kb/821268 (KB821268)

However, there is no throttling for concurrent requests in ASP.NET on IIS 6.0.

WCF Request Throttling and Tuning

Because of lacking the throttling for concurrent requests in ASP.NET, WCF would get unlimited concurrent requests pushed up from ASP.NET if WCF had implemented asynchronous HTTP module/handler on IIS 6.0. It would cause severe memory usage problem. Because of this concern, WCF implemented the synchronous HTTP module/handler in 3.0 and 3.5 instead.

Here I only describe how the HTTP module works for WCF. The logic for the HTTP handler is similar. When a request comes in from ASP.NET, WCF grabs the request if it is a WCF request (by checking the registered BuildProvider of the extension to see whether it is WCF service BuildProvider). After that, it immediately switches to a new I/O thread and hold the ASP.NET worker thread until the request is completed. You may ask, why doesn’t WCF reuse the ASP.NET worker thread to handle the request until it is completed? Here are the reasons:

· As long as the ASP.NET worker thread is not returned back to ASP.NET, the request is not able to be completed. The client would be blocked.

· When the WCF operation is one-way, the request has to be completed before the service operation is invoked. So the ASP.NET worker thread has to return earlier.

· WCF supports people to complete a two-way operation earlier in a service operation when you call RequestContext.Close.

All of these require that WCF should not hold the ASP.NET worker thread once the request is claimed to be “completed”. At the same time, WCF has to hold the ASP.NET worker thread in a waited state before the request is completed due to the requirement of synchronous request processing logic. WCF relies on this ASP.NET thread throttling logic to limit concurrent requests.

Intuitively this seems to have scalability issue. Fortunately for most WCF services, especially high-throughput ones, this has quite low impact as long as the server is busy processing requests. This is demonstrated as the CPU usage of the server.

On multi-proc/multi-core servers, the default limit of concurrent requests may not be enough for the server to achieve best throughput. This would require larger limits for ASP.NET worker threads to be used to handle requests.

Throttling Tuning

In order to achieve high throughput on high-end servers, you would need to make the following throttling tuning:

· Increase ASP.NET thread throttling to allow more concurrent worker threads to handle requests

· Increase WCF service throttling to allow more concurrent requests

The latter reflects the change of the known WCF throttling settings such as MaxConcurrentCalls, MaxConcurrentInstances, and MaxConcurrentSessions. The former would require fine tuning of ASP.NET thread throttling as documented in the KB article KB821268 mentioned above. Basically ASP.NET does not execute more than the following number of concurrent requests:

(maxWorkerThreads*number of CPUs)-minFreeThreads

This is 12 by default on a single CPU machine. In order to get more threads to process requests, you need to increase maxWorkerThreads. Once you can get CPU fully loaded with the settings, setting “minWorkerThreads” to 2 or 3 would allow WCF to have best throughput. Here is sample setting in machine.config:

The above settings work for most WCF applications. However, when the service has very slow operations which are I/O bound (for example, they further connects to backend database or network or file system to perform high-latency operations), you would need to use many outstanding ASP.NET worker threads to achieve parallel requests. This is bad when the number of requests is huge, for example, several thousands or more. Each thread would hold a fair amount of system resource, i.e., memory. It would cause the service to be very slow to respond. This really requires a real asynchronous design to solve the problem. Because of this, ASP.NET introduced a new throttling logic for concurrent requests in IIS 7.0 to support this scenario for WCF.

ASP.NET Throttling on IIS 7.0 in Integrated Mode

As Thomas mentioned in his blog entry (TM0707) mentioned above, the following registry value is introduced to throttle concurrent requests:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0]

“MaxConcurrentRequestsPerCpu”=dword:0000000c

The default value is 12 per CPU which means is similar as the default setting for ASP.NET worker threads. This was introduced in ASP.NET 2.0 SP1 (shipped with .NET 3.5) and thus it is available in Windows 2008. It only applies to the IIS 7.0 Integrated mode, which is exactly where WCF is registered on Windows 2008.

Scalable Solutions for High Latency Services

Though the above throttling support was added to Windows 2008, WCF still uses synchronous module/handler instead of asynchronous ones in the released versions due to timing constraint. However, the internal WCF design does allow asynchronous request processing and it is just one step away from the real asynchronous logic. Here I provide a simple wrapper for that.

Custom HttpModule Using Reflection

The internal WCF types support asynchronous HTTP module. Since they are internal, we have to rely on reflection to create a real one.

First of all, you would need to register the async events for your HttpModule as following:

context.AddOnPostAuthenticateRequestAsync(

MyHttpModule.beginEventHandler,

MyHttpModule.endEventHandler);

This is registered to the PostAuthenticateRequest for the same reasons as the default WCF HttpModule.

Secondly, you would need to use reflection to create the instance of the internal WCF type System.ServiceModel.Activation.HostedHttpRequestAsyncResult:

Type hostedHttpRequestAsyncResultType = typeof(ServiceHostingEnvironment).Assembly.GetType("System.ServiceModel.Activation.HostedHttpRequestAsyncResult");

ConstructorInfo hostedHttpRequestAsyncResult = hostedHttpRequestAsyncResultType.GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.Public,

null, new Type[] { typeof(HttpApplication), typeof(bool), typeof(AsyncCallback), typeof(object) }, null);

return (IAsyncResult)hostedHttpRequestAsyncResult.Invoke(new object[] { application, false, cb, extraData });

Once you have the custom HttpModule, you can replace it with the default one that is registered in the WAS configuration %windir%\system32\inetsrv\config\applicationhost.config:

<system.webServer>

<system.webServer>

To do that, you can also define local settings in web.config:

<system.webServer>

</modules>

</system.webServer>

The sample code is attached.

Improvement in .NET 3.5 SP1

In .NET 3.5 SP1, we will provide the real asynchronous HTTP module/handler for WCF to better solve this problem.

Why changing SendTimeout does not help for hosted WCF services?

wdong — Mon, 10 Mar 2008 22:54:00 GMT

In .NET 3.0, you would handle two different timeouts:

· Binding.SendTimeout

This is the timeout that specifies how long the client can wait for the transport to complete data writing until throwing exception. It is client side setting. If the request would likely take longer than the default (1 minute), you would need to increase it.

· Binding.ReceiveTimeout

This is the timeout that specifies how long the service can wait from the beginning of receiving a request until the message is processed. It’s server-side setting. When you send a large message to the service and the service needs long time to process, you would need to increase this setting.

Ideally, using these two timeouts should solve most timeout problems. However, when a WCF service is hosted in IIS/ASP.NET, another setting would also control the lifetime of the request:

· HttpRuntimeSection.ExecutionTimeout

The default of this value is 110 seconds. When you have very slow service operations which would cause timeouts to happen, the request would be aborted and you can find an ASP.NET EventLog entry that tells that the request has timed out. As per the link, you can configure this setting through web.config as following:

<system.web>

</system.web>

</configuration>

This would set the timeout to be 600 seconds (10 minutes). From code, you can use the following API to achieve the same:

· HttpApplication.Server.ScriptTimeout

If you use ASMX services, you would hit this exact problem too.

Fortunately this has been enhanced in .NET 3.0 SP1 so that this is taken care of internally. The ScriptTimeout was set to be Int.MaxValue for WCF requests. In this way, WCF has the full control of the lifetime of the requests.

Profiling WCF/WF Applications with VS Performance Profiler

wdong — Tue, 04 Dec 2007 10:49:00 GMT

There are tons of good articles and blogs out there regarding how to use Visual Studio Team System (VSTS) performance tools (vsperfcmd.exe etc under "%ProgramFiles%\Microsoft Visual Studio 8\Team Tools\Performance Tools" if you have installed VSTS) to generate profiles for native and managed applications. Nothing is actually special for WCF applications here. The reason why I write this short blog is to simply give you a starting point on how to investigate performance of a Windows application (not just WCF/WF) with a common performance language – VSTS profiler.

When you have an application, you would want to know “where does my application spend its time?” and “where are the hot spots or bottlenecks”?

To answer these questions, you need a powerful performance tool. Have you used VSTS performance tools? Have you heard of different modes of profiling (instrumentation vs sampling)? Do you know the command-lines for the VSTS profiler in generating profiling data files? Do you know how to use the cool features of the tools to analyze the performance of your applications? Here are some good articles about how to use this powerful performance tool to investigate performance issues.

How to profile the startup of a managed application?

You can use the “/launch:” feature as following in IanWho’s blog:

http://blogs.msdn.com/ianhu/archive/2005/11/15/493110.aspx

How to profile an application by skipping the startup?

You can use the “/attach:” feature as following in Tim Cahill’s blog:

http://blogs.msdn.com/timothyc/archive/2006/03/08/546437.aspx

How to profile an ASP.NET application?

You can find useful “/globalxxx” flags in David Gray’s blog:

http://blogs.msdn.com/graycode/articles/aspnetoffroadprofilingarticle.aspx

It’s also summarized in the VSTS’s TechNotes:

http://msdn2.microsoft.com/en-us/teamsystem/aa718860.aspx

How to profile a Windows Service?

You can find details in Richard Wurdack’s blog:

http://blogs.msdn.com/angryrichard/articles/Profiling_Windows_Services.aspx

It’s also summarized in the VSTS’s TechNotes:

http://msdn2.microsoft.com/en-us/teamsystem/aa718870.aspx

How to profile a self-hosted WCF service application?

This is not special topic. Here are simple steps:

· Set the profiling environment with “VSPerfCLREnv.cmd /sampleon”.

· Start the WCF service application.

· Disable the profiling environment with “VSPerfCLREnv.cmd /off”.

· Start the WCF client application.

· Attach the profiler to the WCF service application:

VSPerfCmd.exe /output:C:\TEMP\trace.vsp /start:sample /attach:<your service app> [/timer:<sampling timer>]

· Shutdown profiler etc.

A Sample for WCF Client Proxy Pooling

wdong — Wed, 14 Nov 2007 09:43:00 GMT

Introduction

Ideally we should not need to pool WCF client proxies as I mentioned in my previous blog entry. From some customer feedback, however, I got to know that reusing proxies is not ideal because:

· There may be some unknown contention cost when one proxy is used by multiple threads.

· There is some concern about associating contextual data to a proxy when the proxy is reused at the same time by multiple threads.

So people would generally think about implementing a proxy pool in their systems. As pointed out in my previous blog entry, you need to be aware of the following when you want to implement a proxy pool:

· You need to implement the right synchronization logic for managing the proxies.

· You need to make sure the proxies are used equally. Sometimes, you may want to implement a round-robin pattern for the proxies.

· You need to handle exceptions and retries for the pool.

· The pool size needs to be limited and configurable.

· You may need to be able to create proxies even if when no proxy is available from the pool.

Here are some common ways to implement a pool in my mind:

· Load-balancing model: Whenever you want to send a request, you would pick a proxy from the pool (in a round-robin pattern) and use it to handle the request. The proxy can be used by multiple threads concurrently.

· Take-return-create model: You take the proxy from the pool so that other threads cannot get the same one. After using it, you return it back to the pool. When the pool is empty, you would create a new proxy. When the pool is full when you return it back, you just discard the proxy.

· Take-return-wait model: Similar as above. The difference is that, when the total number of proxies being used reaches the size of the pool and all proxies are being used (i.e., the pool is empty), the thread would wait until a proxy is returned back to the pool.

I will show a sample to implement the second approach “take-return-create model”. It is easy to implement and it solves performance issues nicely. As what I have written in my other blog entries, the sample code and the ideas in this blog imply no warranties and confer no rights and they are provided AS-IS.

The Pool<T> Class

The generic Pool<T> class that I used in the sample implements the simple Take-Return logic. It is actually the one that is taken from the internal calls System.ServiceModel.Pool<T>. You can disassembly the source from System.ServiceModel.dll with the .NET Reflector. Here is the source:

class Pool<T> where T : class

{

T[] items;

int count;

public Pool(int maxCount)

{

items = new T[maxCount];

}

public int Count

{

get { return count; }

}

public T Take()

{

if (count > 0)

{

T item = items[--count];

items[count] = null;

return item;

}

else

{

return null;

}

public bool Return(T item)

{

if (count < items.Length)

{

items[count++] = item;

return true;

}

else

{

return false;

}

public void Clear()

{

for (int i = 0; i < count; i++)

items[i] = null;

count = 0;

}

The logic is very simple. It uses an array-based simple stack to manage the items. Note that the class does not provide thread-safe logic. So we have to add that logic when we use it.

The Pooling Logic

Once we have the pool, we can implement the pooling logic. Here I follow the best practice of closing a proxy as mentioned in the MSDN article http://msdn2.microsoft.com/en-us/library/aa355056.aspx. Here is the code:

void CallWithPool(string message)

{

MyHelloServiceProxy proxy = null;

// Trying to take the proxy from the pool

lock (pool)

{

proxy = pool.Take();

}

// Remove bad proxies (Faulted etc). Note that checking the proxy state has

// some performance cost here as it accesses the Transparent proxy of the

// channel.

if (proxy != null && proxy.State != CommunicationState.Opened)

{

proxy.Abort();

proxy = null;

}

// Create the proxy if not found from the pool.

if (proxy == null)

{

Console.WriteLine("Creating new proxy.");

proxy = CreateProxy();

}

try

{

// Call the service with the proxy

proxy.Hello(message);

}

catch (Exception)

{

// You need to better handle the exception in real production

// environments.

proxy.Abort();

throw;

}

// Trying to put the proxy back to the pool

bool shouldClose = false;

lock (pool)

{

shouldClose = !pool.Return(proxy);

}

// Close the proxy if it's not put back into pool

if (shouldClose)

{

try

{

proxy.Close();

}

catch (Exception)

{

proxy.Abort();

throw;

}

Comparisons

In the sample, I also implemented the other two approaches for comparison purpose:

· Per call: Creating/disposing proxies per call

· Singleton: Using singleton proxies

Data shows that the performance between pooling and singleton is quite close. When BasicHttpBinding with default settings is used, the “per call” approach has comparable performance as the other two. However, when WSHttpBinding is used, you would see much slower results when “per call” model because of the high cost in re-constructing security sessions when each proxy is created.

Configuration Setting Helper

Now that I have provided all the details, why don’t I be a good person to the end? One thing that we have not mentioned is about configuring the pool size. In the sample, I also provided a simple configuration section <channelPoolSettings> that is implemented by the type “ChannelPoolSettingsSection” . In this way, you can easily change the setting from the configuration without touching the code. This can be easily converted to ASP.NET web.config setting. To make the configuration work, you need to register the section handler as following:

</configSections>

Here is the sample configuration:

</configuration>

You can find the sample code attached.

Performance Improvement for WCF Client Proxy Creation in .NET 3.5 and Best Practices

wdong — Sat, 27 Oct 2007 03:39:00 GMT

Introduction

In .NET 3.0 SP1, which will be shipped together with .NET 3.5, there is significant performance improvement in WCF client proxy creation. For BasicHttpBinding, the performance is close to that of ASMX proxy creation.

ASMX Proxy vs WCF Proxy

ASMX proxy is much simpler than WCF proxy. The former is a wrapper the type System.Web.Services.Protocols.SoapHttpClientProtocol. In ASMX world, the programming model is “horizontal” in two ways:

· There is no ServiceContract concept. The signatures of all service operations (WebMethod) are duplicated in the client proxy. The calls to those operations on the client side go through SoapHttpClientProtocol.Invoke. So there is no separation between service interface and the underlying message processing system.

· There is no clear channel stack. The SoapHttpClientProtocol wraps everything on the client-side. It directly invokes System.Net API HttpWebRequest.

Instead WCF provides a “vertical” programming model. You get the same rich programming features on the client side as that you get on the service-side. Internally WCF proxy is a .NET Remoting transparent proxy which is wrapped inside the type System.ServiceModel.Channels.ServiceChannel. The transparent proxy allows you to perform desired type casting and thus you can use the proxy and call service operations just like you are using the ServiceContract interface.

WCF Proxy Creation Cost

As pointed out in my last blog entry, there are two common ways of creating WCF client proxies (or channels):

1) Using ClientBase<T> which is the default WCF support and you can use svcutil.exe tool to generate the proxy code.

2) Using ChannelFactory<T>.CreateChannel().

A common pattern of using ClientBase<T> styled proxies is to perform creation/disposing on each call:

foreach (string msg in myList)

{

// You may add try/catch block here ...

HelloWorldProxy proxy = new HelloWorldProxy("MyEndpoint", new EndpointAddress(address)))

proxy.Hello(msg);

proxy.Close();

// Error handling code here ...

}

Here HelloWorldProxy is auto-generated by svcutil.exe and it’s actually a subclass of ClientBase<IHelloWorld> with the ServiceContract type IHelloWorld.

In .NET 3.0, creating/disposing WCF client proxies is a very expensive operation for both approaches. It’s much worse than that for ASMX proxies. The approach 1) involves creating ChannelFactory<T> internally. Each proxy holds a ChannelFactory object as a private field. The lifetime of the ChannelFactory is fully controlled by the proxy. Approach 2) is much better since you have your own control of the ChannelFactory<T> object and thus you may only create it once and thus save the cost for further proxy creation.

Why is creating/disposing ChannelFactory so expensive? This is because it involves the following major operations:

Constructing the ContractDescription tree
Reflecting all of the required CLR types
Constructing the channel stack
Disposing all of the resources

In most cases, these data are identical between different proxies which are created in the same way.

In .NET 3.0 SP1, which will be shipped together with .NET 3.5, there are two major performance improvements in this area:

· ChannelFactory caching inside ClientBase<T>. This significantly improves the performance for approach 1).

· Channel management logic is improved and thus both 1) and 2) benefits from this.

With these improvements, creation of WCF proxie have quite comparable performance as ASMX ones.

Improvements

The idea of improving the performance of ClientBase<T> is simple: caching ChannelFactory objects. This is similar as the client type cache in ASMX proxy.

ChannelFactory Caching for ClientBase<T>

The key idea is to cache the ChannelFactory object at AppDomain level so that the lifetime of a ChannelFactory object is not controlled by the proxy. The cache is a most recently used (MRU) cache. The cache size is hard-coded as 32 so that the least used ChannelFactory objects are purged from the cache.

With the ChannelFactory cache, the rough process of creating a ClientBase<T> object is as following:

· In the constructor of ClientBase<T>, a lookup is performed to find a matched ChannelFactory in the cache.

· If found, the ref-count of the ChannelFactory is incremented. Otherwise, a new ChannelFactory is created based on the settings.

· Before the inner channel (the transparent proxy) of ClientBase<T> is created, the caching logic for the current ClientBase<T> can be disabled if other public properties (such as ChannelFactory, Endpoint, and ClientCredentials) are accessed.

· Once the innerchannel if created successfully, the ChannelFactory object for the ClientBase<T> is added to the cache if it’s not grabbed from the cache and caching is not disabled.

What is Matched ChannelFactory?

Did I say “matched” ChannelFactory in the last section? Yes, you have to make sure that you can differentiate different ChannelFactory objects based on the input settings from ClientBase<T>’s constructors. There are two types of constructors: those have Binding as a parameter and those don’t. Here are those that don’t:

ClientBase();
ClientBase(string endpointConfigurationName);
ClientBase(string endpointConfigurationName, string remoteAddress);
ClientBase(string endpointConfigurationName, EndpointAddress remoteAddress);
ClientBase(InstanceContext callbackInstance);
ClientBase(InstanceContext callbackInstance, string endpointConfigurationName);
ClientBase(InstanceContext callbackInstance, string endpointConfigurationName, string remoteAddress);
ClientBase(InstanceContext callbackInstance, string endpointConfigurationName, EndpointAddress remoteAddress);

For these constructors, all arguments (including default ones) are in the following list:

· InstanceContext callbackInstance

· string endpointConfigurationName

· EndpointAddress remoteAddress

As long as these three arguments are the same when ClientBase<T> is constructed, we can safely assume that the same ChannelFactory can be used. Fortunately, String and EndpointAddress types are immutable, i.e., we can make simple comparison to determine whether two arguments are the same. For InstanceContext, we can use Object reference comparison. The type EndpointTrait<TChannel> is thus used as the key of the MRU cache.

Here are the two constructors which have Binding as a parameter:

ClientBase(Binding binding, EndpointAddress remoteAddress);
ClientBase(InstanceContext callbackInstance, Binding binding, EndpointAddress remoteAddress);

Since Binding object is mutable, i.e., people can use it for one proxy, modify it and use it for another one, we can not safely detect whether there is a change to a binding object or not. Thus when the Binding is supplied in the constructors, the caching mechanism is disabled for that proxy.

Disabling caching

WCF supports extensibility in the way that people can change the settings of the ChannelFactory and the ContractDescription before the ChannelFactory is opened. Such change to a newly created proxy would prevent it from sharing the same ChannelFactory with other proxies. Same logic applies to other properties for the ClientBase<T>:

ChannelFactory
Endpoint
ClientCredentials

When any of these properties of ClientBase<T> is accessed before the inner channel is created and when ClientBase<T>.Open is called, the caching is disabled for this proxy. This is natural since you don’t want one proxy reuses the ChannelFactory for another proxy which has different contract or security settings.

Channel Management

Each channel that is created internally is added to the System.ServiceModel.Channels.CommunicationObjectManager. The performance of adding/removing an item from this class is improved in .NET 3.5. The fix was to switch from List<T> to Hashtable internally. This is why you will see performance improvement in approach 2) above.

Best Practices

Now that I have talked about the internals, I want to summarize how you can achieve best performance using WCF client proxies.

Reuse the same proxy

In many cases, you would want to reuse the same proxy. This has the best performance. It is especially true when you use security features since the initial security negotiation can have high cost.

Don’t forget to open the proxy explicitly before using it as I mentioned that in my previous blog entry.

Use proxy that enables caching

As mentioned above, you can either use ChannelFactory<T>.CreateChannel to create your proxy or you can use auto-generated proxies. If you use the latter, you need to be aware of the following in order to get ChannelFactory cached:

· Don’t use the constructors of the proxy that takes Binding as an argument.

· Don’t access the public properties ChannelFactory, Endpoint, and ClientCredentials of the proxy.

Disabling caching

If you really want to disable caching, you can simply do the reverse of above. For example, you can access the ChannelFactory property of the proxy before it is first used.

Proxy/Channel pooling

The last resort of achieving high performance is through proxy caching. In some scenarios, you may not want all threads to use the same proxy due to the following two possible reasons:

· The context of the channels does not allow multiple threads to access.

· There is some bottleneck in the Channel stack that hurts performance when a single proxy is used.

The logic of pooling can be from very simple to very complicated. Some ideas to keep in mind are as following:

· You need to implement the right synchronization logic for managing the proxies.

· You need to make sure the proxies are used equally. Sometimes, you may want to implement a round-robin pattern for the proxies.

· You need to handle exceptions and retries for the pool.

· The pool size needs to be limited and configurable.

· You may need to be able to create proxies even if when no proxy is available from the pool.

I don’t have a concrete example about why you need to implement the pool. If you have such a scenario, please do let me know.

Best Practice: Always open WCF client proxy explicitly when it is shared

wdong — Fri, 26 Oct 2007 06:25:00 GMT

In order to provide symmetric programming model for the client-side as for the server-side, WCF leverages .NET Remoting transparent proxy technique so that the service contract interface can be used seamlessly as on the server-side. The svcutil.exe tool can generate WCF client proxy code for you. If you take a closer look at the generated code, you will find out that the proxy class is a subclass of ClientBase<T>. By using ChannelFactory<T>, you can create your proxy (or called channel) directly without going through ClientBase<T>.

Creating a WCF proxy is quite a heavy-weighted operation. So sometimes, you would want to create a single proxy and let multiple threads to use it. This works quite well if you don’t want the proxy to keep specific context (such as security credential) on each call.

The best practice in this case is that: you should always open WCF client proxy explicitly before you are making any calls. Here is the sample code if you use auto-generated proxy from svcutil.exe:

MyHelloServiceClient proxy = new MyHelloServiceClient();

proxy.Open();

// Make a call with the proxy

proxy.Hello("Hello world!");

Here is the sample code if you use ChannelFactory<T> to create a proxy:

ISimpleContract proxy = factory.CreateChannel();

((IClientChannel)proxy).Open();

// Make a call with the proxy

proxy.Hello("Hello world!");

If you don’t call the “Open” method first, the proxy would be opened internally when the first call is made on the proxy. This is called auto-open.

Why? When the first message is sent through the auto-opened proxy, it will cause the proxy to be opened automatically. You can use .NET Reflector to open the method System.ServiceModel.Channels.ServiceChannel.Call and see the following code:

if (!this.explicitlyOpened)

{

this.EnsureDisplayUI();

this.EnsureOpened(rpc.TimeoutHelper.RemainingTime());

}

When you drill down into EnsureOpened, you will see that it calls CallOnceManager.CallOnce. For non-first calls, you would hit SyncWait.Wait which waits for the first request to complete. This mechanism is to ensure that all requests wait for the proxy to be opened and it also ensures the correct execution order. Thus all requests are serialized into a single execution sequence until all requests are drained out from the queue. This is not a desired behavior in most cases.

To avoid such “serializing” artifact, the best practice is to open the proxy explicitly as above. Once you get to this point, you will be able to share the same proxy object among multiple threads.

How to use performance counters to diagnose performance of WCF applications

wdong — Fri, 07 Sep 2007 06:28:00 GMT

There are many ways to diagnose the performance of WCF applications. Here is a simple one that uses performance counters to get some rough ideas. There are two categories of performance counters that you can use:

· ASP.NET counters: http://msdn2.microsoft.com/en-us/library/fxk122b4(vs.71).aspx

· WCF counters: http://msdn2.microsoft.com/en-us/library/ms735098.aspx.

ASP.NET Counters

For web-hosted applications, you can use ASP.NET counters to monitor the overall request processing status. They are enabled by default and they are of type PerformanceCounterInstanceLifetime.Global. So you can open the PerfMon tool to monitor the counters without doing any extra work. Here are the two interesting counters that I want to mention here:

· Requests/sec

· Request Execution Time

The first one tells the throughput of the whole web server for ASP.NET requests (including WCF ones). You may have many clients hitting the same server and thus you will see the overall throughput.

The second counter tells you how long (in milliseconds) it takes for the server to process each request. It measures the latency of the request processing. If the requests are processed sequentially, the value (converted into seconds) of this counter is the inverse of the first one. In general when the requests are processed concurrently, this is not true.

Performance counters are a mechanism for monitoring services that you can get without

WCF Counters

WCF performance counters are quite different. Here are major ones:

· These counters are not enabled by default in V1. You have to do that through the config file.

· All counters are of type PerformanceCounterInstanceLifetime.Process. All of the three different categories (Service, Endpoint, and Operation) are of this type which is also called “instanced based”. This means that in order to monitor the counters from PerfMon, you have to start the WCF application with the counters enabled. When the application exits, you cannot monitor the counter anymore.

· The WCF counters are more finer grained than ASP.NET ones. You can monitor the performance of different service instances at different levels. To recap, the counter instance names have the following pattern:

(ServiceName).(ContractName).(OperationName)@(first endpoint listener address)

Here is the way of how to enable WCF performance counters through configuration:

<system.serviceModel>
<diagnostics performanceCounters="All" />
</system.serviceModel>

The possible settings for the scope of service counters are Off, ServiceOnly, and All for WCF V1.

Here are the two interesting performance counters that we can use for performance analysis:

· Calls Per Second

· Calls Duration

These counters have similar properties as those for ASP.NET. But they are scoped to WCF service (endpoint or operation) level. The value of the second counter is a floating number in seconds. The accuracy is 0.001 second. So if your service is faster than 1000 requests/second in sequential processing, you would always see 0.000.

How to Use the Values of the Counters

Once you have the data captured in PerfMon, you will be able to tell which WCF service is the performance bottleneck. This is useful when you have very slow service operations. If you want advanced performance analysis, you would need more granular tools such as ETW tracing, performance profiles, custom tracing, and debuggers.

Lastly, before adding the performance counters to PerfMon, make sure that 1) you have them enabled from the configuration file, and 2) you have started the WCF application. For web-hosted service, you can ping the service from the browser first to make sure the service has been started.

Trade Benchmark: .NET 3.0 vs IBM WebSphere 6.1

wdong — Fri, 10 Aug 2007 10:55:00 GMT

Microsoft designed WCF seriously with all major basics taken into account: performance, security, and robustness/stress. I am proud as being part of the team. You might have seen the data that Saurabh, our performance Program Manager, posted in his whitepaper. We have competed ourselves for many of our existing technologies.

So it was not a big surprise to me when Greg Leake first showed us the data that he got with his implementation of the Stock Trader performance benchmark using WCF three months ago. Yes, we are faster than WebSphere, significantly!

Greg’s .NET StockTrader sample was recently published here. You can also find his detailed analysis document from here.

Service Instances and Concurrent Execution

wdong — Fri, 10 Aug 2007 10:15:00 GMT

Kenny has a very good blog entry on this topic. I want to add some more points to this from performance perspective.

InstanceContextMode

By default, the default value of InstanceContextMode for a WCF service is PerSession. This means different things for different bindings. For sessionful bindings such as NetTcpBinding and NetNamedPipeBinding with default settings, this means that a single service instance is used to serve requests from the same client for the lifetime of the connection. For non-sessionful bindings such as BasicHttpBinding, the session is not enabled and thus one service instance is created per request. So for the latter, PerSession has the same effect of PerCall.

InstanceContextMode Performance Impact

Suppose if you have a single client for the service, then for sessionful bindings, you would observe the slightly different throughput of different modes from fastest to slowest:

· PerSession, Single, PerCall

The reason why Single is slower than PerSession is because the service needs to maintain the global locking mechanism, no matter how many clients call the service. It is reasonable that PerCall is the slowest because the overhead of creating a service instance on each call is pretty high. However, the above order is not the same for non-sessionful bindings. Here is the order from fastest to slowest for non-sessionful bindings:

· Single, PerCall or PerSession

This is the case because “PerSession” is treated as “PerCall” for non-sessionful bindings.

Scalability and Server CPU Utilization

What would happen if you have multiple clients call the service concurrently? Well, we need to understand how WCF runtime dispatches the requests on the server-side. As Kenny pointed out, this is controlled by another property “ConcurrencyMode” of the ServiceBehavior attribute of the service. The default value is Single. It means that WCF uses a single thread to dispatch all requests sequentially to each service instance. For InstanceContextMode as PerCall and PerSession, this is very natural. However, for Single instance mode, this means that all of the client requests are queued up to be served by the same thread and being dispatched to the same service instance. The advantage for this is that you don’t need to worry about synchronization issue in your service operation implementation. However, this also means that the service is not scalable to handle many client requests concurrently.

For the last case, actually I had a short story. Once I was testing the server throughput for singleton service with BasicHttpBinding. There are 24 clients calling the servicing at the same time. I noticed that the server throughput is low and the CPU utilization was only around 60%. When I debugged the service process, I noticed that there was only one thread actively processing the requests and other requests were in the queue. Suddenly I realized that I used the default setting of ConcurrencyMode.

To conclude this, make sure to set ConcurrencyMode to Multiple if you want to achieve high throughput for your singleton service and make sure to perform your own synchronization in your service operation!

WF Performance Test Sample: SimpleStateMachineWorkflow

wdong — Sat, 07 Jul 2007 02:42:00 GMT

As promised, here I attach the sample code for the “SimpleStateMachineWorkflow” performance test. It is not the same one that is used for the whitepaper (http://msdn2.microsoft.com/en-us/library/Aa973808.aspx). But the testing logic is very close. If you find any significant disparity between the numbers that you get and those posted in the whitepaper, please let me know.

Wenlong Dong's Blog : Performance

Crack Throttling: Debugging WCF TimeoutException for a Middle-Tier Application

Problem

Debugging

ASP.NET/WCF Settings

Trace Log

WinDBG Dump

NetMon Capture

Client Latency Distribution

Performance Counter Data

Configuration Rechecking

Resolution

WCF 4: Higher Default Throttling Settings for WCF Services

Scale WCF Application Better with Asynchronous Programming

Why Asynchronous Pattern?

Basic Asynchronous Pattern with IAsyncResult

Service Asynchronous Pattern

Asynchronous Database Access

Client Asynchrony

Exception Handling

Sample Setup

Prerequisite

Steps to Run the Test

Why Only Two Concurrent Requests for WCF Load Testing?

Orcas SP1 Improvement: Asynchronous WCF HTTP Module/Handler for IIS7 for Better Server Scalability

Introduction

Asynchronous WCF HTTP Module/Handler

Asynchronous HTTP Module

Asynchronous HTTP Handler Factory

Default Installation – Synchronous Version

WCF Module/Handler Registration Tool

WCF Request Throttling and Server Scalability

Two Threads per Request

Asynchronous Request Processing

ASP.NET Threading on IIS 6.0

WCF Request Throttling and Tuning

Throttling Tuning

ASP.NET Throttling on IIS 7.0 in Integrated Mode

Scalable Solutions for High Latency Services

Custom HttpModule Using Reflection

Improvement in .NET 3.5 SP1

Why changing SendTimeout does not help for hosted WCF services?

Profiling WCF/WF Applications with VS Performance Profiler

How to profile the startup of a managed application?

How to profile an application by skipping the startup?

How to profile an ASP.NET application?

How to profile a Windows Service?

How to profile a self-hosted WCF service application?

More Links

A Sample for WCF Client Proxy Pooling

Introduction

The Pool<T> Class

The Pooling Logic

Comparisons

Configuration Setting Helper

Performance Improvement for WCF Client Proxy Creation in .NET 3.5 and Best Practices

Introduction

ASMX Proxy vs WCF Proxy

WCF Proxy Creation Cost

Improvements

The idea of improving the performance of ClientBase<T> is simple: caching ChannelFactory objects. This is similar as the client type cache in ASMX proxy.

ChannelFactory Caching for ClientBase<T>

What is Matched ChannelFactory?

Disabling caching

Channel Management

Best Practices

Reuse the same proxy

Use proxy that enables caching

Disabling caching

Proxy/Channel pooling

Best Practice: Always open WCF client proxy explicitly when it is shared

How to use performance counters to diagnose performance of WCF applications

ASP.NET Counters

WCF Counters

How to Use the Values of the Counters

Trade Benchmark: .NET 3.0 vs IBM WebSphere 6.1

Service Instances and Concurrent Execution

InstanceContextMode

InstanceContextMode Performance Impact

Scalability and Server CPU Utilization