Wenlong Dong's Blog

Two bloggers for performance

Wenlong Dong1 — Wed, 05 Jan 2011 19:51:46 GMT

Dustin Metzgar and Sajay Antony, our performance developers, have all started up their blogs on MSDN, which is great news to developers who are interested in WCF/WF/AppFabric performance or other fun things covered by their blogs.

Dustin has just started up his blog on MSDN. Dustin has great expertise on performance tools (such as xperf, VSTS profiler, CLR Profiler, etc). He also wrote the WF 4 performance whitepaper (see my previous blog).

Sajay Antony has his blog up a while back. He has in-depth experience on WCF performance and has done some solid performance improvements to the WCF/WF stacks. Here is one example.

Enjoy!

Why does machine.config contain invalid Xml content after installing .Net 3.5 patches?

Wenlong Dong1 — Tue, 23 Nov 2010 18:47:45 GMT

For quite a few times, I heard customers would hit this issue after installing .Net 3.5 patches or repair 3.5 on Windows Vista or Windows 2008 Server. Basically the machine.config file contains some invalid Xml content and applications using configuration do not work, especially for IIS-hosted applications. The main problem is that the WCF 3.5 installer (WFServiceReg.exe) did not handle the different cases very well.

Problem Statement

There are three different cases that I have heard:

Issue 1: .Net 3.0 is removed but .Net 3.5 is on the box

On Windows Vista and Windows 2008 Server, .Net 3.0 is installed through Component-Based Setup (CBS). However, .Net 3.5 is installed through Windows Installer (MSI). Thus .Net 3.5 does not have a strong dependency on .Net 3.0. People could accidentally uninstall .Net 3.0 from the box. This would cause the section handlers (for <system.serviceModel> etc) for WCF removed from machine.config. However, any further .Net 3.5 patch would cause the WCF installer to run and it would install the following dangling elements into machine.config:

<system.serviceModel>

...

</extensions>

...

</client>

</system.serviceModel>

This would cause the application to fail with the following error:

System.Configuration.ConfigurationErrorsException: Unrecognized configuration section system.serviceModel. (c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Config\machine.config line 146)

Issue 2: .Net 3.0 is on the box but the WCF section handlers are removed

For some unknown reason, .Net 3.0 is not uninstalled from the machine. However, the WCF section handlers are accidentally removed when different orders of install/uninstall operations happened. The application would also fail with the same error as Issue 1 above.

Issue 3: Redundant Xml elements when configSource is used

The WCF 3.5 installer does not recognize “configSource” attribute of <client/> or <extensions/> elements and still add more Xml content when it runs.

Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.
Parser Error Message: A section using 'configSource' may contain no other attributes or elements.
Source Error:

Line 173: <client configSource="client.config">

Line 174: <metadata>

Line 175: <policyImporters>

Source File: C:\Windows\Microsoft.NET\Framework\v2.0.50727\Config\machine.config Line: 174

Solutions

Here are two different solutions that would help to resolve the above issues:

Solution 1: Repair .Net 3.0

For issues 1 and 2, you can repair .Net 3.0 on the machine. Here are the rough steps on Windows 2008 Server:

· Start -> Control Panel

· Programs -> Turn Windows features on or off -> Features

· Check “.Net Framework 3.0” component to uninstall and reinstall it

Solution 2: Run WFServicesReg.exe Tool

For issues 1 and 2, you can run the WFServicesReg.exe tool (on the box) to fix the problem:

1) For Issue 1, i.e., .Net 3.0 is not installed ("C:\windows\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\ServiceModelReg.exe" does not exist), run the following:

%windir%\Microsoft.NET\Framework\v3.5\WFServicesReg.exe /r /b

2) For Issue 2, i.e., .Net 3.0 is installed ("C:\windows\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\ServiceModelReg.exe" exists), run the following:

%windir%\Microsoft.NET\Framework\v3.5\WFServicesReg.exe /r /b

"%windir%\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\ServiceModelReg.exe" /i

%windir%\Microsoft.NET\Framework\v3.5\WFServicesReg.exe /c /b

Solution 3: Run attached javascript file

The above solutions do not fix the Issue 3. So I created a javascript FixServiceModel30Reg.js to fix the problem. You can run it as following to fix all of the above issues. Steps:

· Download the script FixServiceModel30Reg.txt and save it to the root of c: drive.

· Rename it to FixServiceModel30Reg.js.

· Open the Command Prompt window and run the following command:

Cscript.exe c:\FixServiceModel30Reg.js

WF 4 Performance Whitepaper Published

Wenlong Dong1 — Thu, 11 Nov 2010 00:55:00 GMT

We have just published the performance whitepaper for WF 4 at the following location:

http://msdn.microsoft.com/en-us/library/gg281645.aspx

We have done quite some component-level performance measurement for different activities and scenarios. Here is one snapshot that I would like to share with you:

As you can see, WF 4 is 10X – 100X faster than WF 3 at component level. So building a business application using these build blocks has very minimal overhead from the Framework itself. This is also one of the main reasons why we re-designed WF in this version.

Using ASP.NET Sessions from WCF

wdong — Sun, 21 Feb 2010 04:23:00 GMT

This is not a new topic. But interestingly I could not find a good example on the web while I wanted to find one for some other investigation. So I decided to create a sample here to show the following scenario:

· A WCF service has the access to a ASP.NET session. Different WCF client proxies can connect to the per-call WCF service and share the same ASP.NET session state.

Here I will show a simple example on how to achieve this.

ASP.NET Sessions vs WCF Sessions

WCF sessions are very different from ASP.NET Sessions. In short,

· WCF sessions are represented as service instances (as CLR objects) and the states are part of each service instance. The sessions are initiated by the calling WCF client. WCF relies on special context to provide correlation for the sessions: sessionful channels, secure conversations, reliable-messaging, etc.

· ASP.NET sessions are like shared data storage across different requests. The sessions are always initiated by the server. ASP.NET relies on cookies or special Uri goo to provide correlation for the sessions.

You can find more details for the comparisons of the two from the MSDN article “Using Sessions”.

While these two concepts are quite different and the usages are also drastically different, the WCF sample “HttpCookieSession” tried to use the similar idea as cookie-based ASP.NET sessions to provide WCF session support. The sample is a little bit complex and it does not use real ASP.NET sessions though.

There are many samples on how to use WCF sessions, for example, “How to: Create a Service That Requires Sessions” is a good one.

So I will only show how to use ASP.NET sessions from WCF below. It uses PerCall service to handle this. So different service instances have access to the same ASP.NET session state.

Using ASP.NET Session States

Storing Session States

It is known that ASP.NET session states can be stored in different places: 1) In-Memory (thus In-process), 2) Out-of-proc state service, 3) SQL server, 4) a custom session-state store provider, etc. More details can be found in “ASP.NET Session State Overview”.

For simplicity, I will just show the in-memory case. Here is a very simple session state:

class MySessionState

{

int counter;

public MySessionState() {}

public int Counter { get { return this.counter; } }

public int Touch() { return Interlocked.Increment(ref this.counter); }

}

The state just contains a single counter. This state is stored in the session object as following:

MySessionState state = (MySessionState)HttpContext.Current.Session["MySessionState"];

if (state == null) {

state = new MySessionState();

HttpContext.Current.Session["MySessionState"] = state;

}

When the service receives a request, the sample increments the counter to demonstrate the sharing effect:

int counter = state.Touch();

Enabling Session States

It is very straightforward to enable the ASP.NET session states. You only need to run WCF service in the ASP.NET Compatibility mode as being specified in the web.config:

Retrieving Session Id on the Client

Once the session states are enabled and there is session data populated as above, we can retrieve the Session Id from the cookie on the client side:

using (new OperationContextScope((IContextChannel)proxy))

{

Console.WriteLine(proxy.Greet("Hello"));

HttpResponseMessageProperty responseProperty = OperationContext.Current.IncomingMessageProperties[HttpResponseMessageProperty.Name]

as HttpResponseMessageProperty;

helper = HttpSessionCookieHelper.Create((string)responseProperty.Headers[HttpResponseHeader.SetCookie]);

}

Here the type “HttpSessionCookieHelper” is just a helper class that is used to retrieve the ASP.NET Session Id from the cookie string (with the cookie name “ASP.NET_SessionId”) and append the Session Id to the request. It is defined as following:

class HttpSessionCookieHelper

{

const string AspNetSessionIdCookieName = "ASP.NET_SessionId";

string aspNetSessionId;

public static HttpSessionCookieHelper Create(string cookieString);

public static HttpSessionCookieHelper CreateFromSessionId(string sessionId);

public void AddSessionIdToRequest(HttpRequestMessageProperty requestProperty);

public string AspNetSessionId { get { return this.aspNetSessionId; }

}

Sending Session Id

Once we get the Session Id retrieved from the response of the first request, we can share it for the subsequent client calls. Here is the example:

using (new OperationContextScope((IContextChannel)proxy))

{

HttpRequestMessageProperty requestProperty = new HttpRequestMessageProperty();

helper.AddSessionIdToRequest(requestProperty);

OperationContext.Current.OutgoingMessageProperties[HttpRequestMessageProperty.Name] = requestProperty;

Console.WriteLine(proxy.Greet("Howdy"));

}

The implementation of the HttpSessionCookieHelper.AddSessionIdToRequest does nothing but the following:

requestProperty.Headers[HttpRequestHeader.Cookie] = string.Format("{0}={1}", AspNetSessionIdCookieName, this.aspNetSessionId);

With this logic, the second request would share the same ASP.NET Session State on the server side.

Sample Setup

Here are the steps on how to use it:

1) Unzip the attachment locally.

2) Create a virtual application with name “WCFAspNetSession” and the physical path points to the “WCFAspNetSession” directory.

3) Compile the client code and run it.

Here is the output of the sample:

[Session '1vgfq255e1lpe255whwprf55'] You said: Hello (counter: 1)

[Session '1vgfq255e1lpe255whwprf55'] You said: Howdy (counter: 2)

As you can see, the Session Ids for the two requests are the same and the counter for the shared state has increased.

Here is the sample code: [sample attachment].

Why Does WCF Become Slow After Being Idle For 15 Seconds?

wdong — Thu, 11 Feb 2010 08:08:00 GMT

Customers reported the following WCF performance issue recently:

· A WCF service over TCP or Named Pipe idles for over 15 seconds and the first request after that becomes very slow. Why?

This can be easily reproduced. Basically, if you send two consecutive requests to a WCF service over TCP or Named Pipe transport, you would see something as following:

First message invocation:

Latency for thread 0: 0.0758616 seconds

Sleep for 16 ms

Latency for thread 0: 0.0030432 seconds

However, if it sleeps for 15 seconds, you would get:

First message invocation:

Latency for thread 0: 0.0774229 seconds

Sleep for 15000 ms

Latency for thread 0: 0.4179955 seconds

Here it takes about 418 milliseconds to complete the second request. Sometimes this delay can be even longer. Why does this happen?

The main reason for this is that the CLR ThreadPool has a 15-second timeout for idle threads. It releases all of the I/O threads except for one so that this last one can process I/O requests immediately. If this last I/O thread is used somehow, the ThreadPool would take time to create new threads to handle new WCF requests and also introduced random delays when creating a new thread, even if the number of active threads is smaller than the MinIOthreads setting (can be set through ThreadPool.SetMinThreads). This is a known CLR issue and it is unfortunately that it is still not fixed in 4.0. For WCF services over TCP or Named Pipe, however, WCF does hold one I/O thread to manage timers for the sessionful channels internally. If you attached the debugger to the service, you would see the following active I/O thread:

0:009> !clrstack

OS Thread Id: 0x1b90 (9)

Child-SP RetAddr Call Site

000000001c3aedc0 000007fef4e0c29c System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean)

000000001c3aee20 000007fef4d9713a System.ServiceModel.Channels.IOThreadTimer+TimerManager.OnWaitCallback(System.Object)

000000001c3aee80 000007fef73395a9 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke2()

000000001c3aef00 000007fef4d9708f System.Security.SecurityContext.Run(System.Security.SecurityContext, System.Threading.ContextCallback, System.Object)

000000001c3aef40 000007fef4d96fb0 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke()

000000001c3aef90 000007fef4d96e32 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.ProcessCallbacks()

000000001c3af000 000007fef4d96dc1 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.CompletionCallback(System.Object)

000000001c3af050 000007fef8028815 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

000000001c3af080 000007fef730a71e System.ServiceModel.Diagnostics.Utility+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)

000000001c3af0e0 000007fef8e1d502 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

Fortunately there is a workaround for this issue, as provided by Eric Eilebrecht. Basically you need to keep another I/O thread idle so that it won’t get timed out (in a 15-second internal). It is not ideal but it solves the problem. You can add the following code to the WCF service side:

static class ThreadPoolTimeoutWorkaround

{

static ManualResetEvent s_dummyEvent;

static RegisteredWaitHandle s_registeredWait;

public static void DoWorkaround()

{

// Create an event that is never set

s_dummyEvent = new ManualResetEvent(false);

// Register a wait for the event, with a periodic timeout. This causes callbacks

// to be queued to an IOCP thread, keeping it alive

s_registeredWait = ThreadPool.RegisterWaitForSingleObject(

s_dummyEvent,

(a, b) => {

// Do nothing

null,

1000,

false);

}

Now you can invoke the above workaround method anywhere in your code:

ThreadPoolTimeoutWorkaround.DoWorkaround();

This workaround uses a wait thread to wait for an event which is never set. When the thread times out (in one second), it queues up a packet to the IOCompletionPort. The packet does nothing but causes the IOCompletionPort to wake up and thus keeps the I/O thread from being timed out (with 15 seconds timeout as mentioned above). Since this is a repeating wait that will always time out, we’ll keep queueing packets to the IOCP, and keep at least one thread alive forever. As long as a single I/O thread stays “free” in this manner, a new I/O thread would be created quickly. Note that for .NET 3.5, you also need the QFE that I mentioned in my previous blog entry to be installed.

With this workaround, you would notice the following result:

First message invocation:

Latency for thread 0: 0.0300781 seconds

Sleep for 15000 ms

Latency for thread 0: 0.0032901 seconds

The sample code is attached.

Why Are WCF Responses Slow and SetMinThreads Does Not Work?

wdong — Thu, 11 Feb 2010 06:53:00 GMT

Customers reported the following WCF performance issue recently:

· A WCF client sends 10 requests concurrently to a self-hosted WCF service on 10 different threads. The service operation is very simple and it simply sleeps for 1 second. But the latencies for the 10 requests on the client side were distributed from 1 to 3.3 seconds. Why is this?

First of all, WCF uses managed I/O threads to handle requests. The CLR ThreadPool keeps a certain number of idle I/O threads from being destroyed. When more I/O threads are needed, they are created by the ThreadPool, which is kind of expensive. The number of idle threads is specified by the “MinIOThreads” setting. You can use the ThreadPool.GetMinThreads() API to check what settings you have for your application. By default in a standalone application, this setting is the number of the CPUs that you have on the machine. For example, on my laptop with 2-core, this setting is 2. I observed the following delay numbers in the above scenario:

Thread 0 takes: 1009 ms

Thread 2 takes: 1286 ms

Thread 3 takes: 1799 ms

Thread 4 takes: 2016 ms

Thread 6 takes: 2241 ms

Thread 5 takes: 2256 ms

Thread 7 takes: 2752 ms

Thread 8 takes: 2766 ms

Thread 9 takes: 2967 ms

Thread 1 takes: 3315 ms

Because of this, you would want to bump up this MinIOThreads setting with the ThreadPool.SetMinThreads() API on the service side, for example:

ThreadPool.SetMinThreads(Environment.ProcessorCount, 10);

You may notice that this still does not solve the above problem with .NET 3.5. Why?

Eric, who is the CLR ThreadPool expert, told me that there was a bug in .NET 3.5. ThreadPool does not honor what SetMinThreads() provides. So the above logic does not work. Fortunately this has been fixed in .NET 4.0 and the Microsoft has provided the following QFE for 3.5:

http://support.microsoft.com/kb/976898

Once you have this QFE installed, you would get much better result:

Thread 1 takes: 1026 ms

Thread 0 takes: 1028 ms

Thread 3 takes: 1081 ms

Thread 4 takes: 1074 ms

Thread 2 takes: 1084 ms

Thread 6 takes: 1072 ms

Thread 5 takes: 1073 ms

Thread 8 takes: 1029 ms

Thread 9 takes: 1021 ms

Thread 7 takes: 1154 ms

The sample code is attached.

Crack Throttling: Debugging WCF TimeoutException for a Middle-Tier Application

wdong — Sun, 06 Sep 2009 10:04:00 GMT

Recently I helped a customer with a few friends to debug a TimeoutException issue happened in their middle-tier which uses WCF proxies. The issue turned out to be simple but the experience was quite interesting and I would like to tell you how we proceeded with it.

Problem

The customer’s scenario is a multi-tier application as following:

ASPX clients [AC] ó ASPX invokes WCF clients [FE] ó WCF service (Basic Http with Binary) [BE] ó Backend services (over NamedPipe/HTTP) [BS]

The “ASPX clients” ([AC]) are web clients for a load testing. They send about 60 – 100 concurrent requests to the frontend ASP.NET web pages ([FE]) which are hosted on a front-end server machine. The web pages use WCF proxies to invoke the backend WCF service ([BE]) which is hosted on a back-end server machine. And then the WCF service invokes other backend services ([BS]). All is based on request/reply message exchange pattern.

After a while, the front-end ([FE]) starts to get TimeoutException for some requests as showing in the WCF trace log:

<ExceptionType>System.TimeoutException, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>

<Message>The HTTP request to 'http://co1wacwb201/Conversion.svc' has exceeded the allotted timeout of 00:00:20. The time allotted to this operation may have been a portion of a longer timeout.</Message>

at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)

at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.End()

at System.ServiceModel.Channels.RequestChannel.EndRequest(IAsyncResult result)

at System.ServiceModel.Dispatcher.RequestChannelBinder.EndRequest(IAsyncResult result)

at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishSend(IAsyncResult result, Boolean completedSynchronously)

The SendTimeout of the binding was set to be 20 seconds. However, based on the statistics, the backend service ([BS]) should not take 20 seconds. The number of concurrent requests is not very high. So why did this happen?

Debugging

ASP.NET/WCF Settings

By working with the customer, we first checked the different throttling settings for both ASP.NET and WCF to see whether everything is running as expected. The following is the checklist that we have used.

1) WCF Service Throttling

The first step is to make sure that the backend has big enough WCF service throttling settings. It turns out that the backend has used quite large settings:

· MaxConcurrentCalls: it’s set to 100.

· MaxConcurrentSessions: it’s set to 100.

· MaxConcurrentInstances: it’s set to 200.

These settings should be big enough for the scenario given that number of concurrent clients. Actually the binding is BasicHttpBinding. So MaxConcurrentSessions should not matter at all here.

2) InstanceContextMode/ConcurrencyMode

As per my early blog, the default InstanceContextMode is PerSession and the default for ConcurrencyMode is Single. However, PerSession is equivalent to PerCall for BasicHttpBinding. So there should be no contention here if everything is used as default. After checking the code snippet, it turns out that the settings are not default:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single,

ConcurrencyMode = ConcurrencyMode.Multiple)]

public sealed class MyService

This usage allows concurrent requests for the same service operation and it is also fine.

3) ASP.NET Threading Settings

Now the next thing that I thought was to check the ASP.NET threading settings as per the whitepaper. The server machine is dual-proc/4-core and thus 8-way box. So the following default should be enough:

<system.web>

</system.web>

In this case, ASP.NET would be able to execute 12 * 8 = 96 concurrent requests. They should be also big enough.

4) MaxConcurrentRequestsPerCPU

As per Thomas Marquardt, this setting has a default value of 12 and it’s available through the following registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0

“MaxConcurrentRequestsPerCPU”=dword:0000000C

By default, this registry does not exist, but the value is 12. It’s also settable from the ASP.NET application pools setting file %windir%\Microsoft.NET\{Framework|Framework64}\v2.0.50727\Aspnet.config:

<system.web>

</system.web>

This setting again is good enough for the scenario as the throttling setting is actually 96.

5) Client Settings

To this point, it does not look like any other server settings would matter here. So we need to check whether all client requests were pushed to the backend WCF service. So we asked the customer to try to increase the “maxconnection” setting as per my previous blog entry:

<system.net>

</connectionManagement>

</system.net>

The customer did that but the problem did not go away after this setting.

To this point, I have run out of the throttling settings that I can think of. So the next step is to gather more data for the system.

Trace Log

We asked the customer to enable Warning and Error level WCF tracing to see whether the WCF service has generated any exception and silently fails the client, though it is quite unlikely to happen in usual case. As expected, the log did not show anything interesting. Only the front-end log showed the TimeoutException and the callstack that I pasted above. So the log did not help here.

WinDBG Dump

Process dump can usually tell us a lot of information of the server process. We can examine the managed objects, states, threads, and call stacks. The caveat is that a dump only reflects a single data point of the process and it does not have overall progress. So we ask the customer to generate the process dump with WinDBG.

We examined the objects in the heap and searched for the corresponding WCF objects such as ThreadPool, FlowThrottle, InputQueue, etc by using the SOS debugging extension. We did not find anything wrong with those. We noticed that there were 9 I/O threads:

00000000`0961f160 000007fe`f8aaf0ac mscorwks!ThreadpoolMgr::CompletionPortThreadStart(void * lpArgs = 0x00000000`00000000)+0xee

00000000`0961f200 00000000`7737466d mscorwks!Thread::intermediateThreadProc(void * arg = 0x00000000`00000000)+0x78

There are totally 48 CLR worker threads:

00000000`0497fd70 00000000`7737466d mscorwks!Thread::intermediateThreadProc(void * arg = 0x00000000`00000000)+0x78

And there are about 41 other threads. With this data, we could not figure out the root cause of the timeout issue yet, especially when the process dump is only at a single time stamp. So at this moment, we moved on to look for other data first.

NetMon Capture

In many cases, Network Monitor capture data can provide very detailed information and can be very useful for analyzing network performance and reliability issues. We can analyze the TCP packages and frames and examine the conversations to find out how the data were transferred between the client and the server. The tricky part is that it took about an hour to reproduce the problem and the captured data size was pretty big (around 500MB). Searching for correlated requests between the client and the service was quite a challenge. This is another hard route as using WinDBG to debug the problem. So we skipped this step for this investigation for now.

Client Latency Distribution

We further gathered the distribution data for the client latency as the following graph. It shows that that system worked just fine in the first 92 seconds. All requests completed in sub-seconds. Then it started to show erratic behavior with longer latencies from 4 up to 12 seconds in the subsequent half minute. We need to understand better why this started to happen. So next we really needed to check the distribution for other data types on the front-end and the back-end machines.

Performance Counter Data

A more useful view of the system is to use the data for the ASP.NET and WCF performance counters. Performance counters are very light-weighted and it’s easily to gather the data without affecting the product performance much.

ASP.NET performance counters are turned on by default. We just need to enable WCF performance counters. In order to enable WCF ServiceOnly performance counters, you can add the following to the application configuration file (such as web.config in this case):

<system.serviceModel>

</system.serviceModel>

The following is a list of interesting performance counters that we used to investigate the issue:

Perf Counter Category	Perf Counter Name
ASP.NET	Request Execution Time
ASP.NET	Requests Queued
ASP.NET	Request Wait Time
ASP.NET Applications	Requests Executing
ASP.NET Applications	Pipeline Instance Count
ServiceModelService 3.0.0.0	Call Duration
ServiceModelService 3.0.0.0	Calls Outstanding

To find out more details, you can check the following two articles:

1) ASP.NET Performance Monitoring guide: http://msdn.microsoft.com/en-us/library/ms972959.aspx

2) WCF Performance Counters: http://msdn.microsoft.com/en-us/library/ms735098.aspx

Here is the data that we got from the customer:

This graph gives us very interesting data:

1) The CPU usage on both the front-end server FESERVER1 and the back-end server BESERVER002 is low. So there shouldn’t be traffic congestion/starvation problem.

2) There are no queued requests in the ASP.NET queues because “Requests Queued” counters are 0 on both machines.

3) Both the “Pipeline Instance Count” and the “Requests Executing” counters are pretty high on the front-end server FESERVER1. They are 91 and 69 in average respectively and they matched the number of concurrent clients.

4) However, the above counters are very low on the back-end server BESERVER002. They are 11 and 7 in average respectively. This is a very interesting data point.

From the above data, basically all of the requests are accumulated in the front-end server and they cannot be pushed all the way to the backend. The requests did not reach to the ASP.NET layer on the backend server. Something must be set incorrectly. What could have blocked the requests from being pushed to the backend server?

We checked the code usage of WCF client proxies in the front-end. The code did not use cached proxies. In this case, there should be no contention in the code. We have to check the configuration again.

Configuration Rechecking

Based on the analysis, we believed that that requests were blocked by the throttling in the front-end. We still have the doubt of the System.Net’s “maxconnection” setting.

My friend Mauro Ottaviani spotted an incorrect setting in the machine.config:

</section>

The highlighted line was incorrect placed in the section handler definition part of machine.config instead of under the <system.net> node. The system simply ignored the above <add> element and thus used the default values.

Resolution

After fixing this setting, the customer re-ran the scenario and the test passed without getting into the TimeoutException again. Here is the updated performance counter data:

As from this new graph, we noticed the following:

1) The number of concurrent clients has been increased. The max "Requests Executing" now is 457.

2) The system reached much higher concurrent requests on both the front-end and the backend servers, which is great. For example, the average “Requests Executing” is 42 now. This is a good symptom.

3) There was some high latency in the backend server. The max of “Calls Duration” for WCF service is 5.05 seconds. It is not far from the client timeout setting 20 seconds. For the front-end, it can reach 11 seconds. This may be a problem for the long running system. The backend service must have done some heavy-weighted work. We need to either increase the timeout or optimize the backend service logic more to make it faster.

4) The good news is that much higher CPU usage on both frontend and backend were observed. Apparently the system runs much more efficiently. Also higher throughput was observed from the system which was not recorded in the performance counters.

At this point, we have got the problem solved.

WCF 4: Higher Default Throttling Settings for WCF Services

wdong — Sun, 26 Jul 2009 09:36:00 GMT

It has been quite a while that I did not write anything. This is because I was fully focusing on WCF 4. Now that we have got most of the work done, it is good time to continue my writing. Starting from this blog entry, I will show you performance related changes in WCF 4. Please let me know what you think about WCF performance and what you would like to see in future releases.

When WCF was first released in .NET Framework 3.0, the default values for WCF Service throttling settings such as MaxConcurrentSessions and MaxConcurrentCalls were set very conservatively. For example, here are the default settings in 3.0:

MaxConcurrentSessions (default: 10)
MaxConcurrentCalls (default: 16)
MaxConcurrentInstances (default: 26)

These settings are not enough for many real production environments. With limited concurrency due to the low settings, the power of the production server is normally not well utilized. So people have to bump up these values for enterprise applications. However, it is hard for people to figure out how to set these values correctly. Most people would set very high numbers or even int.MaxValue to workaround the throttling which defeats the purpose of the throttling. The main purpose for the throttling settings can be classified into the following two aspects:

1) Controlled resource usage: With the throttling of concurrent execution, the usage of resources such as memory or threads can be limited to a reasonable level so that the system works well without hitting reliability issues.

2) Balanced performance load: Systems always work in a balanced way when the load is controlled. If there are too much concurrent execution happening, a lot of contention and bookkeeping would happen and thus it would hurt the performance of the system.

In WCF 4, we have revised the default values of these settings so that people don’t have to change the defaults in most cases. Here are the main changes:

· MaxConcurrentSessions: default is 100 * ProcessorCount

· MaxConcurrentCalls: default is 16 * ProcessorCount

· MaxConcurrentInstances: default is the total of the above two, which follows the same pattern as before.

Yes, we use the multiplier “ProcessorCount” for the settings. So on a 4-proc server, you would get the default of MaxConcurrentCalls as 16 * 4 = 64. The basically consideration is that, when you write a WCF service and you use the default settings, the service can be deployed to any system from low-end one-proc server to high-end such as 24-way server without having to change the settings. So we use the CPU count as the multiplier.

We also bumped up the value for MaxConcurrentSessions from 10 to 100. Based on customer’s feedback, this change fits the need for most applications.

Please note, these changes are for the default settings only. If you explicitly set these settings in either configuration or in code, the system would use the settings that you provided. No “ProcessCount” multiplier would be applied.

Scale WCF Application Better with Asynchronous Programming

wdong — Mon, 09 Feb 2009 09:53:00 GMT

WCF is a framework that is asynchronous inside out. Internally, asynchronous patterns are used to achieve non-blocking calls between different layers inside WCF. Externally WCF exposes the asynchronous patterns for both client and service programming.

Why Asynchronous Pattern?

From performance perspective, it’s not always ideal to use asynchronous programming if you don’t have to. Any asynchronous pattern involves at least some thread switches and thus causes delays to requests. So you should not abuse it if you don’t have to.

However, for blocking calls such as I/O operations or remote calls, you should always thinking about using asynchronous patterns. In those cases, you would be able to reduce number of threads used in the system while waiting for operations to complete. Here are a few cases where you would like to use asynchronous patterns:

1) A WCF service operation is waiting for a database operation to complete.

2) A WCF service operation calls into another WCF client proxy of a backend WCF service.

3) A UI thread waits for data from WCF client calls.

4) Any other case when the calling thread is blocked to wait for a slow operation but non-CPU bound operation to complete.

If you are using synchronous calls for the above cases, you would clearly see the threads are blocked from a dump of the process. This is wasting resources and you are preventing more requests from being processed if the system is capable of. This becomes more serious if you are holding hundreds or thousands of threads. So whenever you are seeing a thread blocked waiting for something, you would have to think about why that is the case? Why can’t the thread go away and come back with a different thread when there is real work to do?

Of course in reality, there are a small number of threads doing bookkeeping work, for example, they are doing event wait handling, finalizing, garbage collection, AppDomain unloading, ThreadPool gating, timer handling, etc. The rest of threads are worker threads or I/O threads that do the real work and they are what you need to pay attention to.

Basic Asynchronous Pattern with IAsyncResult

There are a few common asynchronous programming patterns with .NET Framework. Here is a rough list:

1) Begin/End pattern with IAsyncResult

2) Raw Callback model using a delegate as in C++ world

3) Implicit callback model with synchronous APIs

WCF advocates the first approach from the high level and uses it in most of its implementation. For performance critical parts, WCF uses 3) to avoid extra GC costs dealing with IAsyncResult objects. Here I just want to describe the basic idea of how to use 1).

The basic pattern of using IAsyncResult is to expose a pair of Begin/End operations from your type as following:

public class IAsyncWork

{

public IAsyncResult BeginDoWork(MyPropertyBag propertyBag, AsyncCallback callback, object state);

public WorkResult EndDoWork(IAsyncResult result);

}

When a caller invokes the methods of this interface, it supplies some custom arguments (here it’s “propertyBag”), a callback delegate, and the state object that will be consumed by the callback. The callback delegate wraps a callback method of the caller with the following type:

public delegate void AsyncCallback(IAsyncResult ar);

For best performance, here are two principles when you call/implement the above asynchronous pattern:

· Principle 1: Do not do heavy-weighted work inside the Begin method (which is BeginDoWork above).

The reason for this is that you should return the calling thread as soon as possible so that the caller can schedule other work. If it’s a UI thread, the application needs to use the thread to respond to user inputs. You should always put heavy operations in a different thread if possible.

· Principle 2: Avoid calling End method (which is EndDoWork above) on the same thread of the Begin method.

The End method is normally blocking. It waits for the operation to complete. If you implement the End method, you would see that it actually calls IAsyncResult.WaitHandle.WaitOne(). On the other hand, as a normal implementation (for example, the sample attached in this Blog entry), this WaitHandle is a delay allocated ManualResetEvent. As long as you don’t call it, it would be not allocated at all. For fast operations, this is pretty cheap. However, once End is called, you would have to allocate it. The right place to call End is from the callback of the operation. When the callback is invoked, it means that the blocking work is really completed. At this point, you can call End to get data retrieved without sacrificing performance.

In online WCF samples, you would find a typical implementation IAsyncResult. I also included one in the sample attached.

Service Asynchronous Pattern

WCF provides asynchronous service operation contract support. An asynchronous service operation can be defined as following:

[ServiceContract

public interface IAsyncOrderService

{

[OperationContract(AsyncPattern=true)]

IAsyncResult BeginGetOrders(int numOrders, AsyncCallback callback, object state);

Order[] EndGetOrders(IAsyncResult result);

}

As you can see, other than the ServiceContract/OperationContract attributes, the interface is exactly the same as standard asynchronous patterns. One thing to note is that it is not your code but WCF that calls into these methods. So the operation names have to follow the Begin/End convention.

Another minor thing to note is that, if you also have a synchronous operation for the service that matches the signature of the asynchronous operation as following, WCF would call this synchronous operation instead of the asynchronous operation:

Order[] GetOrders(int numOrder);

You may have control of which one to call in future releases.

Asynchronous Database Access

Now it’s time for you to think about your own service implementation. In order to leverage the asynchronous advantage of WCF, you need to implement your service to be purely asynchronous so that no threads are blocking.

In the sample that I attached to this blog, I used SQL database access with System.Data API as an example. For simplicity, I used SQL Express which is available for free. To get asynchronous support, you need to use the “Asynchronous Processing” property of the connection string as following:

const string ConnectionString = @"Server=LOCALHOST\SQLEXPRESS;Database=MyOrderDatabase;

Integrated Security=True;

Asynchronous Processing=True";

Once you have the asynchronous support from the SqlConnection, you can invoke the asynchronous API SqlCommand.BeginExecuteReader as following:

IAsyncResult sqlAsyncResult = this.command.BeginExecuteReader(sqlGetOrdersCallback, this);

With this, you would have a good asynchrony for your service side.

Client Asynchrony

On the client side, you can also asynchronous pattern to access the service. When you invoke the service operations, you would call into the asynchronous methods of the service contract. For example, the sample calls it in the following way:

IAsyncResult result = client.BeginGetOrders(driver.numOrders, onGetOrders, this);

It supplies a callback delegate that handles the response when the operation is completed.

One important thing to note is that WCF is a fully decoupled platform. This means that the client-side asynchrony has nothing to do with the service-side asynchrony. So an asynchronous client can basically invoke either a synchronous or an asynchronous service. The decoupling happened at the transport layer when data is serialized or deserialized as byte arrays and being transmitted through the transport.

If you are building N-tier WCF services, you would have WCF service operations invoking WCF client proxies for other backend services. In this case, you would need to make sure that the middle-tier (routing layer) has asynchronous service operation invoking asynchronous WCF proxy operations. In this way, your middle-tier won’t run out of threads when processing many slow operations.

Exception Handling

The logic of exception handling for asynchronous programming is more complicated than synchronous programming. Basically when an exception is thrown from a callback thread, you should not let it bubble up to the caller, which is almost the top of the callstack of a worker thread or an I/O thread. If you do so, your application would blow up as no other code would catch the exception and handle it.

To best handle an exception happened from the callback thread, you would need to record the exception as a state somewhere, for example, in the AsyncResult as showing in the sample. At some point, when the End method is invoked, this exception will be thrown on that thread and thus causes the exception to be thrown on the caller’s thread.

In the sample attached, we showed this once in SqlGetOrdersAsyncResult. In its callback “OnSqlGetOrdersCallback”, it catches the exception and passes to the AsyncResult.Complete method which then stores the exception there. When the SqlGetOrdersAsyncResult.End is invoked from WCF, this exception would be rethrown from AsyncResult.End. In this way, WCF would be able to catch the exception internally and convert it into a fault and send it to the client side.

Sample Setup

The attached sample implements a simple performance test that demonstrates the 3-tier model as showing above: WCF client, WCF service, and SQL database. The code is pretty self-explanatory and I copy the readme file here:

Prerequisite

1) It’s better to use Windows 2008 so that you can leverage the asynchronous WCF HttpModule/HttpHandler that were released in .NET 3.5 SP1. Check here.

2) Visual Studio 2008 or above installed.

3) SQL Express or SQL server installed

4) IIS, IIS Management tools, and WCF HTTP Activation components installed.

Steps to Run the Test

1) Open the solution file "WCFOrderService.sln" with VS 2008 and compile the whole solution with "Release" type.

2) Run "SQL\SetupSQL.bat" to create database "MyOrderDatabase".

3) Run "OrderCreator\bin\Release\OrderCreator.exe 1000" to insert 1000 distinct orders into database.

4) Open IIS Manager, create a virtual application that points to the full path of "OrderService".

5) Use IIS Manager to configure the default ApplicationPool to have identity "OrderPerfUser". This user was created in step 2) above.

6) Browse the OrderService.svc file in the browser. Make sure that you can see the WCF service correctly responds to the request from browser.

7) Run the client "OrderClient\bin\Release\OrderClient.exe"

8) Tune different parameters in the OrderService\web.config and OrderClient\bin\Release\OrderClient.exe.config

9) Use the tool "Tools\WcfAsyncWebUtil.exe" to configure WCF to use the async HttpModule and increase the ASP.NET registry "MaxConcurrentRequestsPerCpu". Run the test again to see the difference.

10) Open PerfMon to monitor requests such as "Calls Outstanding" counter.

Why Only Two Concurrent Requests for WCF Load Testing?

wdong — Sun, 08 Feb 2009 10:05:00 GMT

This is not a new topic in the world of traditional web service usage. But I was still asked by people who use WCF. So I think it would be valuable to post a blog entry here. One of our customers asked me the following question:

“We are using WCF in our application on Windows 2008 Server. We found the web role instance only have two concurrent threads to handle the incoming requests no matter how many requests we throw into it. We tried 50, 100, 200 clients sending requests to the WCF service at the same time. But web role only have two threads to handle it. We used the following settings in the web.config to set up the number of threads for WCF. But it seems not work. Why is that?”

<serviceThrottling maxConcurrentCalls="32" maxConcurrentInstances="100" maxConcurrentSessions="20"/>

When I looked at this question, my first response is that: the client did not really send enough requests to the server. Why is that? Here are the reasons:

1) If you use the synchronous WCF HttpModule/HttpHandler (installed by default), you would get the maximal number of concurrent requests (held by that number of ASP.NET worker threads) as 12 * [Number of CPU for the Server].

2) WCF throttling is specified above.

There is no other place that has a throttling of “2” on the server side. If you really want to know how many outstanding requests are processed by WCF, you can enable WCF performance counters and check “Calls Outstanding”. It will tell you that information.

Then the real problem should be on the load testing client. There is an apparent throttling on the client side that has default value 2: “maxconnection” from System.Net. This setting is according to RFC 2616, the RFC for Hypertext Transfer Protocol -- HTTP/1.1:

"Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion."

So for load testing purpose, you should increase this setting to an appropriate limit. You can find a lot of documents on how to set this setting. Let me simply re-iterate it here.

1) Using <connectionManagement> configuration

You can use the following configuration to tune this setting:

<add address="www.contoso.com"

maxconnection="16"/>

</connectionManagement>

You can use address=”*” to set the same setting for all remote addresses.

2) Setting it from code with ServicePointManager.DefaultConnectionLimit

You can also set the setting from code before you call into any web service:

System.Net.ServicePointManager.DefaultConnectionLimit = 16;

After this, you would see your client application generate more concurrent requests to the server.