Sajay

WCF Performance Counters

Sajay — Tue, 15 Sep 2009 09:50:38 GMT

Note: Cross posted from Sajay.
Permalink

We generally need to have a quick set of performance counters to identify a performance issue with a service. Shown below are three new counters that you will find with WCF 4.0. I also want to emphasize on the Calls outstanding counter here since this is one very u

PercentOfMaxCalls	Reports # of active requests as % of max calls	Reports the numbers of messages being processed + in the wait queue as a percentage of MaxConcurrentCalls throttle
PercentOfMaxSessions	Reports # of active requests as % of max sessions	Reports the numbers of active instances + calls in the wait queue waiting for an instance as a percentage of MaxConcurrentInstances throttle
PercentOfMaxInstances	Reports # of active requests as % of max instances	Reports the numbers of messages in the wait queue due to MaxConcurrentSessions throttle
CallsOutstanding ( from 3.0)	Reports the # of calls waiting to be completed	Reports the number of in-progress calls to this service.

Here you see that the %Instance throttle has maxed out. What do you think we should do? You can guess what throttle we are hitting here.

Some common questions when trouble shooting performance problems:

Why are my clients timing out, the service is showing 100% for % of MaxConcurrentCalls?

% of MaxConcurrentCalls gives you an indication of how much closer you are to hitting your max throttle value. This means that if you have a max concurrent call of 10 and you have 10 outstanding calls then you have utilized 100% of your throttle and there is no more work that your service can do. So if you see that your % maxConcurrentCalls is very high it probably indicates you have a very low throttle value. Remember the default is 16 till v3.5 and 16* proc count for v4.0. Before 4.0 you needed to work this out by checking the calls outstanding throttle and your web.config to figure out this value. These new % performance counters would help you identify if you are hitting the max values from now.

What if % MaxConcurrentSessions is showing 100% ?

We have bumped up the default number of sessions for service throttling behavior to 100 per CPU. But then again if you are seeing that %MaxConcurrentSessions is very high this means that you have exhausted all your session. This is probably because your clients are not closing either proxies and terminating sessions when you have less clients or possibly due to a very small MaxConcurrentSessions configuration.

How do I get the Message Body as an XElement?

Sajay — Thu, 03 Sep 2009 08:41:20 GMT

Note: Cross posted from Sajay.
Permalink

This was one was interesting as the service was exposed as a fully typed service, but the client wanted to modify some parts of the xml. Ideally you can plug into any part to perform these operations, but the general requirement here was that they needed a simple pointer to the Body and didn't care much about anything else.

For example you can get the XElement from the body using a simple operation and use the Message object on the client to get a pointer to the body.

[code:c#]

[ServiceContract(Namespace="http://www.sajay.com/")]
public interface IClientProxy
{
    [OperationContract(Action="http://www.sajay.com/getdata",
        ReplyAction="http://www.sajay.com/getdataresponse")]        
    Message DoWork();        
}
   
IClientProxy proxy = cf.CreateChannel();
Message msg = proxy.DoWork();                                    
XNode node = XElement.ReadFrom(msg.GetReaderAtBodyContents());

[/code]

So the service can return a typed object, but the client doesn't have to handle the message in the same way. Understanding the Message class here helps you appreciate how the serializer and encoder is decoupled from the message representation. Basically WCF does not constrain the native format of your message on either the producer's or the consumer's side. It merely conforms to some rules of invoking a set of abstract classes classes which you wire up by virtue of specifying the encoder and the serializer. The fact that we have XML semantics around with abstract classes like the XmlReader/XmlDictionaryWriter etc doesn't actually mean that WCF converts everything to XML before it can process the message. It merely uses these to invoke the required parts to obtain things like the header/body and other properties. The Message however can be represented natively in any form like binary/json/atom/ or any whacky type as long as you wire them up properly and can retrieve them in some manner.

This should give you some insight into how much more powerful the message model is compared to simple RPC style contracts. http://msdn.microsoft.com/en-us/library/ms734675.aspx With 4.0 and workflow services, you will see this becoming more and more the default way of thinking about messages and services.

HTTP polling duplex - Performance

Sajay — Fri, 28 Aug 2009 18:41:42 GMT

Note: Cross posted from Sajay.
Permalink

Really good article on Performance of HTTP polling duplex server-side channel in Microsoft Silverlight 3 by Tomek from the ServiceModel/SL team.

http://tomasz.janczuk.org/2009/08/performance-of-http-polling-duplex.html

Applying behaviors on the client channel

Sajay — Fri, 28 Aug 2009 10:58:00 GMT

Note: Cross posted from Sajay.
Permalink

When we need to modify or see the message before its sent or after its received, we generally can use a Message inspector. However sometimes we want to be a bit more granular. One such requirement was to perform some operations for 2-way calls and a different set of operations for one way calls. For this you can probably use extensibility points like the IParameterInspector and attach the required inspector to those appropriate operations.

public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
{
    foreach (ClientOperation operation in clientRuntime.Operations)
    {
        if (!operation.IsOneWay)
        {
            operation.ParameterInspectors.Add(new TwoWayBehavior());
        }
        else
        {
            operation.ParameterInspectors.Add(new OneWayBehavior());
        }
    }
}

Here is the source.(3.13 kb)

WCF 4: AddDefaultEndpoints

Sajay — Tue, 18 Aug 2009 03:38:00 GMT

Note: Cross posted from Sajay.
Permalink

Here is another little goodie we have which will help in reducing configuration. You can use the ServiceHostBase.AddDefaultEndpoints which will pretty much probe your service implementation for contracts and expose them as shown below. Notice that I don't need to specify anything more than the base address and the type of the service and I have actually commented out the call to the API. You can find more details about the simplified configuration here.

const string address = "net.tcp://localhost:8080/";
const string address2 = "http://localhost:8081/";
ServiceHost host = new ServiceHost(typeof(Service1), new Uri(address), new Uri(address2));
//host.AddDefaultEndpoints(); 
host.Open();

How do I get the workflow InstanceId?

Sajay — Sat, 15 Aug 2009 05:43:09 GMT

Note: Cross posted from Sajay.
Permalink

You can use a CodeActivity that returns a single result.

    public sealed class GetWorkflowInstanceId : CodeActivity<Guid>
    {
        protected override Guid Execute(CodeActivityContext context)
        {
            return context.WorkflowInstanceId;
        }
    }

Hello World Workflow Service 4.0

Sajay — Fri, 14 Aug 2009 07:53:00 GMT

Note: Cross posted from Sajay.
Permalink

Building a workflow service using 4.0 gives a very neat set of capabilities from both WF and WCF. Similar to WCF we can fully define a workflow either in code or otherwise just using Xaml. Here I chose a fully code based approach. If you just want the source you can download it here.

The Service Contract

We generally can take a workflow first approach or a contract first. To keep things simple, I have a very bare service contract as shown below.

[ServiceContract]
interface IService1
{
[OperationContract]
void GetData();
}

I also hold on to the contract description. This is just convenience and to avoid some constants like service name etc.

ContractDescription description = ContractDescription.GetContract(typeof(IService1));

The Operation

The idea here is to build the service bottom up from operation we need to perform. I can think of my workflow as composition of activities. The functionality of my service is quite simple here. All I do is write a message to the console. So the body of my operation will look something like this.

new WriteLine
{
TextWriter = Console.Out,
Text ="Hello Workflow."
},

The Implementation

Now I need to make this an operation. An operation has an associated message exchange pattern, commonly referred to as MEP. In my case it's a simple receive reply pattern. This means I have to compose my work in between a Receive and a SendReply. I am also going to tie this all up in Sequence as shown below.

Sequence sequence = new Sequence
{
    Activities =
    {
        receive,
        new WriteLine
        {
            TextWriter = Console.Out, Text ="Hello Workflow."
        },
        reply
    }
};

The Message Exchange

The next thing is to lay out my Receive and Reply which will define the wire format of the messages.

The receive needs to know the operation that the client can invoke. Also I need to make sure that the workflow can be instantiated by this particular operation. For this I need to set the CanCreateInstance property. Also the serializer used by default is the DataContractSerializer.

Receive receive = new Receive
{
OperationName = description.Operations[0].Name,
CanCreateInstance = true
};

The Reply is send back to the client using my SendReply activity. This reply is a part of a conversation and it needs to know which request it would respond to. I can do this by pointing the SendReply to the instance of the corresponding Receive. Also I don't have any data here so it is an empty parameter list. (the operation is not Messages contract based in my case but this is generally a better option for more control and performance.)

SendReply reply = new SendReply
{
Request = receive,
Content = new SendParametersContent { }
};

I now need to define the workflow service which would hold this Sequence and also need to specify the full name of the service definition. For this I can use the description object to get the name, namespace etc.

WorkflowService serviceDefinition = new WorkflowService
{
Body = sequence,
Name = XName.Get(description.Name, description.Namespace)
};

The Host

The next step is to give this definition to the workflow service host so that it can start the service. Here I self host the workflow as shown below.

WorkflowServiceHost host = new WorkflowServiceHost(serviceDefinition,
new Uri("http://localhost:8080"));
host.AddServiceEndpoint(serviceDefinition.Name, new BasicHttpBinding(), "");
host.Open();

The Test

Finally I can test the workflow using a simple proxy.

ChannelFactory<IService1> cf = new ChannelFactory<IService1>(new BasicHttpBinding(),
"http://localhost:8080");
IService1 proxy = cf.CreateChannel();
proxy.GetData();
((IChannel)proxy).Close();

Channel9: Monitoring WF Services

Sajay — Wed, 05 Aug 2009 00:27:56 GMT

Note: Cross posted from Sajay.
Permalink

10-4 Episode 24: Monitoring Workflow Services

by Ron Jacobs -

One of the great advantages to building services with WCF and Windows Workflow 4 is that the environment is instrumented with loads of events that allow you to track what exactly is happening. This is useful for health monitoring, troubleshooting and other scenarios like auditing and compliance. In this episode I'll take you through the monitoring lab from the Visual Studio 2010 training kit and show you how you can take advantage of these powerful capabilities.

Concurrent Receives - MaxPendingReceives

Sajay — Tue, 04 Aug 2009 01:15:30 GMT

Note: Cross posted from Sajay.
Permalink

One of the performance improvements we did with WCF 4.0 was to enable concurrent receives. This greatly helps scenarios where we need to do some kind of work like DB authentication with username passwords or custom channels that may need to log something. Primarily if any path during message receive requires some concurrency, then this is the knob you need to consider.

Here are some common questions and answers regarding concurrent receives and should also help you decide if you need this and if you do what value would benefit you.

<endpointBehaviors>
      <behavior name="MyEndpointBehavior">
                  <dispatcherSynchronization maxPendingReceives="5" />
      </behavior>
</endpointBehaviors>

ServiceThrottlingBehavior defaults for .NET 4

Sajay — Tue, 21 Jul 2009 23:59:04 GMT

This is one of the new updates that you will find with the .NET 4 installation. This was a decision we reached after understanding the fact that almost all customers had to bump this number up to a sweet spot of around 100 sessions. However the initial decision of keeping a very low number was a conscious one. We wanted to make sure this throttle would be hit really early on the development cycle and would cause a forced learning of service throttling behavior and how you need to think about performance of your service.

	3.0 to .NET 3.5 Sp1	.NET 4
MaxConcurrentCalls	16	16* Processor Count
MaxConcurrentSessions	10	100* Processor Count
MaxConcurrentInstances	Sum (26)	Sum (116*Processor Count)

There are few more goodies like more indicative performance counters to show you if your close to hitting this throttle value and if you need to bump this value up any more. For example if your 80% usage then you know that you have about 20% more throughput. But if your CPU is already fully loaded then probably you need to lower the values since you might actually have maxed out a much lower throughput value and it might be more optimal to process lesser calls.

Note: Cross posted from Sajay.
Permalink

What to Measure?

Sajay — Sat, 13 Jun 2009 08:37:02 GMT

Note: Cross posted from Sajay.
Permalink

If it doesn't run fast at first, it will run even slower later.

Performance Testing for Web application - NeXpert

Sajay — Tue, 09 Jun 2009 19:07:02 GMT

Note: Cross posted from Sajay.
Permalink

Here is a quick way to get an idea of how long your request/response takes and a ton of more data regarding your payload.

neXpert is an add-on to Fiddler Web Debugger which aids in performance testing web applications. neXpert was created to reduce the time it takes to look for performance issues with Fiddler and to create a deliverable that can be used to educate development teams.

Blog - http://blogs.msdn.com/nexpert/

Video- http://msdn.microsoft.com/en-us/dd573302.aspx - (4 minutes)

Download -http://www.microsoft.com/downloads/details.aspx?FamilyId=5975DA52-8CE6-48BD-9B3C-756A625024BB&displaylang=en

Introduction - http://blogs.msdn.com/nexpert/archive/2009/02/10/introducing-nexpert.aspx

Note: Cross posted from Sajay.
Permalink

The Channel Pump

Sajay — Tue, 09 Jun 2009 19:04:38 GMT

Note: Cross posted from Sajay.
Permalink

Deep dive - Nicholas Allen's talk on WCF performance and Scale - PowerPoint Deck - Webcast

From the above talk you get an idea of the pull and the push model and a combination of these for certain channels. This defines how the message will be received on the transport and how they will be dispatched. Consider a pump as a loop doing some work. A pump works off a queue which it would push work on for the next layer or pulls work off when there is some work to be done.

Considering a very common scenario like basic http where a message would arrive on the transport. The flow would consist of 2 loops.

Dispatcher/Message Pump

1. Dispatcher gets a message. It has available resources and will dispatch the operation(Application)
2. Dispatcher then tells the pump that it can accept the next message.

Channel Pump

1. Pump registers that it can take pull a message.
2. Message arrives and Transport pushes the message to the dispatcher.

All channels affect how messages arrive and how they would be dispatched. We make a fundamental assumption in WCF that channels would always be fast. This is because of the fact that the channel is your infrastructure and it makes sense to put any heavy work to the application layer and not the channel layer.

So general rule of thumb - Do not do blocking or heavy work in your channels. Pass on the work to the upper application layer.

You should think of your channel as gate to your dispatcher and you don't want to block the gate from letting the next person in. Hence bottom line is that a slow channel is a suboptimal channel.

AsyncResults and IAsyncResults

Sajay — Tue, 09 Jun 2009 19:02:46 GMT

Note: Cross posted from Sajay.
Permalink

Here is a deep dive on of how to implement your AsyncResult

http://www.sajay.com/page/Implementing-IAsyncResult.aspx

Quick Pointers

How to use an AsyncResult?

Sajay — Tue, 09 Jun 2009 19:01:51 GMT

Note: Cross posted from Sajay.
Permalink

I have a much longer article ~~in mind for this and will be publishing it out soon~~. But to quickly answer this let us use an implementation from the framework itself using a delegate. Please note, this is for a usage pattern sample only since all that a delegate does is put the work on the CLR's thread pool which means more latency for non-blocking work. ...