BizTalk Server Performance

MaxDequeueThread and MaxReceiveInterval in the adm_Service table

2008-02-08T22:36:00Z

One of the questions we were recently asked for was about the MaxDequeueThread and MaxReceiveInterval settings. Unfortunately, there is no useful description on them in the BizTalk Server R2 RTM documentation, and the question was 1) how to adjust them, 2) what they are for, and 3) when they could be used .
Both of the switches are used to mainly control throttle of pumping messages from the MessageBox to the Message Agent for outbound processing in the following image:

1) How to adjust them
You can customize the settings through the adm_Service table in the BizTalk management database. The table contains one record for each service type like Xlang, MSMQT, and “Messaging InProcess”. So you should be careful when you change this setting because this is a service-type-wide change.

2)      What they are for
1.The MaxDequeueThread setting is used to limit the number of threads in the NT thread pool for the dequeuing operation. The dequeuing operation calls a stored procedure, bts_DeQueueMesssages in the current implementation, to retrieve messages in a batch from the message box and pushes them into the in-flight message queue, which is an in-memory list. The number of threads that a host instance uses won’t exceed the MaxDequeueThread setting. The default is 5, and the minimum value will be technically 1.
2.The MaxReceiveInterval setting is used
    a.To specify the time interval in millisecond for which the throttle controller checks the system status to decide whether a throttle controlling is needed.
    b.To specify the time interval in millisecond for which the throttle controller delays to dispatch messages to the services under certain throttling conditions where the batch containing the messages was already created before the throttling condition starts. Throttle controller incrementally delays the message dispatch for this setting at a time up to for a time interval, which is dynamically adjusted depending on its stress condition.
    c.To specify the maximum time interval in millisecond for creating a thread for the dequeuing operation. In fact, if the MessageBox has remaining messages unprocessed, it starts new dequeuing operation thread right away without waiting. So, In high volume systems, this interval won’t be used at all. Once this value is used, the time interval will be dynamically changed between MaxReceiveInterval/10 and MaxReceiveInterval. The default value is 500 and the minimum value is 0.

3) When they could be used
They could be used for the low latency scenario, but if your SQL server can’t take the extra load from more frequent polling the server would cause then it would be negative impact. We don’t recommend modifying these setting as far as possible. But, if you have to, remember it would reduce latency only with the right deployment, solution, and hardware, and you have to test your solutions to make sure your assumptions.

How to manually recreate missing BizTalk performance counters

2007-10-01T08:00:36Z

To identify performance issues in a BizTalk solution, you should rely on the performance counters that BizTalk Server uses. All the BizTalk performance counters should be added to the registry when you run the configuration wizard. If the registry keys were overwritten somehow, those performance counters were gone. You could use the configuration wizard to recreate those missing BizTalk performance counters, but it requires reconfiguring the features that use the performance counters. To do so you should unconfigure them first, which is something you might not want to do. Even though it is undocumented, you can manually populate those resister keys for the missing performance counters.

There are two types of the performance libraries that BizTalk Server uses:

1. BTSPerfMonExt.dll for the performance objects that the unmanaged code of BizTalk Server uses.

2. netfxperf.dll for the performance objects that the managed code of BizTalk Server uses.

Note: netfxperf.dll is a .NET framework component. It is shared for all .NET performance counters. For more information about netfxperf.dll see .NET Performance Counters.

The following tables shows which performance library is used for each BizTalk performance object:

Object	Performance Counter Library	Installer
BizTalk:Message Box:*	netfxperf.dll	Microsoft.BizTalk.MsgBoxPerfCounters.dll
BizTalk:Message Agent BizTalk:File Receive Adapter BizTalk:File Send Adapter BizTalk:HTTP Receive Adapter BizTalk:HTTP Send Adapter BizTalk:POP3 Receive Adapter BizTalk:FTP Receive Adapter BizTalk:FTP Send Adapter BizTalk:MSMQ Receive Adapter BizTalk:MSMQ Send Adapter BizTalk:SOAP Receive Adapter BizTalk:SOAP Send Adapter BizTalk:SMTP Send Adapter BizTalk:SQL Receive Adapter BizTalk:SQL Send Adapter BizTalk:Messaging BizTalk:Messaging Latency	BTSPerfMonExt.dll	BTSPerfMonExt.ini BTSPerfMonExt.h
BizTalk: BAS TPM Management Web Service	netfxperf.dll	Microsoft.BizTalk.KwTpm.TPMgmtWSPerf.dll
BizTalk: BAS TPM Publishing Web Service	netfxperf.dll	Microsoft.BizTalk.KwTpm.TPPubWSPerf.dll
BizTalk:Windows SharePoint Services Adapter	netfxperf.dll	Microsoft.BizTalk.KwTpm.WssAdapter.Runtime.dll
BizTalk:TDDS	netfxperf.dll	Microsoft.BizTalk.Bam.EventBus.dll

To recreate the performance counters using netfxperf.dll, you can use InstallUtil.exe in the .NET Framework. If the assembly is installed in the GAC, you can run InstallUtil.exe as follows:

Installutil /i /assemblyname "<InstallerFileName>, Version=<VersionNumOfTheInstaller>, Culture=neutral,
PublicKeyToken=<PublicKeyOfTheInstaller>"

For example,

InstallUtil /i /assemblyname "Microsoft.BizTalk.MsgBoxPerfCounters, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"

If it is not, you can run InstallUtil.exe as follows:

InstallUtil /i <Installer>

For example,

InstallUtil /i Microsoft.BizTalk.KwTpm.TPMgmtWS.dll

To recreate the performance counters using BTSPerfMonExt.dll, you can use lodctr.exe and unlodctr.exe as follows:

Note: lodctr.exe is a tool for registering new performance counters, and unlodctr.exe is for removing performance counters. For more information about these tools, see How to manually rebuild Performance Counter Library values.

1. Copy the BTSPerfMonExt.ini and BTSPerfMonExt.h files to the %windir%\system32 folder.
2. At a command prompt, go to the the %windir%\system32 folder, and then type the following command to uninstall the damaged performance counters.

unlodctr "BTSSvc.3.0"

3. Then, type the following command to reinstall the BizTalk performance counters:

lodctr BTSPerfMonExt.ini

Thanks,
Young

Threads, DB sessions, and throttling

2007-09-28T19:52:00Z

[Note: This text-wrapping due to the size of the images and the site layout may make this post hard to read. If so, you should have a better experience with the copy pasted to http://biztalkperformance.members.winisp.net/BlogPages/MaxWorkerThreads.htm ].

I recently worked on a puzzling problem where spikes in load to BizTalk in a large environment would cause some unpleasant SQL Server problems. This turned out to be directly related to the .NET thread-pool size in the BizTalk hosts, which is configurable via the registry key ‘MaxWorkerThreads’, and can be set as described in http://msdn2.microsoft.com/en-us/library/aa561380.aspx . I’ll attempt to outline the issue here from the perspective of understanding and addressing the root cause, rather than a blow-by-blow chronology of troubleshooting the issue.

The internals

The orchestration engine in BizTalk does most of its work on worker threads from the .NET thread-pool. This is easily verified by doing some sample based profiling of the running XLANG host using the excellent Visual Studio performance tools. Below you can see the bulk of the time spent in a call stack that goes from the thread pool’s “_ThreadPoolWaitCallback” method into the XLANG scheduler’s “RunSomeSegments” method. If you drill further into this call stack you’ll see much time is spent in “BTSDBAccessor.dll”. What this tells us is that XLANG does most of its work, including accessing the BizTalk databases to send and receive messages, on .NET thread-pool worker threads.

Due to this, the actual number of running orchestrations at any one time is directly limited by the size of the .NET thread pool. You can verify this with the performance counter “\XLANG/s Orchestration(hostname)\Running Orchestrations” and see if it ever exceeds your “thead-pool size * number of CPUs”.

Now it should be no surprise to anyone who has worked with BizTalk that one of your key performance limiters is the backend SQL Server database – specifically the MessageBox. Typically your running orchestrations will be spending a lot of time reading and writing to the MessageBox. So what happens if your existing pool of database sessions are all busy serving other instances and your current instance needs to interact with the message box? Well, generally it will open another database session. Thus, under load, your .NET thread-pool size directly relates to your running orchestrations, which directly relates to how many database sessions the host could potentially open.

What about throttling based on DB Sessions you ask!? Well the way this works is that if you are using more database sessions than the number at which throttling is configured to kick in, then it will gradually introduce delays prior to database requests. This gives the existing connections in use a chance to become free, but doesn’t effectively block processing if above the throttling limit (this is a throttle after all, and not a hard limit). What this means is that (a) If the database requests are taking a long time (which is common under load) you will still open up more sessions, and (b) Under a quick spike in load you may open up a bunch of sessions for your running instances before throttling takes effect.

The testing

To demonstrate the effects a large number of suddenly busy database connections to the MessageBox can have, let’s do some testing. For the testing I created a scenario that uses MQSeries exclusively as the message transport (a common enterprise queuing platform). It includes some correlation with a delay (to ensure the subscriptions built up a little) and a couple of dynamic send ports. I also made the dynamic send ports run in parallel within an atomic scope, and constructed a new message before another final send. This is a simple scenario without any compensation, business rules, etc…, but includes a few of the complexities that exist in real business processes and should be plenty to keep the MessageBox busy. The solution is downloadable from this blog if you need further details.

For generating the load I again used some of the great new features put into Visual Studio - this time the Load Testing component included in the Team Suite or Test editions. I created a simple unit test to send a message with a unique OrderId to my MQSeries queue which is the receive location for my XLANG activation port. I then wrapped this inside a Load Test which allowed me to specify the number of concurrent users and a warm up time, as well as what performance counters I wanted to gather on what machines during the run. Note that a “user” in Visual Studio Load Testing effectively equals a thread which will loop through your unit test. In order to produce spikes in load I had each thread “sleep” for a fair amount of time in the unit test while simulating low load, then for one minute out of every ten I’d reduce this sleep considerably. This was a crude – but effective – method for simulating periodic “bursts” of traffic.

Attached to this blog is a link to my solution containing the BizTalk project, the Test project, and a helper class library for working with the MQSeries queues. (Note: You’ll need the MQSeries client bits installed on the Visual Studio box to run the load generation – or just change the projects to use a different transport for your test environment).

My environment was highly scaled out. My MessageBox was running on an 8-way SQL Server (with BAM/DTA/SSO on a separate SQL Server). I also had 10 dual-proc machines; 2 running the receive hosts, 2 running the send hosts, 4 running the XLANG hosts, one dedicated to running MQSeries, and one dedicated to Visual Studio (generating the load, monitoring the counters, etc…).

The results

For the first test the idea was to push SQL beyond its limits with the volume of database sessions created by BizTalk. For this I set the MaxWorkerThreads for all the hosts to 200. If you multiply this number of threads and by the number of active host instances across the deployment – this has the potential for a lot of connections! Running the scenario a few key counters can been seen below. (Note: If wondering what tool this graph is from, this is a screen shot of a section of the Visual Studio Load Test results UI).

200 ‘MaxWorkerThreads’, default (0) ‘DB Connections Per CPU’ throttling

It can be seen from the counters that I dropped ~59,000 test messages, averaged 31.4% SQL CPU on my MessageBox server, and peaked at 1,093 SQL connections! That’s a lot of connections. Of all the test messages dropped ~20,000 were received by the receive locations during the run, and only ~5,000 XLANGs instances completed.

The big problem here though is SQL. Why does it keep dropping to 0% CPU? Looking in the SQL event log gives the reason:

Event Type: Information

Event Source: MSSQLSERVER

Event Category: (2)

Event ID: 17888

Date: 9/21/2007

Time: 11:12:08 AM

User: N/A

Computer: BPI8X32PERF01

Description:

All schedulers on Node 0 appear deadlocked due to a large number of worker threads waiting on LCK_M_S. Process Utilization 0%

(Note: For details on what the SQL Server Scheduler is and the various causes of the above event, see the article at http://www.microsoft.com/technet/prodtechnol/sql/2005/diagandcorrecterrs.mspx ).

The above message appears several times. So SQL Server is effectively completely locked up to the point where it can’t do anything for extended periods. This causes various problems on the BizTalk hosts where you may see events such as the below:

Event Type: Error

Event Source: BizTalk Server 2006

Event Category: BizTalk Server 2006

Event ID: 6912

Date: 9/26/2007

Time: 11:13:21 AM

User: N/A

Computer: BPI2X64PERF04

Description:

The following stored procedure call failed: " { call [dbo].[bts_UpdateMsgbox_XlangHost]( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}". SQL Server returned error string: "Transaction (Process ID 378) was deadlocked on lock | thread resources with another process and has been chosen as the deadlock victim. Rerun the transaction.".

.. or ..

Event Type: Error

Event Source: XLANG/s

Event Category: None

Event ID: 10034

Date: 9/26/2007

Time: 11:13:22 AM

User: N/A

Computer: BPI2X64PERF04

Description:

Uncaught exception (see the 'inner exception' below) has suspended an instance of service 'BtsLoadScenario.SingleOrch(8d7a5913-ce72-0c75-7e41-a8729c4c50c7)'.

The service instance will remain suspended until administratively resumed or terminated.

If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.

InstanceId: ca11bfcf-b5d7-4115-b12d-d21b082b3094

Shape name: ConstructMessage_1

ShapeId: dfbab00f-9e42-4905-bd22-cb046262a86c

Exception thrown from: segment 1, progress 22

Inner exception: Exception from HRESULT: 0xC0C01B13

Exception type: COMException

Source: Microsoft.BizTalk.Interop.Agent

Target Site: Microsoft.BizTalk.Agent.Interop.IBTMessage ReadReferencedMessageByID(System.Guid ByRef)

The following is a stack trace that identifies the location where the exception occured

at Microsoft.BizTalk.Agent.Interop.IBTInstanceState.ReadReferencedMessageByID(Guid& msgID)

at Microsoft.BizTalk.XLANGs.BTXEngine.BTXMessageState._retrieveBTMessage(IStateManager ctx)

at Microsoft.BizTalk.XLANGs.BTXEngine.BTXMessageState._retrieveBTMessage()

at Microsoft.BizTalk.XLANGs.BTXEngine.BTXMessageState.GetPart(Int32 partIndex)

at Microsoft.XLANGs.Core.XMessage.GetUnderlyingPart(Int32 partIndex)

at Microsoft.XLANGs.Core.Part._slowProtectedRegisterWithValueTable()

at Microsoft.XLANGs.Core.Part.ProtectedRegisterWithValueTable()

at Microsoft.XLANGs.Core.Part.CopyFrom(Part part)

at Microsoft.XLANGs.Core.XMessage.CopyFrom(XMessage srcMsg)

at BtsLoadScenario.SingleOrch.segment1(StopConditions stopOn)

at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)

.. or ..

Event Type: Error

Event Source: XLANG/s

Event Category: None

Event ID: 10041

Date: 9/26/2007

Time: 11:16:39 AM

User: N/A

Computer: BPI2X64PERF04

Description:

Suspending due to exception occurring during dehydration of instance d1c19376-7008-4b37-ad95-ce2a6af85964.

Exception type: DehydrationFailedException

Additional error information:

Exception occurred when persisting state to the database.

Exception type: PersistenceException

Source: Microsoft.XLANGs.BizTalk.Engine

Target Site: Void Commit()

The following is a stack trace that identifies the location where the exception occured

at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.Commit()

at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.Dehydrate(Boolean fCommit)

at Microsoft.XLANGs.Core.Service.Persist(Boolean dehydrate, Context ctx, Boolean idleRequired, Boolean finalPersist, Boolean bypassCommit, Boolean terminate)

at Microsoft.BizTalk.XLANGs.BTXEngine.BTXService.PerformDehydration(Boolean wait)

at Microsoft.XLANGs.Core.Service.Dehydrate()

.. and ultimately perhaps ..

Event Type: Error

Event Source: Service Control Manager

Event Category: None

Event ID: 7031

Date: 9/26/2007

Time: 11:17:33 AM

User: N/A

Computer: BPI2X64PERF04

Description:

The BizTalk Service BizTalk Group : XlangHost service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.

Obviously SQL Server is not in a good way with so much work for the MessageBox over so many connections.

200 ‘MaxWorkerThreads’, 20 ‘DB Connections Per CPU’ throttling

After the above test I reset the environment and tested the “DB Connections Per CPU” throttling setting. I set this to 20, and as it’s a per CPU setting, this equates to 40 for the host instances on the test machines. The results of this test are below:

Here 58,400 test messages were sent to the receive queue. SQL CPU averaged a little better at 40.1% (though still dropped to 0% a few times), and SQL still peaked at 1,037 connections! A lot more of the messages were received at around 37,900, however the system still only got through 10,200 XLANG instances (double the previous test – but not great). Still worrying is the fact that SQL still often locked itself to the point of 0% CPU utilization, and I still suffered some of the same event log errors highlighted in the previous run.

The below chart showing different counters for this same run highlights a potential problem with this solution. If you’re throttling on “DB Connections”, then any time you get more active XLANG instances than this you may start to throttle (if they’re all trying to access the database at the same time – such as a burst of traffic). The counters below show that the sessions are much higher than the threshold, and thus “Message publishing throttling state” is pretty much stuck on ‘8’ (“Throttling due to high session count”). Due to this the publishing delay rises and rises until it maxes out at 5 minutes (the default 300,000ms). It resets around the 26 min mark when the host restarts. While marginally better than the first run, this is still a far from ideal result.

100 ‘MaxWorkerThread’s, default (0) ‘DB Connections Per CPU’ throttling

The next test is to drop the number of worker threads per host (the MaxWorkerThreads value in the registry for the host service). The results of this run are shown below:

Now this is more like it! ~57,700 test messages were dropped and this time the CPU utilization on SQL was a respectable 76.6%. The max SQL connections value is down to 694. Most importantly all dropped docs were received by BizTalk by the end of the run, and a whopping 39,400 XLANGs instances have completed! The throttling picture looks much healthier, with the publishing delay never getting above a few seconds for very brief periods due to rate imbalance (and a little for DB growth at the end). Delivery throttling for inbound/outbound rate imbalance and in-memory messages is respectable, and to be expected with such bursts of load.

Most importantly however in this last test – there was no SQL lock-ups and no errors on any of the BizTalk hosts! This environment is pretty solid even under extreme loads.

Default (25) ‘MaxWorkerThreads’, default (0) ‘DB Connections Per CPU’ throttling

As a final test everything was run with just the default settings. Without specifying the registry keys the .NET thread-pool will have a maximum of 25 threads. The results for this run are below:

The numbers for this are very similar to the previous test. ~58,600 test messages were dropped. SQL CPU utilization is a little higher at an average of 81.7%. Due to the lower number of threads the number of SQL connections has peaked at a much lower 480. Again, all dropped docs were received, and 36,700 XLANGs instances have completed. This is a touch lower (~7.4%) for completed XLANG instances than the prior test. The throttling graph for this last test was also very similar to the previous test – so I won’t include it here.

Summary

The final conclusion from this testing is that you need to be mindful of the load you are putting on the back-end as you scale out your environment. There may be valid reasons why you need more threads in your thread pool, (KB article http://support.microsoft.com/kb/900455 describes for example needing to increase this if using orchestrations that can’t dehydrate), but be aware of the equation “number of hosts * max thread-pool size * number of CPUs = potential DB sessions”. Too many connections all trying to read and write to your MessageBox at the same time can overwhelm SQL Server, locking up the system and reducing your throughput dramatically.

Be aware of the type of work your host is doing and the settings that control the threads used. The MaxWorkerThreads setting outlined here controls the threads available to the XLANG engine, and the SOAP & HTTP send adapters. The messaging engine however uses its own thread pool which may be set as described in the documentation at http://msdn2.microsoft.com/en-us/library/aa577604.aspx . If you do need a large thread-pool, then consider throttling based on the number of DB Sessions in order to reduce the volume of requests on those sessions under high load – this will only add limited stability however, and may not be sufficient protection in environments with large sudden bursts of work. Ideally, start with the defaults and only adjust your thread-pool if testing shows this to be beneficial and stable. In a well designed system, more threads often just means more context switching and locking, not more completed work!

We all know that testing is important for a successful implementation, but testing with a realistic environment is key to avoid unpleasant surprises in production. This means testing with realistic hardware (thread-pool size – as with many settings – is per CPU), an appropriately sized test environment (the number of running hosts, not just the work they do, is highly relevant – nothing scales linearly!), a realistic load pattern (throttling state, current sessions, etc… all behavior much differently under bursts than under steady load), and realistic response times (do correlated or response messages really return instantly in production? This directly effects how long your orchestrations run for and the number of active subscriptions at any one time – both key to performance).

I hope this has provided some insight and saved a few headaches.

- Bill Ticehurst

Back in the saddle

2007-09-28T04:19:00Z

The BizTalk Server Performance blog has been a little quite of late, but all that is about to change. We have a new release out of the door in BizTalk Server 2006 R2, and the team is fired up to give our customers the best advice to squeeze as much as possible out of their BizTalk solutions!

Leave us a comment if you have any areas of the product you think the Performance team should be writing about.

Regards,

- Bill Ticehurst

Some Links To BizTalk 2006 Performance

2006-10-20T19:00:00Z

With BTS2006 in full force, the BTS team is frequently fielding performance related questions. There is a lot of information already available for customers to reference; the product documentation is a great place to start.

The Performance and Capacity section in the core docs covers a wide range of topics. The 64-Bit Support article delves into components of BizTalk that are supported in 64 bit and also answers some basic FAQ regarding this topic.

There is a section that covers planning for sustained performance. It contains articles that provide useful guidance on how to include performance considerations during the design, implementation and release phases of BTS2K6 design. It also highlights how to set realistic performance goals and how to measure them. The documentation also contains guidance on scaling solutions as well as tips and tricks to improve performance. In addition to this, there is a paper available on how to manage a successful performance lab here.

Once a solution is deployed, there are also articles in msdn that cover the performance counters available in BizTalk to detect bottlenecks.

And if you just want to look at hard numbers, a comparative adapter study compares the performance of BizTalk Server 2004 against BizTalk Server 2006 adapters.

BizTalk Server 2006 Documentation Now Available

2006-03-09T04:00:00Z

Curious about BizTalk Server 2006 features and functionality? A preview of the final product documentation in .chm format is available here: http://blogs.msdn.com/luke/archive/2006/02/03/524534.aspx

Check out the performance content under the topic "Planning for Sustained Performance". Under this topic, we provide maximum sustainable throughput numbers with and without tracking, plus tips and tricks, scaling guidence, project planning, throttling, and more!

New BizTalk Load Generation Tool Now Available

2005-08-27T05:40:00Z

Overview

This tool is intended for developers and IT professionals to simulate load on a BizTalk Server. Using this tool, you can simulate load to instrument performance and stress against a BizTalk deployment. In addition, this tool may also be extended by developers to simulate load for custom transports. This tool should be used in a test environment only, and should not be used in a production environment. This tool is provided "as-is" and is not supported.

Download LoadGen Here: http://www.microsoft.com/downloads/details.aspx?FamilyId=C2EE632B-41C2-42B4-B865-34077F483C9E&displaylang=en

Get Connected to Free Product Support and Tremendous Online Collaboration

2005-07-14T20:51:00Z

Have you ever wanted to speak to Microsoft developers of a specific feature of BizTalk Server? I am sure your answer was “Yes let me at them”, so the Business Process Integration Division is extending an invitation to all customers to join our key feature developers, program managers, and testers in the following newsgroups:

microsoft.public.biztalk.accelerator.forsuppliers

microsoft.public.biztalk.newuser

microsoft.public.biztalk.accelerator.rosettanet

microsoft.public.biztalk.admin

microsoft.public.biztalk.appintegration

microsoft.public.biztalk.framework

microsoft.public.biztalk.general

microsoft.public.biztalk.library

microsoft.public.biztalk.nonxml

microsoft.public.biztalk.orchestration

microsoft.public.biztalk.sdk

microsoft.public.biztalk.server

microsoft.public.biztalk.setup

microsoft.public.biztalk.tools

microsoft.public.biztalk.xlangs

microsoft.public.biztalk.xsharp

We’ve been working very hard over the past year to connect with folks just like you and want to include you in our community of Most Valuable Professionals (MVP), developers, information technology professionals, chief information officers, chief executive officers, or any other role within large, medium, and small companies that hang out in our online newsgroup communities. We want to have you join in this vibrant online community to ask those questions you always wanted to ask but did not know where to go. Well, now you know where to go, we want you to come on in and join us!

If you are new to BizTalk Server, try out the NewUser newsgroup, Microsoft.public.biztalk.newuser.

We’re offering two levels of interaction with Microsoft Corporation employees as follows:

Managed Newsgroup Support
Unmanaged Newsgroup Support

Managed Newsgroup Support

MSDN managed newsgroups are available in English to MSDN Universal, Enterprise, Professional and Operating Systems subscribers to receive free technical support on select Microsoft technologies as well as to share ideas with other subscribers. MSDN managed newsgroups provide:

Unlimited on-line technical support - keep your PSS incidents
A commitment to respond to your post within two business days
Over 200 newsgroups to choose from
Spam protection for your e-mail address when posting items

Go to the following URL to sign up: http://msdn.microsoft.com/newsgroups/managed . These newsgroups are monitored by Microsoft support engineers and product group team member as described above.

Unmanaged Newsgroup Support

MSDN unmanaged newsgroups are available to all individuals.

Go to the following URL to participate: http://msdn.microsoft.com/newsgroups. These newsgroups are monitored by Microsoft product group members, other customers like you, most valuable professionals, and various other individuals.

Questions, suggestions, and direct feedback can be sent to me.

James Fort

BPI Community Lead

mailto:jfort@microsoft.com

Memory Growth in BizTalk Messaging

2005-04-16T16:03:00Z

By Kartik Paramasivam and Raied Malhas

In this document we list various reasons that might lead BizTalk to get into an Out Of Memory situation and then suggest mitigations/solutions for such conditions.

1. Introduction to BizTalk Hosts:
Before discussing Out of Memory conditions, let us talk about hosts and how adapters are mapped to hosts.

Adapters (receive and send) and orchestrations run under the BizTalk NT Service. An instance of the BizTalk NT Service is called a host instance. There can also be receive adapters (HTTP/SOAP) which can run under in any other process (e.g. IIS) instead of the BizTalk NT Service. Such adapters are called isolated adapters.
Hosts in BizTalk have 1 or more host instances (process). The default BizTalk installation has 2 hosts.
1) Default host
2) Isolated Host.

Note: You can only have 1 host instance per host on a given server.

The following is the mapping between end points (receive or send port) and hosts:
Send Port -> Send Handler -> Send Host
Receive location -> Receive Handler-> Receive Host

Typically you should create separate receive hosts for receive adapters and send hosts for all send adapters. Doing this gives each adapter a separate process to live in. This guarantees that one adapter will not adversely affect another. Also if your BizTalk host/process goes Out of Memory, you will know which components are running in that process.

2. The Obvious Suspects for Out of Memory conditions:

When the process goes Out Of Memory (OOM), you need to 1st find out what components are running in that process. If possible, follow the recommendations listed above for separating out receive/send adapters and orchestrations into different hosts. Once you have done that then you can check if one of the following 3 conditions exists in the process that is going OOM.

1) You are executing transforms/maps on relatively large messages in receive/send port or XLANG. The point here is that XSL transforms load the whole message in memory to transform them.

Solution 1: Decrease the number of messages that your process operates on concurrently (section on Send Host below might give you some idea).
Solution 2: Decrease the size of your XML message that you are trying to transform.

2) You are executing XML receive/send pipeline on a document that contains one or more items such as the following:
- Large attribute values
- Large element values
- Large attribute or element tags.

If one of the above applies to your XML document then that item is fully loaded in memory.

Solution 1: Try to limit the size of the above entities.
Solution 2: If you can’t limit the size, then make sure your process doesn’t concurrently operate on multiple documents. (See sections below on limiting concurrency)

3) You have a custom pipeline component or adapter, which loads the whole document in memory. Most of the components shipped with BizTalk (except transforms) support streaming as opposed to loading the whole document in memory, and hence have a low memory foot print. However, we have observed that custom pipeline components written by customers may or may not support streaming.

3. Out of Memory in Send Host:

The Send Host of BizTalk Adapters can run Out of Memory when it is processing a large number of messages (i.e. when the system is under high stress loads). This can cause the Send Host memory utilization to rise rapidly causing the system to get into an Out Of Memory condition. Usually, the larger the messages being processed the faster the memory utilization of BizTalk’s SendHost process will be.

Calculating the Maximum number of messages that will be loaded in-memory by BizTalk’s Send Host process:
Under high stress loads, the BizTalk send host process will load as many instances of messages into memory as it can until the number of messages exceeds the “HighWatermark”. By default in BizTalk Server 2004, SP1, this value is 200 for messaging. Below you’ll learn how to configure this value, more details on Watermark settings are available in the “BizTalk Server Performance Characteristics” document at: http://www.gotdotnet.com/team/wsservers/). So, if you have a Dual processor server that is processing a large number of messages, the BizTalk host will load a maximum of:
[HighWatermark Value] * NumberOfProcessors
= 200 * 2
= 400 messages in-memory

For a Hyper-threaded server:
If you have a hyper-threaded server, then the number of processors perceived by the BizTalk send host doubles the actual number of processors. So if you had a dual processor BizTalk server and if we assume that the default value for the maximum number of message instances processed at once is 200 (so, ”HighWatermark” value is equal to 200), then the BizTalk process hosting the Send Host will load a maximum of:
[HighWatermark Value]*[NumberOfProcessors for a Hyper-threaded machine]
= 200 * [NumberOfProcessors * 2]
= 200 * [2 * 2]
= 200 * 4
= 800 messages in-memory

So, in a case where you have a dual processor hyper-threaded BizTalk server with 2GigaBytes of RAM that’s under high stress, there is a possibility that SendHost will get into an Out of Memory state. The process in this case will be loading 800 messages (as calculated above) in memory. The following graph shows an example of such a case where the memory usage of the send host process grew very rapidly to 1.5 Giga Bytes in less than 15 minutes:

Graph1: An example of a test that hit an Out of Memory condition in a SendHost containing FILE Adapter

Note: The memory footprints of BizTalk hosts can be monitored using Performance Monitor to view “Private Bytes” counter for the required host (receive, send, etc…) instance under the “Process” object.

Note: On a 32 bit box, the process can only grow to around 1.5 GB Max (2 GB in some rare cases) even if you have more physical memory to spare (i.e. 8 GB RAM for e.g.)

How to avoid Out of Memory in the SendHost:
Fortunately, the Out of Memory condition can be avoided by configuring the setting that controls the maximum number of messages that can be loaded into memory concurrently.

To configure the maximum number of message instances, in BizTalk’s Management Database (default name is: “BizTalkMgmtDB”), open the “adm_ServiceClass” table. You’ll see for “Messaging InProcess” row, by default, the value for LowWatermark is set to 100 and for HighWatermark to 200. The HighWatermark value is what dictates the maximum number of message instances that can be loaded in-memory of the Send Host process. Change the Low and High Watermark values to lower numbers, an example of lowered values are:
LowWatermark = 25
HighWatermark = 50

If those values (25-50) still show OOM conditions, you should decrease those values more until you stop hitting the Out of Memory condition.

The following graph shows an example of the same test case that the graph above shows. In the graph below, the Low and HighWatermark values were changed to 25-50, respectively. Please note how the memory remained constant without growing to get into an Out of Memory:

Graph2: An example of a test that avoided hitting an Out of Memory in a SendHost containing FILE Adapter

Note: Decreasing the Watermark values can decrease Performance. So, only decrease those values if you run into an Out of Memory condition.

4. Out of Memory in Receive Hosts:

It is fairly rare to see a host containing a receive adapter to go out of memory. If that happens it is typically due to one of the cases listed in section 2.

If none of the suggested recommendations in Section 2 can be performed, then reducing the number of messages processed concurrently can help protect the receive host from getting into an out of memory condition.

Note: Remember however that by doing so, we decrease the concurrency and hence the throughput/performance will decrease.

1) Reduce the messaging engine thread pool size: A user can control the number of threads used by the messaging engine to publish messages into the message box. By reducing the number of threads used by the messaging engine we reduce the rate at which the receive adapter will receive messages into the message box. Remember this setting only needs to be done for the host corresponding to the receive handler for the adapter.

By default the messaging engine creates 10 threads/cpu for
publishing messages.

To specify the messaging thread pool size
1. Click Start, and then click Run.
2. In the Run dialog box, in the Open box, type regedit, and then click OK.
3. In Registry Editor, expand HKEY_LOCAL_MACHINE, expand SYSTEM, expand CurrentControlSet, expand Services, right-click BTSSvc* (you would first need to determine which of the BTSSvc* regkeys correspond to the host corresponding to the receive handler), point to New, click DWORD Value, type MessagingThreadPoolSize, and then press ENTER.
4. In Registry Editor, double-click MessagingThreadPoolSize.
In the Edit DWORD Value dialog box, in the Value data box, type a number between 1 and 30, and then click OK.

5. Note for Custom Adapters:

If you have verified all of the above conditions and if you think your custom receive/send adapter is causing the OOM conditions, then you need to look at the source code of your adapter and verify the following:

The following only applies to Adapters written in Managed code:
The adapters get an object of type IBTTransportBatch using the GetBatch() API on the TransportProxy object. Once the adapter is done using the IBTTransportBatch object the adapter needs to do a Marshal.ReleaseComObject( batch) in a loop to release the object.

Basically the .net GarbageCollector does not kick in in time to release the unmanaged memory. Hence failure to call ReleaseComObject() will result in what looks like a memory leak.

Understanding BizTalk Server 2004 SP1 Throughput and Capacity

2005-04-07T20:05:00Z

By Wayne Clark

What is Sustainable?

Of primary concern when planning, designing, and testing business solutions built on BizTalk Server 2004 SP1 is that the solutions must be able to handle the expected load and meet required service levels over an indefinite period of time. Given the number of solution architectures, configurations, and topologies possible on BizTalk Server 2004 SP1, there are many things to consider when evaluating a proposed or existing deployment. The purpose of this, our inaugural BizTalk Performance blog posting, is to provide guidance on:

Understanding BizTalk Server SP1 throughput and backlog capacity behavior and how to observe the behavior of your system.
Critical success factors when planning for capacity.

Let me start off by defining some terms and concepts:

Maximum Sustainable throughput is the highest load of message traffic that a system can handle indefinitely in production. Typically this is measured and represented as messages processed per unit time. Solution design choices such as the choice of adapters, pipeline components, orchestrations, and maps will all have a direct effect on system performance. In addition, BizTalk offers scale-up and scale-out options that provide flexibility when sizing a system. Often overlooked, however, are things like standard operations, monitoring, and maintenance activities that have an indirect effect on sustainable throughput. For example, Performing queries against the messagebox database (e.g., from HAT) will require cycles from SQL and effect overall throughput depending on the type and frequency of the query. Backup, archiving, and purging activities on the database also have an indirect effect on throughput, and so on.

Engine Capacity, also known as backlog capacity, is the number of messages that have been received into the messagebox, but have not yet been processed and removed from the messagebox. This is easily measured as the number of rows in the messagebox database table named spool.

BizTalk Server 2004 SP1 Backlog Behavior

BizTalk 2004 SP1 implements a variety of store-and-forward capabilities for messaging and long and short running orchestrations. The messagebox database, implemented on SQL server, provides the storage for in-flight messages and orchestrations. As messages are received by BizTalk 2004 SP1, they are en-queued, or published into the messagebox database so they can be picked up by subscribers to be processed. Subscribers include send ports and orchestrations. Some of the arriving messages activate new subscriber instances. Other messages arrive and are routed, via a correlation subscription, to a waiting instance of an already running subscriber such as a correlated orchestration.

In order for correlated orchestrations to continue processing, arriving correlated messages must not be blocked. To facilitate this, BizTalk does its best to make sure messages (both activating and correlated) continue to be received, even under high load, so that subscribers waiting for correlated messages can finish and make room for more processes to run. This means it is possible to receive messages faster than they can be processed and removed from the messagebox, thus building up a backlog of in-process messages. Being a store-and-forward technology, it is only natural for BizTalk to provide this type of buffering.

Every message that is received by, or created within, BizTalk 2004 SP1 is immutable. That is, once it has been received or created, its content cannot be changed. In addition, received messages may have multiple subscribers. Each subscriber of a particular message, references the same, single copy of that message. While this approach minimizes storage, a ref-count must be kept for each message and garbage-collection must be performed periodically to get rid of those messages that have a ref count of 0. There are a set of SQL Agent jobs in BizTalk 2004 SP1 that perform garbage collection for messages:

MessageBox_Message_Cleanup_BizTalkMsgBoxDb – Removes all messages that are no longer being referenced by any subscribers.
MessageBox_Parts_Cleanup_BizTalkMsgBoxDb – Removes all message parts that are no longer being referenced by any messages. All messages are made up of one or more message parts, which contain the actual message data.
PurgeSubscriptionsJob_BizTalkMsgBoxDb – Removes unused subscription predicates left over from things like correlation subscriptions.
MessageBox_DeadProcesses_Cleanup_BizTalkMsgBoxDb – Called when BizTalk detects that a BTS server has crashed and releases the work that that server was working on so another machine can pick that work up.
TrackedMessages_Copy_BizTalkMsgBoxDb – Copies tracked message bodies from the engine spool tables into the tracking spool tables in the messagebox database.
TrackingSpool_Cleanup_BizTalkMsgBoxDb – Flips which table TrackedMessages_Copy_BizTalkMsgBoxDb job writes to.

The first two from the above list are the ones responsible for keeping the messagebox cleared of garbage messages on a regular basis. To do their work, they sort through the messages and message parts looking for messages with a ref count of 0 and for parts that are not referenced by any messages, respectively, and removing them.

So, what does all this have to do with throughput and capacity?

When a system is at steady state, that is, processing and collecting garbage as fast as messages are received, this is clearly indefinitely sustainable. However, if for some length of time the system is receiving faster than it can process and remove, messages start to build up in the messagebox. As the amount of this backlog builds up, the amount of work that the cleanup jobs have to do increases and they typically start taking longer and longer to complete. In addition, the cleanup jobs are configured to be low priority in the event of a deadlock. As a result, when running under high load, the cleanup jobs may start to fail as the result of being the victim of deadlocks. This allows the messages being en-queued to have precedence and not be blocked.

As an example, let’s take a look at a system that we have driven at various throughput levels and investigate the observed behavior. The system is configured as follows:

Two BizTalk Servers – These servers are HP DL380 G3, equipped with dual 3GHz processors with 2GB of RAM. BizTalk 2004 SP1 is running on these two servers. Local Disks.
One SQL Server Messagebox – This server is an HP DL580 G2, equipped with quadruple 1.6GHz processors with 4GB of RAM. This server is connected to a fast SAN disk subsystem via fiber. The server is dedicated to the messagebox database and the data and transaction log files for the messagebox database are on separate SAN LUNs.
One SQL Server All Other Databases – This server is an HP DL580 G2, equipped with quadruple 1.6Ghz processors with 2GB of RAM. This server is also connected to the SAN. This server houses all BizTalk databases other than the messagebox, including the management, SSO, DTA, and BAM databases.
Load Driver Server – This server is an HP DL380 G3, equipped with dual 3GHz processors with 2GB of RAM. This server was used to generate the load for testing the system using an internally developed load generation. This tool was used to send copies of a designated file to shares on the BizTalk servers to be picked up by the file adapter.

The Test Scenario

The test scenario is very simple. The load generation tool distributes copies of the input file instance evenly across shares on both BizTalk servers. Using the file adapter (we’ll explore other adapters in subsequent blog entries), files are picked up from the local share and en-queued into the messagebox. A simple orchestration containing one receive and one send shape, subscribes to each received message. Messages sent back into the messagebox by the orchestration are picked up by a file send port and sent to a common share, defined on the SAN. Files arriving on the output SAN share are immediately deleted in order to avoid file buildup on that share during long test runs.

There are 4 hosts defined for the scenario, one to host the receive location, one to host orchestrations, one to host the send port, and one to host tracking. For the purposes of observing engine backlog behavior, tracking is completely turned off during the test runs. Turning tracking off involves more than just stopping (or not creating) and tracking host instance. To turn tracking completely off, use the WMI property MSBTS_GroupSetting.GlobalTrackingOption. For more information on turning tracking on and off using this property, please see: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/ht m/ebiz_sdk_wmi_msbts_groupsetting_fqxz.asp.

Both BizTalk servers are identical in that they each have instances of the receive host, orchestration host, and send host. No instances of the tracking host were created since tracking was turned off to isolate core messagebox behavior for these tests.

A simple schema was used and the instance files used for the test were all 8KB in size. No mapping or pipeline components were used inbound or outbound in order to keep the test scenario simple to implement and keep the behavior observations focused on the messagebox.

As a first step, the system is driven at a level near, but below, it’s maximum sustainable throughput so that observations of a healthy system can be made. The growth rate of the messagebox backlog is a key indicator of sustainability. Clearly, the messagebox cannot continue to grow indefinitely without eventually running into problems. So, the depth of the messagebox database backlog, monitored over time, is used to evaluate sustainability. The messagebox table named spool contains a record for each message in the system (active or waiting to be garbage collected). Monitoring the number of rows in this table, and the number of messages received per second, while increasing system load provides an easy way to find the maximum sustainable throughput. Simply increase the input load until either (a) the spool table starts to grow indefinitely or (b) the number of messages received per second plateaus, whichever comes first, and that is your maximum sustainable throughput. Note that if you are not able to drive a high enough load to cause the spool table to grow indefinitely, it simply means that the slowest part of your system is on the receive side, rather than the processing/send side. The following graph shows key indicators after using this approach to find the maximum sustainable throughput of our test system (described above).

The blue line shows the total messages received per second by the system (i.e., for both BizTalk servers), the pink line shows the running average of the messages received per second, and the yellow line shows the spool table depth (x 0.01) for the test duration time provided on the X axis. What this graph shows is that, for the hour of the test, the spool was stable and not growing and making the sustainable throughput equal to the number of messages received per second, in this case 150 msgs/sec.

Part of any analysis of a BizTalk deployment performance should include checking some key indicators to understand resource bottlenecks. The key measures and their values used for this deployment running under maximum sustainable throughput (i.e., the test in the graph above) were as follows:

CPU Utilization:

BTS Servers (each): Avg CPU Utilization = 59%

MsgBox DB Server: Avg CPU Utilization = 54%

Mngmt DB Server: Avg CPU Utilization = 13%

Physical Disk Idle Time:

MsgBox DB Server, Data File Disk: Avg Disk Idle Time = 69

MsgBox DB Server, Transaction Log File Disk: Avg Disk Idle Time = 98

SQL Locks:

MsgBox DB Server: Avg Total Lock Timeouts/Sec = 1072

MsgBox DB Server: Avg Total Lock Wait Time (ms) = 40

Cleanup Jobs:

MessageBox_Message_Cleanup_BizTalkMsgBoxDb: Typical Runtime = 30 Sec

MessageBox_Parts_Cleanup_BizTalkMsgBoxDb: Typical Runtime = 15 Sec

Evnet Log:

No errors in any of the server application event logs.

From these data, we can draw the following conclusions: There are no obvious resource bottlenecks in our system. All of these indicators are well within healthy limits. CPU and Disk Idle Times show that there is plenty of headroom and they are not even close to being pegged. The SQL lock indicators look good. Lock Timeouts/sec doesn’t start to become an issue until around 5000 or so (depending on your SQL Server) and Lock Wait times under .5 – 1 second are also healthy. Finally, the cleanup jobs are completing successfully every time and are taking 30 seconds or less to run. If the cleanup jobs start failing often, or start taking over a minute, this is an indication that the system is being over-driven and will typically be accompanied by an increasing spool depth.

TIP: You can expose the number of rows in the spool table by using the user defined counter capability provided by SQL Server. Create a stored procedure (in your own database) as follows:

CREATE PROCEDURE [dbo].[SetSpoolCounter] AS

SET NOCOUNT ON

SET TRANSACTION ISOLATION LEVEL READ COMMITTED

SET DEADLOCK_PRIORITY LOW

declare @RowCnt int

select @RowCnt = count(*) from BizTalkMsgBoxDB..spool with (NOLOCK)

execute sp_user_counter1 @RowCnt

GO

By running this stored procedure periodically (e.g., once per minute) as a SQL Agent job, you can add the depth of the spool table to your counters in Performance Monitor. For more information, search on sp_user_counter1 in the SQL books on line.

For additional useful messagebox queries, check out Lee Graber’s paper on advanced messagebox queries: http://home.comcast.net/~sdwoodgate/BizTalkServer2004AdvancedMessageBoxQueries.doc?

Now that we have shown how to find the maximum sustainable throughput and seen what the key indicators look like for a sustainable, healthy system, let’s explore some behavior associated with a system that is receiving faster than it is processing and collecting garbage.

To simulate a continuously overdriven system, we configured the load generation tool to send in about 175 msgs/sec, 25 msgs/sec more than our measured maximum sustainable throughput. The test was designed not only to over drive the system, but to get an idea of how long it would take to recover from a spool backlog depth of around 2 million messages. To accomplish this, we drove the system at the increased rate until the spool depth was around 2 million and then stopped submitting messages altogether. The following graph shows the same indicators as in the sustainable graph above.

As can be seen from the graph, the spool depth started building up immediately, peaking at just above 2 million records. At this rate, it took just under 3 hours to get to the targeted 2 million record backlog. After the load was stopped, it took around 4.5 hours for the cleanup jobs to recover from the backlog.

Note that, even though the receive rate started out at 175 msgs/sec, it didn’t take long for it to degrade to an average less than our maximum sustainable throughput. This is primarily due to the throttling that BizTalk provides and to increased lock contention on the message box. During the overdrive period of the test, BizTalk throttled the receiving of messages (by blocking the thread the adapters submit messages on) based on the number of database sessions opened by a host instance and the messagebox database. This throttling is indicated by messages in the application event log that indicate when BizTalk starts and stops throttling.

Taking a look at our other key indicators during this test, we see some interesting trends. Consider the following graph showing the physical disk idle time for the messagebox data file disk, average CPU utilization (%) for the messagebox server, and average lock timeouts per second on the messagebox database ( x 0.01).

Comparing this graph to the one above it, we can see that, while the system is being over driven and the spool is building up, the disk gets more and more saturated (i.e., disk idle time is trending down). Also notice that, once the cleanup jobs are given “free reign” after the load is stopped, disk idle time drops to near zero. If it wasn’t for the fact that the cleanup jobs are configured for low deadlock priority, they would take much more of the disk I/O bandwidth even earlier in the cycle. Instead, what we see from the job histories is that they are failing nearly every time they are executed because of lock contention while the load is still underway (as indicated by the avg lock timeouts/sec). Once the lock contention is reduced (at the point the load is stopped), the cleanup jobs are able to succeed and begin removing messages from the spool. It’s interesting to note that the message cleanup job ran only twice after the load was stopped, but ran for hours each time in order to collect all the unreferenced messages.

Not shown in the above graph, the lock wait times were also quite high, averaging 7 seconds during the load period, and then dropping to normal sub-100ms levels during the recovery period.

Floodgate Scenarios

QUESTION: “But what if I only have one or two “floodgate” events per day? Do I really have to build a system that will handle these peaks with no backlog, only for it to sit idle the rest of the time?

ANSWER: Of course not, as long as the system can recover from the backlog before the next floodgate event, you will be fine.

There are a number of scenarios where there are just a few large peaks (a.k.a., “floodgate events”) of messages that arrive at the system each day. Between these peaks, the throughput can be quite low. Examples of these types of scenarios include equity trading (e.g., market open and market close) and banking systems (e.g., end of day transaction reconciliation).

Other types of events cause backlog behavior simlar to floodgate events. For example, if a partner send address goes off line so that messages destined for that address must be re-tried and/or suspended, this can result in backlog building up. When the partner comes back on line, there may be a large number of suspended messages that need to be resumed, resulting in another type of floodgate event.

To illustrate how this works, consider a third test of our system as follows. We drove the system at around half the maximum sustainable throughput. This was, of course, very stable. Then, to simulate a floodgate event, we dropped 50,000 additional messages all a once (as fast as we could generate them) and monitored the system. The graph below provides our now familiar indicators of messages received per second and spool depth.

Note from the graph that the spool indeed built up a backlog during the floodgate event. However, because the event was relatively short lived and the subsequent receive rate after the event was below the maximum sustainable rate, the cleanup jobs were able to run and recover from the event without requiring a system receive “outage”.

Of course, every system is different, so “your mileage will vary”. The best way to verify that you can recover is to test with a representative load before going into production.

Findings and Recommendations

Know your load behavior profile: As our three examples above have shown, it is critical to know the profile of your load in terms of messages processed over time. The better this is understood, the more accurately you can test and adjust your system capacity. If all you know is your peak throughput requirement, then the most conservative approach would be to size your system so that your maximum sustainable throughput is the same or higher than your peak load. However, if you know that you have predictable peaks and valleys in your load, you can better optimize your system to recover between peaks, resulting in a smaller, less expensive overall deployment.

Test performance early: A common situation that we encounter at customers is that they have invested significant effort in designing and testing the functionality of their scenario, but have waited until the last minute to investigate its performance behavior on production hardware. Run performance tests on your system, simulating your load profile, as early as you possibly can in your development cycle. If you have to change anything in your design or architecture to achieve your goals, knowing this early will give you time to adjust and test again.

Emulate your expected load profile when testing performance: There are two primary components to this: 1) emulate the load profile over time and 2) run the test long enough to evaluate if it is sustainable. If, like many customers, your cycles are daily in nature, you should plan to run performance tests for at least one day to validate sustainability. The longer the tests, the better.

Emulate the production configuration: For example, the number and type of ports, the host and host instance configuration, database configuration, and adapter setup. Don’t assume that changes in the configuration will not be significantly different from a performance standpoint.

Use real messages: Message sizes and message complexity will have an affect on your performance, so be sure an test with the same message schemas and instances that you plan to see in production.

Emulate your normal operations during performance tests: Though the examples above did not include them, standard operations activities such as periodic database queries, backups, and purging will affect your sustainable throughput, so make sure you are performing these tasks during your performance and capacity test runs. This includes both DTA and BAM tracking, if you plan to use them in production.

The speed of the I/O subsystem for the messagebox is a key success factor: Remember that, for this scenario, we are using a fast SAN for the messagebox database files that is dedicated to this build-out. Even in this case, the cleanup jobs were able to drive the idle time to near zero for the SQL data file. The I/O subsystem is the most common bottleneck we have seen in customer engagements. A common strategy to optimize SQL I/O, for example, is to place the database data file(s) and log file(s) on separate physical drives, if possible.

Make sure the SQL Agent is running on all messagebox servers: Clearly, the cleanup jobs will never run if the SQL Agent is not running, so make sure these are running.

Spool depth and cleanup job run time are key indicators: Regardless of other indicators, these two measures will give you a quick and easy way to assess if your system is being over-driving or not.

Acknowledgements

I would like to thank the following contributors to this blog entry:

Mitch Stein: Thanks for helping set up the test environment and generate the test data!

Binbin Hu, Hans-Peter Mayr, Mallikarjuna rao Nimmagadda, Lee Graber, Kevin Lam, Jonathan Wanagel, and Scott Woodgate: Thanks for reviewing and providing great feedback that improved the content!

Disclaimer: This posting is provided "AS IS" with no warranties, and confers no rights.