Ewan Fairweather

My friend and colleague Paolo Salvatori has just started a blog at http://blogs.msdn.com/paolos/ and began with a great post on Synchronous to Asynchronous Flows Without an Orchestration http://blogs.msdn.com/paolos/archive/2008/07/21/msdn-blogs.aspx.  As you'll see from the post Paolo loves to delve into technical issues and always provides comprehensive report and great source code samples to back up his findings.

I'd definitely recommend adding this blog to your feed/favourites as I can guarantee from Paolo's previous work that you will see lot's of interesting content.

I've just completed the BizTalk 2006 R2 Hyper-V Guide project working with the BizTalk UE team and Clint Huffman from Premier Field Engineering. The purpose of this guide is to provide you with the information, techniques and best practices necessary to evaluate BizTalk Server performance in a Hyper-V environment.

Overall the results were very positive.  In most scenarios we were able to process about 75% of the load in a virtualized environment compared to an equivalent physical environment. The Hyper-V team have done a great job at improving this when compared to Hyper-V.  Pass-through storage provides fast storage access and they have done a great job at ensuring that CPU and memory overhead of virtualization is kept to a minimum.  You can even install the Hyper-V role on Server Core which minimizes the overhead further.

The guide provides full details of our experiements, results and findings.  I hope that you find this information useful.

The guide is available on MSDN (http://msdn.microsoft.com/en-us/library/cc768518.aspx) and TechNet (http://technet.microsoft.com/en-us/library/cc768518.aspx )

Launch Mail 

The BizTalk Customer Advisory Team and BizTalk UE team are pleased to announce the first edition of the “Microsoft BizTalk Server 2006 R2 Hyper-V Guide”.

The Microsoft BizTalk Server 2006 R2 Hyper-V Guide is the third installment in a series of guides intended to provide easily accessible, hands-on guidance to our customer and partner community. This 145 page guide is available on MSDN, TechNet and as a separate DOCX, PDF or CHM download alongside the already available “Microsoft BizTalk Server Operations Guide”  and “Microsoft BizTalk Server Performance Optimizations Guide”

The guide provides relevant information to IT professionals to enable them to make educated decisions about the advantages and tradeoffs of using Windows Server 2008 Hyper-V to virtualize BizTalk Server environments. This guidance was derived from a 6 week performance lab conducted by the BizTalk Customer Advisory Team and Premier Field Engineering.  It will be refreshed for future versions of Microsoft BizTalk Server.

The key sections of the guide are:

• Getting Started: provides conceptual information about Hyper-V, the virtualization technology introduced with Windows Server 2008, and an introduction to the Hyper-V architecture.
• Deploying BizTalk Server on Hyper-V: describes the steps that were followed to set up the lab environment, which was used to compare the performance of a BizTalk Server solution running on Hyper-V environment to the same BizTalk Server solution running on comparable physical hardware.
• Evaluating BizTalk Server Performance on Hyper-V: details the important considerations when measuring the performance of a BizTalk Server solution running on a Hyper-V virtualized environment.
• Testing BizTalk Server Performance on Hyper-V: provides detailed results of four distinct testing scenarios that compare the performance of a BizTalk Server solution running on Hyper-V environment to the same BizTalk Server solution running on comparable physical hardware.

The target audience for this guide is Microsoft field, partner organizations, and customers who plan, deploy, and maintain BizTalk Server installations.

Full MSDN URL:     http://msdn.microsoft.com/en-us/library/cc768518.aspx

It's been a long time since my last post.  Since the last post a lot of things have happened.  I took a new position as a Program Manager in the BizTalk Customer Advisory Team (CAT) in Redmond in April.  I'll be giving more details on my move and what my new team does in a later post.

The BizTalk CAT was previously known as BizTalk Rangers.  The main project I've been working on recently has been the BizTalk Server Performance Optimizations Guide.  We launched this on MSDN and Technet a few weeks ago and have had very positive feedback so far.  I've included the launch mail below which will tell you more.  I hope this is of use to you, if you have any feedback on the document please let me know!

 Ewan

Launch Mail

The BizTalk Customer Advisory Team and BizTalk UE team are pleased to announce the first edition of the “Microsoft BizTalk Server Performance Optimization Guide”.

The Microsoft BizTalk Server Performance Optimization Guide is the second installment in a series of guides intended to provide easily accessible, hands-on guidance to our customer and partner community. This 228 page guide is available on MSDN, TechNet and as a separate DOCX or CHM download alongside the already available “Microsoft BizTalk Operations Guide”  

The guide is based on real-world experience from BizTalk CAT (Rangers), Premier Field Engineering, MCS and other customer engagements. This guide is intended to provide regularly maintained prescriptive guidance & best practices on optimizing BizTalk Server performance for demanding production environments. 

The key sections of the guide are: 

Getting Started: Provides an overview of the BizTalk Server functional components that can affect performance. It also describes the phases of a BizTalk Server performance assessment.

Finding and Eliminating Bottlenecks: The Finding and Eliminating Bottlenecks section describes various types of performance bottlenecks as they relate to BizTalk Server solutions and information about how to resolve the bottlenecks.

Automating Testing: Describes how to implement an automated build process and how to automate functional and load testing using Visual Studio Team System, BizUnit and Loadgen.

Optimizing Performance: The Optimizing Performance section provides guidance for optimizing performance of specific components in a BizTalk Server environment

The target audience for this guide is Microsoft field, partner organizations, and customers who plan, deploy, and maintain mission critical BizTalk Server installations. The guide was created from the key learnings, processes and methodology that have been developed by the Rangers to effectively run Performance Labs for our customers.

The guide has been carefully reviewed and vetted by experts from the community of BizTalk Server, whom we gratefully acknowledge. We believe that the information presented here will help BizTalk Server users optimize their solutions

Full MSDN URL:        http://msdn.microsoft.com/en-us/library/cc558617.aspx 

Full TechNet URL:     http://technet.microsoft.com/en-us/library/cc558617.aspx

I'm doing some work with a customer this week who had an excessively large staticstateinfo table (more than 60,000 rows) in their BizTalk Message Box.  The table contains static Orchestration information, to be loaded at runtime, it has a row for
each deployed orchestration.  While the size of the table posed no immediate concerns the customer wanted to understand why the table grows bigger and how they could stop the growth.

The reason this customer had such a large number of rows is because they had several hundred Orchestrations deployed to their environment, but they ALSO had over 100 versions of each of these Orchestrations deployed to their environment.  These Orchestrations were no longer used. Because BizTalk only removes the information in the static state info table when the Orchestration version is actually removed from the group, their table was getting bigger and bigger and bigger....

Versioning of your Orchestrations is a very common and often required scenario, when implementing this you should ensure that once a version is no longer used or required it is removed from the system.  Doing this will decrease the size of the static state info table and also reduce the number of artifacts you have to manage within BizTalk.  Reducing the number of artifacts you have to manage makes administrating your system easier (especially in BizTalk 2004 where there is no 'application' grouping).   Therefore please make sure you consider how and when you are going to perform this undeployment right from the moment you decide to implement versioning of your Orchestrations.

I've recently been doing some performance tuning work for a customer.  The customer wanted to achieve a throughput of 100 messages per second on their solution.  The message flow was quite straightforward so it should have been achievable.  Unfortunately when I turned up on site they were getting less than 5 messages per second throughput!

The hardware used for the BizTalk Servers (of which there were 2) was very new (Dual proc, dual core machines with plenty of RAM :-) ) but the performance was still not there.  It became clear that we were not going to get the improvement through 'incremental' changes.

The difficulty in these scenarios is that there is no standard 'benchmark' that you can use to compare your system to. This is because your combination of BizTalk components (Orchestration, Messaging, custom components, etc) and hardware is unlikely to have been tested.  In this case it was not clear whether the problem was been caused by the custom code or the hardware.

Therefore to 'bench mark' the system I set up a simple pass thru pipeline receive location and a pass thru pipeline send port (which subscribed to all messages on the inbound receive location).  I then used Loadgen to load the system with 3KB messages at a rate of 500 per second (at this time we had one BizTalk box started, with a separate SQL Server with SAN attached storage). On modern hardware the BizTalk engine should be capable of easily achieving 500 messages per second in this scenario.  When the test was run on the customers environment we were getting less than 50!!

 By this time we had set up a duplicate system using similar hardware in Microsoft's netlabs (which we use to replicate customers environments for Proof of Concepts and Performance Labs). We were easily getting 500 messages per second in the netlab, therefore the issue had to be with the customers environment.

 In this scenario the SAN should be your first point of investigation, the SQLIO tool can be used to bench mark your disk configuration, this tool is produced and used by MIcrosoft Product Support Services (PSS).  Running this tool indicated that the SAN was not the culprit (the results from SQLIO were broadly the same as the environment we had in the netlabs).  Further investigation eventually indicated that it was the speed of the BizTalk disks that was the culprit for this problem!  The Avg Disk Writes per second physical disk performon counter was the counter ultimately used to determine this.  In the netlabs we were getting approximately 2000 writes per second, the customer was only getting 200!

 Replacing the disks in the BizTalk Server massively improved performance and the customer is now well on their way to achieveing their numbers.

 The ultimate point of this post is that:

a) You should performance test your solution early - many people try for the 'finger in the air' approach but this rarely turns out well in my experience

b) You should bench mark your environment using simple BizTalk Content Based Routing and pass thru pipelines, if you are not able to achieve a minimum of 250 messages per second in this scenario, perform further analysis to determine what the hardware bottleneck is.

Welcome to my blog! 

 My name is Ewan Fairweather and in this blog I'll be blogging about BizTalk and the other technology I work with at Microsoft.  Expect about 80% of the content to be BizTalk related, I'll include posts on SQL and .NET as appropriate.

I've been working for Microsoft for three years now and currently work in the Premier Field Engineering team. In this role I get to work with enterprise customers, helping them to maintain and optimise their BizTalk applications.  This involves providing both proactive (i.e. making sure it doesn't break) and reactive (helping them fix it when its broken) onsite assistance within the UK and the rest of Europe. I've also worked in a dedicated capacity on some of the world’s largest BizTalk deployments, predominantly within financial services.  I hope to use this blog as a means to pass on my knowledge and experience to a wider audience than I alone can reach!

Most recently I've had the honour of writing the administration section for the upcoming Professional BizTalk Server 2006 book by Darren Jefford, Kevin Smith and myself.  Although I'm very biased I must say that the content is looking great.  The reviews that we've had back of the final versions of each chapter have been very positive. Yossi, one of the key reviewers for the book, has posted his comments here and Darren's latest post on the book is here.

 Now that my intro is complete, lets move onto my first post.

 BizUnit

This is a flexible and extensible test framework which enables rapid automated testing of distributed systems.  This has been used on many of the world's biggest BizTalk projects and has been key to their sucess.  It was developed by Kevin Smith whilst he was working on mission critical BizTalk projects for Microsoft Consulting Services.  In my opinion this is a  must do for all large BizTalk projects, I've seen many projects that have avoided this type of automated testing fail.  This was originally hosted on gotdotnet, but Kevin has now moved this to codeplex and has added some really great intro text.  I encourage you all to review it at http://www.codeplex.com/bizunit