I had the pleasure of taking part in Richard Seroter's interview series this week http://seroter.wordpress.com/2009/04/02/interview-series-four-questions-with-ewan-fairweather/
Richard is one of our most prolific CSD bloggers and is a real asset to the MVP community. I've had the pleasure of reviewing his upcoming book "SOA Patterns with BizTalk Server 2009" which is available for pre-order now. For those of you who are architecting SOA applicaitons that are using BizTalk Server and related technologies Richard's book is unmissable reading. I will be publishing a full review of the book shortly.
Richard has an overview of the book available on his blog at http://seroter.wordpress.com/2009/03/19/my-book-soa-patterns-with-biztalk-server-2009-is-up-for-pre-order/
I discovered a must install KB for Hyper-V BizTalk installations if you are using NLB today.
I'm setting up the lab for our BizTalk 2009 performance boot camp next week. We're comparing the performance of common Orchestration scenarios and optimizations that we've discovered on the R2 platform on 2009 physical and 2009 running in a Hyper-V environment. The results of this will be published in the BizTalk 2009 Performance Optimization Guide and the BizTalk 2009 Hyper-V Guide (see the previous versions http://technet.microsoft.com/en-us/library/cc558617.aspx and http://technet.microsoft.com/en-us/library/cc768518.aspx.
Because we're using WCF receive locations I set up NLB on the Hyper-V boxes. Everytime I logged onto the boxes and try and install BizTalk 2009 the boxes would Blue Screen of Death.
After much troubleshooting that KB953828 resolves the issue.
http://support.microsoft.com/kb/953828/en-us
I've just got back from a week working with a customer. The customer was a large retailer using BizTalk to perform processing over AS2. They had just installed a new QA environment and wanted the operational environment to be optimized for their application. I had a gotcha with SQL Clustering which I wasn't aware of and I wanted to share.
As with any BizTalk installation, I started from the ground up and switched their SQL Storage from Raid 5 (which is very bad for write performance - which BizTalk does a lot of) to RAID 10. Over time in the lab I've also found that changing the NTFS cluster size can gain performance improvements. Typically we use 64KB for data LUNS (which host the BizTalk Database MDF files) and 4KB NTFS cluster size for log file LUNS (LDF's).
Optimizing TEMPDB is another of the optimizations that we tend to do. Specifically splitting temp db into multiple equal size files, 1 per Logical Proc available on the server. To reduce contention further on SQL Server and prevent almost all single page allocations Trace Flag T1118 is then applied.
I used SQL 2005 configuration manager to make the change by opening the tool, clicking SQL Server 2005 Services and then double clicking SQL Server (instance name). In the SQL Server (Instance name) properties dialog box on the advanced tab I added the following text at the end of the existing string: -T1118. Crucially what I'd forgotten to include was the semi colon required. So what I'd actually done was change the following
...-lc:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\mastlog.ldf
to
...-lc:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\mastlog.ldf-T1118
In other words the SQL Server would not attempt to load a log file for the Master database with name mastlog.ldf-T1118. Still unbeknown that I had made this error we rebooted the cluster so an OS hotfix could take affect.
When the server rebooted SQL Server would not start and the following error was in the SQL Error log
Can't open file "c:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\mastlog.ldf -T1118"
We removed the T1118 from the Configuration Manager properties used previously and attempted to bring the SQL Service back online. It failed again with the sam error in the error log. We examined the SQL configuration manager properties and the -T1118 is still appended to the Master log file name. We try this several times even going into the registry directly (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\Parameters) to try and change it. But no matter what we do it stayed there.
Those of you who are cluster gurus reading this are probably laughing because you know what has happened. If SQL is clustered it is the WIndows Clustering Service which controls its execution and manages based on the clsuter config which node it will start on. In order that SQL Server functions correctly when failed over between servers, the cluster service stores a checkpoint registry on the cluster quorum disk.
In this case when the change was made and the cluster was rebooted, this incorrect configuration was stored on the quorum disk. Because changes to the checkpoint can only be made when the service (in this case SQL) is online the change I was making to remove T1118 was never getting saved and the incorrect config was consistently erasing our right config. Talk about frustrating.
Fortunately there is a work around on this, which one of my colleagues in the Clustering Support Team was able to help me out with.
1. Disable the cluster checkpoint for the specific registry key:
c:\> cluster res "SQL Server" /removecheck: "Software\Microsoft\Microsoft SQL Server\MSSQL.\MSSQLSERVER"
2. Correct the trace flag using SQL configuraiton manager or the registry
3. Re-enable Cluster checkpoint for the specific registry key:
c:\> cluster res "SQL Server" /addcheck: "Software\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLSERVER"
My friend and colleague Paolo Salvatori has just started a blog at http://blogs.msdn.com/paolos/ and began with a great post on Synchronous to Asynchronous Flows Without an Orchestration http://blogs.msdn.com/paolos/archive/2008/07/21/msdn-blogs.aspx. As you'll see from the post Paolo loves to delve into technical issues and always provides comprehensive report and great source code samples to back up his findings.
I'd definitely recommend adding this blog to your feed/favourites as I can guarantee from Paolo's previous work that you will see lot's of interesting content.
I've just completed the BizTalk 2006 R2 Hyper-V Guide project working with the BizTalk UE team and Clint Huffman from Premier Field Engineering. The purpose of this guide is to provide you with the information, techniques and best practices necessary to evaluate BizTalk Server performance in a Hyper-V environment.
Overall the results were very positive. In most scenarios we were able to process about 75% of the load in a virtualized environment compared to an equivalent physical environment. The Hyper-V team have done a great job at improving this when compared to Hyper-V. Pass-through storage provides fast storage access and they have done a great job at ensuring that CPU and memory overhead of virtualization is kept to a minimum. You can even install the Hyper-V role on Server Core which minimizes the overhead further.
The guide provides full details of our experiements, results and findings. I hope that you find this information useful.
The guide is available on MSDN (http://msdn.microsoft.com/en-us/library/cc768518.aspx) and TechNet (http://technet.microsoft.com/en-us/library/cc768518.aspx )
Launch Mail
The BizTalk Customer Advisory Team and BizTalk UE team are pleased to announce the first edition of the “Microsoft BizTalk Server 2006 R2 Hyper-V Guide”.
The Microsoft BizTalk Server 2006 R2 Hyper-V Guide is the third installment in a series of guides intended to provide easily accessible, hands-on guidance to our customer and partner community. This 145 page guide is available on MSDN, TechNet and as a separate DOCX, PDF or CHM download alongside the already available “Microsoft BizTalk Server Operations Guide” and “Microsoft BizTalk Server Performance Optimizations Guide”
The guide provides relevant information to IT professionals to enable them to make educated decisions about the advantages and tradeoffs of using Windows Server 2008 Hyper-V to virtualize BizTalk Server environments. This guidance was derived from a 6 week performance lab conducted by the BizTalk Customer Advisory Team and Premier Field Engineering. It will be refreshed for future versions of Microsoft BizTalk Server.
The key sections of the guide are:
- Getting Started: provides conceptual information about Hyper-V, the virtualization technology introduced with Windows Server 2008, and an introduction to the Hyper-V architecture.
- Deploying BizTalk Server on Hyper-V: describes the steps that were followed to set up the lab environment, which was used to compare the performance of a BizTalk Server solution running on Hyper-V environment to the same BizTalk Server solution running on comparable physical hardware.
- Evaluating BizTalk Server Performance on Hyper-V: details the important considerations when measuring the performance of a BizTalk Server solution running on a Hyper-V virtualized environment.
- Testing BizTalk Server Performance on Hyper-V: provides detailed results of four distinct testing scenarios that compare the performance of a BizTalk Server solution running on Hyper-V environment to the same BizTalk Server solution running on comparable physical hardware.
The target audience for this guide is Microsoft field, partner organizations, and customers who plan, deploy, and maintain BizTalk Server installations.
Full MSDN URL: http://msdn.microsoft.com/en-us/library/cc768518.aspx
Full TechNet URL: http://technet.microsoft.com/en-us/library/cc768518.aspx
It's been a long time since my last post. Since the last post a lot of things have happened. I took a new position as a Program Manager in the BizTalk Customer Advisory Team (CAT) in Redmond in April. I'll be giving more details on my move and what my new team does in a later post.
The BizTalk CAT was previously known as BizTalk Rangers. The main project I've been working on recently has been the BizTalk Server Performance Optimizations Guide. We launched this on MSDN and Technet a few weeks ago and have had very positive feedback so far. I've included the launch mail below which will tell you more. I hope this is of use to you, if you have any feedback on the document please let me know!
Ewan
Launch Mail
The BizTalk Customer Advisory Team and BizTalk UE team are pleased to announce the first edition of the “Microsoft BizTalk Server Performance Optimization Guide”.
The Microsoft BizTalk Server Performance Optimization Guide is the second installment in a series of guides intended to provide easily accessible, hands-on guidance to our customer and partner community. This 228 page guide is available on MSDN, TechNet and as a separate DOCX or CHM download alongside the already available “Microsoft BizTalk Operations Guide”
The guide is based on real-world experience from BizTalk CAT (Rangers), Premier Field Engineering, MCS and other customer engagements. This guide is intended to provide regularly maintained prescriptive guidance & best practices on optimizing BizTalk Server performance for demanding production environments.
The key sections of the guide are:
Getting Started: Provides an overview of the BizTalk Server functional components that can affect performance. It also describes the phases of a BizTalk Server performance assessment.
Finding and Eliminating Bottlenecks: The Finding and Eliminating Bottlenecks section describes various types of performance bottlenecks as they relate to BizTalk Server solutions and information about how to resolve the bottlenecks.
Automating Testing: Describes how to implement an automated build process and how to automate functional and load testing using Visual Studio Team System, BizUnit and Loadgen.
Optimizing Performance: The Optimizing Performance section provides guidance for optimizing performance of specific components in a BizTalk Server environment
The target audience for this guide is Microsoft field, partner organizations, and customers who plan, deploy, and maintain mission critical BizTalk Server installations. The guide was created from the key learnings, processes and methodology that have been developed by the Rangers to effectively run Performance Labs for our customers.
The guide has been carefully reviewed and vetted by experts from the community of BizTalk Server, whom we gratefully acknowledge. We believe that the information presented here will help BizTalk Server users optimize their solutions
Full MSDN URL: http://msdn.microsoft.com/en-us/library/cc558617.aspx
Full TechNet URL: http://technet.microsoft.com/en-us/library/cc558617.aspx
I'm doing some work with a customer this week who had an excessively large staticstateinfo table (more than 60,000 rows) in their BizTalk Message Box. The table contains static Orchestration information, to be loaded at runtime, it has a row for
each deployed orchestration. While the size of the table posed no immediate concerns the customer wanted to understand why the table grows bigger and how they could stop the growth.
The reason this customer had such a large number of rows is because they had several hundred Orchestrations deployed to their environment, but they ALSO had over 100 versions of each of these Orchestrations deployed to their environment. These Orchestrations were no longer used. Because BizTalk only removes the information in the static state info table when the Orchestration version is actually removed from the group, their table was getting bigger and bigger and bigger....
Versioning of your Orchestrations is a very common and often required scenario, when implementing this you should ensure that once a version is no longer used or required it is removed from the system. Doing this will decrease the size of the static state info table and also reduce the number of artifacts you have to manage within BizTalk. Reducing the number of artifacts you have to manage makes administrating your system easier (especially in BizTalk 2004 where there is no 'application' grouping). Therefore please make sure you consider how and when you are going to perform this undeployment right from the moment you decide to implement versioning of your Orchestrations.
I've recently been doing some performance tuning work for a customer. The customer wanted to achieve a throughput of 100 messages per second on their solution. The message flow was quite straightforward so it should have been achievable. Unfortunately when I turned up on site they were getting less than 5 messages per second throughput!
The hardware used for the BizTalk Servers (of which there were 2) was very new (Dual proc, dual core machines with plenty of RAM :-) ) but the performance was still not there. It became clear that we were not going to get the improvement through 'incremental' changes.
The difficulty in these scenarios is that there is no standard 'benchmark' that you can use to compare your system to. This is because your combination of BizTalk components (Orchestration, Messaging, custom components, etc) and hardware is unlikely to have been tested. In this case it was not clear whether the problem was been caused by the custom code or the hardware.
Therefore to 'bench mark' the system I set up a simple pass thru pipeline receive location and a pass thru pipeline send port (which subscribed to all messages on the inbound receive location). I then used Loadgen to load the system with 3KB messages at a rate of 500 per second (at this time we had one BizTalk box started, with a separate SQL Server with SAN attached storage). On modern hardware the BizTalk engine should be capable of easily achieving 500 messages per second in this scenario. When the test was run on the customers environment we were getting less than 50!!
By this time we had set up a duplicate system using similar hardware in Microsoft's netlabs (which we use to replicate customers environments for Proof of Concepts and Performance Labs). We were easily getting 500 messages per second in the netlab, therefore the issue had to be with the customers environment.
In this scenario the SAN should be your first point of investigation, the SQLIO tool can be used to bench mark your disk configuration, this tool is produced and used by MIcrosoft Product Support Services (PSS). Running this tool indicated that the SAN was not the culprit (the results from SQLIO were broadly the same as the environment we had in the netlabs). Further investigation eventually indicated that it was the speed of the BizTalk disks that was the culprit for this problem! The Avg Disk Writes per second physical disk performon counter was the counter ultimately used to determine this. In the netlabs we were getting approximately 2000 writes per second, the customer was only getting 200!
Replacing the disks in the BizTalk Server massively improved performance and the customer is now well on their way to achieveing their numbers.
The ultimate point of this post is that:
a) You should performance test your solution early - many people try for the 'finger in the air' approach but this rarely turns out well in my experience
b) You should bench mark your environment using simple BizTalk Content Based Routing and pass thru pipelines, if you are not able to achieve a minimum of 250 messages per second in this scenario, perform further analysis to determine what the hardware bottleneck is.
Welcome to my blog!
My name is Ewan Fairweather and in this blog I'll be blogging about BizTalk and the other technology I work with at Microsoft. Expect about 80% of the content to be BizTalk related, I'll include posts on SQL and .NET as appropriate.
I've been working for Microsoft for three years now and currently work in the Premier Field Engineering team. In this role I get to work with enterprise customers, helping them to maintain and optimise their BizTalk applications. This involves providing both proactive (i.e. making sure it doesn't break) and reactive (helping them fix it when its broken) onsite assistance within the UK and the rest of Europe. I've also worked in a dedicated capacity on some of the world’s largest BizTalk deployments, predominantly within financial services. I hope to use this blog as a means to pass on my knowledge and experience to a wider audience than I alone can reach!
Most recently I've had the honour of writing the administration section for the upcoming Professional BizTalk Server 2006 book by Darren Jefford, Kevin Smith and myself. Although I'm very biased I must say that the content is looking great. The reviews that we've had back of the final versions of each chapter have been very positive. Yossi, one of the key reviewers for the book, has posted his comments here and Darren's latest post on the book is here.
Now that my intro is complete, lets move onto my first post.
BizUnit
This is a flexible and extensible test framework which enables rapid automated testing of distributed systems. This has been used on many of the world's biggest BizTalk projects and has been key to their sucess. It was developed by Kevin Smith whilst he was working on mission critical BizTalk projects for Microsoft Consulting Services. In my opinion this is a must do for all large BizTalk projects, I've seen many projects that have avoided this type of automated testing fail. This was originally hosted on gotdotnet, but Kevin has now moved this to codeplex and has added some really great intro text. I encourage you all to review it at http://www.codeplex.com/bizunit
