One of the primary purposes behind creating the BizTalk Terminator tool was to allow users the ability to easily resolve many of the common database integrity issues identified by the BizTalk MsgBoxViewer tool. Also, with BizTalk 2010, the BizTalk product group added a new SQL Agent Job to the product called Monitor BizTalk Server that identifies many of these same database integrity issues. The purpose of this post is to illustrate how BizTalk Terminator can be used to resolve database integrity issues identified by MBV or the BTS2010 Monitor BizTalk Server job.
Looking for info on where to download Terminator? Go here
Terminator Not Cleaning Up Caching Items? Go here
Resolving issues identified by the BTS2010 Monitor BizTalk Server Job
One of the new features introduced in BizTalk 2010 is the Monitor BizTalk Server SQL Agent job. This job was added to the product to provide an out-of-the-box way to identify database integrity issues that we previously had to use MBV or manual SQL scripts to identify. If this job identifies a database integrity issue, the job itself will fail with a custom error message that describes the problem that was identified. A database admin or a monitoring tool like SCOM can take a look at SQL Agent to identify when a problem has occurred. See the BizTalk Server Monitoring section at http://msdn.microsoft.com/en-us/library/aa577454 (BTS.70).aspx for more information about this job.
Terminator provides the easiest way to resolve most of these issues once they have occurred. Use the steps below to resolve issues identified by the Monitor BizTalk Server Job.
1. Before running Terminator, always make sure you have a BizTalk Backup of your databases, all the BTS hosts have been stopped, BTS SQL Agent jobs have been disabled.
2. Use Table 1 below to determine the Terminator View and Resolution Tasks as well as the Terminator Task Category for the tasks associated with the issue identified by the BizTalk Monitor Job
3. Open Terminator to enter the initial screen.
4. Specify the SQL server and database information for the BizTalk Management Database.
5. Check the checkboxes to acknowledge that you have completed step #1 above and click Connect.
6. Once connected, click the appropriate button depending on the Terminator Task Category
7. Select the appropriate task from the list, click the Parameters tab and populate any parameter that needs to be set and click the Execute button.
Table 1 provides a summary of all the common database integrity issues identified by the Monitor job and the associated task in Terminator that is used to resolve or view the issue. Note that some issues do not have any resolution options and will require troubleshooting by Microsoft Support.
(in Manage task category)
View Count of Messages Without RefCounts
DELETE Orphaned Messages in Specific Host
(both in Delete task category)
View Count of Orphaned Messages by Host
(in Delete task category)
View Count of Orphaned DTA Service Instances
View Count of Orphaned DTA Service Instance Exceptions
Table 1. BTS Monitor job-identified Issues and Associated Terminator Tasks
Resolving issues identified by BizTalk MsgBoxViewer
First, MBV is always being improved and updated so make sure you are always running the latest version. Below are some relevant MBV links:
- MBV Blog - http://blogs.technet.com/jpierauc/ - MBV FAQ - http://blogs.technet.com/b/jpierauc/archive/2008/07/22/msgboxviewer-mbv-q-a.aspx- MBV Download location - http://blogs.technet.com/jpierauc/pages/msgboxviewer.asp x
There are 2 ways to use Terminator to resolve issues identified by MBV. This document will outline both options in detail:
1. MBV Integration – With a minimum of MBV version 10.13, MBV will output MBVCleanupScripts.xml which Terminator can consume to automatically provide the user with a list of Tasks that can be used to resolve identified issues. Not all issues identified by MBV and addressable with Terminator are supported by MBV integration.
2. Manual Task Selection – The user can look at the MBV html output and manually select the appropriate tasks in Terminator
Table 2 provides a summary of all the common database integrity issues identified by MBV and the associated task in Terminator that used to resolve or view the issue. Note that some issues require manual integration and some do not have any integration options and will require troubleshooting by Microsoft Support.
Issue Identified by MBV
Resolution Options
Terminator Resolution Task
Terminator View Task
Root Cause
Orphaned RFRs
MBV Integration or Manual Task Selection
DELETE Orphaned RFRs in all Hosts
DELETE Orphaned RFRs in Specific Host
View Count of RFRs By Host
This is due to a known bug and there is a hotfix available. See KB 941690 for details.
Orphaned Cache Instances
Terminate Caching Instances
View Count of Cache Messages in All Host Queues
View Count of Cache Instances in All Hosts
This is due to a known bug and there is a hotfix available. See KBs 944426 & 936536 for details.
Orphaned DTA Service Instances
Repair Orphaned DTA Service Instances
This is due to a known bug and there is a hotfix available. See KB 978796 for details. This can also be due to running a cleanup script against trackingdata_x_x or instances table.
Orphaned DTA Service Instance Exceptions
DELETE Orphaned DTA Service Instance Exceptions
No known root cause
Message Refs w/out Spool rows
None
No cleanup task – contact MS support.
No view task – contact MS support.
Messages w/out RefCounts
Repair Refcounts for All Messages
This is due to an issue in BTS reference counting logic.
Message Refs w/out Instances
DELETE Orphaned Messages in All Hosts
Only known root cause are Orphaned RFRs but we have seen other types of message refs w/o instances and there is no know root cause for those
Subscriptions w/out Instances
Manual Task Selection
DELETE Orphaned Instance Subscriptions
View Count of Orphaned Instance Subscriptions
InstanceState w/out Instances
Missing Control Messages
Repair Control Messages
View Missing Control Messages
Large number of Suspended Messages
Manual Task Selection Only
Resume Instances or Terminate Instances and select Host, Class, and Status as appropriate.
(Resume in Manage task category and Terminate in Delete task category)
Count Instance (and Save Messages)
Could be due to various causes that lead instances to get suspended. BTS Admin will need to look at the error message associated with the suspended items to investigate root cause.
Large MarkLog Table
PURGE Marklog Table
View Table Row Count (No SQL Statistics)
Due BTS backup job’s backup on mark logic. Mark is created across all DBs involved in backup so restores can be done in a transactionally consistent state.
Table 2. MBV-identified Issues and Associated Terminator Tasks
MBV Integration
This feature allows the user to resolve many of the common database integrity issues MBV identifies without having to analyze the MBV output or determine which task in Terminator is associated with which issue. MBV 10.13 or above is required for this functionality and will produce an MBVCleanupScripts.xml file used by Terminator for MBV integration.
How to use Terminator's MBV mode for MBV Integration:
1. Run MBV and verify that an MBVCleanupScripts.xml file has been created in the output folder.
2. Before running Terminator, always make sure you have a BizTalk Backup of your databases, all the BTS hosts have been stopped, BTS SQL Agent jobs have been disabled.
3. Open Terminator to enter the initial screen (Figure 1).
5. Check the checkboxes to acknowledge that you have completed step #2 above and click Connect.
6. Once connected, click the Import MBV button (Figure 2) and browse to the MBVCleanupScripts.xml file created in step #1.
7. Notice that the dropdown list says "Select a MBV task from this drop down" (Figure 3) and clicking the dropdown provides a list of all tasks that have been selected to resolve all the issues that MBV identified (and support MBV integration mode).
8. Execute each task from the list to resolve the issues identified by MBV. Note that the Parameters tab must be clicked to enable the Execute button. Make sure all params are populated correctly before executing a task. The user can re-run MBV or use the Terminator View tasks mentioned in Table 2 to verify that the issues have been resolved.
Important considerations for MBV Integration:
· MBV only creates an MBVCleanupScripts.xml file if it identifies an issue that is addressable with Terminator using MBV Integration.
· MBVCleanupScripts.xml will not be created if MBV only identifies issues that require manual task selection to resolve and do not support MBV Integration.
· Once the MBV file has been imported, use the MBV Task Category button and other Task Category buttons to go in and out of MBV mode
· There are currently 8 Terminator tasks that can be pre-selected by MBV Integration:
1. DELETE Orphaned RFRs in all Hosts
2. Terminate Caching Instances
3. Repair Orphaned DTA Instances
4. DELETE Orphaned DTA ServiceInstance Exceptions
5. Repair Refcounts for All Messages
6. DELETE Orphaned Messages in All Hosts
7. Repair Control Messages
8. PURGE Marklog Table
In some cases, the user prefers to analyze the MBV output html and select the Terminator tasks in a manual fashion.
Manual Task Selection Steps:
2. Use the below sections of this document to identify an issue mentioned in the Warnings and Summary Report section of the MBV output html and note down the Terminator View and Resolution Tasks as well as the Terminator Task Category.
Orphaned RFRs in Critical Warnings section of Warning and Summary Report:
Click Query Report button to see further details:
MsgBox Dbs : MsgBox Integrity Details (SQL Query, 4 Rules)
- Messages count with no associated instances grouped by classID
Terminator View Task:
· View Count of RFRs By Host (in View category)
Terminator Resolution Tasks:
· DELETE Orphaned RFRs in all Hosts (in Delete category)
· DELETE Orphaned RFRs in specific Host (in Delete category)
MsgBox Dbs : Active Cached Msg Ref in Spool (SQL Query, 2 Rules)
- MSGBOX DB 1 (MASTER) "BizTalkMsgBoxDb" on SERVER
· View Count of Cache Messages in all Hosts Queues (in View category).
· View Count of Cache Instances in all Hosts (in View category). Note that while MBV shows the number of orphaned cache messages, this Terminator task shows the number of cache instances. One cache instances can have thousands of cache messages associated with it.
Terminator Resolution Task:
· Terminate Caching Instances (in Delete category). Note that while MBV shows the number of orphaned cache messages, Terminator terminates cache instances. One cache instances can have thousands of cache messages associated with it.
DTA Db : Orphaned Svc Instances (SQL Query, 2 Rules)
· View Count of Orphaned DTA Service Instances (in View category)
· Repair Orphaned DTA Service Instances (in Manage category)
DTA Db : Orphaned serviceInstanceExceptions (SQL Query, 1 Rule)
· View Count of Orphaned DTA Service Instance Exceptions (in View category)
· DELETE Orphaned DTA Service Instance Exceptions (in Delete category)
Messages w/out RefCounts in Non Critical Warnings section of Warning and Summary Report:
· View Count of Messages Without RefCounts (in View category)
· Repair Refcounts for All Messages (in Manage category)
Click Query Report button to see further details. There are 20053 messages without instances not including the 10991 orphaned RFRs:
Go to MsgBox Dbs : MsgBox Integrity (SQL Query, 3 Rules) to see more details. This a total count including orphaned RFRs and orphaned MSMQT messages:
· View Count of Orphaned Messages by Host (in View category)
· DELETE Orphaned Messages in All Hosts (in Delete category). Note that this task will also clean up orphaned RFRs and orphaned MSMQT messages.
· DELETE Orphaned Messages in Specific Host (in Delete category). Note that this task will also clean up orphaned RFRs and orphaned MSMQT messages.
·View Missing Control Messages (in View category)
· Repair Control Messages (in Manage category).
MsgBox Dbs : BizTalk Queues (SQL Query, 4 Rules)
- MSGBOX DB 1 (MASTER) "BizTalkMsgBoxDb3" on SERVER
· Count Instances (and Save Messages) (in View category)
· Resume Instances (in Manage category). User must verify suspended items need to be resumed before executing.
· Terminate Instances (in Delete category). User must verify suspended items need to be terminated before executing.
·View Table Row Count (No SQL Statistics) (in View category). Find MarkLog table in the output list.
·PURGE Marklog Table (in View category)