"From the Field"- Neil Thompson

Event IDs 7076, 6482, 6398 in Appplication Log

Symptom

Event Type: Error
Event Source: Windows SharePoint Services 3
Event Category: (964)
Event ID: 6398
Date: <DATE>
Time: <TIME>
PM User: N/A
Computer: <SERVERNAME>
Description:
The Execute method of job definition Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob
(ID GUID) threw an exception. More information is included below.

Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

You may also get just a blank page when you try to manage IIS

Solution/Recommendation

Check for the hotfix at http://support.microsoft.com/?id=946517

Commentary

SP1 seems to help a bit, but the hotfix closes the loop

Configuration Wizard - Failed to Connect

When specifying the SQL server, Configuration Database name and account for SharePoint you may see the following error

Failed to connect to the database server or the database name does not exist. Ensure the database server is a SQL server, and that you have appropriate permissions to access the database server. To Diagnose the problem, review the extended error information located at c:\program files\common files\microsoft shared\web server extensions\12\logs\PSCDiagnostics <DATETIME>.log. Please consult the SharePoint Products and Technologies Configuration Wizards for help or additional information regarding the databse server security configuration and network access.

When you look at the log you will probably see some kind of timeout exception at the bottom, right before it all exploded.

LOTS of things could be causing this.  Below is a list of item that I have seen in my travels.

SQL Server Problems

1. SQL Server was OFF,
2. the SQL Server service was stopped,
3. the remote TCP/IP connections were not configured for SQL 2005
4. SQL was sysprepped and the name was changed and it was just generally messed up and had to be rebuilt (Life Lesson: Just install SQL normally, perhaps record an unattended, but don't bring an image of a Sysprepped SQL and try to rename the computer [regardless if it has domain membership] )

Network/Security Problems

1. Router was melted,
2. firewall was blocking access,
3. logged in account really did not have the right permissions on the SQL server (see technet for details of required access),
4. Cluster name was wrong
5. Cluster was offline
6. Account was local (not domain)
7. Recent DNS/IP change, needed to flushdns on SharePoint server
8. Admin password was changed while logged on, needed AD refresh between DCs

This is not likely a bad username/password problem. You will see a different message for that.

Technet Snippit

Important

The server farm account is used to access your configuration database. It also acts as the application pool identity for the SharePoint Central Administration application pool, and it is the account under which the Windows SharePoint Services Timer service runs. The SharePoint Products and Technologies Configuration Wizard adds this account to the SQL Server Logins, the SQL Server Database Creator server role, and the SQL Server Security Administrators server role. The user account that you specify as the service account must be a domain user account, but it does not need to be a member of any specific security group on your Web servers or your back-end database servers. We recommend that you follow the principle of least privilege, and specify a user account that is not a member of the Administrators group on your Web servers or your back-end servers.

Try this to help diagnose

1. Ping SQL/Cluster name
2. Browse to \\ClusterName\C$ (this will tell you if your logged in account has admin permissions on that box)
3. If SQL is a cluster, be sure which node is active
4. Open the SQL server in SQL Management Studio
5. Check for AV or firewall software on SQL server blocking the access
6. Try user name as domain\username (NOT username@domain)
7. Check the lists of all the stuff that can go wrong (above) and be sure you are not having those problems.
8. If all else fails, just call PSS or your Microsoft Technical Account Manager (if you have one)

SharePoint - Sample Service Level Agreement (SLA)

This is a very basic sample of a service level agreement for SharePoint.  This would be more typical of providing an SLA to an internal department of an organization. Providing an SLA to an external party would probably include an additional set of items around penalties, charges, terms of use etc.

SharePoint - Governance (Part 1)

Call to action - What parts of governance for SharePoint do you want to hear about?  I'll post new topics based on user requests. Just create requests by posting comments at the bottom of this blog. (request it again, even if others have already asked for it. Its kind of like voting and the most votes gets a post first)

This is the first of many rants on Governance.  I'm definitely trying to socialize many facets of Governance with Customers and Partners.  It is essential to success and (evidently) easily overlooked.

What is Governance?
Structured Control.

What if you don't have any Governance?
You don't have control.

Do you need Governance for a successful SharePoint installation?
YES!

What should be in a Governance Plan? (minimal list)

1. Hosting model - Central IT or delegated to business units?
2. Service model - Central service owners or delegated to business units?
3. Taxonomy - What is the structure of the systems that SharePoint will implement? (formal division sites, mysites, collaboration spaces?)
4. Architecture- Sizing, Capacity, Growth Strategy, Logical Architecture
5. Operations - Who runs the infrastructure and what tools/processes to they have? What standards to they have to meet?
6. Availability, DR, and Business Continuity -
7. Service Management - SLA's, monitoring, uptime, service tiers (commodity vs. customized applications)
8. Provisioning - Central vs. Self Service. What can users do on their own? How can you create big picture policies while still allowing freedom and removing strain on IT departments and help desks?
9. Information management policies - Retention, Disposal, Labeling, Bar coding, Auditing, Records Management, Rights Management
10. Security Policies - Authentication providers, Remote users, External Users, etc.
11. Finance Policies - Do you have a chargeback model? How is cost distributed
12. Users - How do you support them? How do your garner adoption?
13. Communication plan - What do you tell the organization? and when?

What if you don't Govern?

1. Site sprawl - sites proliferate like wildfire and you will be without ways to reign them in without upsetting users (who thought they had unlimited storage forever)
2. Technology proliferation - Users don't know where to put content so it ends up replicated all over file shares, public folders, etc.
3. Server Proliferation - Groups are unwilling to consume centralized service so they install their own SharePoint (and probably do it poorly due to lack of budget/mandate/experience)
4. Waste - Content and systems overlap and are needlessly redundant or repetitious wasting tons of money on hardware and software.
5. Inconsistent - Everybody does things differently so everything from backups to site branding is all over the map. Creating a poor and ill managed user experience.
6. Damage - "Home grown" SharePoint installations often lack the rigor of setting up data recovery or other essential items.  This often leads to data loss or compromise
7. Poor user adoption - If you don't articulate how they should use it and how you will support them, user adoption will suffer.

What are the Goals of Governance? (no particular order)

1. Drive Consistency
2. Assigning responsibility to qualified parties (IT, Service Owner, etc.)
3. Operational Excellence
4. Mitigate conflict and provide a framework for Decision making
5. Manage hardware and software assets strategically
6. Set clear expectations of service with users
7. Lower costs
8. Reduce Risk
9. Provide "Roadmap" for approach to intentionally and deliberately plan rollouts

Some Resources

• Governance TechNet Landing Page -
  http://technet.microsoft.com/en-us/office/sharepointserver/bb507202.aspx
  • See this site for links to tools, third parties, community resources etc.
• Governance Tools - http://www.codeplex.com/governance

SharePoint 2007 - Incoming Email

There are many little "gotchas" with setting up incoming email for MOSS.  Just nit picky little things, normally they are all about email infrastructure, SMTP, etc. My advice below is to help you avoid taking your heart medication, it can be easy to set this up, but it can also be a hair pulling, eye rolling, staying up all night and getting no results kind of affair as well.

I promise, this technology does actually work, but if you deviate from the standard build (see list item 3 below for "standard build" as defined by me) you will fall off a cliff unless really understand what you are doing.

I'm not an exchange guru, just a humble BizTalk/Moss guy.

Don't just do this trial and error and play with settings until it works. It won't work. Be deliberate and intentional, setup the standard way and then slowly change things, each time checking to see if the forwarding still works.

This is where I recommend you start, read all this stuff first, before you race off back to your Virtual Machines.

1. Plan incoming e-mail (Office SharePoint Server) <link updated - August 2009>
2. Configure incoming e-mail settings (Office SharePoint Server)<link updated - August 2009>
3. In addition to the technet site, a very good place to start is
1. How to configure email enabled lists in Moss2007 using Exchange 2003
2. How to configure email enabled lists in Moss2007 using Exchange 2007
4. Read these documents and set it up in a lab exactly as described until it works
5. Now introduce incremental changes to the infrastructure and after each one, verify that email is still getting through, or not.

Gotchas and Advice

1. Trust me, for the first run, just make the FQDN of the moss server (SMTP server) the same as the @domain that you are trying to send to
   
   Clarification
   AD domain = example.com,
   moss server name = MOSS2K7
   moss FQDN = MOSS2K7.example.com
   email domain = MOSS2K7.example.com
   example address = myemail@moss2k7.example.com
   
   
   
2. Trust me, for the first run, start off by actually allowing MOSS to own an OU and manage it's own contacts, don't be a cowboy and hope to configure them manually on the first try.
3. Just do everything the default way the first time, use a lab environment if you have to
4. Use the OWA website on your lab email server (or a local instance of Outlook conntected to that server)  to send emails around, don't try to hook-up external MX-Records and send from external emails.
5. Keep and eye out for .eml files showing up in the Queue folder on the MOSS box. This is a sign that the SMTP server is forwarding them elsewhere and that is a sign that there is a fundamental problem with the email coming in.  Most likely the SMTP sever does not recognize that it is the utlimate destination for emails with that @domainname address.

If you try to read from the SMTP queue on the MOSS server, you will get

A critical error occurred while processing the incoming e-mail file C:\Inetpub\mailroot\Queue\NTFS_cab371a601c8166600000011.EML. The error was: The process cannot access the file 'C:\Inetpub\mailroot\Queue\NTFS_cab371a601c8166600000011.EML' because it is being used by another process..

The solution to this is "DON'T READ FROM THE QUEUE". If the emails are in the SMTP queue on your Moss server then your email/smtp settings are probably messed up.  (Perhaps you did not take my advice on keeping the FQDN of the MOSS server the same as the @domain name that you are sending mail to?)

______________________

If you try to "drag and drop" from queue to the drop folder manually, you will get

A critical error occurred while processing the incoming e-mail file C:\Inetpub\mailroot\drop\NTFS_db489bee01c8166600000012.EML. The error was: Bad senders or recipients..

Don't tell me, let me guess.  You copied the .eml files from the Queue folder to the Drop folder and then saw sharepoint delete them. Same point as above, If the emails are in the SMTP queue on your Moss server then your email/smtp settings are probably messed up.  (Perhaps you did not take my advice on keeping the FQDN of the MOSS server the same as the @domain name that you are sending mail to?)

_______________________

BizTalk Server 2006 - Enterprise Production Considerations - Part 4 - BAM

Obligatory Introduction

BAM is such a good name for this technology for a few reasons. The first is that it reminds me of the kid Bambam on the Flinstones (You know, the baby with the club that had a thing for Pebbles). Very powerful and very cool.  Also because "BAM" is the sound my forehead makes while it collides with the palm of my hand as I discover I could have simply used BAM instead of writing a custom reporting app.

Sample to Whet the Appetite

Here is a fictitious BAM portal Aggregation for one of my lab projects

The Basics

This article will be a fairly unstructured summary of my experiences with BAM under BTS 2006.  If there is interest, I will create some more in-depth posts on the topic. I'll be brief here because most of you know this already. The basic way BAM works is the following:

1. Create a Business Activity showing the LOGICAL progression and points of interest in a process (using Excel, or Visio ODBA)
2. Deploy that Activity (as .xls or .xml to the biztalk server) using BM.exe
3. Associate low level technical events (i.e. hitting a shape in an orch or a method in a .net class) with items you defined in the activity (using the BAM API  or the Tracking Profile Editor (TPE)

The end result of this is that you can take the mess of code, orchestrations etc and transform it back into a logical/business view that a CEO could understand.  All in all, very cool.

Some Pragmatic Advice

"DO" List

1. Try both ODBA and Excel for creating activities (I like excel better, but hey, who cares?)
2. Become familiar with the BM.exe utility. You really only need the "deploy-all" and "remove-all" commands to start with
3. Keep your activity files versioned! If you deploy an activity, then change the activity file, then try to undeploy you may have problems. Then you have to undeploy by manually deleting tables and rows in the BAMPrimaryImport table.
4. I recommend that you manipulate your activity definition in Excel. When you are ready to deploy it, export it to xml and then deploy the xml. That way if you fiddle with the Excel file, you can still undeploy the activity using the same xml you used to deploy it.
5. Before you re-export your activity to XML, undeploy the activity. Or even better export your Activity to a new version of the xml (i.e. with a version number in the file name
6. Start with the tracking profile editor, master it, then move onto the BAM API calls.

"DON'T" List

1. Don't try to map milestones from Message Constructs or Loops (or other special cases). That is advanced and you may need to use the BAM API call to make that work as you would expect
2. Don't put spaces or special characters in your Activity Names. It can lead to problems later where the Event Log will say that "stored procedure does not exist" and your BAM Aggregation will not work.
3. Don't have a group shape as the last shape in the orchestration you are tracking in TPE. For some reason it won't work. Just add any shape (i.e. Expression) after the group to fix the issue.

Continuations & Relationships

I can't do these justice in this entry so this will be a bit shallow. I will probably blog on this separately. The more comments i see below requesting this, the sooner I will get around to it.

Continuations allow a single activity defintion to span muliple orchestrations or even other classes/systems seamlessly. It can also allow you to use different unique identifiers accross those systems to  When using TPE to create a continuation. Create a continuation and name it, then create a continuationID and name it the exact same thing.  Map some kind of ID value (PO ID, RequestID, etc.) from a messaging shape (receive, construct, send as described in the above DO List) in the first orchestration to the continuation. Then in the second orchestration map that same ID value to the continuationId. The idea is that the engine will recognize that those to values match and will correlate them so that both instances can contribute data to the same Activity instance

Use Continuations when the Relationship is 1:1
Use Relationships when the Relationship is 1:Many

Real-Time vs. Scheduled Aggregations

There is a little button in the Excel view (if you have installed the BAM addon) that will mark the aggregation as RealTime or Scheduled.  Again, this is complex so I will summarize below.

BAM Alerts

For the most part, real time data is better for driving alerts, as they are usually time sensitive.

Common Problems & Solutions

Problem: The BAM Activity and the Tracking Profile have been deployed, the process has been run, but there is no data displayed for the activity in the BAM portal or LiveData Excel sheet.

Solution: When you deploy a non-Real Time Aggregation (this is default) while using SQL 2005. It will create the <package>_DM and <package>_AN packages. These packages need to be run to populate the BAM tables and get you the BAM Data.  To run these packages you should select "Integration Services" in the SQL Management Console (not database engine). Once you have found them, run them. Hopefully, you're data will appear.

Problem: The BAM portal or LiveData Excel sheet shows "BLANK" for one column and one row in the Aggregation display. It seems like the measures, and the milestones are not being combined into the same activity. Also, the data shows up in two rows (or more) in the BAMPrimaryImport.dbo.bam_<ActivityName_Completed table.

Solution: This could be a number of things, but if it was the same problem that I had the solution is the following. When you are in the TPE and you are associating data from the message, be sure to use the orchestration schedule.  Right click on your receive, Message Construct, or Send shape and select Message Payload and map the items from there. Many people go to the "Select Event Source" and use the "Messaging Payload" from there. YOU ONLY WANT TO DO THAT IF YOU ARE JUST USING SEND RECEIVE PORTS TO MOVE MESSAGES. If you are using an orchestration, map everything from the orchestration schedule. 

Problem: Your BAM aggregation is not working and you see "Could not find stored procedure" in your event log.

Solution: This is probably caused becaues you have a "space" or some other bad character in your Activity Name. Undeploy your activity, rename it so it does not have the space. Redeploy it. Adjust your TPE (if required). This should fix the problem.

Wish List

1. Not being able to have spaces in the Activity Name AND not being warned about it is a serious limitation
2. The length of Activity Names and Item names is too short. I wish for longer names.
3. Measures using Average also seem to not show for RealTime Aggregations (at least they did not in mine).
4. I would like to see this integrated with the Visual Studio Environment to a much greater degree. My vision is that we use Office and VS and nothing else.
5. I also want to be able to deploy BAM stuff with MSI easier

SharePoint Office Server 2007 - Overview WebCasts for Decision Makers

This posting is intented to complement the Training Roadmap for Office SharePoint Server 2007 with additional resources for non-technical staff members.

This are excerpts from webcasts available at msevents.microsoft.com.   These are best for Technical Decision Makers, Business Decisions Makers and Architects...or developers and IT Pros who have to explain this stuff to BDMs,TDMs, or Architects :)

These webcasts cover:

1. Productivity and Communication
2. Enterprise Search
3. Records Management
4. Enterprise Content Management
5. Business Intelligence

Microsoft Webcast: Improve Productivity and Communication Using Office SharePoint Server 2007 (Level 100)

Have you ever thought about how much time is lost by your employees because of miscommunication or by the learning curve for a new product, environment, or policy? Join this webcast to get a better understanding of how Microsoft IT is attacking this problem head-on with the built-in features in Microsoft Office SharePoint Server 2007.We describe using a programmatic approach to improving the quality, effectiveness, and timeliness of the communications and content sent to employees. We also examine how tools, support, availability of critical patches, and access to user education resources can help you boost productivity and communication. 2/22/2007 7:00 PM Pacific Time (US & Canada)- 2/21/2009 12:00 AM | Duration:52 Minutes

Primary Language:   English

Primary Target Audience:   Technology Decision Maker

Microsoft Webcast: Enterprise Search at Microsoft with Office SharePoint Server 2007

Like most large companies, Microsoft’s intranet contains terabytes of structured and unstructured content that employees need to find and use on a daily basis. In the past, it has been difficult to bring such diverse content sets together in a single search-center experience. Join this webcast to see how the recent release of Microsoft Office SharePoint Server 2007 makes it possible for users to search different content repositories with just a few clicks. We explore how Microsoft manages its internal search experience built on SharePoint Server 2007, both on the central corporate portal site and as a search service used by more than 75,000 employees worldwide.2/15/2007 7:00 PM Pacific Time (US & Canada)- 2/14/2009 12:00 AM | Duration:69 Minutes

Primary Language:   English

Primary Target Audience:   Technology Decision Maker

A Web Seminar with AIIM and Microsoft: Records Management in Microsoft Office SharePoint Server 2007

 Microsoft Office SharePoint Server 2007 is a comprehensive Business Productivity platform, offering a rich set of capabilities including Enterprise Content Management. Following an introduction to the principles of electronic records management by Marc Fresko of Cornwell Management Consultants, Rob Gray, Microsoft UK Product Marketing Manager for SharePoint and ECM, will take a closer look at the Records Management capabilities of Microsoft Office SharePoint Server 2007.

11/10/2006 6:00 PM GMT, London- 11/9/2008 12:00 AM | Duration:74 Minutes

Primary Language:   English

Primary Target Audience:   Technology Decision Maker

http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032316620&culture=en-US

A Web Seminar with AIIM and Microsoft: Web Content Management using Microsoft Office SharePoint Server 2007

Microsoft and AIIM present Web Content Management using Office SharePoint Server 2007: Microsoft Office SharePoint Server 2007 includes many features that are useful for designing, deploying, and managing enterprise intranet portals, corporate Internet presence Web sites, and divisional portal sites. These features enable you to author and publish Web content in a timely manner and can ultimately reduce the cost and overhead of managing multiple sites. Microsoft partner cScape deployed the world’s first web content management system on Office SharePoint Server 2007 with Ministry of Sound label HedKandi. Ben Robb, Lead Technical Architect at cScape and James Bacchus, CIO at Ministry of Sound will talk about the choice of SharePoint as a WCM platform and their plans for the future.

1/19/2007 8:00 PM GMT, London- 1/18/2009 12:00 AM | Duration:57 Minutes

Primary Language:   English

Primary Target Audience:   Technology Decision Maker

Microsoft Business Intelligence Solutions Demonstration

Demonstration of Business Intelligence Using Sharepoint Server, Office Excel 2007 and SQL Server 2005

8/23/2007 10:30 PM Hong Kong, Taiwan, Singapore- 8/22/2009 12:00 AM | Duration:25 Minutes

Primary Language:   English

Primary Target Audience:   Business PC User

SharePoint 2007 Training Material Roadmap

This blog posting is a page designed to support basic feedback and management for the Training Material Roadmap that I created for Windows SharePoint Services 3.0 (WSS 3.0) and Microsoft Office SharePoint Server 2007 (MOSS 2007).

(Oct 16 2007) - Training Document Now Released!
Word Document Version - Click Here
WebPage Version (Limited Formatting) - Click Here

You may use this page to:

1. Obtain a copy of the roadmap document (not the accompanying DVD) - click here for doc version!
2. Request additional references for particular topics
3. Get links to supplemental blog postings
1. Click here for webcasts for Business Decision Makers, Technical Decision Makers, and Architects
4. Provide feedback or suggestions regarding future iterations of the roadmap

I welcome your feedback and criticism for how the materials have been organized in the roadmap.  I would like to help you navigate the massive topic of SharePoint 2007 technologies and nobody can give me more insight into your training and rampup needs than you.

BizTalk Server 2006 - Enterprise Production Considerations - Part 3 - Clustering

I have had this recurring conversation with customers and partners.  Clustering is not a simple concept, and neither is BizTalk.  If you put them together you seem to have a perfect storm.  There's lots of confusion, which I hope to be able to dispel with this post.

Also, I would like to have a link to send people, just so that I don't have to have the conversation again :)

The Summary

1. The SQL servers hosting the BizTalk databases can be clustered for high availability.
2. The Enterprise Single-Sign On Master Secret Service (MSS) can be clustered for high availability.
3. The BizTalk hosts can be clustered for the following reasons:
   1. They are hosting something (i.e. Receive Port) that has to be single instance, but still highly available.
   2. If you have BizTalk instances running on the same nodes that are running a clustered instance of the MSS.

FAQ

1. What parts should I cluster? 
   You should cluster the SQL instance(s) holding the biztalk databases, the ENTSSO Master Secret service, and the BizTalk Hosts that require on a single instance running at a time, but still needs to be highly available.
2. Is a non clustered-BizTalk host still Highly Available?
   If you have at least two host instances running in the same group for this host, then YES.  It behaves more like an NLB cluster than an MSCS service.

The Easy Part

I say that the SQL clustering is the "Easy Part" not because it is simple, but because I don't feel compelled to explain the nitty gritty details in this post. I will give some of the BizTalk specifics here, and dedicate the next post to SQL cluster configuration. For now, as a rule, go with SQL 2005 Active/Active/Passive.

1. Separate the MessageBox and Tracking Database among separate LUNs and separate SQL instances.
2. The "Other" databases can often be place on the same node running the tracking DB.  If you are heavy on BAM or BAS you might want to break off the respective databases to a separate instance.
3. All the databases should have at least 2 LUNS (data log), often they have more, but that gets into mount points and that is a tanget.

The Somewhat Easy Part

Clustering the ENTSSO MSS is pretty simple and the procedure is posted here. Basically, this is just a running service of the ENTSSO that also takes on the responsibility of distributing the Master Secret Key. This is the key that all the ENTSSO services running on other machines need to read the encrypted information in the SSO Database.

1. This should be clustered.  Unless you don't care about high availabilty. This can really mess up BizTalk if it goes down.
2. This is not really a resource intensive service (typically) so it often piggy-backs on the SQL server nodes runing the clustered instances of SQL Server.
3. You don't want this clustered on the BizTalk boxes as a rule of thumb. (reasons covered later in this post)

Ok, now the meat.

BizTalk hosts generally have instances on more than one box, as such they are already highly available. This means that clustering a BizTalk host is not generally necessary to ensure high availability. If one goes down the others will pick up the work, out-of-the-box standard config.  The problem comes in when you REALLY don't want to have mutliple instances of a host running at the same time.  The classic example of this is a receive port for an adapter that can not gracefully handle two threads reading it at the same time (i.e. FTP). It also happens when you need to be certain that all of the messages are picked up and processed sequentially and you don't want to code around the problem.

So you want an FTP port receive, but you can not affort duplicate message reads.  That means you can have only one instance of the host, but that kills the out-of-the-box high availability.  Now you need to cluster that host instance, that is the only way to ensure that exactly one instance of the host will be running (so long as at least one of the BizTalk app servers is running)

The Very Wierd Part

This is the one that people scratch their heads over.  It does not happen much, and its not really all that complex.  But it is not intutitive, so it messes with people's heads. 

For this to really make sense, I have to back up a step and explain how the ENTSSO service generally works in a mutli-box BizTalk installation.

1. All of the BizTalk App servers have an instance of the ENTSSO service running on the machine. It is this service that obtains the secret key from the ENTSSO Master Secret Server and then uses it to independently access the info in the SSO Database.
2. One of the ENTSSO servers is setup as the Master Secret Server. This means that it will distribute the secret key to the other ENTSSO service instances, in addition to being the standard ENTSSO service for the local box.  This is the service instances that needs to be configured for High Availablity because all of the other ENTSSO service instances depend on it.
3. The BizTalk host instances have a dependency on the ENTSSO Service running on the same box as them. This is because they need desperately need it to give them info from the SSO DB. This information is typically configurations for their ports that is stored in the SSO DB for safe keeping.

Figure 1 - Standard Multi-Box ENTSSO Strategy

Lets say that you have only 2 production servers. They will be running both SQL and BizTalk.  Like a good boy/girl you decide to cluster SQL and the ENTSSO MSS.

Now you try to configure BizTalk host instances, and you find that they don't want to work quite right. That is because they have a dependency on the ENTSSO service, they always need it.  Now that the ENTSSO service is clustered, only the host instances running on the active (ENTSSO) node actually have access to the service they need. All the other instances will fail based on dependencies.

Figure 2 - BizTalk Hosts with Broken Dependency

An an extra bonus you have messed up the High availability of the MSS as well. The clustering service detects that the host instances are depending on the ENTSSO MSS service and probably will not allow it to failover to other nodes.

...NICE...

Option 1 is to not cluster ENTSSO MSS in a 2 box scenario

Bottom line is that, if you have only budget for two prod boxes, you probably don't want to cluster the ENTSSO MSS. But this is really the only time I can think of that clustering that service is bad.  If it is not clustered you better be a "Johnny on the spot" with MOM and implement a way to have MOM do a poor man's failover when it detects that the ENTSSO MSS server is down.

Option 2 is to also cluster all BizTalk host instances that depend on a clustered ENTSSO service.

If you really do want to cluster the ENTSSO MSS, then you also have to cluster all of the BizTalk hosts as well. This ensures that as the ENTSSO Service "flips" from one box to the next, the Microsoft Clustering Service can ensure that the BizTalk host instances flip at the same time and don't come crashing down.

BizTalk Server 2006 - Enterprise Production Considerations - Part 2 - Host and Host Instances

BizTalk Server 2006 - Enterprise Production Considerations - Part 1 - Production Server Topology

Introduction:

This post is the first in a series of posts that will describe a number of production considerations for Enterprise customers (“Enterprise” loosely defined as those with moderately complex BizTalk implementations)

This is in response to a great deal of confusion that I have encountered in the field and hopefully I can kill about 500 birds with 1 stone by publishing these findings (no offence to bird enthusiasts J ).

Today’s Topic – Starting the Production Infrastructure

BizTalk Server 2006 has a dizzying array of scalability options, but many options can mean complexity.  This post will specifically discuss the most common “starting points” for production enterprise BizTalk systems and compare/contrast them so you can hopefully figure out which one best suits your needs.

The X + Y notation means:

X Dedicated BizTalk boxes
Y SQL Server Nodes

Server Configurations

Before we get started here, I have to emphasize that these are all common STARTING configurations.  Configurations can (and do) scale out and up depending on your requirements.

Also, all of these configurations have some notion of high availability, no single node failure will kill the infrastructure.  Way more options are available if you sacrifice High Availability (but that’s pretty unusual in the Enterprise isn’t it? J )

Comparison Chart

 Call to Action

 I know what I want to write about, but this blog isn't for me!  What topics do you want to see in the next article? Drop in a request in the comments section below and I'll review before I start writing the next article (I promise)

• Configuration
• SAN/Storage
• Troubleshooting/Debugging
• High Availability
• other....

BizTalk gets "General Network Error" under heavy load

This is a fairly common scenario after Windows 2003 SP1 (or greater) is installed, so I thought I better put in my two cents

Scenario: BizTalk seems to lose connectivity to the MsgBox (and all other databases on that SQL instance)

Problem: The operating system (w2k3 sp1) running the SQL server instance is misinterpreting the heavy TCPIP connection traffic from the BizTalk App server as a Denial of Service (DOS) attack.

Clarification: This is an operating system level issue. This is not BizTalk or SQL per se (though they are common victims of this security feature). This is a security feature for the OS that affects how the OS interprets large amounts of incoming connections from individual clients.

Factors:

1.     Windows Server 2003 SP1 (or higher) is installed on the SQL server hosting the BizTalk DBs

2.     SynAttackProtect is still set to “1” on the SQL server

3.     Concurrent TCPIP connection requests from a specific BizTalk app server exceeds the critical threshold (Sorry, don’t know what that threshold is!)

Diagnosing:

1. Look for the following event entries:

Event Type: Error
Event Source: BizTalk Server 2004
Event Category: (1)
Event ID: 6913
Computer: %ServerNameHere%

Description: SQLServer, BizTalkMsgBoxDb, [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation..

Event Type: Warning
Event Source: BizTalk Server 2004
Event Category: (1)
Event ID: 5410
Computer: %ServerNameHere%

Description: An error has occurred that requires the BizTalk service to terminate. The most common causes are an unexpected out of memory error and an inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.

Error message: [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation.

Error source:
BizTalk host name: BizTalkHost
BizTalk Windows service name: BTSSvc$BizTalkHost.

Fixes:

1.     Set the SynAttackProtect registry setting as described in http://support.microsoft.com/kb/899599

2.     ALSO take a look at the Group policy for that SQL server to be sure that the Group Policy does not periodically UNDO the setting change that you did in step 1. I have been in the situation where I fixed the issue, but then the GP Update or reboot undid the fix!  (Put the server in a separate OU if you have to, but make sure you isolate the box from policies that would reset this registry key)

Limitations

1.     I don’t know how to disable this feature for a specific set of clients (i.e. SynAttackProtect is off for the BizTalk app servers but remains on for other servers).  The setting seems to be at the protocol level so I wouldn’t even know where to start looking for network set definition here.

Is there something that I can do to make this blog more helpful to you? I welcome your feedback!

Office 2007 Open Format

Office 2007 users want to see something cool?  Save out a word 2007 .docx format, rename the file to have a .zip extension, and then crack it open. How cool is that?

You can read more here http://msdn2.microsoft.com/en-us/library/aa338205.aspx

Excessive Blocking when using BizTalk SQL Adapter

------------------------------------- 
Addendum – June 22 2007: Under absolutely NO circumstances should you modify the data model of the BizTalk databases in any way.  You should only optimize your own application databases.
------------------------------------- 

I have seen this issue in many places,  which still surprises me because there are a lot of blog entries about this.  For what it's worth, here's one more.

Scenario: BizTalk solution makes moderate to heavy use of SQL adapter in calling stored procedures in a database.

Problem: SQL server shows many blocking (perhaps deadlocking) SPIDs, most of which are owned by the BizTalk host instance account.

Factors:

1. The longer the procedure takes to execute, the worse off you are
1. Big tables, poor logic, bad indexes, bad db maintenance, disk contention etc.
2. The more concurrent procedure executions the worse off you are
1. short polling interval
2. multiple ports
3. multiple host instances

Diagnosing:

1. Enterprise Manager or Management Studio can give some insight to active and blocking spids (check to see who owns the blocking spid and use dbcc inputbuffer to see the sql it is executing) 
2. Try using sp_who,sp_who2,and sp_who3, they may give you lots of insight
3. Call MS PSS (or download pssdiag for yourself from the MS download site) and have them configure a pssdiag trace for you.  Look especially closely at the blocker output and the profiler trace.  Use rml.exe (read80trace) to generate a report of the "worst offenders" sql scripts or procedures.

Fixes:

1. As per documentation, don't use T-SQL transactions in your proc.  The SQL Adapter already has this all wrapped up in a distributed transaction so relax!
2. As per documentation, you can try setting the transaction isolation level to READ COMMITTED, or perhaps REPEATABLE READ.  If you are running SERIALIZABLE that is a great way to tank performance and cause blocking.
3. Speed up the procs (Archive, purge, index, create views, run db maintenance, buy a big SAN, whatever. The locks won't kill you if they don't last) 
4. Don't go silly with coarse locking hints.  Keep your lock few in number and small in granuarity.  If you must manually lock, then please read up on the READPAST hint - especially for queue or "trigger" tables. (Thanks to Kunjal K for this hint)
5. Reduce concurrency - You can try lengthening your polling interval or reducing the number of host instances that run that port.

Didn't find what you were looking for? Try one of these:

1. Known SQL Adapter Issues
2. How to resolve a deadlock
3. Understanding Isolation Levels
4. Contact me through comment below

Is there something that I can do to make this blog more helpful to you?
I welcome your feedback!