"From the Field"- Neil Thompson

Event IDs 7076, 6482, 6398 in Appplication Log

neilth — Thu, 26 Mar 2009 19:42:00 GMT

Symptom

Event Type: Error
Event Source: Windows SharePoint Services 3
Event Category: (964)
Event ID: 6398
Date: <DATE>
Time: <TIME>
PM User: N/A
Computer: <SERVERNAME>
Description:
The Execute method of job definition Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob
(ID GUID) threw an exception. More information is included below.

Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

You may also get just a blank page when you try to manage IIS

Solution/Recommendation

Check for the hotfix at http://support.microsoft.com/?id=946517

Commentary

SP1 seems to help a bit, but the hotfix closes the loop

Configuration Wizard - Failed to Connect

neilth — Fri, 25 Apr 2008 18:49:00 GMT

When specifying the SQL server, Configuration Database name and account for SharePoint you may see the following error

Failed to connect to the database server or the database name does not exist. Ensure the database server is a SQL server, and that you have appropriate permissions to access the database server. To Diagnose the problem, review the extended error information located at c:\program files\common files\microsoft shared\web server extensions\12\logs\PSCDiagnostics <DATETIME>.log. Please consult the SharePoint Products and Technologies Configuration Wizards for help or additional information regarding the databse server security configuration and network access.

When you look at the log you will probably see some kind of timeout exception at the bottom, right before it all exploded.

LOTS of things could be causing this. Below is a list of item that I have seen in my travels.

SQL Server Problems

SQL Server was OFF,
the SQL Server service was stopped,
the remote TCP/IP connections were not configured for SQL 2005
SQL was sysprepped and the name was changed and it was just generally messed up and had to be rebuilt (Life Lesson: Just install SQL normally, perhaps record an unattended, but don't bring an image of a Sysprepped SQL and try to rename the computer [regardless if it has domain membership] )

Network/Security Problems

Router was melted,
firewall was blocking access,
logged in account really did not have the right permissions on the SQL server (see technet for details of required access),
Cluster name was wrong
Cluster was offline
Account was local (not domain)
Recent DNS/IP change, needed to flushdns on SharePoint server
Admin password was changed while logged on, needed AD refresh between DCs

This is not likely a bad username/password problem. You will see a different message for that.

Technet Snippit

Important

The server farm account is used to access your configuration database. It also acts as the application pool identity for the SharePoint Central Administration application pool, and it is the account under which the Windows SharePoint Services Timer service runs. The SharePoint Products and Technologies Configuration Wizard adds this account to the SQL Server Logins, the SQL Server Database Creator server role, and the SQL Server Security Administrators server role. The user account that you specify as the service account must be a domain user account, but it does not need to be a member of any specific security group on your Web servers or your back-end database servers. We recommend that you follow the principle of least privilege, and specify a user account that is not a member of the Administrators group on your Web servers or your back-end servers.

Try this to help diagnose

Ping SQL/Cluster name
Browse to \\ClusterName\C$ (this will tell you if your logged in account has admin permissions on that box)
If SQL is a cluster, be sure which node is active
Open the SQL server in SQL Management Studio
Check for AV or firewall software on SQL server blocking the access
Try user name as domain\username (NOT username@domain)
Check the lists of all the stuff that can go wrong (above) and be sure you are not having those problems.
If all else fails, just call PSS or your Microsoft Technical Account Manager (if you have one)

SharePoint - Sample Service Level Agreement (SLA)

neilth — Wed, 23 Apr 2008 19:45:00 GMT

This is a very basic sample of a service level agreement for SharePoint. This would be more typical of providing an SLA to an internal department of an organization. Providing an SLA to an external party would probably include an additional set of items around penalties, charges, terms of use etc.

SERVICE ITEM

SERVICE COMMITTMENT

Availability

99.9%

Recovery Time Objective

< 4 hrs

Recovery Point Objective

20 minute data loss window

Service Window

(Weekly) Sunday 1am-2am (PST)

Interface

HTTP/HTTPS (http://InternalSharePoint)

Internet Access

HTTPS (https://sharepoint.external.com)

Remote Administration

Not provided

SharePoint Designer Access

Not provided

Self Provisioning

End Users - http://MySite
Power Users - http://team/Sites
Division Heads - http://Divisions/Sites
All other provisioning through Help Desk

Dedicated SSP

Yes, help desk administered

Enterprise Search

Yes, 50 Million items

Exchange Integration

Not Provided

Business Data Catalog

Not Provided

User Profiles

Yes, from (Internal.com) domain

MySites

Yes, centrally hosted, user self provisioned

Office Communicator

Yes (Client needs client software)

Document Management

Yes

Custom Branding

Yes, 2 Master Pages, 4 custom page layouts

3rd Part Web Parts

Yes, but only from approved Enterprise Directory

Custom Code Deployment

Not available

Dedicated Resources

2 Dedicated Web Applications - Shared Infrastructure

Total Storage

25 GB - $5 per additional GB

User Support

Tier 1 - Help Desk, Tier 2 - Technical Team

Support Fees

200 cases per calendar year, $225 per additional case

User Training

Online Courses

SharePoint - Governance (Part 1)

neilth — Wed, 23 Apr 2008 18:40:00 GMT

Call to action - What parts of governance for SharePoint do you want to hear about? I'll post new topics based on user requests. Just create requests by posting comments at the bottom of this blog. (request it again, even if others have already asked for it. Its kind of like voting and the most votes gets a post first)

This is the first of many rants on Governance. I'm definitely trying to socialize many facets of Governance with Customers and Partners. It is essential to success and (evidently) easily overlooked.

What is Governance?
Structured Control.

What if you don't have any Governance?
You don't have control.

Do you need Governance for a successful SharePoint installation?
YES!

What should be in a Governance Plan? (minimal list)

Hosting model - Central IT or delegated to business units?
Service model - Central service owners or delegated to business units?
Taxonomy - What is the structure of the systems that SharePoint will implement? (formal division sites, mysites, collaboration spaces?)
Architecture- Sizing, Capacity, Growth Strategy, Logical Architecture
Operations - Who runs the infrastructure and what tools/processes to they have? What standards to they have to meet?
Availability, DR, and Business Continuity -
Service Management - SLA's, monitoring, uptime, service tiers (commodity vs. customized applications)
Provisioning - Central vs. Self Service. What can users do on their own? How can you create big picture policies while still allowing freedom and removing strain on IT departments and help desks?
Information management policies - Retention, Disposal, Labeling, Bar coding, Auditing, Records Management, Rights Management
Security Policies - Authentication providers, Remote users, External Users, etc.
Finance Policies - Do you have a chargeback model? How is cost distributed
Users - How do you support them? How do your garner adoption?
Communication plan - What do you tell the organization? and when?

What if you don't Govern?

Site sprawl - sites proliferate like wildfire and you will be without ways to reign them in without upsetting users (who thought they had unlimited storage forever)
Technology proliferation - Users don't know where to put content so it ends up replicated all over file shares, public folders, etc.
Server Proliferation - Groups are unwilling to consume centralized service so they install their own SharePoint (and probably do it poorly due to lack of budget/mandate/experience)
Waste - Content and systems overlap and are needlessly redundant or repetitious wasting tons of money on hardware and software.
Inconsistent - Everybody does things differently so everything from backups to site branding is all over the map. Creating a poor and ill managed user experience.
Damage - "Home grown" SharePoint installations often lack the rigor of setting up data recovery or other essential items. This often leads to data loss or compromise
Poor user adoption - If you don't articulate how they should use it and how you will support them, user adoption will suffer.

What are the Goals of Governance? (no particular order)

Drive Consistency
Assigning responsibility to qualified parties (IT, Service Owner, etc.)
Operational Excellence
Mitigate conflict and provide a framework for Decision making
Manage hardware and software assets strategically
Set clear expectations of service with users
Lower costs
Reduce Risk
Provide "Roadmap" for approach to intentionally and deliberately plan rollouts

Some Resources

Governance TechNet Landing Page -
http://technet.microsoft.com/en-us/office/sharepointserver/bb507202.aspx
- See this site for links to tools, third parties, community resources etc.
Governance Tools - http://www.codeplex.com/governance

SharePoint 2007 - Incoming Email

neilth — Fri, 26 Oct 2007 18:44:00 GMT

There are many little "gotchas" with setting up incoming email for MOSS. Just nit picky little things, normally they are all about email infrastructure, SMTP, etc. My advice below is to help you avoid taking your heart medication, it can be easy to set this up, but it can also be a hair pulling, eye rolling, staying up all night and getting no results kind of affair as well.

I promise, this technology does actually work, but if you deviate from the standard build (see list item 3 below for "standard build" as defined by me) you will fall off a cliff unless really understand what you are doing.

I'm not an exchange guru, just a humble BizTalk/Moss guy.

Don't just do this trial and error and play with settings until it works. It won't work. Be deliberate and intentional, setup the standard way and then slowly change things, each time checking to see if the forwarding still works.

This is where I recommend you start, read all this stuff first, before you race off back to your Virtual Machines.

Plan incoming e-mail (Office SharePoint Server) <link updated - August 2009>
Configure incoming e-mail settings (Office SharePoint Server)<link updated - August 2009>
In addition to the technet site, a very good place to start is

Read these documents and set it up in a lab exactly as described until it works
Now introduce incremental changes to the infrastructure and after each one, verify that email is still getting through, or not.

Gotchas and Advice

Trust me, for the first run, just make the FQDN of the moss server (SMTP server) the same as the @domain that you are trying to send to

Clarification
AD domain = example.com,
moss server name = MOSS2K7
moss FQDN = MOSS2K7.example.com
email domain = MOSS2K7.example.com
example address = myemail@moss2k7.example.com
Trust me, for the first run, start off by actually allowing MOSS to own an OU and manage it's own contacts, don't be a cowboy and hope to configure them manually on the first try.
Just do everything the default way the first time, use a lab environment if you have to
Use the OWA website on your lab email server (or a local instance of Outlook conntected to that server) to send emails around, don't try to hook-up external MX-Records and send from external emails.
Keep and eye out for .eml files showing up in the Queue folder on the MOSS box. This is a sign that the SMTP server is forwarding them elsewhere and that is a sign that there is a fundamental problem with the email coming in. Most likely the SMTP sever does not recognize that it is the utlimate destination for emails with that @domainname address.

If you try to read from the SMTP queue on the MOSS server, you will get

A critical error occurred while processing the incoming e-mail file C:\Inetpub\mailroot\Queue\NTFS_cab371a601c8166600000011.EML. The error was: The process cannot access the file 'C:\Inetpub\mailroot\Queue\NTFS_cab371a601c8166600000011.EML' because it is being used by another process..

The solution to this is "DON'T READ FROM THE QUEUE". If the emails are in the SMTP queue on your Moss server then your email/smtp settings are probably messed up. (Perhaps you did not take my advice on keeping the FQDN of the MOSS server the same as the @domain name that you are sending mail to?)

______________________

If you try to "drag and drop" from queue to the drop folder manually, you will get

A critical error occurred while processing the incoming e-mail file C:\Inetpub\mailroot\drop\NTFS_db489bee01c8166600000012.EML. The error was: Bad senders or recipients..

Don't tell me, let me guess. You copied the .eml files from the Queue folder to the Drop folder and then saw sharepoint delete them. Same point as above, If the emails are in the SMTP queue on your Moss server then your email/smtp settings are probably messed up. (Perhaps you did not take my advice on keeping the FQDN of the MOSS server the same as the @domain name that you are sending mail to?)

_______________________

BizTalk Server 2006 - Enterprise Production Considerations - Part 4 - BAM

neilth — Tue, 23 Oct 2007 19:12:00 GMT

Obligatory Introduction

BAM is such a good name for this technology for a few reasons. The first is that it reminds me of the kid Bambam on the Flinstones (You know, the baby with the club that had a thing for Pebbles). Very powerful and very cool. Also because "BAM" is the sound my forehead makes while it collides with the palm of my hand as I discover I could have simply used BAM instead of writing a custom reporting app.

Sample to Whet the Appetite

Here is a fictitious BAM portal Aggregation for one of my lab projects

The Basics

This article will be a fairly unstructured summary of my experiences with BAM under BTS 2006. If there is interest, I will create some more in-depth posts on the topic. I'll be brief here because most of you know this already. The basic way BAM works is the following:

Create a Business Activity showing the LOGICAL progression and points of interest in a process (using Excel, or Visio ODBA)
Deploy that Activity (as .xls or .xml to the biztalk server) using BM.exe
Associate low level technical events (i.e. hitting a shape in an orch or a method in a .net class) with items you defined in the activity (using the BAM API or the Tracking Profile Editor (TPE)

The end result of this is that you can take the mess of code, orchestrations etc and transform it back into a logical/business view that a CEO could understand. All in all, very cool.

Some Pragmatic Advice

"DO" List

Try both ODBA and Excel for creating activities (I like excel better, but hey, who cares?)
Become familiar with the BM.exe utility. You really only need the "deploy-all" and "remove-all" commands to start with
Keep your activity files versioned! If you deploy an activity, then change the activity file, then try to undeploy you may have problems. Then you have to undeploy by manually deleting tables and rows in the BAMPrimaryImport table.
I recommend that you manipulate your activity definition in Excel. When you are ready to deploy it, export it to xml and then deploy the xml. That way if you fiddle with the Excel file, you can still undeploy the activity using the same xml you used to deploy it.
Before you re-export your activity to XML, undeploy the activity. Or even better export your Activity to a new version of the xml (i.e. with a version number in the file name
Start with the tracking profile editor, master it, then move onto the BAM API calls.

"DON'T" List

Don't try to map milestones from Message Constructs or Loops (or other special cases). That is advanced and you may need to use the BAM API call to make that work as you would expect
Don't put spaces or special characters in your Activity Names. It can lead to problems later where the Event Log will say that "stored procedure does not exist" and your BAM Aggregation will not work.
Don't have a group shape as the last shape in the orchestration you are tracking in TPE. For some reason it won't work. Just add any shape (i.e. Expression) after the group to fix the issue.

Continuations & Relationships

I can't do these justice in this entry so this will be a bit shallow. I will probably blog on this separately. The more comments i see below requesting this, the sooner I will get around to it.

Continuations allow a single activity defintion to span muliple orchestrations or even other classes/systems seamlessly. It can also allow you to use different unique identifiers accross those systems to When using TPE to create a continuation. Create a continuation and name it, then create a continuationID and name it the exact same thing. Map some kind of ID value (PO ID, RequestID, etc.) from a messaging shape (receive, construct, send as described in the above DO List) in the first orchestration to the continuation. Then in the second orchestration map that same ID value to the continuationId. The idea is that the engine will recognize that those to values match and will correlate them so that both instances can contribute data to the same Activity instance

Use Continuations when the Relationship is 1:1
Use Relationships when the Relationship is 1:Many

Real-Time vs. Scheduled Aggregations

There is a little button in the Excel view (if you have installed the BAM addon) that will mark the aggregation as RealTime or Scheduled. Again, this is complex so I will summarize below.

	RealTime	Scheduled
Maintenance	Low maintencance - The data is kept in the relational tables and periodically cleaned. All you have to do is set the TimeSlice and TimeWindow values to determine the trimming behavior	High Maintenance - This data is kept in cubes so you need to have Analisys server setup, schedule the packages, and make sure they execute as expected
Amount of Data	Small Amount of Data - Because this is kept in the relational tables the volume of data has to be kept rather trim	Large Amount of Data - Because this is kept in cubes you can have almost unlimited amount of data
Accuracy of Data	Very Accurate - This provides up to the second view of data and can show in progress activities	The data you have is correct, but you will be missing any data collected since the last execution of the packages
Comprehensive Measures	Limited - You can not use MIN,MAX, and apparently AVG functions in RTA	Unlimited - You can use any type of measure in scheduled packages
Performance	This does contend with SQL and BizTalk for resources on the production box.	This is presumably a separate machine serving as a DataWarehouse so it should not contend with the production server for resources. (except to execute the packages)

BAM Alerts

For the most part, real time data is better for driving alerts, as they are usually time sensitive.

Common Problems & Solutions

Problem: The BAM Activity and the Tracking Profile have been deployed, the process has been run, but there is no data displayed for the activity in the BAM portal or LiveData Excel sheet.

Solution: When you deploy a non-Real Time Aggregation (this is default) while using SQL 2005. It will create the <package>_DM and <package>_AN packages. These packages need to be run to populate the BAM tables and get you the BAM Data. To run these packages you should select "Integration Services" in the SQL Management Console (not database engine). Once you have found them, run them. Hopefully, you're data will appear.

Problem: The BAM portal or LiveData Excel sheet shows "BLANK" for one column and one row in the Aggregation display. It seems like the measures, and the milestones are not being combined into the same activity. Also, the data shows up in two rows (or more) in the BAMPrimaryImport.dbo.bam_<ActivityName_Completed table.

Solution: This could be a number of things, but if it was the same problem that I had the solution is the following. When you are in the TPE and you are associating data from the message, be sure to use the orchestration schedule. Right click on your receive, Message Construct, or Send shape and select Message Payload and map the items from there. Many people go to the "Select Event Source" and use the "Messaging Payload" from there. YOU ONLY WANT TO DO THAT IF YOU ARE JUST USING SEND RECEIVE PORTS TO MOVE MESSAGES. If you are using an orchestration, map everything from the orchestration schedule.

Problem: Your BAM aggregation is not working and you see "Could not find stored procedure" in your event log.

Solution: This is probably caused becaues you have a "space" or some other bad character in your Activity Name. Undeploy your activity, rename it so it does not have the space. Redeploy it. Adjust your TPE (if required). This should fix the problem.

Wish List

Not being able to have spaces in the Activity Name AND not being warned about it is a serious limitation
The length of Activity Names and Item names is too short. I wish for longer names.
Measures using Average also seem to not show for RealTime Aggregations (at least they did not in mine).
I would like to see this integrated with the Visual Studio Environment to a much greater degree. My vision is that we use Office and VS and nothing else.
I also want to be able to deploy BAM stuff with MSI easier

SharePoint Office Server 2007 - Overview WebCasts for Decision Makers

neilth — Tue, 16 Oct 2007 21:34:00 GMT

This posting is intented to complement the Training Roadmap for Office SharePoint Server 2007 with additional resources for non-technical staff members.

This are excerpts from webcasts available at msevents.microsoft.com. These are best for Technical Decision Makers, Business Decisions Makers and Architects...or developers and IT Pros who have to explain this stuff to BDMs,TDMs, or Architects :)

These webcasts cover:

Productivity and Communication
Enterprise Search
Records Management
Enterprise Content Management
Business Intelligence

Microsoft Webcast: Improve Productivity and Communication Using Office SharePoint Server 2007 (Level 100)

Have you ever thought about how much time is lost by your employees because of miscommunication or by the learning curve for a new product, environment, or policy? Join this webcast to get a better understanding of how Microsoft IT is attacking this problem head-on with the built-in features in Microsoft Office SharePoint Server 2007.We describe using a programmatic approach to improving the quality, effectiveness, and timeliness of the communications and content sent to employees. We also examine how tools, support, availability of critical patches, and access to user education resources can help you boost productivity and communication. 2/22/2007 7:00 PM Pacific Time (US & Canada)- 2/21/2009 12:00 AM | Duration:52 Minutes

Primary Language: English

Primary Target Audience: Technology Decision Maker

Microsoft Webcast: Enterprise Search at Microsoft with Office SharePoint Server 2007

Like most large companies, Microsoft’s intranet contains terabytes of structured and unstructured content that employees need to find and use on a daily basis. In the past, it has been difficult to bring such diverse content sets together in a single search-center experience. Join this webcast to see how the recent release of Microsoft Office SharePoint Server 2007 makes it possible for users to search different content repositories with just a few clicks. We explore how Microsoft manages its internal search experience built on SharePoint Server 2007, both on the central corporate portal site and as a search service used by more than 75,000 employees worldwide.2/15/2007 7:00 PM Pacific Time (US & Canada)- 2/14/2009 12:00 AM | Duration:69 Minutes

Primary Language: English

Primary Target Audience: Technology Decision Maker

A Web Seminar with AIIM and Microsoft: Records Management in Microsoft Office SharePoint Server 2007

Microsoft Office SharePoint Server 2007 is a comprehensive Business Productivity platform, offering a rich set of capabilities including Enterprise Content Management. Following an introduction to the principles of electronic records management by Marc Fresko of Cornwell Management Consultants, Rob Gray, Microsoft UK Product Marketing Manager for SharePoint and ECM, will take a closer look at the Records Management capabilities of Microsoft Office SharePoint Server 2007.

11/10/2006 6:00 PM GMT, London- 11/9/2008 12:00 AM | Duration:74 Minutes

Primary Language: English

Primary Target Audience: Technology Decision Maker

http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032316620&culture=en-US

A Web Seminar with AIIM and Microsoft: Web Content Management using Microsoft Office SharePoint Server 2007

Microsoft and AIIM present Web Content Management using Office SharePoint Server 2007: Microsoft Office SharePoint Server 2007 includes many features that are useful for designing, deploying, and managing enterprise intranet portals, corporate Internet presence Web sites, and divisional portal sites. These features enable you to author and publish Web content in a timely manner and can ultimately reduce the cost and overhead of managing multiple sites. Microsoft partner cScape deployed the world’s first web content management system on Office SharePoint Server 2007 with Ministry of Sound label HedKandi. Ben Robb, Lead Technical Architect at cScape and James Bacchus, CIO at Ministry of Sound will talk about the choice of SharePoint as a WCM platform and their plans for the future.

1/19/2007 8:00 PM GMT, London- 1/18/2009 12:00 AM | Duration:57 Minutes

Primary Language: English

Primary Target Audience: Technology Decision Maker

Microsoft Business Intelligence Solutions Demonstration

Demonstration of Business Intelligence Using Sharepoint Server, Office Excel 2007 and SQL Server 2005

8/23/2007 10:30 PM Hong Kong, Taiwan, Singapore- 8/22/2009 12:00 AM | Duration:25 Minutes

Primary Language: English

Primary Target Audience: Business PC User

SharePoint 2007 Training Material Roadmap

neilth — Tue, 16 Oct 2007 19:55:00 GMT

This blog posting is a page designed to support basic feedback and management for the Training Material Roadmap that I created for Windows SharePoint Services 3.0 (WSS 3.0) and Microsoft Office SharePoint Server 2007 (MOSS 2007).

(Oct 16 2007) - Training Document Now Released!
Word Document Version - Click Here
WebPage Version (Limited Formatting) - Click Here

You may use this page to:

Obtain a copy of the roadmap document (not the accompanying DVD) - click here for doc version!
Request additional references for particular topics
Get links to supplemental blog postings

Click here for webcasts for Business Decision Makers, Technical Decision Makers, and Architects

Provide feedback or suggestions regarding future iterations of the roadmap

I welcome your feedback and criticism for how the materials have been organized in the roadmap. I would like to help you navigate the massive topic of SharePoint 2007 technologies and nobody can give me more insight into your training and rampup needs than you.

BizTalk Server 2006 - Enterprise Production Considerations - Part 3 - Clustering

neilth — Sat, 22 Sep 2007 05:08:52 GMT

I have had this recurring conversation with customers and partners. Clustering is not a simple concept, and neither is BizTalk. If you put them together you seem to have a perfect storm. There's lots of confusion, which I hope to be able to dispel with this post.

Also, I would like to have a link to send people, just so that I don't have to have the conversation again :)

The Summary

The SQL servers hosting the BizTalk databases can be clustered for high availability.
The Enterprise Single-Sign On Master Secret Service (MSS) can be clustered for high availability.
The BizTalk hosts can be clustered for the following reasons:
1. They are hosting something (i.e. Receive Port) that has to be single instance, but still highly available.
2. If you have BizTalk instances running on the same nodes that are running a clustered instance of the MSS.

FAQ

What parts should I cluster?
You should cluster the SQL instance(s) holding the biztalk databases, the ENTSSO Master Secret service, and the BizTalk Hosts that require on a single instance running at a time, but still needs to be highly available.
Is a non clustered-BizTalk host still Highly Available?
If you have at least two host instances running in the same group for this host, then YES. It behaves more like an NLB cluster than an MSCS service.

The Easy Part

I say that the SQL clustering is the "Easy Part" not because it is simple, but because I don't feel compelled to explain the nitty gritty details in this post. I will give some of the BizTalk specifics here, and dedicate the next post to SQL cluster configuration. For now, as a rule, go with SQL 2005 Active/Active/Passive.

Separate the MessageBox and Tracking Database among separate LUNs and separate SQL instances.
The "Other" databases can often be place on the same node running the tracking DB. If you are heavy on BAM or BAS you might want to break off the respective databases to a separate instance.
All the databases should have at least 2 LUNS (data log), often they have more, but that gets into mount points and that is a tanget.

The Somewhat Easy Part

Clustering the ENTSSO MSS is pretty simple and the procedure is posted here. Basically, this is just a running service of the ENTSSO that also takes on the responsibility of distributing the Master Secret Key. This is the key that all the ENTSSO services running on other machines need to read the encrypted information in the SSO Database.

This should be clustered. Unless you don't care about high availabilty. This can really mess up BizTalk if it goes down.
This is not really a resource intensive service (typically) so it often piggy-backs on the SQL server nodes runing the clustered instances of SQL Server.
You don't want this clustered on the BizTalk boxes as a rule of thumb. (reasons covered later in this post)

Ok, now the meat.

BizTalk hosts generally have instances on more than one box, as such they are already highly available. This means that clustering a BizTalk host is not generally necessary to ensure high availability. If one goes down the others will pick up the work, out-of-the-box standard config. The problem comes in when you REALLY don't want to have mutliple instances of a host running at the same time. The classic example of this is a receive port for an adapter that can not gracefully handle two threads reading it at the same time (i.e. FTP). It also happens when you need to be certain that all of the messages are picked up and processed sequentially and you don't want to code around the problem.

So you want an FTP port receive, but you can not affort duplicate message reads. That means you can have only one instance of the host, but that kills the out-of-the-box high availability. Now you need to cluster that host instance, that is the only way to ensure that exactly one instance of the host will be running (so long as at least one of the BizTalk app servers is running)

The Very Wierd Part

This is the one that people scratch their heads over. It does not happen much, and its not really all that complex. But it is not intutitive, so it messes with people's heads.

For this to really make sense, I have to back up a step and explain how the ENTSSO service generally works in a mutli-box BizTalk installation.

All of the BizTalk App servers have an instance of the ENTSSO service running on the machine. It is this service that obtains the secret key from the ENTSSO Master Secret Server and then uses it to independently access the info in the SSO Database.
One of the ENTSSO servers is setup as the Master Secret Server. This means that it will distribute the secret key to the other ENTSSO service instances, in addition to being the standard ENTSSO service for the local box. This is the service instances that needs to be configured for High Availablity because all of the other ENTSSO service instances depend on it.
The BizTalk host instances have a dependency on the ENTSSO Service running on the same box as them. This is because they need desperately need it to give them info from the SSO DB. This information is typically configurations for their ports that is stored in the SSO DB for safe keeping.

Figure 1 - Standard Multi-Box ENTSSO Strategy

Lets say that you have only 2 production servers. They will be running both SQL and BizTalk. Like a good boy/girl you decide to cluster SQL and the ENTSSO MSS.

Now you try to configure BizTalk host instances, and you find that they don't want to work quite right. That is because they have a dependency on the ENTSSO service, they always need it. Now that the ENTSSO service is clustered, only the host instances running on the active (ENTSSO) node actually have access to the service they need. All the other instances will fail based on dependencies.

Figure 2 - BizTalk Hosts with Broken Dependency

An an extra bonus you have messed up the High availability of the MSS as well. The clustering service detects that the host instances are depending on the ENTSSO MSS service and probably will not allow it to failover to other nodes.

...NICE...

Option 1 is to not cluster ENTSSO MSS in a 2 box scenario

Bottom line is that, if you have only budget for two prod boxes, you probably don't want to cluster the ENTSSO MSS. But this is really the only time I can think of that clustering that service is bad. If it is not clustered you better be a "Johnny on the spot" with MOM and implement a way to have MOM do a poor man's failover when it detects that the ENTSSO MSS server is down.

Option 2 is to also cluster all BizTalk host instances that depend on a clustered ENTSSO service.

If you really do want to cluster the ENTSSO MSS, then you also have to cluster all of the BizTalk hosts as well. This ensures that as the ENTSSO Service "flips" from one box to the next, the Microsoft Clustering Service can ensure that the BizTalk host instances flip at the same time and don't come crashing down.

BizTalk Server 2006 - Enterprise Production Considerations - Part 2 - Host and Host Instances

neilth — Tue, 31 Jul 2007 15:57:00 GMT

I've seen one too many BizTalk groups with poorly configured Host/Host Instances. So I've decided to create a BASIC post on the subject. This is a pretty sophisticated topic having to do with scaling, high availability, resource management etc. so please don't assume that what I'm about to say is universally appropriate. I'll give a few suggestions and more importantly, I'll give the reasons behind those suggestions which should give you the background you need to create your own configuration wisely.

Suggested Starting Point

Enable "Allow host tracking" on the default in-process BizTalk Application Host
1. Do NOT enable "Allow host tracking" on any other Host
  Confirm with the BTS docs if you like, but only one host has to have this enabled
2. Do not use this host for any ports or orchestrations, this is dedicated for host tracking
Create at least 3 in-process hosts for each logical application that you have. (Use at least 2 hosts if you are not using any orchestrations)

One for Send Operations [Bind your send points to this host]
One for Receive Operations [Bind your receive ports to this host, except HTTP or SOAP]
One for Processing [Bind your orchestrations to this host, if you have any]
A case by case examination can be made here for additional hosts, usually for:

High volume orchestrations or those you wish to have granular control over. These get a dedicated host
Out of process hosts for HTTP/SOAP/Custom protocols . These get additional Hosts/Instances

Application Isolation
Having separate hosts for your applications allows you to restart the host instances for that application (perhaps because of a re-deployment) without interrupting the other apps.

Scaling Out
Having a host for Send/Receive/Process allows you to distribute your application accross several servers in your BizTalk group in a flexible manner. You can increase the number of host instances running to "beef up" any combination of these three types of operations as required.

Memory Utilization
As you well know, 32 bit systems have limits regarding how much memory can be used by a single host (process). A single host on a 32GB system will really only use about 1.8GB - 2GB or so. If you want to take full advantage of all of that juicy memory you'll have to create more host instances and let each of them take their 2GB bites out of it. (As a rule of thumb, be sure to leave 4GB for the OS). This is not universally true (i.e. 64bit systems are different)

High Availability
Have at least two host instances running for every host that you have. This way, any single node can fail and the other node will continue to operate. This can become more complicated for Receive nodes so check the HA configuration for whatever adapter types you have running. For instance, FTP does not support locking so having two host instances reading from it can cause duplicate messages. This has to have just one host instance reading from with (which I tend to call a "Singleton" adapter configuration). In this case there is one host that is clustered for high availability

Security
You may also need to create additional hosts so that you can use different credentials to access different systems. The security context is determined by the Host/Instance which can be very helpful in scenarios that are using AD to authenticate your requested operations. (i.e. File Adapter, SQL Adapter, and I believe the MS CRM adapter, among others)

Some Caveats

Too many hosts.

My BizTalk groups seldom have more than 22 hosts or so (7 apps + default host for tracking) So if you begin to experience problems with Groups having more hosts than that I recommend you consider too many hosts as a possible cause. There could be RAM availability issues at these levels.
Configurations in BTSNTsvc.exe.config may give you weird results because you may be assuming that they apply to the whole group, when really they apply to the host instances
Again, in the BTSNTsvc.exe.config, the changes in this file are picked up when the host instance restarts. You may have changed the configurations, restarted only a handful of your hosts. Later down the road, you restart other host instances that then begin to have problems because of the contents of the BTSNTsvc.exe.config. This is VERY hard to trouble shoot and I have two recommendations.
1. Don't change BTSNTsvc.exe.config unless you absolutely MUST.
2. If you change BTSNTsvc.exe.config restart ALL of your host instances immediately so you can quantify the affect of the changes immediately.

Single Box groups
1. If you're running BizTalk all on one box then it hardly helps you to be able to partition out the operations. In this case, I would just create one host per application so you can still have Application Isolation, without worrying about the scale out features or HA (which you can't really use with just one box anyways)

Agree, Disagree, Don't Care? Please comment below

BizTalk Server 2006 - Enterprise Production Considerations - Part 1 - Production Server Topology

neilth — Thu, 26 Jul 2007 17:50:00 GMT

Introduction:

This post is the first in a series of posts that will describe a number of production considerations for Enterprise customers (“Enterprise” loosely defined as those with moderately complex BizTalk implementations)

This is in response to a great deal of confusion that I have encountered in the field and hopefully I can kill about 500 birds with 1 stone by publishing these findings (no offence to bird enthusiasts J ).

Today’s Topic – Starting the Production Infrastructure

BizTalk Server 2006 has a dizzying array of scalability options, but many options can mean complexity. This post will specifically discuss the most common “starting points” for production enterprise BizTalk systems and compare/contrast them so you can hopefully figure out which one best suits your needs.

The X + Y notation means:

X Dedicated BizTalk boxes
Y SQL Server Nodes

Server Configurations

Before we get started here, I have to emphasize that these are all common STARTING configurations. Configurations can (and do) scale out and up depending on your requirements.

Also, all of these configurations have some notion of high availability, no single node failure will kill the infrastructure. Way more options are available if you sacrifice High Availability (but that’s pretty unusual in the Enterprise isn’t it? J )

This configuration has the minimal number of servers involved but it does present some complexity problems because the BizTalk hosts must be clustered if you are running the ENTSSO service as a clustered service. You either cluster the hosts or live with having an un-clustered Enterprise Single Sign- on Master Secret Server. Either option could be best depending on your requirements.

This configuration is a pretty good starting point for the cost-sensitive enterprise. This allows separation of BizTalk from SQL (a good thing) while maintaining redundancy by having two BizTalk nodes that can “cover for each other” in case one of them goes down. The ENTSSO MSS can be installed as a clustered service on the SQL cluster without creating complications for the BizTalk hosts (i.e. they don’t have to be clustered, which is also a good thing in many cases)

This is probably my favorite starting point. It is the lowest cost, and scaling out with more BizTalk servers is really easy. If you’re not sure which one to pick, take this one.

This configuration is an improvement on the 2+2 configuration because it really allows the orchestrations and business logic to become intensive without having much of an impact on the Send/Receive operations (even though the Send/Receive operations can still interfere with each other). This is for enterprise applications with moderate Send/Receive activity and fairly intensive business logic.

Of the configurations being considered in this article, this is the most capable (and most expensive). This allows full separation between BizTalk and SQL, and also between the Send/Receive/Process operations within BizTalk. This configuration is for Enterprises with intensive Send/Receive/Process requirements.

Notice the additional SQL server, a pretty good rule of thumb is a ratio of 3:1 for BizTalk:SQL servers. Although this one is 3:1 with only 2 SQL servers, a failure of one node would change that ratio to 6:1. Seeing as how this configuration is for high volume scenarios, that would be bad. Invest a couple of bucks and have a passive node in the wings to keep things running if a SQL node goes down!

Comparison Chart

	0+2	2+2	4+2	6+3
High Availability	Yes	Yes	Yes	Yes
Performance	Lowest	Medium	Medium	Best
Cost (# Servers)	Lowest	Medium	Medium	Highest
Ease of clustering SSO MSS	Hard	Easy	Easy	Easy
Separation of SQL and BizTalk	No	Yes	Yes	Yes
Dedicated Receive Nodes	No	No	No	Yes
Dedicated Send Nodes	No	No	No	Yes
Dedicated Process Nodes	No	No	Yes	Yes
Easy to Scale out	Complex	Yes	Yes	Yes

Call to Action

I know what I want to write about, but this blog isn't for me! What topics do you want to see in the next article? Drop in a request in the comments section below and I'll review before I start writing the next article (I promise)

Configuration
SAN/Storage
Troubleshooting/Debugging
High Availability
other....

BizTalk gets "General Network Error" under heavy load

neilth — Fri, 06 Jul 2007 00:53:00 GMT

This is a fairly common scenario after Windows 2003 SP1 (or greater) is installed, so I thought I better put in my two cents

Scenario: BizTalk seems to lose connectivity to the MsgBox (and all other databases on that SQL instance)

Problem: The operating system (w2k3 sp1) running the SQL server instance is misinterpreting the heavy TCPIP connection traffic from the BizTalk App server as a Denial of Service (DOS) attack.

Clarification: This is an operating system level issue. This is not BizTalk or SQL per se (though they are common victims of this security feature). This is a security feature for the OS that affects how the OS interprets large amounts of incoming connections from individual clients.

Factors:

1. Windows Server 2003 SP1 (or higher) is installed on the SQL server hosting the BizTalk DBs

2. SynAttackProtect is still set to “1” on the SQL server

3. Concurrent TCPIP connection requests from a specific BizTalk app server exceeds the critical threshold (Sorry, don’t know what that threshold is!)

Diagnosing:

Look for the following event entries:

Event Type: Error
Event Source: BizTalk Server 2004
Event Category: (1)
Event ID: 6913
Computer: %ServerNameHere%

Description: SQLServer, BizTalkMsgBoxDb, [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation..

Event Type: Warning
Event Source: BizTalk Server 2004
Event Category: (1)
Event ID: 5410
Computer: %ServerNameHere%

Description: An error has occurred that requires the BizTalk service to terminate. The most common causes are an unexpected out of memory error and an inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.

Error message: [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation.

Error source:
BizTalk host name: BizTalkHost
BizTalk Windows service name: BTSSvc$BizTalkHost.

Fixes:

1. Set the SynAttackProtect registry setting as described in http://support.microsoft.com/kb/899599

2. ALSO take a look at the Group policy for that SQL server to be sure that the Group Policy does not periodically UNDO the setting change that you did in step 1. I have been in the situation where I fixed the issue, but then the GP Update or reboot undid the fix! (Put the server in a separate OU if you have to, but make sure you isolate the box from policies that would reset this registry key)

Limitations

1. I don’t know how to disable this feature for a specific set of clients (i.e. SynAttackProtect is off for the BizTalk app servers but remains on for other servers). The setting seems to be at the protocol level so I wouldn’t even know where to start looking for network set definition here.

Is there something that I can do to make this blog more helpful to you? I welcome your feedback!

Office 2007 Open Format

neilth — Wed, 20 Jun 2007 06:59:00 GMT

Office 2007 users want to see something cool? Save out a word 2007 .docx format, rename the file to have a .zip extension, and then crack it open. How cool is that?

Excessive Blocking when using BizTalk SQL Adapter

neilth — Wed, 20 Jun 2007 06:00:00 GMT

-------------------------------------
Addendum – June 22 2007: Under absolutely NO circumstances should you modify the data model of the BizTalk databases in any way. You should only optimize your own application databases.
-------------------------------------

I have seen this issue in many places, which still surprises me because there are a lot of blog entries about this. For what it's worth, here's one more.

Scenario: BizTalk solution makes moderate to heavy use of SQL adapter in calling stored procedures in a database.

Problem: SQL server shows many blocking (perhaps deadlocking) SPIDs, most of which are owned by the BizTalk host instance account.

Factors:

The longer the procedure takes to execute, the worse off you are

Big tables, poor logic, bad indexes, bad db maintenance, disk contention etc.

The more concurrent procedure executions the worse off you are

short polling interval
multiple ports
multiple host instances

Diagnosing:

Enterprise Manager or Management Studio can give some insight to active and blocking spids (check to see who owns the blocking spid and use dbcc inputbuffer to see the sql it is executing)
Try using sp_who,sp_who2,and sp_who3, they may give you lots of insight
Call MS PSS (or download pssdiag for yourself from the MS download site) and have them configure a pssdiag trace for you. Look especially closely at the blocker output and the profiler trace. Use rml.exe (read80trace) to generate a report of the "worst offenders" sql scripts or procedures.

Fixes:

As per documentation, don't use T-SQL transactions in your proc. The SQL Adapter already has this all wrapped up in a distributed transaction so relax!
As per documentation, you can try setting the transaction isolation level to READ COMMITTED, or perhaps REPEATABLE READ. If you are running SERIALIZABLE that is a great way to tank performance and cause blocking.
Speed up the procs (Archive, purge, index, create views, run db maintenance, buy a big SAN, whatever. The locks won't kill you if they don't last)
Don't go silly with coarse locking hints. Keep your lock few in number and small in granuarity. If you must manually lock, then please read up on the READPAST hint - especially for queue or "trigger" tables. (Thanks to Kunjal K for this hint)
Reduce concurrency - You can try lengthening your polling interval or reducing the number of host instances that run that port.

Didn't find what you were looking for? Try one of these:

Is there something that I can do to make this blog more helpful to you?
I welcome your feedback!