Jeff Nuckolls {Application Platform and Development Tools} : Biztalk

Biztalk POP3 Adapter and WSS Adapter Example

jeffnuck — Sat, 05 Apr 2008 22:59:00 GMT

After some extensive research I could not find a good example of how I could receive an Email via the POP3 Adapter for Biztalk Server 2006 R2 that completely parsed out the header content (To, From, Subject, CC, etc...) and the Body (I don't parse the attachments in this example, but here is a sample that does a great job of parsing multiple attachments, thanks to Stephen W. Thomas for posting it.).

* For this article I make a general assumption that you already have a working knowledge about the Biztalk for creating and deploying a basic Biztalk application.

Scenario:

College Admissions process requires that a prospective student submit an Essay as part of their application for admittance to the University. Here the student will be submitting their essays in the body of an email. Instead having someone in the Admissions office monitor a huge inbox, we can automate the process of by parsing out the email and submitting it to a Sharepoint List. For this first example that is as far as we'll take it, but the idea is that once anything is submitted to a Sharepoint List... Sharepoint workflows can be started, staff from the Admission Office can easily search/edit/export these essays, they can be automatically be correlated with other application artifacts (ie. Transcripts, Application Info, Financial records, etc...) based on the application ID or student email to ultimately create a single "student profile" once everything has been received for a particular student. (I'll demonstrate the rest of these other pieces in a separate example.)

Setup a custom List in Sharepoint (WSS or MOSS):

(Details about setting up and configuring Sharepoint and Biztalk are beyond the scope of this example.)

Here I have created a custom List called "Essays for Student Admission Applications" with the following Columns:

Changed the Default "Title" Column name to Submitted By
Essay Response (Multiple lines of text)
Application ID (Single line of text)
Date Submitted (Single line of text)

* I left the defaults for Created By and Modified By as is.

Create a new POP3 Mailboxes and setup the Outlook client:

* Here we are assuming that you have already installed/configured POP3 and SMTP Services or your will be leveraging an existing POP3 mail server (ie MS Exchange Server). (The installation/configuration of POP3/SMTP Services is beyond the scope of this example.)

In this example I installed POP3/SMTP on my local machine and configured it to use my local domain (called nuckolls.edu)
Add a new Mailbox to be used by Biztalk (you must know the password of the account associated with this mailbox)

Added a mailbox associate with a user account (called ProcessApplication@nuckolls.edu)

Add a new Mailbox to simulate the Student (you must know the password of the account associated with this mailbox)

Added a mailbox associate with a user account (ie. jeff@nuckolls.edu)
Create a new Account Profile in Outlook or Outlook Express to send and receive email as this Student. (Steps for this were omitted, assuming this was a reasonably simple step... just make sure it works, by send/receiving email or checking the C:\InetPub\mailroot\Mailbox\nuckolls.edu\YourMailbox.mbx)

Create a new blank Biztalk Application project:

Name the Project TestPOP3
Right-click on the TestPOP3 project in Solution Explorer and select Properties
In the Project Properties window ensure you have done the following:
- Common Properties/Assembly:
  - Assembly Key File = (assign your *.snk key file... if you haven't already created it, create one using the VStudio command prompt and sn.exe -k C:\MyKey.snk)
- Configuration Properties/Build:
  - Application Name = TestPOP3
  - Restart Host Instances = True

Create the initial Biztalk application artifacts (Schemas):

In this example we will need two schemas, one for the inbound email message (we will call EmailSchema.xsd) and one for the outbound Sharepoint message (we will call AdmissionApp.xsd).

Create a new Schema called EmailSchema.xsd
1. Rename the Root node to EmailText
2. Create one Child Field Element named Text with a Data Type = xs:string
3. Promote the Text element as a "Distinguished Field" property.
  
  EmailSchema.xsd should appear as follows:
Create another Schema called AdmissionApp.xsd
1. Rename the Root node to Application
2. Create the following Child Field Elements all with a Data Type = xs:string and promote all of them as "Distinguished Field" properties:
  
  AdmissionApp.xsd should appear as follows:
  
  * Ensure all of the Child Field Elements in both schemas are set to "Distinguished Fields" as follows:

Add a (Flat file disassembler) Pipeline component:

This will be used to help parse out the header and body elements of the text-based email we will receive from the POP3 Adapter.

Add a new Receive Pipeline component to your Biztalk project called GetEmailText.btp
Add a Flat file disassembler to the Disassemble shape on the pipeline designer.
Assign the Document Schema property of the Flat file disassembler = TestPOP3.EmailSchema

Create a Biztalk Orchestration:

This orchestration represents the automated Business Process of receiving an Essay as part of a larger Admissions Application process.

Add an Orchestration:

Right-click on TestPOP3 project in Solution Explorer and Add a new Biztalk Orchestration
Name it EmailProcess.odx

* Your Solution Explorer window should now appear as follows:

* Before we start adding/configuring shapes to the Orchestration Designer... let's first open the Orchestration View window and define some Messages and Variables that we will later use within our Orchestration:

Orchestration Messages and Variables:

Select View-->Other Windows-->Orchestration View (otherwise it should already accessible on the tab next to the Solution Explorer window)
Right-click Messages and create the following Messages:
- In
  - Message Type = TestPOP3.EmailSchema
- Out
  - Message Type = TestPOP3.AdmissionApp
Right-click Variables and create the following Variables:

strEmailAddress
- Type = System.String
strStudentID
- Type = System.String
XMLDOM

Type = System.Xml.XmlDocument

* The Orchestration View should appear as follows:

Orchestration Workflow/Process Designer

Back to the Orchestration Designer we can now start adding some shapes and configuring some properties that will ultimately define our workflow process.

Add a Receive shape
- Active = True
- Message = In
Add a Construct Message shape
- Message Constructed = Out
Drag a Message Assignment shape into the Construct Message shape we just added above

Expression = (add the following to the Expression Editor)

XMLDOM = new System.Xml.XmlDocument();

//NOTE: The following xml can be easily generated by right-clicking on the EmailSchema.xsd file and selecting Generate Instance
//IMPORTANT: The following line of XML should all be on a single line, its on several lines here just for reading purposes only!!!
XMLDOM.LoadXml(@"<ns0:Application xmlns:ns0=""http://TestPOP3.AdmissionApp""><StudentID>StudentID_0</StudentID>
<AppDate>AppDate_0</AppDate><SubmittedBy>SubmittedBy_0</SubmittedBy>
<EssayResponse>EssayResponse_0</EssayResponse>
</ns0:Application>");

Out = XMLDOM;
Out.AppDate = In(POP3.Date);

//Here we chop of the first 11 characters of the "Subject" (which is "Application")and return the rest as the StudentID
strStudentID = In(POP3.Subject);
Out.StudentID = strStudentID.Substring(11);

//Here we do some string manipulations to remove "<" and ">" from the email adddress...
strEmailAddress = In(POP3.ReplyTo);
strEmailAddress = strEmailAddress.Remove(0,1);
strEmailAddress = strEmailAddress.Remove((System.Int32)strEmailAddress.Length -1);

Out.SubmittedBy = strEmailAddress;

Out.EssayResponse = In.Text;

Out(FILE.ReceivedFileName) = "StudentID" + In(POP3.Subject) + ".xml";
XMLDOM = null;
XMLDOM = Out;

//The following if for debugging purponses only...
System.Diagnostics.EventLog.WriteEntry("TestPOP3 Debug Complete Msg", System.String.Format("Out Message = {0}", XMLDOM.InnerXml));

Add a Send shape

Message = Out

Create Logical Send and Receive Ports:

Drag a Port shape from the Toolbox to the "Port Surface" on the left side of the Orchestration Designer

Wizard will start, click Next
Name = Port_In, click Next
Port Type = PortType_In, click Next
Port Binding screen:

Port direction of communication = I'll always be receiving messages on this port
Port Binding = Specify Later

Click Next, Finish

Drag another Port shape from the Toolbox to the "Port Surface" on the right side of the Orchestration Designer

Wizard will start, click Next
Name = Port_Out, click Next
Port Type = PortType_Out, click Next
Port Binding screen:

Port direction of communication = I'll always be sending messages on this port
Port Binding = Specify Later

Click Next, Finish

Assign Logical Port Operations with Send and Receive shapes:

From the Port_In port we just created, Drag the green "Port Connector Line" to the Receive shape
From the Port_Out port we just created, Drag the green "Port Connector Line" to the Send shape

* Your Orchestration should now look as follows:

Build and Deploy your Biztalk Application:

At this point we are ready to build, deploy, and test our application...

Right-click on TestPOP3 and select Build (resolve any errors)
Right-click on TestPOP3 and select Deploy (resolve any errors)

Create Physical Biztalk Port and Define Orchestration Bindings:

Once you have successfully been able to Build and Deploy your Biztalk application, you will find it available in the Biztalk Server Administration console under Applications.

Open up the Biztalk Server Administration console
Drill down into Applications --> TestPOP3
Create a new Receive Port called ReceivePOP3 (accept default settings)

Create a new Receive Location called ReceivePop3Loc

Type = POP3
Receive pipeline = GetEmailText
* Receive Location properties should appears as follows:

Configure the POP3 Adapter:

Mail Server = (The name of your mail server or local machine name if you install POP3/SMTP locally)
Authentication Scheme = Basic
Password = (The password of the ProcessApplication@nuckolls.edu account or account your using to be monitored by Biztalk)I
Polling Interval = 5
Polling Interval Unit = Seconds
* POP3 Adapter properties should appear as follows:

Create a new Send Port called SendEssayToWSS

Type = Windows Sharepoint Services
Send pipeline = XMLTransmit
* Send Port should appear as follows:
Configure the WSS Adapter:

Adapter Web Service Port = 8088 (This is the port that your WSS/MOSS site listens on)
Destination Folder URL = Lists/Essays%20for%20Student%20Admission%20Applications
Namespaces Aliases = ns0="http://TestPOP3.AdmissionApp"
SharePoint Site URL = http://btsmoss:8088/
Column 01 = Submitted By
Column 01 = %XPATH=//ns0:Application/SubmittedBy%
Column 02 = Essay Response
Column 02 = %XPATH=//ns0:Application/EssayResponse%
Column 03 = Application ID
Column 03 = %XPATH=//ns0:Application/StudentID%
Column 04 = Date Submitted
Column 04 = %XPATH=//ns0:Application/AppDate%
* The WSS Adapter properties should appear as follows:

Configure the Orchestration Bindings

In the Orchestrations folder under TestPOP3 in the Biztalk Server Administration console...

Double-click TestPOP3.EmailProcess and select Bindings in the properties window

Host = BiztalkServerApplication
Port_In/Receive Ports = ReceivePOP3
Port_Out/Send Ports = SendEssayToWSS
* The Orchestration Binding properties should appear as follows:

Run and Test your Application:

Finally we are ready to test our application...

Right-click on TestPOP3 in the Biztalk Server Administration console and select Start
Click Start

Open your email client (Outlook or Outlook Express):

Create an email

To = ProcessApplication@nuckolls.edu
Subject = Application9959 (You can use any # following "Application"... ie: Application1234567890, recall our Orchestation parses out the first 11 characters of the Subject line, to return only the Application ID that follows... or StudentID as it is referred to in the schema)
Body = (Type in some essay response text here to be submitted as part of the admissions process.)
* Your email should appear as follows:

Click Send

Confirm the message was sent to Sharepoint:

Open up your Sharepoint site to the custom List we created earlier (ie. http://btsmoss:8088/Lists/Essays%20for%20Student%20Admission%20Applications)
* You should see the message appear in the "Essays for Student Admission Applications" List as follows:

Hopefully this was helpful and will save some time! :-)

Biztalk Adapter for SQL - Issues/Resolutions

jeffnuck — Tue, 01 Apr 2008 03:05:00 GMT

Two things that you might encounter while you are attempting to create your schema for your SQL Adapter using the "SQL Transport Schema Generation Wizard":

One thing that seems to be a common problem, is the SQL Transport Schema Generation Wizard diappears or closes unexpectedly right after you have entered your Stored Procedure script or manually added your SQL script and click 'Next'. This KB article speaks to this known issue and how to resolve it, but it might also be because you have been developing/running other local applications that are holding a connection and/or reading/writing to this table otherwise locking it out. In this case you could try closing any other open apps (including VStudio) that may be attached to this database. You might also restart the SQL Service for good measure.
The other issue you may encounter when using the SQL Transport Schema Generation Wizard (perhaps after you've resolved the first issue) is an error message "Failed to execute SQL Statement. Please ensure that the supplied syntax is correct.". Again this error is referenced in the KB article mentioned above, but there are a couple additional ways to resolve this...

Be sure that if your database only accepts Windows authentication only, that you didn't select the 'Use a specific user name and password' on the Data Link Properties pane, instead select 'Use Windows NT Integrated security'.
Check the SQL itself and reference this article "Working with the Biztalk Adapter for SQL" for some tips. I resolved this issue be appends ELEMENTS at the end of my stored procedure as in the following example:

CREATE PROCEDURE SP_GetNewStudentInfo
AS
DECLARE @Process_Date DateTime
SET @Process_Date=GetDate()
Update StudentInfo Set ProcessedDate=@Process_Date Where ProcessedDate is NULL
SELECT StudentID, Lastname, Firstname, Term, GPA, DateofAdmission FROM StudentInfo WHERE ProcessedDate=@Process_Date FOR XML AUTO, ELEMENTS
GO

NOTE: If you add "ELEMENTS" to the Stored Proc as I did here, it will create the schema representing each column as a "Record" node... if you remove "ELEMENTS" it will create the schema representing each column as an "Attribute" node (which may be best if you plan on promoting any of those columns for routing purposes or to inspect their values within an Orchestration).

Hope this helps!

Issue when attempting to deploy a Biztalk 2006 assembly from Visual Studio 2005

jeffnuck — Tue, 19 Sep 2006 05:21:00 GMT

While attempting to deploy a very simple BizTalk Server 2006 (Beta 3 bits) application from my (Windows XP SP2) development machine running VSTS 2005 to my (Windows 2003 SP1) BizTalk 2006/SQL 2005 server, I received several MSDTC errors such as:

MSDTC may not be configured correctly --or-- Access Denied --or-- Failed to add resource (Luid="System.BizTalk:BizTalkAssembly", Type="MyBizTalkProject, version=1.0.0.0,....). Cannot open database BizTalkMgmtDb on server MyBizTalkServer. etc...

To resolve this issue...

First I made sure I could ping both server-->dev machine and dev machine-->server with their NetBios names to that was working. (If that doesn't work you may have to your ICMP setting are set to allow ping requests and also "Enable NetBios over TCP/IP" via your Network adapter's TCP/IP Properties/Advance (button)/WIN (tab).

Second I ensured that both the (biztalk and sql) server and (vsts) developer machine had all the appropriate MSDTC settings as follows:

To enable DTC access for Windows Server 2003 with SP1 or Windows XP with SP2, do the following:

From the Start menu, click Control Panel, double-click Administrative Tools, and then double-click Component Services.
Expand Component Services, and then expand Computers.
Right-click the computer you need to set, and then select Properties.
In the Properties window, select the MSDTC tab.
In the Transaction Configuration area, click Security Configuration.
In the Security Settings area, select the Network DTC Access check box.
In the Transaction Manager Communication area, select the Allow Inbound and the Allow Outbound check boxes.
Click OK to close each open dialog box.
Restart your computer to enable the remote DTC option.

To enable DTC access for Windows Server 2003 without SP1, do the following:

From the Start menu, double-click Control Panel, double-click Administrative Tools, and then double-click Component Services.
Expand Component Services, and then expand Computers.
Right-click the computer you need to set, and then select Properties.
In the Properties window, select the MSDTC tab.
In the Transaction Configuration area, click Security Configuration.
In the Security Settings area, select the Network DTC Access check box.
In the Security Settings area, select the Network Transactions and Network Clients check box.
Ensure Allow Remote Clients is selected, and ensure No Authentication Required is selected.
Click OK to close each open dialog box.
Restart your computer to enable the remote DTC option.

When I completed this task, my MSDTC errors were still not resolved and could not deploy the BizTalk assembly, so I attempted to test my DTC connections from both the server and developer machine using DTCPing.exe which did little but reconfirm that I was have Access Denied and DTC binding issues.

Finally After playing around with disabling my firewall and antivirus (which worked, but temporarily left my computer vulnerable), I completed the following steps to get everything working:

Configure Windows Firewall to include the MSDTC program and to include port 135 as an exception. To do this, follow these steps:

a.	Click Start, and then click Run.
b.	In the Run dialog box, type Firewall.cpl, and then click OK.
c.	In Control Panel, double-click Windows Firewall.
d.	In the Windows Firewall dialog box, click Add Program on the Exceptions tab.
e.	In the Add a Program dialog box, click Browse, and then locate the Msdtc.exe file. By default, the file is stored in the `Installation drive`:\Windows\System32 folder.
f.	In the Add a Program dialog box, click OK.
g.	In the Windows Firewall dialog box, click to select the msdtc option in the Programs and Services list.
h.	Click Add Port on the Exceptions tab.
i.	In the Add a Port dialog box, type 135 in the Port number text box, and then click to select the TCP option.
j.	In the Add a Port dialog box, type a name for the exception in the Name text box, and then click OK.
k.	In the Windows Firewall dialog box, select the name that you used for the exception in step j in the Programs and Services list, and then click OK.

This worked and I was able to successfully deploy my BizTalk application!!!