SSIS Team Blog : Connectivity

Something new for SSIS in SQL Server 2008 R2 November CTP

mmasson — Thu, 12 Nov 2009 19:29:00 GMT

Ok, so it turns my previous post about R2 wasn’t entirely correct.

SSIS does have one new feature which appears in the latest SQL Server 2008 R2 CTP. If you open up the ADO.NET Destination UI, you’ll notice a new “Use Bulk Insert when Possible” check box.

Previously, the ADO.NET Destination did all of its inserts row by row (we do some batching internally, which is why the component has a BatchSize property, but the underlying ADO.NET provider will always do single row inserts). With this new feature enabled, SSIS will use a bulk insert interface (like enabling “FastLoad” for OLEDB Destination). Unfortunately, there isn’t a generic Bulk Load interface for ADO.NET, so this functionality is currently only supported by SQL Server (through the SqlBulkCopy API). Hopefully we can extend support to other ADO.NET providers in the future.

The main reason for implementing this functionality was to improve our support for SQL Azure. As you might already know, ADO.NET is the primary way to communicate with SQL Azure, and adding support for SqlBulkCopy greatly increased the transfer speed. It also speeds up things when you’re working with regular SQL Server systems -- although OLEDB with FastLoad is still the preferred way of doing SQL data loads.

Be sure to try it out and provide feedback before the final release.

Writing to a MySQL database from SSIS

mmasson — Thu, 08 Jan 2009 04:32:54 GMT

A couple of users reported being unable to use the ADO.NET destination to insert data into their mysql databases. When I originally tried this out, it worked, but it seems like changes made since the early 2008 CTPs have made us incompatible with MySQL. We do have a bug logged to make this more flexible, but the good news is that in the meantime there is a workaround when using the ODBC connector.

For the ADO.NET Destination to work properly, the MySQL database needs to have the ANSI_QUOTES SQL_MODE option enabled. This option can be enabled globally, or for a particular session. To enable it for a single session:

Create an ADO.NET Connection Manager which uses the ODBC driver
Set the connection manager’s RetainSameConnection property to True
Add an Execute SQL Task before your data flow to set the SQL_MODE – Ex. set sql_mode='STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ANSI_QUOTES'
Make sure that your Execute SQL Task and your ADO.NET Destination are using the same connection manager.

Setting the RetainSameConnection property to True will ensure that your Execute SQL Task and ADO.NET Destination are in the same session.

Note, I recommend using the ODBC Driver when writing to the MySQL database, because the MySQL .NET Connector has an additional blocking issue. If you try it out, you’ll get an error which looks something like this:

Error: 2009-01-05 12:03:47.79
   Code: 0xC020844B
   Source: Data Flow Task 1 Destination - Query [28]
   Description: An exception has occurred during data insertion, the message returned from the provider is: You have an
error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use nea
r '"name", "date", "type", "remark") VALUES (p1, p2, p3, p4), (p1,p2,p3,p4), (p1,p2' at line 1
End Error

Note that the “VALUES” portion has parameter names, and not the actual values. This appears to be an issue with the value the MySQL provider returns for its ParameterMarkerFormat. I did find a bug that was opened against them, but it looks like they decided not to fix it. I’ve heard that the DevArt dotConnect drivers do not have this problem, but I haven’t been able to try them out myself.

New connectivity white papers

mmasson — Wed, 03 Dec 2008 22:07:08 GMT

Two new SSIS connectivity white papers have been published on MSDN.

Connectivity Options for Microsoft SQL Server 2008 Integration Services by Martin Ellis

This paper talks about the various connectivity options that SSIS supports, as well as the new connectors for Oracle, Teradata and SAP BI that were recently released. I was also happy to see that it contains sections on how to connect to SAP R/3, Siebel eBusiness Applications, and DB2.

Using SQL Server 2008 Integration Services with SAP BI 7.0 by Chunhui Zhu

This white paper is a walkthrough on how to use the new SAP BI connectors. It covers the two core scenarios of extracting data from SAP BI into SQL Server, and how to load non-SAP data into SAP BI.

Attunity connectors for Oracle and Teradata are now available

mmasson — Fri, 03 Oct 2008 01:05:29 GMT

The high speed connectors for Oracle and Teradata are now available for download.

	Supported versions	Platforms
Oracle	9.2.0.4 or higher	x86, x64, ia64
Teradata	2R6.0, 2R6.1, 2R6.2 and 12.0	x86, x64

Attunity has done a great job putting these together, and from what I’ve seen, the performance is really impressive. Be sure to check them out!

Accessing OLEDB Connection Managers in a Script

mmasson — Sat, 23 Aug 2008 02:05:29 GMT

Accessing ADO.Net Connection Managers from an SSIS script task / script component is pretty easy – you just need to cast the object returned from AcquireConnection() to the appropriate class (i.e. SqlConnection if you’re using SQL Native Client).

SqlConnection conn = (SqlConnection)Dts.Connections["adonet"].AcquireConnection(null);

If you can’t use ADO.Net for some reason, and are using OLEDB connection managers, it’s a little trickier. Since the AcquireConnection() method of the OLEDB connection manager returns a native COM object, I didn’t think there was a way to make this work, but today someone showed me how to do it!

By casting the Connection Manager’s InnerObject to the IDTSConnectionManagerDatabaseParameters100 interface (IDTSxxx90 in 2005), you can call the GetConnectionForSchema() method to return an OleDbConnection object.

2008 (C#):

ConnectionManager cm = Dts.Connections["oledb"];
IDTSConnectionManagerDatabaseParameters100 cmParams = cm.InnerObject as IDTSConnectionManagerDatabaseParameters100;
OleDbConnection conn = cmParams.GetConnectionForSchema() as OleDbConnection;

2005 (VB):

Dim cm As ConnectionManager
Dim cmParam As Wrapper.IDTSConnectionManagerDatabaseParameters90
Dim conn As OleDb.OleDbConnection

cm = Dts.Connections("oledb")
cmParam = CType(cm.InnerObject, Wrapper.IDTSConnectionManagerDatabaseParameters90)
conn = CType(cmParam.GetConnectionForSchema(), OleDb.OleDbConnection)

Note, you’ll need to add a reference to the Microsoft.SqlServer.DTSRuntimeWrap assembly to get the IDTSConnectionManagerDatabaseParameters100 interface. If you’re doing this in a script task, you’ll need to prefix the Microsoft.SqlServer.Dts.Runtime.Wrapper namespace (or use fully qualified names) so that it doesn’t conflict with the namespace for the VSTA proxy classes.

Keep in mind that there are a couple of limitations with this approach:

You won’t be able to enlist in the current transaction
This connection doesn’t honor the “retain same connection” setting

ADO.Net is still the recommended connection manager type for scripts, but I found this to be a nice work around.

SSIS and SAP

mmasson — Mon, 30 Jun 2008 04:40:08 GMT

Douglas already beat me to it, but I figured it was important enough to repeat incase you missed it the first time around.

I still don't fully understand why SAP is so popular. It's probably because I don't know enough about it, but it seems the more I get to know, the less appealing it is. Maybe there is a sudden "a ha!" moment when your SAP knowledge reaches a critical mass. Regardless, it was the biggest topic of conversation when I was helping out at TechReady last February, which is why I'm glad Microsoft has created an SAP Technical Guidance site.

The site has a lot of good content on making the most of your SAP ERP installation with SSIS and Office, as well as other Microsoft technologies. There are PDF white papers, power point slide decks, and also some video presentations.

I'm sure you can expect some content about using the new SAP BI connectors appearing when they are released later this year.

It should be noted that although the SAP R/3 connector comes through the "BizTalk Adapter Pack", you don't actually need a BizTalk installation to make use of them.

Improved error messages about missing OLEDB providers

mmasson — Thu, 22 May 2008 20:54:09 GMT

We've modified the OLEDB_NOPROVIDER error to return the name of the provider we failed to find. This will appear in the RC0 release of SQL 2008.

SSIS Error Code DTS_E_OLEDB_NOPROVIDER_ERROR. The requested OLE DB provider SQLNCLI.1 is not registered. Error code: 0x0000000

This should help in debugging issues related to connection string upgrades.

Package upgrade - connection strings

mmasson — Thu, 22 May 2008 02:53:00 GMT

We tried to make our package upgrade as seamless as possible, but one problem we've seen people running into is with their connection strings. As I've previously noted, the provider name for SQL Native Client was changed in 2008 from SQLNCLI to SQLNCLI10. This will could cause your connection managers to fail at runtime, as we wouldn't be able to locate the correct OLEDB provider.

Our package upgrade process will modify your connection strings automatically (this is a configurable option, that is on by default). However, there are a couple of scenarios that we don't handle.

Expressions

If you find your connections look right at design time, but fail at runtime, check whether your connection manager is using expressions to set its connection string. The package upgrade process will not modify values stored in variables or expressions.

Data Source (.ds) files

The package upgrade process doesn't modify connection strings contained within Data Source files. Although we modify the package itself, you'll be prompted to synchronize the connection string the first time you open the package in the designer. This will reset the Provider back to the old value.

A workaround for both of these scenarios is to do a find/replace on all of the files in the solution, changing "SQLNCLI.1" to "SQLNCLI10.1".

Some notes about SQL Native Client in 2008

mmasson — Sat, 22 Mar 2008 07:10:26 GMT

The SQL Native Client (SNAC) provider was renamed from SQLNCLI to SQLNCLI10 in 2008. The means that connection strings that used the Yukon provider (SNAC9) won't work if you only have the Katmai provider (SNAC10) installed.

This will affect most SSIS users, as we use SNAC as the default connection manager for a lot of things. To resolve this issue, you have two options:

Install SNAC9 side by side with SNAC10. The redist can be found on the 2005 installation CD.
Change the "Provider" portion of the connection string from "SQLNCLI.1" to "SQLNCLI10.1"

Starting in the February CTP (CTP6), SSIS can automatically update your connection strings for you.

Note, while SNAC10 is backwards compatible with SNAC9, there are some slight behavior differences.

New SSIS packages in Katmai will default to use SNAC10, but you can always switch the provider to SNAC9 if you have it installed on the machine.

What's New in SQL Server Native Client 10.0

Updating an Application to SQL Server 2008 Native Client from SQL Server 2005 Native Client

New connectivity options in 2008

mmasson — Tue, 11 Mar 2008 02:52:00 GMT

Update: 2008-08-13 - The connector pack will be available two to three months from the GA of SQL 2008

Update: 2008-10-02 - The Oracle and Teradata connectors are now available

Update: 2008-11-26 - The SAP BI connections are available

Darvey Lavender, group program manager for SSIS, has updated the connectivity wiki with some information about our new connectivity options in SQL Server 2008. The three new connectors (for Oracle, Teradata and SAP BW) will be available as a downloadable feature pack for Enterprise Edition customers two to three months from the general availablity of SQL 2008.

Here is some of the interesting info from the FAQs:

Attunity Oracle Connector

What Features Does The Connector Support?

The Attunity Oracle Connector supports the following functionality:

Bulk Load Destination using OCI Direct Path
Incremental Load Destination using OCI Array Binding
Bulk Extract Source using OCI Array Binding
Oracle Connection Manager

What Oracle Versions Are Supported?

The connector will support Oracle V9 or later.

Attunity Teradata Connector

What Features Does The Connector Support?

The Attunity Teradata Connector supports the following functionality:

Bulk Load Destination using TPT FastLoad
Incremental Load Destination using TPT TPUMP
Bulk Extract Source using TPT
Teradata Connection Manager

What Teradata Versions Are Supported?

The connector will support Teradata versions ~~V2R4.11 and V2R5 or later~~ V2R6.0, V2R6.1, V2R6.2 and 12.00.

SAP BI Connector

What Features Does The Connector Support?

The SAP BI Connector supports both source and destination connections for SAP BI.

What SAP BW Versions Are Supported?

The connector will support ~~both 3.5 (with InfoSpokes) and~~ BI 7.0 (with OHS Destinations).

Will The Connector Be Certified by SAP?

Yes, the current plans will be to have both the SAP BI source and destination features certified by SAP BI. The timing of the certification may not coincide with the initial release.

Update: 2008-04-17 - The SAP BI connector will support SAP BW 7.0, and not 3.5.

Update: 2008-07-09 - Updated Teradata version information.

Connecting to MySQL from SSIS

mmasson — Tue, 04 Mar 2008 09:23:00 GMT

Update: See this follow-up post on writing data to MySQL.

I've recently seen a bunch of questions about connecting to MySQL from SSIS, so I thought I'd give it a try. Our connectivity wiki currently doesn't have anything about MySQL on it, but I'll try and get it update with this information.

My overall findings was that while there were quirks, both the ODBC and ADO.Net drivers that I tried worked fine. Both drivers work with the ADO.Net Source (DataReader Source in 2005), and ADO.Net destination (Katmai only). For ease of use and install, I'd recommend using the ADO.Net driver.

The screen shots in this post were taken with an early February CTP Katmai build.

ODBC - Connector/ODBC 5.1, Connector/ODBC 3.51

Our connectivity white paper briefly mentions using MySQL's ODBC drivers, so they were the first thing I tried. I didn't so extensive testing, but it looked like both the 5.1 (beta) and 3.51 (release) drivers worked the same when connecting to my MySQL 5.0.45 server.

To use an ODBC connection in an SSIS data flow, create a new ADO.NET Connection and select the "Odbc Data Provider".

The ODBC drivers didn't show on the list of choices in the windows "ODBC Data Source Administrator" dialog (I'm running Vista x64... not sure if I have to do something special to make them show), so I entered the connection string directly instead of using a DSN.

DRIVER={MySQL ODBC 5.1 Driver};SERVER=<host>;DATABASE=mydb;UID=root
DRIVER={MySQL ODBC 3.51 Driver};SERVER=<host>;DATABASE=mydb;UID=root

Once the connection is created, you can pull data from the database using an ADO.Net Source in the data flow (DataReader Source in 2005). Trying to retrieve the tables using the drop down list resulted in an error:

Switching to use a SQL query instead, and that worked just fine. I was able to pull back both the correct metadata, with one small problem - the varchar(50) columns came back with a length of 51. This resulted in some warnings, but the package ran correctly.

I should note that the first time I ran the package, I got the classic 64bit problem -

[ADO NET Source [1]] Error: System.Data.Odbc.OdbcException: ERROR [IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified
   at Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSConnectionManager100.AcquireConnection(Object pTransaction)
   at Microsoft.SqlServer.Dts.Pipeline.DataReaderSourceAdapter.AcquireConnections(Object transaction)
   at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostAcquireConnections(IDTSManagedComponentWrapper100 wrapper, Object transaction)

I had only installed the 32bit drivers on my machine (it doesn't look like they will let you install both 32bit and 64bit MySQL drivers at the same time). Switching the Run64BitRuntime project setting to False fixed the issue.

ADO.NET - Connector/Net 5.1

The provider was very easy to install. Setting up the connection manager was fairly straightforward, although I had some weirdness when setting the "Persist Security Info" value to true. For some reason it didn't save my login information the first time I hit OK - I had to open it and save it again.

Like we saw with the ODBC drivers, the tables and views didn't show up in the drop down list (I didn't get a chance to debug too deep into it, so I'm not sure if that's a problem on the SSIS side, or something about the metadata being returned by the provider).

I switched it to SQL query mode ("SELECT * from Customers"), and that brought back the metadata correctly.

The ADO.Net Destination worked fine, although I had to type in the table name as the list wasn't auto-populated, and I didn't run into any problems with the simple data set I was using. The transfer speed seemed reasonable as well. Eventually I'd like to benchmark various ADO.Net providers, but I'll leave that as a topic for another post.

----

NOTE: I did this investigation about a month ago, but didn't have time to complete the write up. It looks like Connector/Net 5.2 has been released since then. It has some interesting features (including VS 2008 integration), so I'll give it a try and update the post if anything has changed.

SQL Native Client 10 - provider name change

mmasson — Thu, 07 Jun 2007 19:40:27 GMT

The provider name for SQL Native Client (SNAC) connections has changed in Katmai. Since this is the default provider for OLEDB connections, you'll probably be affected by this change if you're running SQL Server 2005 packages in Katmai.

Version	Provider name
SQL Server 2005 (Yukon)	SQLNCLI.1
SQL Server 2008 (Katmai)	SQLNCLI10.1

When the full upgrade solution is in place, we'll prompt you on whether or not to upgrade your connections, but for this CTP, you have two options. You can install the SNAC9 redist that comes with SQL Server 2005 (it should work fine side by side with the Katmai SNAC10), or you'll need to manually edit your connection strings when opening Yukon packages in Katmai.

To edit the connections yourself:

Open the package in the designer
Select the connection in the Connection Managers window
Right click, select Properties
Edit the ConnectionString text field, changing the "Provider=SQLNCLI.1" portion to "Provider=SQLNCLI10.1"

You'll also encounter a similar issue for Analysis Services connections.

Version	Provider name
SQL Server 2005 (Yukon)	MSOLAP.3
SQL Server 2008 (Katmai)	MSOLAP.4

SNAC9 and SNAC10 should be fully backwards compatible, although there might be small changes in behavior due to bug fixes. For example, writing a DT_DBTIMESTAMP value (that contains datetime value with milliseconds) to a smalldatetime column using an OLE DB Destination used to work with SNAC9, but results in an error with SNAC10 ("The value could not be converted because of a potential loss of data"). These differences will be compiled into a KB article for the release.

SSIS connectivity white paper

mmasson — Wed, 31 Jan 2007 21:20:00 GMT

A connectivity white paper writen by Bob Beauchemin has been added to the connectivity wiki site. Definitely worth checking out.