SSIS Team Blog : Katmai

Competitive Comparison of SQL Server 2008 Integration Services

mmasson — Wed, 20 Aug 2008 02:03:00 GMT

Did you know about this white paper comparing SSIS to IBM Information Server (DataStage), Informatica PowerCenter, Oracle Warehouse Builder, and Oracle Data Integrator? I had heard rumors of its existence, but I hadn’t actually seen the links for it until now.

I saw a lot of good points in there, but looking at the chart on page 2 (which I’ve included below), it seems to me that it may have a slight bias towards SSIS…

What do you think? Are white papers like this an effective way to get our message out there?

EDIT: The spell checker in Windows Live Writer suggests “overeater” for “PowerCenter” … I found that amusing.

SQL Server 2008 RTM is Live

mmasson — Wed, 06 Aug 2008 21:09:20 GMT

The RTM version of SQL Server 2008 is now available for MSDN and Technet subscribers.

Hurray!

SSWUG Business Intelligence Virtual Conference

mmasson — Thu, 31 Jul 2008 00:00:30 GMT

I’ve been invited to speak at the SSWUG BI Virtual Conference in September. Like John Welch mentioned in his blog, the current speaker lineup is very impressive. I’m honored (and a little intimidated) to be on the presenters list! I’ll be flying out to Tucson in early September to record three SSIS related sessions:

What’s new in SSIS in SQL Server 2008
Beyond Scripting – Developing reusable extensions for SSIS
Advanced lookup scenarios in Integration Services with SQL Server 2008

The conference will take place online September 24th – 26th. The session list hasn’t been posted yet, but judging by the list of speakers, I’m sure it will be interesting.

First two community samples for 2008 now on Codeplex

mmasson — Thu, 24 Jul 2008 01:34:43 GMT

The Integration Services Community Samples project is now active on Codeplex. These samples are being created by the SSIS product team to supplement the content in books online, and the Product Samples for 2008. They will (hopefully) provide useful functionality that you can then take an customize for your ETL solutions.

The first two samples are pipeline components. We have an XML Destination, which takes one or more input streams and outputs XML documents, and a RegEx Flat File Source, which allows you to parse flat files using regular expressions.

From the releases page:

XML Destination

This sample includes source and binary for a simple XML Destination pipeline component. Use this sample to learn more about how to:

Create custom data flow destination components for use with SSIS
Build component user interfaces
Support multiple inputs on a single component

Regular Expression Flat File Source

This sample includes source and binary for a regular expression based flat file parsing source. Use this sample to learn more about how to:

Create custom data flow sources components for use with SSIS
Support multiple outputs from a single component
Define output columns
Validate metadata

These components are pretty easy to use, but I’ll follow up with a couple of posts (with screen shots!!@) that show how to use them.

Reporting on execution logs using Reporting Services

mmasson — Mon, 23 Jun 2008 21:56:00 GMT

Microsoft published a set of canned reports for RS a while back. The pack includes some reports for Integration Services that give you execution results and statistics for your packages (if they are using SQL Server logging). The reports are very handy, and provide a good starting point if you want to create your own custom reports.

Here are some screen shots of runs I did recently (those of you familiar with Project REAL might recognize the package names).

Summary

Analysis

Log Details

I recently tried the reports out with SQL 2008, and was happy to see that everything still worked – with one additional step. Since the SSIS logging table name has been changed from sysdtslog90 (2005) to sysssislog (2008), you’ll need to either update the reports to use the new name, or create a View which maps to the new table.

CREATE VIEW [dbo].[sysdtslog90]
AS
    SELECT [id]
          ,[event]
          ,[computer]
          ,[operator]
          ,[source]
          ,[sourceid]
          ,[executionid]
          ,[starttime]
          ,[endtime]
          ,[datacode]
          ,[databytes]
          ,[message]
      FROM [dbo].[sysssislog]

Note, if you’re upgrading from 2005 to 2008, this view is created for you automatically in MSDB. You’ll still need to create it yourself if you’re logging to a separate database, however.

Use 32bit runtime option for SQL Agent

mmasson — Fri, 13 Jun 2008 00:59:34 GMT

To add to Doug's recent post about 64bit platform considerations, here's a screen shot (fresh from RC0) of the new "Use 32bit runtime" option on the SQL Agent job configuration screen. You'll find it on the "Execution options" tab.

SQL Server 2008 RC0 is now available

mmasson — Wed, 11 Jun 2008 02:33:40 GMT

Incase you missed it through the other channels, RC0 is now available for download.

Also see:

Improved error messages about missing OLEDB providers

mmasson — Thu, 22 May 2008 20:54:09 GMT

We've modified the OLEDB_NOPROVIDER error to return the name of the provider we failed to find. This will appear in the RC0 release of SQL 2008.

SSIS Error Code DTS_E_OLEDB_NOPROVIDER_ERROR. The requested OLE DB provider SQLNCLI.1 is not registered. Error code: 0x0000000

This should help in debugging issues related to connection string upgrades.

Package upgrade - connection strings

mmasson — Thu, 22 May 2008 02:53:00 GMT

We tried to make our package upgrade as seamless as possible, but one problem we've seen people running into is with their connection strings. As I've previously noted, the provider name for SQL Native Client was changed in 2008 from SQLNCLI to SQLNCLI10. This will could cause your connection managers to fail at runtime, as we wouldn't be able to locate the correct OLEDB provider.

Our package upgrade process will modify your connection strings automatically (this is a configurable option, that is on by default). However, there are a couple of scenarios that we don't handle.

Expressions

If you find your connections look right at design time, but fail at runtime, check whether your connection manager is using expressions to set its connection string. The package upgrade process will not modify values stored in variables or expressions.

Data Source (.ds) files

The package upgrade process doesn't modify connection strings contained within Data Source files. Although we modify the package itself, you'll be prompted to synchronize the connection string the first time you open the package in the designer. This will reset the Provider back to the old value.

A workaround for both of these scenarios is to do a find/replace on all of the files in the solution, changing "SQLNCLI.1" to "SQLNCLI10.1".

Integration Services Samples for SQL Server 2008

mmasson — Thu, 22 May 2008 01:48:02 GMT

Now that things are winding down for SQL Server 2008, we've had some time to focus on samples. I'm especially interested in publishing sample extensions (tasks, data flow components, etc) out on codeplex, but we're also working on SSIS package samples that make use of some new functionality in 2008.

Here are some of the custom extensions we have so far:

XML Destination
RegEx Source - parse a text file using regular expressions
Partition Management Task - based on an existing codeplex project
Database Population Task - populate a database/table with random data
Random Data Source

In the next couple of months we'll be making these available via Codeplex. I'll also try and provide some early binary preview versions if there is enough interest.

I thought I'd post and see if there was any specific features / functionality we have in our stock extensions that you'd like to see exposed in a sample. For example:

Persisting sensitive fields (passwords in connection managers)
Filtering on connection manager type in a properties grid
Handling multiple inputs in a data flow component
Custom UIs

What else would you like to see?

Package Upgrade Wizard

mmasson — Sat, 22 Mar 2008 07:50:11 GMT

SSIS packages created in 2005 have to go through an upgrade process to run in 2008. 2008 packages created prior to the February CTP also have to be upgraded. The upgrade kicks in automatically once the package is loaded by the runtime - whether in the designer, dtexec, or through the API. To make the upgrade permanent, the package has to be saved (so although dtexec will upgrade the package when it's being executed, it will do so every time the package is run). Note that once the package is upgraded, it is no longer compatible with 2005.

The February CTP introduced a new Package Upgrade Wizard which allows you to do batch upgrades. It can be launched in a number of ways:

In BIDS, by right clicking on the "SSIS Packages" folder of an SSIS project and selecting "Upgrade All Packages"
Running "ssisupgrade.exe" from the command line
In SSMS connected to the Integration Services service, by right clicking on a package folder and selecting "Upgrade Packages..."

The wizard will also launch automatically if you open an SSIS project in BIDS and one or more 2005 packages are detected in the project directory.

Running the wizard

The first thing to do is select the source location of the packages you want to upgrade. This will recursively pick up all packages from all sub directories. You can select a path from the file system, SSIS Service (Package Store), or SQL Server.

Things to note:

When the wizard is launched from BIDS, the location is automatically set to the project directory
The wizard can only connect to the 2008 SSIS Service (however, you can configure the 2008 service to manage packages stored on a 2005 server)

The next page of the wizard shows all packages contained under the source location you selected. Here you can select which packages you want to upgrade, and set passwords for packages with EncryptSensitiveWithPassword or EncryptAllWithPassword protection levels. Select multiple packages, type in the password in the lower box and click the "Apply to Selection" button to set a password on a bunch of packages in one shot.

Things to note:

Both 2005 and 2008 packages will be listed here. Running the upgrade wizard on a 2008 package will succeed without changing anything.
You can rename the package by changing the "Upgrade package name" value in the grid
You can set the password directly in the grid

Once the packages have been chosen, the next page asks you to pick the output location. This can be:

Same as the source location
File System
SSIS Service (Package Store)
SQL Server

If you choose "Save to source location", you'll be given the option to backup the packages on the next page of the wizard.

Things to note:

Launching the wizard from BIDS will select "Save to source location" by default
The wizard does not support changing the package protection level - which means you'll get an error if you're trying to save packages with "ServerStorage" to the file system
The package hierarchy / directory structure is maintained between the source and destination

The options that appear on the next wizard page will depend on what you choose as your destination.

The options:

Update connection strings to use new provider names

If this option is selected (true by default), connection managers which use MS providers that were renamed in 2008 will be automatically updated to use the new values. These include SQL Native Client (SQLNCLI -> SQLNCLI10), and MSOLAP.3 -> MSOLAP.4.

See my blog post about SQL Native Client 10.0 for more details.

Validate upgraded packages

This will force validation of the package after it has been upgraded. If validation fails, the upgraded package will not be saved.

Create new package ID

If this option is checked, the package ID GUID will be regenerated after the package is upgraded.

Continue upgrade process when a package upgrade fails

Packages will be upgraded one by one. If this option is set, the wizard will stop when it encounters an error.

Backup original packages

This option appears when you are using the same source and destination locations. It will create a backup folder in the source directory (SSISBackupFolder), and copy the original packages into it to prevent them from being overwritten.

Package name conflicts

This option tells the wizard what to do if a package with the same name already exists at the destination. The options are:

Overwrite existing package files
Add numeric suffixes to upgrade package names
Do not upgrade packages

The final page of the wizard displays the upgrade status. Each package will have it's own line, and progress indicator. If a package has warnings (common if connection strings are being updated), or errors, you can click the "Messages..." link to see more details. Click the "Report" button to see messages for all of the packages.

Reasons why upgrade might fail

Package has validation errors

If your package doesn't validate before the upgrade, your upgrade may fail if you have the "Validate upgraded packages" option set. See the details above.

Password protected fields

If the package has a protection level that encrypts with a user key (and the wizard is run as a different user), the upgrade will fail with the following error:

Error 0xc0016016: : Failed to decrypt protected XML node "PackagePassword" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available.

The best way to upgrade this package is to run the wizard with the account that created the package. If you don't mind losing the password information stored in the package, you can also upgrade it by opening it in the designer.

Packages with SQL Server configurations

The current CTP has a bug that causes the runtime to try to acquire connections for any SQL Server configurations in the package. If the connection fails, the package will fail to upgrade. This will hopefully be resolved for RTM.

Script upgrade failures

If your script task / component fails to upgrade for some reason (missing assemblies, early version of VSTA, compilation error), it will block the package upgrade. Manually upgrade the package by opening it in the designer, instead.

See my post about upgrading scripts from early Katmai builds.

Upgrading scripts from early 2008 CTPs

mmasson — Sat, 22 Mar 2008 07:44:40 GMT

The February CTP has been out almost a month now, but just incase you hadn't checked it out yet, I thought I'd post a heads up. The February CTP introduced Visual Studio 2008 (Orcas), replacing Visual Studio 2005 as our Business Intelligence Designer Studio (BIDS).[ side note - why is it I always misspell the word "Intelligence" ] Bringing in VS2008 also changed our scripting engine from VSTA 1.0 to VSTA 2.0, which broke compatibility with earlier Katmai CTPs.

From the February CTP Readme:

4.4.9 Package with Microsoft Visual Studio 2005 Tools for Applications (VSTA) Scripts

In the February CTP and earlier CTP releases of SQL Server 2008, the Script task and Script component used a version of Visual Studio for Applications (VSTA) that is no longer supported. Therefore, the scripts that the Script task and Script component contain are not supported in the following cases:

You used the February CTP or an earlier CTP release of SQL Server 2008 to open a SQL Server 2005 package, convert the existing scripts, and then save the converted package.
You used the February CTP or an earlier CTP release of SQL Server 2008 to open a SQL Server 2005 package and add a script to the package.
You used the February CTP or an earlier CTP release of SQL Server 2008 to create a new package and add a script.

If the script is not supported, SQL Server 2008 Integration Services will display an error message when you open the package. To run the package, create a new Script task or Script component and create a new VSTA script.

When you migrate a SQL Server 2005 package to SQL Server 2008, the migration process automatically converts the scripts in the package to the current version of VSTA. (The package migration feature for SQL Server 2005 packages will be available in a future release of SQL Server 2008.)

Some notes about SQL Native Client in 2008

mmasson — Sat, 22 Mar 2008 07:10:26 GMT

The SQL Native Client (SNAC) provider was renamed from SQLNCLI to SQLNCLI10 in 2008. The means that connection strings that used the Yukon provider (SNAC9) won't work if you only have the Katmai provider (SNAC10) installed.

This will affect most SSIS users, as we use SNAC as the default connection manager for a lot of things. To resolve this issue, you have two options:

Install SNAC9 side by side with SNAC10. The redist can be found on the 2005 installation CD.
Change the "Provider" portion of the connection string from "SQLNCLI.1" to "SQLNCLI10.1"

Starting in the February CTP (CTP6), SSIS can automatically update your connection strings for you.

Note, while SNAC10 is backwards compatible with SNAC9, there are some slight behavior differences.

New SSIS packages in Katmai will default to use SNAC10, but you can always switch the provider to SNAC9 if you have it installed on the machine.

What's New in SQL Server Native Client 10.0

Updating an Application to SQL Server 2008 Native Client from SQL Server 2005 Native Client

Data profiling task improvements

mmasson — Tue, 11 Mar 2008 19:37:36 GMT

There's been some great feedback coming in about the new data profiling task. Jamie Thompson has a set of blog posts that cover all of the different profile options, and yesterday I read an interesting post from John Welch which describes a clever way to profile an entire database.

I wanted to mention that we've been incorporating some of the feedback we've been getting, and list some of the improvements/changes that should be showing up in CTP Refresh.

Quick profile page now has "All Tables" option
Clarification that the task uses ADO.Net Connection managers
Button to create ADO.Net connection managers directly from the task
Progress events to show task completion percentage

Keep the feedback coming!

Custom Extensions in SQL Server 2008

mmasson — Sat, 02 Feb 2008 00:57:00 GMT

The introduction of new SSIS features in the SQL Server 2008 release and beyond made it necessary for the SSIS team to introduce some breaking changes that affect SSIS custom extensions built for SQL Server 2005. This post is intended to clarify what those breaking changes are and how to migrate your SQL Server 2005 SSIS custom extensions to SQL Server 2008.

Updating Your Extension

Update references for SSIS assemblies from 9.0.242.0 to 10.0.0.0
Rename IDTSxxxx90 objects to IDTSxxxx100
Follow any deprecated code warnings
(Optional) Reversion your extension (see below)

You can check out my previous post for more details (including a simple regex you can use in visual studio to do the interface name changes).

Deploying Your Extension

Deploying your extension is similar to 2005. The only difference is that you would copy your extension to a directory under %ProgramFiles%\Microsoft SQL Server\100\DTS, instead of 90\DTS.

Whether or not you choose to reversion (changing the extension's assembly version for managed code, or the ProgID/CLSID for native code) will affect the way your packages that consume the extension will be upgraded.

Keeping the same Version

If the version number of your extension doesn't change, there are no special steps needed for upgrade. Your 2005 packages should upgrade to 2008 format with no additional modifications.

Changing the Version

If you decide to reversion your extension (you'd do this if you want to be able to install Yukon and Katmai versions side by side), you'll need to provide an upgrade mapping file so the SSIS upgrade engine knows how to map your assembly.

Upgrade Mappings

The next Katmai CTP will add a new "UpgradeMappings" directory. SSIS will read the XML mapping files placed in this directory when upgrading a package.

Sample mapping file:

<?xml version="1.0" encoding="utf-8"?>
<Mappings xmlns="http://www.microsoft.com/SqlServer/Dts/UpgradeMapping.xsd">
  <!-- Connection Managers -->
  <ConnectionManagerMapping tag="MSOLAP - Analysis Services connections" 
                            oldValue="MSOLAP90"
                            newValue="MSOLAP100" />
  
  <!-- Extensions -->
  <ExtensionMapping tag="mycustom extension"
      oldAssemblyStrongName="MyCustomAssembly.MyCustomTask, MyCustomAssembly,
 Version=1.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91"
      newAssemblyStrongName="MyCustomAssembly.MyCustomTask, MyCustomAssembly,
 Version=2.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" />

  <!-- Providers -->
  <ProviderMapping tag="SQL Native Client - Version dependent" 
                   oldValue="sqlncli.1"
                   newValue="sqlncli10.1" />
  <ProviderMapping tag="SQL Native Client - Version independent"
                   oldValue="sqlncli"
                   newValue="sqlncli10" />
  <ProviderMapping tag="MSOLAP - Analysis Services connections"
                   oldValue="msolap.3"
                   newValue="msolap.4" />
</Mappings>

The interesting one for most users will be the <ExtensionMapping> element.

Attribute	Description
tag	Text describing your extension (used for logging)
oldAssemblyStrongName	The strong name of your extension's assembly in 2005
newAssemblyStrongName	The strong name of your extension's assembly in 2008

You have two options when mapping assemblies - you can use the fully qualified class name (like in the example - <assembly name>, <class>, Version=<version>, Culture=<culture>, PublicKeyToken=<key>), or you can provide the strong name of the assembly itself, without the class. This will map all classes from the old assembly to the new assembly.

For example:

<ExtensionMapping tag="mycustom extension"
    oldAssemblyStrongName="MyCustomAssembly.MyCustomTask, 
Version=1.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91"
    newAssemblyStrongName="MyCustomAssembly.MyCustomTask,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" />

Once you've deployed a mapping file to the UpgradeMappings directory, SSIS will be able to upgrade packages containing your custom extensions. Note, the mappings are only needed during package upgrade - once all of your packages have been upgraded, you can remove them.

----

Update: 2008-03-04 - Darren Green contacted me about a bug in the February CTP which causes mapping to fail if you're using the assembly name, and not the fully qualified class name. This will be resolved for the next CTP!

Update: 2008-06-13 - A topic covering this has been added to Books Online.