SSIS Team Blog : SSIS

Something new for SSIS in SQL Server 2008 R2 November CTP

mmasson — Thu, 12 Nov 2009 19:29:00 GMT

Ok, so it turns my previous post about R2 wasn’t entirely correct.

SSIS does have one new feature which appears in the latest SQL Server 2008 R2 CTP. If you open up the ADO.NET Destination UI, you’ll notice a new “Use Bulk Insert when Possible” check box.

Previously, the ADO.NET Destination did all of its inserts row by row (we do some batching internally, which is why the component has a BatchSize property, but the underlying ADO.NET provider will always do single row inserts). With this new feature enabled, SSIS will use a bulk insert interface (like enabling “FastLoad” for OLEDB Destination). Unfortunately, there isn’t a generic Bulk Load interface for ADO.NET, so this functionality is currently only supported by SQL Server (through the SqlBulkCopy API). Hopefully we can extend support to other ADO.NET providers in the future.

The main reason for implementing this functionality was to improve our support for SQL Azure. As you might already know, ADO.NET is the primary way to communicate with SQL Azure, and adding support for SqlBulkCopy greatly increased the transfer speed. It also speeds up things when you’re working with regular SQL Server systems -- although OLEDB with FastLoad is still the preferred way of doing SQL data loads.

Be sure to try it out and provide feedback before the final release.

SSIS Team Blog – Now in Chinese

mmasson — Tue, 10 Mar 2009 19:26:00 GMT

Xiaochen Wu, a member of our Shanghai team, has started a Chinese SSIS blog. He’s started it off by translating some of the posts I’ve done, and I’m sure he’ll find plenty of content in the future.

I don’t know what the following text means, but I assume it’s the title of the blog :)

SSIS 技术博客

分享SSIS使用心得和技巧

Matthew Roche on RunAs Radio

mmasson — Wed, 28 May 2008 21:08:53 GMT

Matthew recently did an interview for RunAs radio where he gives a really good overview of SSIS, DTS, and ETL in general. It's great if you're just getting started with SSIS, or need to sell someone else on the idea of using it!

SSIS sets an ETL world record

mmasson — Thu, 28 Feb 2008 00:58:55 GMT

They mentioned the TPC benchmark results at the SQL Server 2008 launch today. The final paragraph mentions a world record for ETL load performance. The fine folks who maintain the SQLPerf blog have posted details about the result, explaining how they were able to load over 1 TB of TPC-H data in under 30 minutes. It makes for a very interesting read!

What's new in SQL Server 2008 for SSIS - Part two

mmasson — Wed, 23 Jan 2008 04:03:39 GMT

Change Data Capture (CDC)

While not a direct SSIS feature, the new Change Data Capture functionality in 2008 can be taken advantage of in SSIS to do incremental loads. CDC captures changes to SQL Server tables (inserts, updates, deletes), and makes them available in an "easily-consumed, relational format".

CDC was added to Katmai in one of the earlier CTPs, and I've heard that the most recent CTP added some new stored procedures to make using it in SSIS even easier. I've heard that the CDC team is working on posting some samples on Codeplex, and I'll post more details once I get a chance to try them out.

BOL - Improving Incremental Loads with Change Data Capture

Merge

Another engine feature you can take advantage of in SSIS is the merge statement, which allows you to perform insert, update or delete operations on a target table based on the results of a join with a source table.

John Welch had a couple of good posts on Merge, and how to use it from SSIS awhile back that are definitely work checking out.

I've also seen some samples in the works that I'll post about once they are up on Codeplex.

Data Profiling

The new Data Profiling Task let's you analyze SQL tables to determine (and maintain) data quality.

The way it works is you configure the task to do a set of analysis on columns/tables, such as:

Candidate keys
Column length distribution
Null ratio
Pattern detection
Value distributions and stats
Functional dependencies
Value inclusion

The results are dumped out to an XML file, which you can then load using a special Viewer application.

Combined with a script task, you can use the profiling task to do a quality check on your data before passing it off to your regular data flow ETL.

BOL - Profiling Data with the Data Profiling Task and Viewer

Visual Studio 2008 - Orcas

The next Katmai CTP will introduce Visual Studio 2008 as our Business Intelligence Design Studio. This doesn't change much functionally, but does allow for side by side installations of BIDS - meaning you can edit Yukon packages (with Visual Studio 2005) and Katmai packages (with Visual Studio 2008) on the same machine.

Package Upgrade Wizard

The next CTP will also introduce the bulk of our upgrade work, including our Package Upgrade Wizard. While a 2005 package is automatically upgraded when opened in BIDS in 2008, this wizard lets you upgrade a set of packages all at once. It can read packages from a file share, SQL Server database, or Katmai SSIS service, upgrade them, and dump them to a location of your choosing.

The wizard can be launched from the command line (SSISUpgrade.exe), from BIDS, and from the menu in Management Studio. It will also launch automatically in BIDS when you open up a project which contains Yukon packages.

Improved Memory Dumps

A lot of work has gone into improving our supportability, and part of this was enhancing our memory dump capability for debugging crashes and hung packages.

When a dump is created (the directory is configurable, but defaults to %ProgramFiles%\Microsoft SQL Server\100\Shared\ErrorDumps), we also generate a text file which contains useful debugging information that SSIS developers and support can make use of. We've been using the dumps internally to debug tough pipeline deadlocks and crashes, and so far it's been really helpful.

These dumps will be created automatically during a crash, but can also be triggered by the user.

Dumping with DTExec

Two dumping options have been added to DTExec - /Dump, which takes a semi-colon separated list of error codes (HResults) to dump on, and /DumpOnError, which will trigger a memory dump anytime an error is encountered.

Dumping with DTUtil

The /Dump option added to DTUtil allows you to create a memory dump for a running package without disrupting its execution (at least not for very long). This is very useful if you suspect your package has hung.

[ EDIT: adding a link to the BOL content ]

----

If I've missed something, I'll follow up with a Part 3, but I think I've covered all of the main features. On top of this, there have been many bug fixes and minor improvements. I'll continue covering some of the more important ones later on.

What's new in SQL Server 2008 for SSIS - Part one

mmasson — Fri, 11 Jan 2008 04:04:30 GMT

Improved Scripting

SQL Server 2008 (Katmai) introduces VSTA as our new scripting engine, which replaces the old VSA engine used in SQL Server 2005 (Yukon). There are a number of advantages to the new engine, but I think the feature users are most excited about is that it allows you to use C# as a scripting language (as well as VB.NET). It also makes it easier to reference all .Net assemblies, which enables a number of common scenarios (such as easier web service integration).

Data Flow Task Improvements

Michael Entin provided a great write-up for the enhancements to our data flow engine. It essentially boils down to smarter thread allocation and processing of our execution trees. Long chains of synchronous transforms are optimized, and most packages will see a performance increase on multi-processor machines.

One CPU usage graph I really like is the one shown at PASS this year:

This shows Yukon package execution on a quad CPU machine. You can see CPU #4 is used steadily, while two others spike occasionally. It also looks like usage goes down on #2 when it goes up on #3, and vice versa.

Running the same package using the Katmai engine looks like this:

This graph shows much better CPU usage distribution. In this case, the package received an 80% performance increase, but your mileage may vary.

Enhanced ADO.Net Support

Katmai now has an ADO.Net source (replacing the DataReader source) and destination adapters. They function a lot like the OLEDB ones, supporting a custom UI, drop down list of tables/views, and query builder. Its properties are also expression-able. This means we now support ODBC as a destination, as well.

Import/Export Wizard

The SQL Server Import/Export Wizard (which generates SSIS packages) has been improved as well.

ADO.Net support (which gives you ODBC)
Data type conversion - mappings are read from customizable (xml) files, and data convert transforms will be inserted into the data flow if required
Scales better - large numbers of tables will be split across multiple data flows (up to five sources per data flow)

Here is an example of the data conversion page:

Cached Lookup

As I mentioned in an earlier post, Jamie Thompson provided a very good write up of the new lookup component. The main feature is that you can now create a lookup cache in a separate data flow using the new Lookup Cache Transform and Cache Connection Manager, and persist it to disk. This cache can be created from any data source supported by SSIS, which removes the need to stage your lookup table in a database.

This persisted cache is especially useful in scenarios where reference data is updated periodically - say, once a week. You can run a package to generate your cache when the update comes in, and re-use it in the rest of your packages until it needs to be updated again.

Another scenario is managing large lookup tables, where a small number of keys are used a large percentage of the time. For example, 5% of your customer IDs show up in 80% of your rows. Instead of caching the entire lookup table, you can use a cached lookup for the top 5%, and cascade another lookup in partial / no-cache mode for the infrequent rows.

As you can see from the screen shot, Lookup also has a new "No Match" output - rows with no matches aren't directly sent to the error output anymore. We also maintain a "Miss Cache" internally, so missing keys aren't constantly re-queried.

More to follow...

SSIS Survey

mmasson — Wed, 21 Nov 2007 01:43:38 GMT

The SQL User Experience (SQLUX) and Integration Services team are currently conducting a survey on ETL practices. We hope to use this feedback in our planning for the Katmai end game, and follow up releases.

If you fill out the survey before December 15th, 2007, you'll be entered in a draw to win one of five Zunes.

More info here: https://mscuillume.smdisp.net/Collector/Survey.ashx?Name=SqlETLSurvey2

Running packages from custom applications

mmasson — Sat, 15 Sep 2007 03:33:51 GMT

There are a lot of different ways to run SSIS packages, and many situations call for a custom package launcher application. If you're coding your own launcher, one thing to keep in mind is that certain SSIS functionality (such as the Script task / component) requires a multithreaded apartment (MTA), and can fail or behave incorrectly when run in a single threaded apartment (STA). By default, VB.NET console applications run in STA.

I worked on a customer issue where their custom package launcher started failing on parent/child packages after they changed their loading mechanism. The parent package was failing with an obscure E_NOTIMPLEMENTED error code when trying to retrieve a child package from SQL Server. We also noticed the issue only occurred when both packages were loaded from SQL using a username and password - loading either one from file, or using windows integrated authentication made the problem disappear.

The issue turned out to be related to the threading apartment. When loading packages from SQL using a username/password, the Execute Package Task attempts to reuse the parent package's connection managers. When this is done, the runtime also tries to reuse any open recordset objects as an optimization. The underlying OLEDB code we use to retrieve the package requires the thread apartment to be MTA, and was returning the obscure error code when it failed.

Once we modified the app to run in MTA, everything starting working again! Doing this for a VB.NET app is really simple - just prefix your Main() function with MTAThread.

   1: <MTAThread()> _

   2: Sub Main()

   3:     Console.WriteLine("Running...")

   4:     ....

   5: End Sub

First public Katmai CTP

mmasson — Tue, 05 Jun 2007 18:06:00 GMT

The first public community technology preview for SQL Server 2008 (code named Katmai) was announced at TechEd yesterday, and is now available for download. You can register here to download the developer editions, and post questions / comments / concerns in the special Katmai MSDN forums they have setup.

You'll see a number of bug fixes and minor improvements to SSIS in this first CTP... mostly stuff that we weren't able to fit into the 2005 SP2 release. For major improvements, the old VSA script engine was replaced with VSTA, which allows you to use C# in your script tasks and script components.

There was also a pretty major rewrite to the way we schedule threads in the pipeline which should give a pretty big performance boost to certain types of data flows. I'm hoping we can post more details about this soon (specifically what types of packages are affected, and how you can take advantage of the new threading model), either here or on Michael Entin's blog.

I'll continue posting new information about Katmai throughout the CTP period.

Real world scenarios

mmasson — Fri, 01 Jun 2007 20:14:00 GMT

May has been a busy month!

The first Microsoft Business Intelligence conference was earlier in the month, and it seemed to be a big success. I was amazed at how many people were there, and I met a lot of interesting people when I was manning the demo booth and Ask The Experts table we had setup in one of the pavilions. I got to see some interesting talks, and previews of the new BI offerings MS has in the works (PerformancePoint Server 2007 looks very slick).

The SSIS team has been really busy working on Katmai, and we're now approaching our first public CTP. Typically before each release we hold an internal "Bug Bash", where the whole team spends the day running through user scenarios trying to find any last minute issues. This time we decided to do something a bit more involved...

The group was split up into various teams, and each team was given a real world scenario that we had to solve using SSIS. The idea was to give the product a workout, as well as give us a better point of view on some of the customer pain points. We were given a limited amount of time, so most solutions probably won't have full error handling, logging or other fail safe mechanisms, such as transactions and checkpoints. We hope to publish the scenarios and packages to Codeplex however, and continue adding to them until they can be used as "Best Practice" type examples for common problems.

Over the next little while I'll post some of the projects and go over the thought process behind the solutions.

SQL Server SP2 – What’s new for SSIS

mmasson — Wed, 28 Feb 2007 06:47:55 GMT

You can see all of this information on MSDN in the SP2 Readme, but I thought I'd highlight a couple of things.

Diagnostic Logging

A lot of trace logging was added in SP2 to help with debugging connectivity issues. These messages are logged using the "Diagnostic" event. A side effect of this is that if you had Diagnostic level logging enabled before, you are now getting a lot more messages. In SP1 there was one Diagnostic message per run – a package level event about the max concurrent threads. In SP2, you typically get two messages per database API call (pre and post call messages). If you're seeing any performance decrease in package execution after upgrading to SP2, you should check your logging settings to make sure that you don't have Diagnostics enabled.

It is recommended you only enable Diagnostic logging for debugging purposes – specifically when you're hitting connectivity issues.

BypassPrepare

This property in the Execute SQL Task is now set to True by default, which should prevent a lot of confusion when using parameterized queries. Preparing the statement can give a minor performance increase for certain statements, but should only be set to false when you're executing a statement without any parameters.

ParameterSize

A new column was added to the Parameter Mapping tab of the Execute SQL Task for the new ParameterSize field.

This was a fairly big change to address a number of issues with variable length output parameters (the main issue being they didn't work). I had previously posted about using output parameters as an alternative to a result set, and provider syntax issues aside, they work fairly well with certain types. However, differences in the way each provider handles output parameters led to a number of problems with the original implementation when dealing with variable length types (strings, mostly).

With this change you are able to specify a length, or use the default value (-1) to let SSIS guess at the correct size. When the default is used, we retrieve the size information directly from the provider if supported; otherwise we use a reasonable default. In most cases the default value should be good enough, but there are some situations where you might want to use a specific value – 1) our "reasonable default" isn't reasonable enough, 2) you want to truncate the returned value to a certain size/length. It should be noted that some providers (ADO for example) don't support truncation, and will fail if the parameter size isn't large enough.

SSIS connectivity white paper

mmasson — Wed, 31 Jan 2007 21:20:00 GMT

A connectivity white paper writen by Bob Beauchemin has been added to the connectivity wiki site. Definitely worth checking out.

Stored procedures with output parameters

mmasson — Thu, 09 Nov 2006 03:07:00 GMT

When executing Stored Procedures with the Execute SQL Task, SSIS passes the statement to the underlying provider without parsing it. While efficient, the different ways of handling the input between the various providers can be a little frustrating. I was working on a couple of issues found in SP1, and realized that when dealing with stored procedures with output parameters, each provider wanted the SQL in a different format.

I'll stick with a simple stored procedure example that returns a single value.

CREATE PROC intoutput
@value INT OUTPUT
AS
SELECT @value = 10

I want my Execute SQL Task to run the stored procedure, and store the result in a package variable (varInt). For different providers, you'll need to change two main fields – the SQL Statement, and the Parameter Name value on the Parameter Mapping page.

OLE DB

SQL Statement: exec intoutput ? output
Parameter Name: 0

You should also set BypassPrepare to True, as the SQL can't be properly parsed if it contains a parameter marker.

1 – OLE DB General

2 – OLE DB Parameter Mapping

ODBC

SQL Statement: {call intoutput (?)}
Parameter Name: 1

The SQL statement for ODBC is very different than the others. You need to use "call" instead of "exec", and wrap the whole thing with curly braces. Also, the parameter numbering starts at 1 instead of 0.

3 - ODBC General

4 - ODBC Parameter Mapping

ADO

SQL Statement: intoutput
Parameter Name:<parameter name>

For ADO, you can use the stored procedure name by itself, and set the IsQueryStoredProcedure attribute to True. For parameter name, you use the name you gave the parameter in the stored procedure (in this case, "value").

5 - ADO General

6 - ADO Parameter Mapping

ADO.NET

SQL Statement: intoutput
Parameter Name:@<parameter name>

ADO.NET is similar to ADO. Set IsQueryStoredProcedure to True, and set your statement to the name of the SP you want to execute. Like ADO, you need to use the name of the parameter in the stored procedure for the Parameter Name column, prefixed with @ (in this case, "@value"). Note: The parameter mapping seems to work without the use of the @ sign, but I'm not sure if that works for all providers.

7 - ADO.NET General

8 - ADO.NET Parameter Mapping

Summary

Running stored procedures with the Execute SQL Task is tricky due to the differences in the way the providers handle the input SQL. The main differences are with the SQL Statement, and the Parameter Name value. The following table summarizes the differences (assumes a stored procedure of name StoredProc with a single output parameter named Param):

Provider	SQL Statement	Parameter Name	Notes
OLE DB	exec StoredProc ? output	0	Set BypassPrepare to True
ODBC	{call StoredProc (?)}	1
ADO	StoredProc	Param	Set IsQueryStoredProcedure to True
ADO.NET	StoredProc	@Param	Set IsQueryStoredProcedure to True

For more info about the Execute SQL Task, you can check out the books online entry, and the nice overview of the task on SQLIS.com.

Conditional Constraints

mmasson — Thu, 02 Nov 2006 03:06:00 GMT

Here's a more practical example of using constraints based on variables.

I have a package that requires an existing database file to be attached to the local SQL Server. The package is synced out to multiple machines from a source control system and run frequently. I don't want to manually setup the database on each machine, nor do I want to store the large database file in our source control system… so I decided to make the setup steps part of the package.

My first attempt looked something like this:

1 - Initial attempt

I created a package with a Script task, two File System tasks, and an Execute SQL task. The Script task branches the execution path based on whether or not the DB file exists on the local file system. This sort of worked – the file was copied, but the "Attach Database" task failed to run.

2 - Execute SQL Task does not execute

I soon realized that if you branch your logic like this, you need to set the constraints on the task where your execution merges again (in this case, "Attach Database") to OR's instead of AND's.

3 - Change constraint to logical OR

Now when I run, the file is copied, and the database task is executed. On subsequent runs, the file copying tasks are skipped and "Attach Database" is called immediately.

4 - It works... kind of

Although I could live with an error appearing the first time the script is run, it threw off some of our scripts, and people who weren't familiar with the package would always freak out the first time they saw it.

Error: 0x4 at Check if DB Files Exist: The Script returned a failure result.

Task failed: Check if DB Files Exist

While my initial solution (have my Script task print out the message "You can safely ignore the following error" when the file didn't exist) calmed the masses, it was still a problem for the execution scripts (not to mention just bad form!). This is how I discovered how to create constraints with conditional expressions.

I added a new Boolean variable to the package, and modified my script to set its value instead of setting success or failure.

5 - Package variables

My script:

Public Sub Main()

Dim strFile As String = Dts.Variables("dbFile").Value.ToString()

Dts.Variables("varFileExists").Value = File.Exists(strFile)

Dts.TaskResult = Dts.Results.Success

End Sub

(Be sure to add the variables you use to the script task to the "ReadOnlyVariables" and "ReadWriteVariables" property of the script task. Also note that you'll need to import System.IO for the File.Exists() method)

I edited the constraints connected to the script task, set "Evaluation operation" to "Expression", and added an expression based on the varFileExists variable.

6 - Setting expressions

This gives us a much cleaner solution! Now we don't see any errors the first time the package is run, and our files are copied only once. We're good to go!

NOTE: After I finished writing this up, I stopped by the SSIS forums and noticed a post where someone had encountered a similar problem, and wrote a blog entry about it. I thought I'd link it here.

Using Constraints and Expressions

mmasson — Tue, 31 Oct 2006 01:49:00 GMT

Besides the regular Success and Failure constraints, you can create precedence constraints that are linked to an expression.

In this example, I have a Script Task that determines the course for the workflow.

Besides the pass and fail, I also want to handle a third case that will be executed whenever a special condition is met in the Script Task.

To set this up, I create a new Boolean variable (varSpecialCase), and set the value in my Script Task. Double clicking the “Handle Special Case” constraint brings up the Precedence Constraint Editor, where I can change the way the constraint is evaluated.

Here I’ve set “Evaluation operation” to “Expression”, and used my variable in the expression text. If varSpecialCase has been set to true by the Script Task, the “Handle Special Case” script task will be executed.

One thing to keep in mind: = != ==

(The equals sign is not the same as double equals in the constraint expression!)

I had originally set my expression up using a single equal sign.

As this is still a valid expression, it will validate successfully when you click the Test button.

When you run however, it always fails. After spending a while trying to figure out why my task wasn’t being executed, I finally looked it up the operators in the Integration Services Expression Reference. I’ll definitely look there first, next time!