SSIS Team Blog : Performance

Tutorial: SSIS Performance Videos

mmasson — Tue, 26 May 2009 19:01:22 GMT

This post is coming a little late (these have already been announced here and here), but incase you haven’t seen these before, I wanted to bring attention to four new performance related created by the SQLCAT and SSIS teams. They mostly focus on larger scale/Enterprise level package deployments, but there are good tips for all levels of SSIS usage.

From the SQLCAT post:

Measuring and Understanding the Performance of Your SSIS Packages in the Enterprise (SQL Server Video)
Link | Watch this video

Author: Denny Lee, Microsoft Corporation

This video demonstrates how to measure and understand the performance of packages, based on lessons learned from enterprise customers. In this video, you will learn the following guidelines for improving performance:

How the limits of the source system affect performance.
Why disk I/O is important.
Why you should establish a package performance baseline

Tuning Your SSIS Package Data Flow in the Enterprise (SQL Server Video)
Link | Watch this video

Author: David Noor, Microsoft Corporation

This video demonstrates how to improve the performance of the data flow in an Integration Services package. In this video, you will learn how to tune the following phases of the data flow:

Extraction
Transformation
Loading

You can apply these performance tuning tips when you design, develop, and run the data flow.

Understanding SSIS Data Flow Buffers (SQL Server Video)
Link | Watch this video

Author: Bob Bojanic, Microsoft Corporation

This video looks at the memory buffers that transfer data to the data flow of an Integration Services package. The video includes the following items:

Demonstration of a simple package that shows you how Integration Services divides the data along the data flow paths and how buffers carry data along those paths.
This simple package performs the following operations:
- Data extraction
- Character mapping
- Creation of new columns by using a synchronous transformation
- Multicasting
- Sorting by using a blocking (asynchronous) transformation.
Design guidelines for building and maintaining the data flow of a package.

Designing Your SSIS Packages for Parallelism (SQL Server Video)
Link | Watch this video

Author: Matt Carroll, Microsoft Corporation

This video demonstrates how to increase the performance of your Integration Services packages by designing for parallelism.

------

These four videos join the growing list of SSIS tutorial videos in the SQL Server Video series. Others that might interest you:

We Loaded 1TB in 30 Minutes with SSIS, and So Can You

mmasson — Wed, 04 Mar 2009 04:16:02 GMT

A popular SSIS talk at recent conferences has been about the ETL Record that we announced last year. They’ve converted the talk to a white paper, and it is now available on MSDN. It’s a very interesting read.

Summary: In February 2008, Microsoft announced a record-breaking data load using Microsoft® SQL Server® Integration Services (SSIS): 1 TB of data in less than 30 minutes. That data load, using SQL Server Integration Services, was 30% faster than the previous best time using a commercial ETL tool. This paper outlines what it took: the software, hardware, and configuration used. We will describe what we did to achieve that result, and offer suggestions for how to relate these techniques to typical scenarios. Even for customers who don't have needs quite like this benchmark, such efforts can teach a lot about getting optimal performance.

Performance: Top 10 Best Practices

mmasson — Fri, 05 Dec 2008 20:25:41 GMT

The SQL CAT team published a Top 10 SQL Server Integration Services Best Practices list back in October, and I thought I’d draw some more attention to it incase anyone missed it the first time around. The list contains performance related tips that were proven during the ETL world record performance run that was published earlier this year.

Performance profiling your custom extensions

mmasson — Thu, 13 Nov 2008 04:05:57 GMT

The Team Development Edition and Team Suite versions of Visual Studio 2008 comes with a performance profiling feature that can used to find bottlenecks in your custom extensions (tasks, data flow components, etc) for SSIS. Although having your extension assemblies in the GAC make profiling a little more involved, it can still be done with a few additional steps, and I’ve found it to be a very effective way to improve the performance of your code.

In this post I’ll walk through how to use this tool on a custom Data Source Component (in this case, the source component for the SQL Data Services connectors on Codeplex).

The basic steps are:

Configure your extension for debugging
Create a profile
Delay sign and disable strong name validation for your extension assembly(s)
Manually instrument your assembly(s) and add them to the GAC
Run the profile
Enjoy the results

Read on for more details.

Configure your extension for debugging

The first step is to setup your extension project to be easy to debug.

1. Create an SSIS package which makes use of your extension.

I’ve created a simple package which uses the SSDS Source, and pipes it to a row count transform.

2. Open the solution for your extension. Right click on the Project, and configure the debugging settings to launch the package using dtexec.exe.

Using dtexec to run your package makes it a lot easier to debug than it would be running it inside of BIDS. If you’re on a 64bit platform, be sure to select the 32bit version of dtexec (under Program Files (x86)), as the performance profiler has issues with 64bit executions.

Specify the path to your package using the /f switch. If you need to specify other command line parameters, you can use dtexecui to build your command line for you.

Once you have your project configured for debugging

Configuring the Performance Profiler

1. Launch the Performance Wizard (Analyze –> Launch Performance Wizard…)

2. Select your extension project, and click next.

3. On the next page you can select the type of profiling you’d like to do.

Both can be useful, but “Instrumentation” will give you a detailed report of which functions are being called most, and how much time is being spent in them. For SSIS extensions, this is probably what you’ll want to see.

After finishing the wizard, you’ll see the Performance Explorer window, with a new Target added for your project. If your extension relies on additional assemblies, you’d add them as additional targets by right clicking on the Targets folder, and selecting one of the Add options. In my case, the SSDS Source component makes heavy use of a Connectivity layer assembly. To get a full performance profile, I’ll instrument both of them.

Instrumenting your assemblies

If you tried to do a profiled run now (by clicking the Launch with Profiling button, or Analyze –> Profiler –> Launch with Profiling from the menu), you’d get the following warning dialog:

When running in Instrumentation mode, the profile will need to modify your assembly to insert the hooks it needs. This will invalidate the signature on your assembly. This becomes an issue if your assembly needs to be in the GAC (like with SSIS), as the GAC will only accept properly signed assemblies.

If you click ok and run anyways, dtexec will run without error, but no data will be collected because your extension assembly in the GAC wasn’t profiled.

To get around this, we’ll need to do a few things.

Delay sign our assemblies
Disable strong name validation for your assembly
Manually instrument the assembly(s) using the vsinstr.exe application
Add the instrumented assemblies to the GAC

To delay sign the assembly, check the “Delay sign only” box on the Signing tab of the project properties page for your extension project. Do the same for any other extensions required by your

To disable strong name verification for your assembly (which should only be done in your dev/test environment, and never in production) we’ll use the Strong Name Tool (sn.exe) which comes with the .NET framework SDK. If you’ve installed SQL Server 2008 on your machine, you’ll find it under C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin.

From the command line (must be elevated on Vista), run sn.exe -Vr <extension>.dll

C:\code\source\bin\release>sn -Vr Microsoft.Samples.DataServices.Source.dll

Verification entry added for assembly 'Microsoft.Samples.DataServices.Source,DA625E43F8E8D37E'

Next we’ll instrument our assembly(s) using the vsinstr.exe tool. From the command line, run vsinstr.exe <extension>.dll

File to Process:
C:\code\source\bin\release\Microsoft.Samples.DataServices.Source.dll -->
C:\code\source\bin\release\Microsoft.Samples.DataServices.Source.dll
Original file backed up to C:\code\source\bin\release\Microsoft.Samples.DataServices.Source.dll.orig

Warning VSP2001 : C:\code\source\bin\release\Microsoft.Samples.DataServices.
Source.dll is a strongly named assembly. It will need to be re-signed before it can be executed.
Successfully instrumented file C:\code\source\bin\release\Microsoft.Samples.DataServices.Source.dll.
Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process. The CLR header flags have been updated to reflect this.

Now add the instrumented assembly to the GAC. This can be done using the Global Assembly Cache Tool (gacutil.exe), which can be found in the same directory as sn.exe. From the command line, run gacutil.exe /i <extension>.dll

C:\code\source\bin\release>gacutil /i Microsoft.Samples.DataServices.Source.dll

Assembly successfully added to the cache

These steps should be repeated for each assembly you’d like to profile.

Note: since you’ll most likely need to run the profiling more than once, I’d suggest you create a batch file which runs the VS instrumentation tool and adds the files to the GAC.

Launching with profiling

Once the instrumented assemblies have been placed in the GAC, you can run launch a profiled run (click the Launch with Profiling button, or Analyze –> Profiler –> Launch with Profiling from the menu). You’ll still get an instrumentation warning, but this time when you click ok, you’ll get the profiling results.

From the summary page you can drill down to see which functions are being called most, and which are taking the most time. From there you can begin to optimize your code and fix your bottlenecks.

Remember that you need to re-instrument and re-GAC your assembly anytime you modify it.

Be sure to check out the Performance Profiler MSDN article for more details on the features of the profiler.

SSIS sets an ETL world record

mmasson — Thu, 28 Feb 2008 00:58:55 GMT

They mentioned the TPC benchmark results at the SQL Server 2008 launch today. The final paragraph mentions a world record for ETL load performance. The fine folks who maintain the SQLPerf blog have posted details about the result, explaining how they were able to load over 1 TB of TPC-H data in under 30 minutes. It makes for a very interesting read!