SSIS Team Blog : Extensions

Will converting my script to a custom component make it faster?

mmasson — Wed, 18 Feb 2009 23:30:30 GMT

Recently this question came up on an internal SSIS mailing list:

We have to process hundreds of thousands of rows through a fairly complex script component. Would converting it to a custom SSIS component give us any performance gain?

I’ve seen this question come up a couple of times, and I thought it would make a good post.

The main reason to move from script -> custom component is make re-use and maintenance of the code easier. I wouldn't recommend doing it just for performance reasons.

People usually see a minor performance gain when making the switch, provided that the component is written properly (minimizing calls to the COM interfaces and caching all metadata in PreExecute() are key!). The nice thing about the Script Component is that it wraps all of the implementation details from you. Custom components don’t have these wrappers, which makes it easier to make coding mistakes that can drastically decrease your performance.

You would get a performance gain by writing the custom component using native code (C++), as it skips all of the managed layers and interop. I wouldn't recommend it for anything other than really low level operations, however, as the COM interfaces have minimal documentation and can be pretty hard to use.

You typically get the biggest gain by making sure the code is well written. The quickest way to identify performance bottlenecks it to run it through a profiler. Custom components have an advantage here, as you can’t hook up a debugger to a script component, but you can achieve the same results by moving your script outside of the SSIS, and unit testing it directly.

How to create an installer for your custom extension

mmasson — Fri, 05 Dec 2008 04:16:47 GMT

The simplest way to create an installer for your custom SSIS extension is by using a Setup Project in Visual Studio. Here are the steps we take when we’re creating installers for our Codeplex projects.

1. In Visual Studio, create a new Setup Project – File -> New Project…

2. Under Other Project Types | Setup and Deployment, select Setup Project

3. Delete User’s Desktop and User’s Project Menus folders from File System (Setup) screen

4. On Application Folder, set the DefaultLocation property to the root DTS directory:

Ex. [ProgramFilesFolder]Microsoft SQL Server\100\DTS

5. Right click on the Application Folder and add the sub folder you need to put your sample in (ie. Tasks, PipelineComponents, Connections)

6. Right click on the new folder, and add “Project Output” to the folder to place binaries from your sample.

7. If you need to place files in the GAC, right click on File System on Target Machine -> Add Special Folder -> Global Assembly Cache Folder

It will now show up as an additional folder, and you can add project output to it.

8. Exclude all dependencies by highlighting all of the files under the Detected Dependencies Folder, and selecting Exclude

9. If you’d like to add a readme .rtf file, place it in the Application Folder.

10. You’ll need to edit the UI pages to display a readme after the install. Right click on the Setup project, View -> User Interface to bring up the UI page.

11. For both the Install and Administrative Install, right click on End and select Add Dialog. Select the Read Me dialog. Drag the “Read Me” page to be above the “Finished” page.

12. Set the ReadmeFile property to your readme file.

Performance profiling your custom extensions

mmasson — Thu, 13 Nov 2008 04:05:57 GMT

The Team Development Edition and Team Suite versions of Visual Studio 2008 comes with a performance profiling feature that can used to find bottlenecks in your custom extensions (tasks, data flow components, etc) for SSIS. Although having your extension assemblies in the GAC make profiling a little more involved, it can still be done with a few additional steps, and I’ve found it to be a very effective way to improve the performance of your code.

In this post I’ll walk through how to use this tool on a custom Data Source Component (in this case, the source component for the SQL Data Services connectors on Codeplex).

The basic steps are:

Configure your extension for debugging
Create a profile
Delay sign and disable strong name validation for your extension assembly(s)
Manually instrument your assembly(s) and add them to the GAC
Run the profile
Enjoy the results

Read on for more details.

Configure your extension for debugging

The first step is to setup your extension project to be easy to debug.

1. Create an SSIS package which makes use of your extension.

I’ve created a simple package which uses the SSDS Source, and pipes it to a row count transform.

2. Open the solution for your extension. Right click on the Project, and configure the debugging settings to launch the package using dtexec.exe.

Using dtexec to run your package makes it a lot easier to debug than it would be running it inside of BIDS. If you’re on a 64bit platform, be sure to select the 32bit version of dtexec (under Program Files (x86)), as the performance profiler has issues with 64bit executions.

Specify the path to your package using the /f switch. If you need to specify other command line parameters, you can use dtexecui to build your command line for you.

Once you have your project configured for debugging

Configuring the Performance Profiler

1. Launch the Performance Wizard (Analyze –> Launch Performance Wizard…)

2. Select your extension project, and click next.

3. On the next page you can select the type of profiling you’d like to do.

Both can be useful, but “Instrumentation” will give you a detailed report of which functions are being called most, and how much time is being spent in them. For SSIS extensions, this is probably what you’ll want to see.

After finishing the wizard, you’ll see the Performance Explorer window, with a new Target added for your project. If your extension relies on additional assemblies, you’d add them as additional targets by right clicking on the Targets folder, and selecting one of the Add options. In my case, the SSDS Source component makes heavy use of a Connectivity layer assembly. To get a full performance profile, I’ll instrument both of them.

Instrumenting your assemblies

If you tried to do a profiled run now (by clicking the Launch with Profiling button, or Analyze –> Profiler –> Launch with Profiling from the menu), you’d get the following warning dialog:

When running in Instrumentation mode, the profile will need to modify your assembly to insert the hooks it needs. This will invalidate the signature on your assembly. This becomes an issue if your assembly needs to be in the GAC (like with SSIS), as the GAC will only accept properly signed assemblies.

If you click ok and run anyways, dtexec will run without error, but no data will be collected because your extension assembly in the GAC wasn’t profiled.

To get around this, we’ll need to do a few things.

Delay sign our assemblies
Disable strong name validation for your assembly
Manually instrument the assembly(s) using the vsinstr.exe application
Add the instrumented assemblies to the GAC

To delay sign the assembly, check the “Delay sign only” box on the Signing tab of the project properties page for your extension project. Do the same for any other extensions required by your

To disable strong name verification for your assembly (which should only be done in your dev/test environment, and never in production) we’ll use the Strong Name Tool (sn.exe) which comes with the .NET framework SDK. If you’ve installed SQL Server 2008 on your machine, you’ll find it under C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin.

From the command line (must be elevated on Vista), run sn.exe -Vr <extension>.dll

C:\code\source\bin\release>sn -Vr Microsoft.Samples.DataServices.Source.dll

Verification entry added for assembly 'Microsoft.Samples.DataServices.Source,DA625E43F8E8D37E'

Next we’ll instrument our assembly(s) using the vsinstr.exe tool. From the command line, run vsinstr.exe <extension>.dll

C:\code\source\bin\release>vsinstr.exe Microsoft.Samples.DataServices.Source.dll
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

File to Process:
C:\code\source\bin\release\Microsoft.Samples.DataServices.Source.dll -->
C:\code\source\bin\release\Microsoft.Samples.DataServices.Source.dll
Original file backed up to C:\code\source\bin\release\Microsoft.Samples.DataServices.Source.dll.orig

Warning VSP2001 : C:\code\source\bin\release\Microsoft.Samples.DataServices.
Source.dll is a strongly named assembly. It will need to be re-signed before it can be executed.
Successfully instrumented file C:\code\source\bin\release\Microsoft.Samples.DataServices.Source.dll.
Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process. The CLR header flags have been updated to reflect this.

Now add the instrumented assembly to the GAC. This can be done using the Global Assembly Cache Tool (gacutil.exe), which can be found in the same directory as sn.exe. From the command line, run gacutil.exe /i <extension>.dll

C:\code\source\bin\release>gacutil /i Microsoft.Samples.DataServices.Source.dll

Assembly successfully added to the cache

These steps should be repeated for each assembly you’d like to profile.

Note: since you’ll most likely need to run the profiling more than once, I’d suggest you create a batch file which runs the VS instrumentation tool and adds the files to the GAC.

Launching with profiling

Once the instrumented assemblies have been placed in the GAC, you can run launch a profiled run (click the Launch with Profiling button, or Analyze –> Profiler –> Launch with Profiling from the menu). You’ll still get an instrumentation warning, but this time when you click ok, you’ll get the profiling results.

From the summary page you can drill down to see which functions are being called most, and which are taking the most time. From there you can begin to optimize your code and fix your bottlenecks.

Remember that you need to re-instrument and re-GAC your assembly anytime you modify it.

Be sure to check out the Performance Profiler MSDN article for more details on the features of the profiler.

Creating a custom task with a default UI

mmasson — Sat, 19 Jul 2008 04:20:05 GMT

The Developing a User Interface for a Custom Task entry in Books Online describes how to build your UI from scratch, but there is an easier way to create a UI that has the same look and feel as the stock tasks that ship with SSIS.

The Microsoft.DataTransformationServices.Controls assembly exposes a base class that most of the stock SSIS tasks inherit from – DTSBaseTaskUI. With minimal coding, you too can have a beautiful Task UI (that looks something like this:)

After creating your task, you can link it to a UI assembly using the UITypeName parameter of the DtsTask property. The recommended practice is to create a separate assembly for your UI to separate the UI code from the code that gets loaded at runtime, but this is optional. It can all work in the same assembly as well.

We’re going to create three classes for our “HelloWorldTask”

HelloWorldTaskUI – Implements IDtsTaskUI
HelloWorldTaskMainWnd – Extends DTSBaseTaskUI
GeneralView – Implements IDTSTaskUIView

HelloWorldTaskUI is the entry point for the UI functionality, and is responsible for initializing and displaying your UI.

Extending DTSBaseTaskUI in HelloWorldTaskMainWnd gives you the form in the screen shot above. Each page of properties you want to display is a separate class that implements IDTSTaskUIView. You get the “Expressions” page for free.

Let’s walk through what you need to do.

First you’ll need to add some additional references:

Microsoft.DataTransformationServices.Controls
Microsoft.SqlServer.Dts.Design
System.Drawing
System.Windows.Forms

Note, that Microsoft.DataTransformationServices.Controls.dll is in the GAC, but doesn’t show up in the “.NET” tab of the “Add Reference…” dialog.

Add new User Control – GeneralView.cs. On the designer, add a property grid. To keep the same look and feel as other SSIS tasks, the InitializeComponents() should look something like this:

// 
// propertyGrid
// 
this.propertyGrid.Dock = System.Windows.Forms.DockStyle.Fill;
this.propertyGrid.Location = new System.Drawing.Point(0, 0);
this.propertyGrid.Name = "propertyGrid";
this.propertyGrid.PropertySort = System.Windows.Forms.PropertySort.Categorized;
this.propertyGrid.Size = new System.Drawing.Size(150, 150);
this.propertyGrid.TabIndex = 0;
this.propertyGrid.ToolbarVisible = false;

In your GeneralView class, define an internal class which contains all of the properties you’d like to display in the UI – ie. GeneralViewNode

internal class GeneralViewNode
{
    // Properties variables
    private string displayText = string.Empty;
    private string name = string.Empty;
    private string description = string.Empty;

    internal TaskHost taskHost = null;
    internal IDtsConnectionService connectionService = null;

    internal GeneralViewNode(TaskHost taskHost, IDtsConnectionService connectionService)
    {
        this.taskHost = taskHost;
        this.connectionService = connectionService;

        // Extract common values from the Task Host
        name = taskHost.Name;
        description = taskHost.Description;

        // Extract values from the task object
        HelloWorldTask.HelloWorldTask task = taskHost.InnerObject as HelloWorldTask.HelloWorldTask;
        if (task == null)
        {
            string msg = string.Format("Type mismatch for taskHost inner object.");
            throw new ArgumentException(msg);
        }

        displayText = task.DisplayText;
    }

    #region Properties

    [Category("General"), Description("Task name")]
    public string Name
    {
        get
        {
            return name;
        }
        set
        {
            string v = value.Trim();
            if (string.IsNullOrEmpty(v))
            {
                throw new ApplicationException("Task name cannot be empty");
            }
            name = v;
        }
    }

    [Category("General"), Description("Task description")]
    public string Description
    {
        get
        {
            return description;
        }
        set
        {
            description = value.Trim();
        }
    }

    [Category("General"), Description("Text to display")]
    public string DisplayText
    {
        get
        {
            return displayText;
        }
        set
        {
            displayText = value;
        }
    }

    #endregion
}

As you can see, we have the single property for the task (DisplayText), and two of the common Task properties – Name, and Description.

Create a member variable for this node class:

private GeneralViewNode generalNode;
internal GeneralViewNode GeneralNode
{
    get { return generalNode; }
}

Implement the IDTSTaskUIView interface. These methods are called when the UI is opened, and when the UI clicks OK.

#region IDTSTaskUIView Members

public void OnCommit(object taskHost)
{
    TaskHost host = taskHost as TaskHost;
    if (host == null)
        throw new ArgumentException("Arugment is not a TaskHost.", "taskHost");

    HelloWorldTask.HelloWorldTask task = host.InnerObject as HelloWorldTask.HelloWorldTask;
    if (task == null)
    {
        throw new ArgumentException("Arugment is not a HelloWorldTask.", "taskHost");
    }

    host.Name = generalNode.Name;
    host.Description = generalNode.Description;

    // Task properties
    task.DisplayText = generalNode.DisplayText;
}

public void OnInitialize(IDTSTaskUIHost treeHost, TreeNode viewNode, object taskHost, object connections)
{
    this.generalNode = new GeneralViewNode(taskHost as TaskHost, connections as IDtsConnectionService);
    this.propertyGrid.SelectedObject = generalNode;
}

public void OnLoseSelection(ref bool bCanLeaveView, ref string reason)
{
}

public void OnSelection()
{
}

public void OnValidate(ref bool bViewIsValid, ref string reason)
{
}

#endregion

Add new Windows Form – HelloWorldTaskMainWnd.cs. Change the inheritance from Form to DTSBaseTaskUI.

Note, after inheriting from DTSBaseTaskUI, you’ll get an error when trying to open the code in Design view in Visual Studio 2008. This is caused by DTSBaseTaskUI not having a default constructor. We’re aware of the issue, and will hopefully be able to resolve it in a cumulative update. It won’t cause issues at runtime.

Add some members for the UI properties we’ll use to initialize DTSBaseTaskUI.

private const string Title = "Hello World Task";
private const string Description = "Displays a message box";
private static Icon TaskIcon = new Icon(typeof(HelloWorldTask), "Task.ico");

Add a member for GeneralView class.

private GeneralView generalView;
public GeneralView GeneralView
{
    get { return generalView; }
}

Add a constructor.

public HelloWorldTaskUIMainWnd(TaskHost taskHost, object connections) :
    base(Title, TaskIcon, Description, taskHost, connections)
{
    InitializeComponent();
    
    // Setup our views
    generalView = new GeneralView();
    this.DTSTaskUIHost.FastLoad = false;
    this.DTSTaskUIHost.AddView("General", generalView, null);
    this.DTSTaskUIHost.FastLoad = true;
}

Note, the first parameter you use for AddView() is the text that gets displayed on the left of the main window.

Finally, add a new class that implements IDtsTaskUI to control the launching of your new window.

public class HelloWorldTaskUI : IDtsTaskUI
{
    private TaskHost taskHost = null;
    private IDtsConnectionService connectionService = null;

    #region IDtsTaskUI Members

    public void Delete(IWin32Window parentWindow)
    {
    }

    public ContainerControl GetView()
    {
        return new HelloWorldTaskUIMainWnd(taskHost, connectionService);
    }

    public void Initialize(TaskHost taskHost, IServiceProvider serviceProvider)
    {
        this.taskHost = taskHost;
        this.connectionService = serviceProvider.GetService(typeof(IDtsConnectionService)) as IDtsConnectionService;
    }

    public void New(IWin32Window parentWindow)
    {
    }

    #endregion
}

And there you go! Be sure to add the task UI assembly to the GAC before trying it out in Visual Studio.

I should mention that Kirk’s book has a chapter on creating custom tasks that gives a lot more details than I have here. If you’re interested in extending the SSIS platform, Kirk’s book (or the “SSIS White Book” as I like to call it) is an invaluable resource.

I’ve uploaded the source for the highly useful Hello World Task (and its UI) to my Sky Drive public folder. I developed this using SQL Server 2008, but it should also work the same way in 2005.

I’ll also be turning this into a community sample to be published on Codeplex (along with our other new development samples) sometime soon.

Enjoy!