Something new for SSIS in SQL Server 2008 R2 November CTP

mmasson — Thu, 12 Nov 2009 19:29:27 GMT

Ok, so it turns my previous post about R2 wasn’t entirely correct.

SSIS does have one new feature which appears in the latest SQL Server 2008 R2 CTP. If you open up the ADO.NET Destination UI, you’ll notice a new “Use Bulk Insert when Possible” check box.

Previously, the ADO.NET Destination did all of its inserts row by row (we do some batching internally, which is why the component has a BatchSize property, but the underlying ADO.NET provider will always do single row inserts). With this new feature enabled, SSIS will use a bulk insert interface (like enabling “FastLoad” for OLEDB Destination). Unfortunately, there isn’t a generic Bulk Load interface for ADO.NET, so this functionality is currently only supported by SQL Server (through the SqlBulkCopy API). Hopefully we can extend support to other ADO.NET providers in the future.

The main reason for implementing this functionality was to improve our support for SQL Azure. As you might already know, ADO.NET is the primary way to communicate with SQL Azure, and adding support for SqlBulkCopy greatly increased the transfer speed. It also speeds up things when you’re working with regular SQL Server systems -- although OLEDB with FastLoad is still the preferred way of doing SQL data loads.

Be sure to try it out and provide feedback before the final release.

SQL Server Data Services connectors now on Codeplex

mmasson — Fri, 26 Sep 2008 01:16:55 GMT

When I was visiting the SSIS development team in Shanghai a couple of months ago, we started working on a side project to create Source and Destination components for SQL Server Data Services (SSDS). We developed them as samples, and I was surprised at how popular they became internally. David Robinson beat me to the announcement, but after months of delay, I’ve finally been able to get it all together and publish the project on Codeplex.

If you’re interested in the source code, I particularly like the way we got around a limitation in the Destination. SSDS currently supports a “batch” query mode for retrieving multiple entities (“rows” in relational terms) – you can pull back up to 500 entities in a single query -but there is no batch functionality for inserts. That means that each entity is inserted with a single query. To increase performance we do many inserts in parallel using multiple threads. You have to tweak your .NET HTTP connection settings (these adapters use the SOAP interface that SSDS provides) to increase the maximum number of concurrent connections to a server, but it ended up giving us a 10-50x performance increase. Definitely something to keep in mind if you are creating your own web service based data flow component.

I thought I’d include some additional instructions and images here here that didn’t fit in the project readme.

Install

Set the path the DTS folder of your SQL Server installation. This should be %ProgramFiles%\Microsoft SQL Server\100\DTS.

On 64bit installs, place the files under the “Program Files (x86)” directory.

The installer will place the following files:

PipelineComponents\Microsoft.Samples.DataServices.Destination.dll

PipelineComponents\Microsoft.Samples.DataServices.Source.dll

Connections\Microsoft.Samples.DataServices.ConnectionManager.dll

Microsoft.Samples.DataServices.Connectivity.dll (GAC only)

All of the DLLs also go in the GAC.

Creating a connection manager

Right click on the Connection Managers window

Select New Connection…

Choose “SSDS”

Enter connection information

· Authority

· User Name

· Password

Note: You only need to enter your Authority ID and not the full URL to the service.

Test Connection to see if it worked. Click OK when done.

Setting up the Data Flow

Add a new Data Flow Task

Right click on Toolbox, chose “Choose Items…”

Check the boxes for SSDS Destination and SSDS Source

Source

The source component currently doesn’t have a custom UI. All editing is done through the common Advanced Editor UI.

1. Drag an SSDS Source component from the toolbox onto the Data Flow canvas

2. Double click the source component

3. Click “Yes” asking if you want the component to fix itself. The advanced editor will display.

4. Select the Connection Manager you created in the previous step.

5. On the Component Properties tab, set the following properties:

Property	Description
ContainerID	Must be set to a valid container ID within your Authority
EntityKind	Optional. Use this to limit the entities retrieved to a specific kind
PreviewCount	Used to limit the number of rows pulled back to determine the column metadata. Note: the current functionality is currently limited. We will always pull back up to 500 entities, but will only look at “PreviewCount” number of entities to determine metadata.

6. Once the ContainerID is set, click the Column Mappings tab to select which columns you want to use, and set the output column names.

Destination

The destination has a custom UI, but it currently doesn’t allow you to specify column mappings. This has to be done separately on the Advanced UI.

1. Drag an SSDS Destination object from the Toolbox onto the design surface

2. Connect the pipeline flow to the destination

3. Double click the destination to bring up the custom UI

4. Set the SSDS connection manager from the drop down, the Container ID you want to store the results in, and the Entity Kind value. These fields are all required. To generate a new ID for each entity being inserted (as a GUID), select the “Create New ID” option. To use a value from a column in the pipeline, select “Use Existing ID” and type in the column name.

5. Click the “Columns” tab to select the columns from the data flow you want to send to SSDS. Each row becomes a new entity. Each column becomes a property.

Uninstall

Remove the “Data Services Connectors for SSIS” entry from Add/Remove Program Files

SSIS Team Blog : SSDS