SSIS Team Blog : Lookup

API Sample – Lookup Transform

mmasson — Fri, 02 Jan 2009 21:34:50 GMT

This sample creates a data flow package with an OLEDB Source component feeding into a Lookup Transform. The Lookup transform is set to Full Cache mode, and uses [DimCustomer] as its reference table.

Items of interest:

CustomerKey and GeographyKey are used as the index (join) columns. This is configured by using the JoinToReferenceColumn property
The FirstName column is being overwritten by the value retrieved by the lookup transform
The LastName2 column is being added as a new output column

static void Main(string[] args)
{
    Package package = new Package();

    // Add Data Flow Task
    Executable dataFlowTask = package.Executables.Add("STOCK:PipelineTask");

    // Set the name (otherwise it will be a random GUID value)
    TaskHost taskHost = dataFlowTask as TaskHost;
    taskHost.Name = "Data Flow Task";

    // We need a reference to the InnerObject to add items to the data flow
    MainPipe pipeline = taskHost.InnerObject as MainPipe;

    //
    // Add connection manager
    //

    ConnectionManager connection = package.Connections.Add("OLEDB");
    connection.Name = "localhost";
    connection.ConnectionString = "Data Source=localhost;Initial Catalog=AdventureWorksDW2008;Provider=SQLNCLI10.1;Integrated Security=SSPI;Auto Translate=False;";

    //
    // Add OLEDB Source
    //

    IDTSComponentMetaData100 srcComponent = pipeline.ComponentMetaDataCollection.New();
    srcComponent.ComponentClassID = "DTSAdapter.OleDbSource";
    srcComponent.ValidateExternalMetadata = true;
    IDTSDesigntimeComponent100 srcDesignTimeComponent = srcComponent.Instantiate();
    srcDesignTimeComponent.ProvideComponentProperties();
    srcComponent.Name = "OleDb Source";

    // Configure it to read from the given table
    srcDesignTimeComponent.SetComponentProperty("AccessMode", 0);
    srcDesignTimeComponent.SetComponentProperty("OpenRowset", "[DimCustomer]");

    // Set the connection manager
    srcComponent.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(connection);
    srcComponent.RuntimeConnectionCollection[0].ConnectionManagerID = connection.ID;

    // Retrieve the column metadata
    srcDesignTimeComponent.AcquireConnections(null);
    srcDesignTimeComponent.ReinitializeMetaData();
    srcDesignTimeComponent.ReleaseConnections();

    // Add transform
    IDTSComponentMetaData100 lookupComponent = pipeline.ComponentMetaDataCollection.New();
    lookupComponent.ComponentClassID = "DTSTransform.Lookup";
    lookupComponent.Name = "Lookup";

    CManagedComponentWrapper lookupWrapper = lookupComponent.Instantiate();
    lookupWrapper.ProvideComponentProperties();

    // Connect the source and the transform
    IDTSPath100 path = pipeline.PathCollection.New();
    path.AttachPathAndPropagateNotifications(srcComponent.OutputCollection[0], lookupComponent.InputCollection[0]);

    //
    // Configure the transform
    //

    // Set the connection manager
    lookupComponent.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(connection);
    lookupComponent.RuntimeConnectionCollection[0].ConnectionManagerID = connection.ID;

    // Cache Type - Full = 0, Partial = 1, None = 2
    lookupWrapper.SetComponentProperty("CacheType", 0);
    lookupWrapper.SetComponentProperty("SqlCommand", "select * from [DimCustomer]");

    // initialize metadata
    lookupWrapper.AcquireConnections(null);
    lookupWrapper.ReinitializeMetaData();
    lookupWrapper.ReleaseConnections();

    // Mark the columns we are joining on
    IDTSInput100 lookupInput = lookupComponent.InputCollection[0];
    IDTSInputColumnCollection100 lookupInputColumns = lookupInput.InputColumnCollection;
    IDTSVirtualInput100 lookupVirtualInput = lookupInput.GetVirtualInput();
    IDTSVirtualInputColumnCollection100 lookupVirtualInputColumns = lookupVirtualInput.VirtualInputColumnCollection;

    // We are joining on CustomerKey and GeographyKey
    // Note: join columns should be marked as READONLY
    var joinColumns = new string[] { "CustomerKey", "GeographyKey" };
    foreach (string columnName in joinColumns)
    {
        IDTSVirtualInputColumn100 virtualColumn = lookupVirtualInputColumns[columnName];
        IDTSInputColumn100 inputColumn = lookupWrapper.SetUsageType(lookupInput.ID, lookupVirtualInput, virtualColumn.LineageID, DTSUsageType.UT_READONLY);
        lookupWrapper.SetInputColumnProperty(lookupInput.ID, inputColumn.ID, "JoinToReferenceColumn", columnName);
    }
    
    // Overwrite the existing FirstName column value with the one returned by the Lookup.
    // To do this, we need to flag the column as READWRITE, and set the CopyFromReferenceColumn property on the input
    var overwriteColumns = new string[] { "FirstName" };
    foreach (string columnName in overwriteColumns)
    {
        IDTSVirtualInputColumn100 virtualColumn = lookupVirtualInputColumns[columnName];
        IDTSInputColumn100 inputColumn = lookupWrapper.SetUsageType(lookupInput.ID, lookupVirtualInput, virtualColumn.LineageID, DTSUsageType.UT_READWRITE);

        lookupWrapper.SetInputColumnProperty(lookupInput.ID, inputColumn.ID, "CopyFromReferenceColumn", columnName);
    }

    // First output is the Match output
    IDTSOutput100 lookupMatchOutput = lookupComponent.OutputCollection[0];

    // Add a new LastName2 column from the "LastName" column returned by the lookup
    var newColumns = new Dictionary<string, string>();
    newColumns.Add("LastName", "LastName2");

    foreach (string sourceColumn in newColumns.Keys)
    {
        string newColumnName = newColumns[sourceColumn];
        string description = string.Format("Copy of {0}", sourceColumn);

        // insert the new column
        IDTSOutputColumn100 outputColumn = lookupWrapper.InsertOutputColumnAt(lookupMatchOutput.ID, 0, newColumnName, description);
        lookupWrapper.SetOutputColumnProperty(lookupMatchOutput.ID, outputColumn.ID, "CopyFromReferenceColumn", sourceColumn);
    }
}

Lookup Pattern: Range Lookups

mmasson — Tue, 25 Nov 2008 20:30:28 GMT

Performing range lookups (i.e. to find a key for a given range) is a common ETL operation in data warehousing scenarios. It's especially for historical loads and late arriving fact situations, where you're using type 2 dimensions and you need to locate the key which represents the dimension value for a given point in time.

This blog post outlines three separate approaches for doing range lookups in SSIS:

Using the Lookup Transform
Merge Join + Conditional Split
Script Component

All of our scenarios will use the AdventureWorksDW2008 sample database (DimProduct table) as the dimension, and take its fact data from AdventureWorks2008 (SalesOrderHeader and SalesOrderDetail tables). The "ProductNumber" column from the SalesOrderDetail table maps to the natural key of the DimProduct dimension (ProductAlternateKey column). In all cases we want to lookup the key (ProductKey) for the product which was valid (identified by StartDate and EndDate) for the given OrderDate.

One last thing to note is that the Merge Join and Script Component solutions assume that a valid range exists for each incoming value. The Lookup Transform approach is the only one that will identify rows that have no matches (although the Script Component solution could be modified to do so as well).

Lookup Transform

The Lookup Transform was designed to handle 1:1 key matching, but it can also be used in the range lookup scenario by using a partial cache mode, and tweaking the query on the Advanced Settings page. However, the Lookup doesn't cache the range itself, and will end up going to the database very often - it will only detect a match in its cache if all of the parameters are the same (i.e. same product purchased on the same date).

We can use the following query to have the lookup transform perform our range lookup:

select [ProductKey], [ProductAlternateKey], 
     [StartDate], [EndDate]
from [dbo].[DimProduct]
where [ProductAlternateKey] = ?
and   [StartDate] <= ?
and (
    [EndDate] is null or 
    [EndDate] > ?
)

On the query parameters page, we map 0 -> ProductNumber, 1 and 2 -> OrderDate.

This approach is effective and easy to setup, but it is pretty slow when dealing with a large number of rows, as most lookups will be going to the database.

Merge Join and Conditional Split

This approach doesn't use the Lookup Transform. Instead we use a Merge Join Transform to do an inner join on our dimension table. This will give us more rows coming out than we had coming in (you'll get a row for every repeated ProductAlternateKey). We use the conditional split to do the actual range check, and take only the rows that fall into the right range.

For example, a row coming in from our source would contain an OrderDate and ProductNumber, like this:

From the DimProduct source, we take three additional columns - ProductKey (what we're after), StartDate and EndDate. The DimProduct dimension contains three entries for the "LJ-0192-L" product (as its information, like unit price, has changed over time). After going through the Merge Join, the single row becomes three rows.

We use the Conditional Split to do the range lookup, and take the single row we want. Here is our expression (remember, in our case an EndDate value of NULL indicates that it's the most current row):

StartDate <= OrderDate && (OrderDate < EndDate || ISNULL(EndDate))

This approach is a little more complicated, but performs a lot better than using the Lookup Transform.

Script Component

The third approach uses a custom script component to perform the lookup. I wrote the script in two ways - one that simulates a "Full Cache" type lookup, and one that is similar to partial cache except it pulls back all values for a given natural key, instead of just the one for the given date range. The caching behavior is controlled by the PreCache boolean package variable.

Conclusion

I ran the three packages using the following environment (my laptop):

Dual core Intel 1.8ghz
3gb of RAM
AdventureWorks2008 and AdventureWorksDW2008

~120,000 order rows (SalesOrderDetail)
~600 reference rows (DimProduct)

Here are the results, in rows per second (larger being better):

At 120k+ rows per second, we can see that the custom script (or better yet, a custom transform) is the best alternative here. We can also see that even though the Lookup approach was by far the slowest (3639 rows / second), it is still a viable choice when you're processing a small number of rows.

There's a couple of reasons that the Lookup Transform performs poorly here. First, because it's not able to pre-cache any of the reference data, it has to go to the database often. Second, it matches only on actual parameter values - it doesn't have a concept of ranges. Since it will only find a cache hit if all parameters are the same, it ends up hitting the database for almost every row (120k times). By comparison, the script component will only query once per unique ProductNumber (~600 times max).

So there you have three different approaches for doing range lookups in SSIS. I'm hoping we'll be able to either enhance the Lookup component to support this functionality in the future, or perhaps provide a new transform to handle this case.

In the mean time, please feel free to post / email any alternative approaches you might have.

I've attached the packages used in this post incase you want to try out the different options for yourself.

Lookup Pattern: Incremental persistent cache updates

mmasson — Sun, 23 Nov 2008 23:16:00 GMT

The Cache Transform isn't currently able to update (i.e. append to) an existing persistent cache file. This pattern presents a way to incrementally build up your lookup cache if your data flow process is responsible for adding new rows to your reference table. For an alternative approach, please see the Other Resources section at the end of this post.

The process is split up across multiple packages. Because a Cache Connection Manager reads its cache file once, and keeps the cache in memory until package execution is complete, you can't read and update a cache file in the same package.

Control Package

This is the control package which uses a script task to check if the cache file already exists. If it doesn't, it executes the "Create Cache" package. Once the cache is there, the main data flow package which performs your general ETL logic is executed. The "Data Flow" package will write out any new rows for the reference cache to a temporary file, and update a variable in the control package to indicate how many rows were written out. The parent checks if any rows were written, and if so, runs the "Update Cache" package to update the persistent cache file

Create Cache

The cache creation package reads the source reference data, and then writes it to two separate locations. First it uses the Cache Transform to create a persistent cache, and then it writes the same data out to a RAW file. This RAW file will be incrementally updated by the Data Flow package to store the latest data for our reference set.

Data Flow

In the main ETL package, we use a key generation pattern to generate and insert new values into our reference table. Instead of rejoining the main flow right away, we multicast it out to a Row Count, followed by a RAW File Destination.

The Row Count writes to a local variable. After this data flow, we'll use a script task to copy this variable's value over to a variable in the parent package. You can see Jamie Thomson's post for more about how this works.

The RAW File Destination is able to append to an existing RAW File. Here we use it to add the new rows to the "incremental update" file we created in the "Create Cache" package.

Update Cache

If the main ETL package wrote any new rows to the incremental update RAW file, the control package will run the Update Cache package. This package simply reads the incremental update file (which will contain the entire cache, since it's simply being appended to each time), and recreate the cache file.

The sample packages used in this post can be downloaded from my SkyDrive share.

Other Resources

John Welch has an alternative approach to incremental updates

Lookup Pattern: Case Insensitive

mmasson — Sun, 23 Nov 2008 20:53:24 GMT

The Lookup Transform does case sensitive string comparisons. This means that you need to a little bit of special handling to get it to work in a case insensitive way.

In most cases (and especially if you're using a case sensitive collation on the database/table that holds your reference data), you'll want to add an upper case version of your index column(s) to your data flow (or replace your current column(s) if you don't need it later downstream). This can be done directly at the source, through a Character Map transform, or a Derived Column transform (both transforms should perform about the same).

You'll have to use the UPPER() function on the columns in your lookup transform's SQL statement as well to get them to match. You'll want to use a Full cache mode here, because using a function (like UPPER()) on your index columns will prevent the database from using any indexes it has created on the column.

This is a case where you can also use the new cache connection manager to pre-create your case insensitive cache.

If your lookup columns are indexed, your database has a case insensitive collation, and you're not going to be hitting a large percentage of your reference table, you might want to take an alternate approach. Switch the cache mode to Partial and remove the UPPER() function from your SQL statement in the Lookup. This will allow the query engine to make use of your table indexes.

Other Resources

See a related post and discussion about case insensitive lookups on Jamie Thomson's blog.

Lookup Pattern: Cascading

mmasson — Sun, 23 Nov 2008 07:46:34 GMT

The cascading lookup pattern uses two lookup transforms with different cache modes. A common use of this pattern is when your data flow is inserting new rows into your reference table.

The first lookup in the chain is set to Full cache mode. Since it creates its cache before the data flow begins, it will only have the keys that exist before the package was executed.

A second lookup is hooked up to the No Match output of the first, using a Partial Cache mode. This one will pick up any rows that have been added since the data flow began. We hookup any logic needed to generate the key, or insert the row into the database into the No Match output of the second lookup.

Note, you don’t really need that first lookup – you could accomplish the same thing with a single lookup in a partial cache mode. But if you’re processing a good number of rows, and a large number of your keys already exist, the first lookup will improve your overall performance.

Make sure that you do not enable the Miss Cache ("Enable cache for rows with no matching entries" on the advanced options page). If you do, the partial cache won't go to the database the next time the key value comes in again.

Lookup Pattern: Upsert

mmasson — Sun, 23 Nov 2008 07:31:46 GMT

This is a pretty basic pattern where we use a lookup to determine whether we need to update and existing row, or insert a new one. The lookup checks if a key or set of values exists. If the key isn't found, the row is sent to an OLEDB Destination for the insert. If it is found, it is sent to an OLEDB Command to do the update.

Note, the OLEDB Command transform operates on a row by row basis - so a separate SQL statement will be executed for every row going in. As such, the OLEDB Command can be very slow if you're processing a large number of rows. An alternate approach is to stage the data, and either update your target table using the MERGE statement, or an UPDATE ... FROM batch command.

You can also check out the MERGE Destination or Batch Destination available on Codeplex. John Welch (author of the Batch Destination) has a blog post which compares the two.

Lookup Pattern: Key Generation

mmasson — Sun, 23 Nov 2008 07:28:54 GMT

This pattern is used when you have transformation logic which relies on a key which might not already exist. If the lookup fails to find the key, a new key is generated with a script task so it can be used later on downstream. Optionally, the key could be inserted immediately into the reference table following the script task (multicast to send to an OLEDB Destination).

The way you generate the key will vary depending on the situation. If you don't need to worry about concurrency issues, you could use an Execute SQL Task in the control flow to retrieve the next or current maximum key value, and store it in a variable. You’d then increment it each time you go through the key generation process.

Lookup Patterns

mmasson — Sun, 23 Nov 2008 07:28:29 GMT

From the Lookup presentation I put together for the MS BI conference in October, here is a series of posts which describe different patterns for using the Lookup transform.

Each pattern will contain a diagram with Upstream and Downstream images.

These represent any collection of source / transform / destination components that could appear before/after the Lookup section described in the pattern.

Lookup - Using the cache connection manager

mmasson — Sun, 23 Nov 2008 06:51:00 GMT

Using the Cache Connection Manager (CCM) is a new option for the Lookup transform in SSIS 2008. The CCM provides an alternative to doing lookups against a database table using an OLEDB connection.

This post will suggest some best practices for using the Cache Connection Manager, and illustrate a couple of common scenarios that the CCM was meant to handle.

For more information on the Cache Connection Manager and how to make use of it with the Lookup Transform, please see the books online entries linked at the end of this post.

Best practices

Reuse the cache to reduce database load
Share the cache between lookups to reduce memory usage
Using the CCM is not always faster than OLEDB - the cost of disk access can out weight the benefits of pre-creating the cache
The cache is essentially clear text - do not store sensitive data inside of the cache
In terms of Cache Modes and the best practices that surround them, using a cache connection manager is equivalent to using a Full Cache mode

Reducing database and memory usage

If your reference database is remote, or under heavy load, consider using the Cache Connection Manager instead of an OLEDB connection.

Once a cache is used (or created) in an SSIS package, it will be kept in memory until the package has finished executing. The cache can be reused across multiple data flows, and shared between multiple lookups in the same data flow. It can also be persisted to disk, and reused across package executions.

Create the cache from any SSIS data source

To use the CCM, you need to create a lookup cache in a separate data flow using the Cache Transformation. Because the cache is created in a regular data flow, this means that you can now use any data source that SSIS can connect to as a source for your lookup reference (flat file, excel, SAP, etc).

With SSIS 2005, a common approach when using non-OLEDB accessible lookup sources was to stage the data first. If this data is only being used for your lookups, consider creating a persisted cache instead.

Cache the most common values

Sometimes you might have a large reference table, but the majority of your incoming data only uses a small portion of it. For example, you have a very large custom list, and the top 5% of your customers generate 90% of your transactions. In a scenario like this, you could pre-cache the information of your most active customers. Your data flow could use a cascading lookup pattern where you have one lookup which uses the cache, with its No Match output falling through to a second lookup running in a partial cache mode to hit the database to handle the remaining 10% of rows.

Resources

Calculating the size of your Lookup cache

mmasson — Sun, 19 Oct 2008 02:01:15 GMT

Good news - a couple of Information log events where added to the Lookup Transform in SQL 2008 to help you better understand your lookup cache. When you're running in Full Cache mode, the message will tell you the number of rows in the cache, its total size, and how long it took to create it. When running in partial cache mode you don't get the cache size, but you do get the number of database hits vs. number of cache hits, which can be helpful in determining whether you should be using a full cache lookup instead.

Here are some equations you can use to estimate the amount of memory a cache will use (I say estimate because given memory boundaries, pages, etc, it will always vary). There are separate (but similar) equations for Full and Partial cache modes, as they handle things a little differently internally.

For each row, in bytes:

Full cache

<Row size> + 20 + (4 * # of used columns)

Partial cache

<Row size> + 36 + (4 * # of columns in reference query)

Row size is the total data length of the index and values columns.

The 20/36 number is a constant representing the size of the hash used for comparisons.

The last part is 4 (technically, the size of an int) times the number of used columns for full cache, and total number of columns in the reference query for partial cache.

Example

Our lookup query is:

select ProductKey, ProductName from [Products]

ProductKey is an int - 4 bytes

ProductName is an nvarchar(25) - which comes out to 52 bytes (two bytes per nchar, + 2 for null)

Plugging this into our Partial cache equation, we get:

56 + 36 + 8 = 100 bytes per row

If our reference table had 100,000 rows, we'd need ~10mb to hold the entire data set.

(100,000 * 100) / 1024 / 1024 = ~10mb

(divide by 1024 twice to go from bytes to kilobytes, kilobytes to megabytes)

In my example, I've edited my lookup query to contain only the columns I need - ProductKey, which is my index column (the column I'm matching on), and ProductName, which is my value column (the column I'm adding to my data flow). If I had done a select * (or picked a table/view name from the drop down in the UI, which results in a select *), I'd be increasing the number of columns in my reference query, and my partial cache row size would change. I'd end up with an additional 4 bytes for every column in my query, even though they aren't being used directly in the lookup.

Lookup cache modes

mmasson — Sat, 18 Oct 2008 23:54:16 GMT

Over the past couple of months I've been putting together a presentation on the Lookup Transform. I presented most of it as a Chalk Talk at the MS BI Conference last week, and from the evaluation scores, it seems like it was pretty well received. I'll be splitting up some of its content into a series of blog posts over the next little while. If you're interested in seeing the whole talk, it will also be shown at the SSWUG Virtual Conference in November.

----

The most important setting of the Lookup Transform is the Cache Mode - it can greatly impact your data flow performance, and affects overall package design. Because of its importance, we made it the first thing you see in the new 2008 Lookup UI. I feel this is a great improvement over 2005, where the cache mode was abstracted away - see Michael Entin's post for more details.

This blog post describes the three cache modes, how they work, and best practices around using them. Note that these cache modes apply when you're using an OLE DB connection manager - using the new Cache connection manager is similar to using a Full Cache mode.

Full Cache

The default cache mode for the lookup is Full cache. In this mode, the database is queried once during the pre-execute phase of the data flow. The entire reference set is pulled into memory. This approach uses the most memory, and adds additional startup time for your data flow, as all of the caching takes place before any rows are read from the data flow source(s). The trade off is that the lookup operations will be very fast during execution. One thing to note is that the lookup will not swap memory out to disk, so your data flow will fail if you run out of memory.

When to use this cache mode

When you're accessing a large portion of your reference set
When you have a small reference table
When your database is remote or under heavy load, and you want to reduce the number of queries sent to the server

Keys to using this cache mode

Ensure that you have enough memory to fit your cache
Ensure that you don't need to pick up any changes made to the reference table
- Since the lookup query is executed before the data flow begins, any changes made to the reference table during the data flow execution will not be reflected in the cache

Partial Cache

In this mode, the lookup cache starts off empty at the beginning of the data flow. When a new row comes in, the lookup transform checks its cache for the matching values. If no match is found, it queries the database. If the match is found at the database, the values are cached so they can be used the next time a matching row comes in.

Since no caching is done during the pre-execute phase, the startup time using a partial cache mode is less than it would be for a full cache. However, your lookup operations would be slower, as you will most likely be hitting the database more often.

When running in partial cache mode, you can configure the maximum size of the cache. This setting can be found on the Advanced Options page of the lookup UI. There are actually two separate values - one for 32bit execution, and one for 64bit. If the cache gets filled up, the lookup transform will start dropping the least seen rows from the cache to make room for the new ones.

In 2008 there is a new Miss Cache feature that allows you to allocate a certain percentage of your cache to remembering rows that had no match in the database. This is useful in a lot of situations, as it prevents the transform from querying the database multiple times for values that don't exist. However, there are cases where you don't want to remember the misses - for example, if your data flow is adding new rows to your reference table. The Miss Cache is disabled by default.

When to use this cache mode

When you're processing a small number of rows and it's not worth the time to charge the full cache
When you have a large reference table
When your data flow is adding new rows to your reference table
When you want to limit the size of your reference table by modifying query with parameters from the data flow

Keys to using this cache mode

Ensure that your cache size setting is large enough
Use the Miss Cache appropriately
If the cache size isn't large enough for your rows, sort on lookup index columns if possible

No Cache

As the name implies, in this mode the lookup transform doesn't maintain a lookup cache (actually, not quite true - we keep the last match around, as the memory has already been allocated). In most situations, this means that you'll be hitting the database for every row.

When to use this cache mode

When you're processing a small number of rows
When you have non-repeating lookup indexes
When your reference table is changing (inserts, updates, deletes)
When you have severe memory limitations

Keys to using this cache mode

Ensure that the partial cache mode isn't the better choice

---

To find out more on how to implement the look up transform, please see these books online entries:

Presenting at the Microsoft BI Conference

mmasson — Thu, 02 Oct 2008 00:27:14 GMT

On Monday (October 6th) I’ll be doing a chalk talk presentation at the MS BI Conference. The topic is Advanced Scenarios with the Lookup Transform. Here is the abstract:

Performing lookups is one of the most common operations in the ETL process, and doing them incorrectly can severely affect the performance of your data load. In this talk you’ll learn best practices and design patterns for using the lookup component in SQL Server 2008 Integration Services, and how to take advantage of the new lookup features. If you’ve ever wondered about the differences between full and partial caches, the advantages of cascading lookups, and how to do ranged lookups, you won’t want to miss this talk.

I’ll also be hanging out at the SSIS / SQL BI booths throughout the day. Come by and say hello!

Enum value for Lookup’s NoMatchBehavior property

mmasson — Sat, 23 Aug 2008 02:17:37 GMT

The lookup transform has a new property in SQL 2008 which controls how to handle rows with no matches – NoMatchBehavior. It has two values - “Treat rows with no matching entries as errors” and “Send rows with no matching entries to the no match output”.

From the BOL entry:

When the property is set to Treat rows with no matching entries as errors, the rows without matching entries are treated as errors. You can specify what should happen when this type of error occurs by using the Error Output page of the Lookup Transformation Editor dialog box.

When the property is set to Send rows with no matching entries to the no match output, the rows are not treaded as errors.

In the properties window, this shows up as a drop down list. However, it looks like we forgot to document the actual enumeration values you would use if you are setting the property programmatically.

The enum looks like this:

public enum NoMatchPropertyEnum : int
{
    TreatAsError = 0,
    SendToNoMatchOutput = 1
}

This info should appear in the BOL docs the next time they are updated.

SSWUG Business Intelligence Virtual Conference

mmasson — Thu, 31 Jul 2008 00:00:30 GMT

I’ve been invited to speak at the SSWUG BI Virtual Conference in September. Like John Welch mentioned in his blog, the current speaker lineup is very impressive. I’m honored (and a little intimidated) to be on the presenters list! I’ll be flying out to Tucson in early September to record three SSIS related sessions:

What’s new in SSIS in SQL Server 2008
Beyond Scripting – Developing reusable extensions for SSIS
Advanced lookup scenarios in Integration Services with SQL Server 2008

The conference will take place online September 24th – 26th. The session list hasn’t been posted yet, but judging by the list of speakers, I’m sure it will be interesting.

Lookups with Visual FoxPro

mmasson — Tue, 04 Mar 2008 08:20:23 GMT

Here's a quick tip from a couple other members of the SSIS team, Da Lin and David Noor.

If you have a lookup component using partial cache mode using a Visual FoxPro oledb provider to connect to your reference table, be sure to change the OLE DB Services setting in the connection manager to something other than "default" or "enable all".

If you don't change this setting, you'll get the following error at runtime:

An OLE DB record is available. Source: "Microsoft OLE DB Provider for Visual FoxPro" Hresult: 0x80040E46 Description: "One or more accessor flags were invalid."