Michael Entin's notebook : Katmai

SQL 2008 & VS 2008

michen — Wed, 05 Dec 2007 09:42:00 GMT

Currently SQL Business Intelligence Development Studio (BIDS) and all the project types (AS, IS and RS) live in Visual Studio 2005. So don't try to open a solution that contains IS project in VS 2008 yet. What about final SQL 2008 - now that Visual Studio 2008 is released - what are the plans for BIDS and support of BI projects VS 2008?...(read more)

SQL 2008 November CTP is available

michen — Mon, 19 Nov 2007 20:56:00 GMT

Get it from https://connect.microsoft.com/SQLServer

One of the improvements in this build is the ability to persist Lookup reference data and use non-OLEDB sources for Lookup.

Custom transforms: determining end of rowset

michen — Fri, 31 Aug 2007 21:41:00 GMT

With SQL 2008 CTPs starting to get into the hands of customers and ISVs, we've found that some custom transforms and script transforms use incorrect code to check for the end of rowset (incoming data), which could cause problems when you move the code to SQL 2008. Unfortunately, some Microsoft docs helped to spead this error, so let me clarify what I mean and the correct usage pattern.

Most often the EndOfRowset property of SSIS buffer is being often used incorrectly. The property exists on PipelineBuffer object and ScriptBuffer object, so this applies to custom transforms and script transforms.

The buffer.EndOfRowset property is an indicator that the current buffer is the very last (for particular input), and the component will not see any more buffers (on this input).

Note that:
1) Value of this property does not depend on whether you’ve iterated over rows in that buffer,
2) The value should not be used to check whether the buffer contains any rows.

The last point is the most important here. In SQL 2005, the data flow engine sends an additional empty buffer with EndOfRowset indicator at the end. So the buffer [in SQL 2005] either contains some rows, or contains EndOfRowset indicator. But don't rely on either of this.

In SQL 2008, this dummy empty buffer was deemed too expensive from performance point of view, and the data flow engine would not send an additional empty buffer anymore. Instead it simply sets EndOfRowset flag on the last buffer. Thus you get a series of buffers that contain the data, and at the very end a buffer that contains both the data and the EndOfRowset indicator.

In some samples and forum posts I've seen incorrect usage of EndOfRowset property that relies on SQL 2005 behavior (exclusive OR: either end-of-rowset or some data), i.e. the code does not check the incoming rows if it gets end of rowset. In other cases the code assumes it should receive empty buffer, and uses row count as end-of-rowset indicator. If you have a code that uses incorrect patterns, e.g.:

/* warning - incorrect code */
if (!buffer.EndOfRowset)
{
  while(buffer.NextRow())
  {
    // do something with the row
  }
}
else
{
  // do something at the end
}

/* warning - another incorrect code */
if (buffer.RowCount != 0)
{
  while(buffer.NextRow())
  {
    // do something with the row
  }
}
else
{
  // do something at the end
}

Please change it to:

while (buffer.NextRow())
{
  // do something with the row
}

if (buffer.EndOfRowset)
{
  // do something at the end
}

Otherwise you risk missing contents of last buffer in SQL 2008 (first incorrect sample) or fail to detect end of data (second incorrect sample). The last version works correctly both in SQL 2005 and in SQL 2008.

Update: see this followup blog
http://blogs.msdn.com/michen/archive/2008/10/19/does-buffer-nextrow-skips-the-first-row-in-a-buffer.aspx

SSIS Backpressure Mechanism

michen — Wed, 13 Jun 2007 05:28:00 GMT

One of the mechanisms that SSIS data flow engine utilizes to achieve high performance is “back pressure”.

Let’s consider a simple package with a source and destination. What happens if the source is fast and destination is slow? Say source is huge local raw file and destination is a remote relational database. If we read the source data with full speed, we would need a lot of memory to store all the data we’ve read, but could not write yet. In many cases we’d simply run out of memory or had to swap data to hard drive.

To avoid this, SSIS limits the speed of sources by controlling the number of active buffers inside each execution tree. This way you can process fast sources without running out of memory.

If a source or an async component is too fast (compared to the the transformations or destination down the path), the source is suspended when its execution tree gets too many buffers (currently this is fixed at 5 buffers). If the source is slow (i.e. transforms and destinations can process data faster than sources generate it), the back-pressure mechanism does not get involved and sources can run with full speed.

Unfortunately, in SQL 2005 we did not have any diagnostics that would tell user what is the slow part of the flow - the source or the transforms/destination. In Katmai (SQL 2008) if the back-pressure kicks in during package execution, at the end of package execution we report the total time that the source had to wait because of this mechanism.

What would you do with this information? If you run the package and see that the time a particular source was suspended is zero or relatively low (compared to the total execution time), you know the source is the slowest part of the data flow, and you need to focus on optimizing the source. E.g. you may remove unused columns from the query, simplify SQL statements, create indexes, etc.

But if a source reported that it has been suspected most of the time (considerable part of the total package execution time), you know the source is fast enough and you need to concentrate on performance of transforms and destinations.

Another implication of back pressure is that the source database connection can be opened longer than it is needed to execute the query otherwise. If you see that a source has been suspended for several minites, and you don't want to hold the appropriate database connection for such a long time, consider staging source data in raw files as suggested by Jamie Thomson at
http://blogs.conchango.com/jamiethomson/archive/2006/12/12/SSIS_3A00_-Dropping-data-into-a-raw-file.aspx

Katmai SSIS data flow task improvements

michen — Mon, 11 Jun 2007 10:00:00 GMT

With first Katmai (SQL Server 2008) CTP out, I think it is time to blog about some performance and scalability improvements in this release.

I'll assume readers are familiar with SSIS data flow performance concepts, if not make sure you've read these two articles:
http://www.microsoft.com/technet/prodtechnol/sql/2005/ssisperf.mspx
http://www.simple-talk.com/sql/sql-server-2005/sql-server-2005-ssis-tuning-the-dataflow-task/

The most important concepts for today are:
Asynchronous components - components that create new rows and thus new data flow buffers (compare to synchronous components that modify data in the incoming buffers, but can't create or remove rows). Both blocking and partially blocking components (in terms of ssisperf whitepaper linked above) are asynchronous.
Execution trees - the tree describing the path a data buffer takes through the components, from start to end. The tree starts with a source or async component and usually ends with a destination or sometimes with a transform. It is a tree (not a simple list) because some transforms, e.g. Multicast "split" the data flow into multiple logical paths, but avoid buffer duplication - all sub-trees use the same "physical" buffer.

In SQL 2005 each execution tree was assigned a single OS thread ("worker thread"), and under some conditions (complex packages with small data flow EngineThreads property value) several execution trees could share a thread. One benefit of this approach is that all thread scheduling was done in advance during pre-execution phase, so the data flow did not have to spend any time to assign threads at runtime.

But there were several drawbacks: since the data flow did not know relative amount of work per execution tree, the scheduling could be suboptimal. Also, if you had simple package with just one or two execution trees, you would only use one or two processors, and the package might not benefit from high-end multiprocessor machine. You may have a lot of synchronous components, but if they share the same execution tree, they used to share the thread too. Even if you logically split the data flow using Multicast (synchronous transform), all output paths of Multicast belong to the same execution tree, and thus are executed serially by Yukon data flow task.

To achieve high level of parallelism on a multiprocessor machine, you had to split an execution tree, either by splitting the tree into independent paths, or by inserting async transforms to create a new tree. UnionAll could be used for later, as described in the ssisperf whitepaper. If you insert UnionAll with one input, it does not change the outputs, but splits the execution tree into two new trees - each can be executed on its own processor. The drawback is that the UnionAll is async transform, and thus has to copy data, so it might have a noticeable performance overhead. You should only use this trick if you checked that your package benefits from this extra parallelism. Usually packages already have multiple data flows, or multiple execution trees inside single data flow, so they don't need to do this. But if you have a very high-end machine and a pipeline that takes too long and does not use all the processors, you should try it (make a backup copy of original package and compare the performance before commiting to this change).

Now the good news: you don't need to worry about any of this or use the UnionAll trick anymore! In Katmai, the data flow was redesigned to do dynamic scheduling and can now execute multiple components in parallel, even if they belong to the same execution tree. The overhead of dynamic scheduling is very small. We sometimes saw a very small (~1%) performance loss on single processor machines, but on multiprocessor machines you'll usually see performance improvement, especially if you had to use UnionAll trick to introduce more parallelism and remove it when you move to Katmai.

As most server machines now have two or more processors or at least Hyper Threading, we think most users will see the performance improvements from this change. And you get the best performance automatically, no need to think if you need to introduce more parallelism by adding UnionAll transform, one less bullet to worry about.