Craig Freedman's SQL Server Blog : Parallelism

Partial Aggregation

craigfr — Fri, 18 Jan 2008 22:12:00 GMT

In some of my past posts, I've discussed how SQL Server implements aggregation including the stream aggregate and hash aggregate operators. I also used hash aggregation as an initial example in my introductory post on parallel query execution. In this post, I'll look at a partial aggregation. Partial aggregation is a technique that SQL Server uses to optimize parallel aggregation. Before I begin, I just want to note that I also discuss partial aggregation in Inside Microsoft SQL Server 2005: Query Tuning and Optimization. (See the bottom of page 187.)

Let's begin with a simple scalar aggregation example. Recall that a scalar aggregate is an aggregate without a GROUP BY clause. A scalar aggregate always produces a single output row.

CREATE TABLE T (A INT, B INT IDENTITY, C INT, D INT)
CREATE CLUSTERED INDEX TA ON T(A)

SELECT COUNT(*) FROM T

Not surprisingly, this query yields a trivial stream aggregate plan:

|--Compute Scalar(DEFINE:([Expr1004]=CONVERT_IMPLICIT(int,[Expr1005],0)))
|--Stream Aggregate(DEFINE:([Expr1005]=Count(*)))
|--Clustered Index Scan(OBJECT:([T].[TA]))

Now, suppose that we want to parallelize this query. Because this query outputs a single row, we cannot simply put an exchange (i.e., parallelism operator) at the root of the plan and divide the work of counting among multiple threads. Such a strategy would yield one output row per thread which is clearly not the correct result.

Alternatively, we could put a gather streams exchange between the stream aggregate and clustered index scan operators. This strategy would permit us to use a parallel scan while still counting in a single thread and outputting a single row. However, we would end up moving every row from the scan through the exchange - a rather costly operation. Thus, this option, while valid, would not yield nearly the performance we'd like to see.

Fortunately, there is a third option. We can use a parallel scan, divide the work of counting among multiple threads (as in the first option), use an exchange to gather the per-thread counts into a single thread, and finally sum the per-thread counts to generate the grand total. This strategy is much more efficient as we need only move a single row per thread through the exchange. To get the optimizer to generate this plan, we need to add lots of data to the table. To save time, I'm going to use UPDATE STATISTICS to trick the optimizer into thinking that we've added rows to the table:

UPDATE STATISTICS T WITH ROWCOUNT = 1000000, PAGECOUNT = 100000
SELECT COUNT(*) FROM T OPTION (RECOMPILE)

We need the RECOMPILE query hint to force the optimizer to generate a new plan with the new statistics. Here is the plan we get:

|--Compute Scalar(DEFINE:([Expr1004]=CONVERT_IMPLICIT(int,[globalagg1006],0)))
       |--Stream Aggregate(DEFINE:([globalagg1006]=SUM([partialagg1005])))
            |--Parallelism(Gather Streams)
                 |--Stream Aggregate(DEFINE:([partialagg1005]=Count(*)))
                      |--Clustered Index Scan(OBJECT:([T].[TA]))

We call the bottommost aggregate operator a partial aggregate because it computes only part of the result. We also sometimes refer to this operator as a local aggregate because it computes the portion of the result that is local to the thread where it executes. We refer to the topmost aggregate as a global aggregate because it computes the full result.

SQL Server is able to use partial aggregation for most aggregate functions including the standard built-ins: COUNT, SUM, AVG, MIN, and MAX. While partial aggregation is necessary to parallelize scalar aggregates, it is also useful even for aggregates with a GROUP BY clause. Whether the optimizer chooses to use partial aggregation depends on the number of unique groups and the size of these groups. If the optimizer anticipates that a query will generate few large groups (such as in the scalar aggregation case), it will use partial aggregation. However, if the optimizer expects that a query will generate many small groups, it may choose to use a single level aggregation. With small groups, a partial aggregate cannot reduce the number of rows significantly and merely adds overhead to the query. Moreover, with many groups it is easy to parallelize the aggregation by hashing on the GROUP BY keys and distributing different groups to different threads.

Let's see how the optimizer makes this choice. Column B in our example has the IDENTITY property. Although we have no real data, this property is sufficient to trick the optimizer into concluding that this column is mostly unique. (Without a unique index, the optimizer cannot be certain that the column is indeed unique and must assume that it is not.) Suppose we aggregate on this column:

SELECT COUNT(*) FROM T GROUP BY B

|--Parallelism(Gather Streams)
       |--Compute Scalar(DEFINE:([Expr1004]=CONVERT_IMPLICIT(int,[Expr1007],0)))
            |--Hash Match(Aggregate, HASH:([T].[B]) DEFINE:([Expr1007]=COUNT(*)))
                 |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([T].[B]))
                      |--Clustered Index Scan(OBJECT:([T].[TA]))

Notice that this query yields a normal single level, albeit parallel, aggregate operator. Now suppose we aggregate on column C which does not have the IDENTITY property:

SELECT COUNT(*) FROM T GROUP BY C

|--Compute Scalar(DEFINE:([Expr1004]=CONVERT_IMPLICIT(int,[globalagg1006],0)))
       |--Parallelism(Gather Streams)
            |--Stream Aggregate(GROUP BY:([T].[C]) DEFINE:([globalagg1006]=SUM([partialagg1005])))
                 |--Sort(ORDER BY:([T].[C] ASC))
                      |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([T].[C]))
                           |--Hash Match(Partial Aggregate, HASH:([T].[C]), RESIDUAL:([T].[C] = [T].[C]) DEFINE:([partialagg1005]=COUNT(*)))
                                |--Clustered Index Scan(OBJECT:([T].[TA]))

This time we get a partial aggregate. Also, observe that the partial aggregate is a hash aggregate while the global aggregate is a stream aggregate. The optimizer is free to choose either physical aggregation operator (stream or hash) for the partial and global aggregates in a plan with partial aggregation. The decision of which operator to use is cost based.

Finally, it is worth noting that, while a stream aggregate behaves identically whether it is computing a partial or global aggregate, a partial hash aggregate does differ slightly from a normal hash aggregate. First, a partial hash aggregate requests only a fixed minimal memory grant as it presumes that it will be computing a relatively small number of groups. Second, a partial hash aggregate never spills rows to tempdb. If a partial hash aggregate runs out of memory, it simply stops aggregating and begins returning non-aggregated rows. This behavior is safe since the global aggregate will always compute the correct final results regardless of what the partial aggregate does. The partial aggregate is merely a performance optimization and the goal is to prevent it from stealing resources from the global aggregate.

Parallel Query Execution Presentation

craigfr — Wed, 18 Apr 2007 01:47:00 GMT

For those of you readers who've been wondering whatever happened to me, I've been rather busy. Among other activities, I've been writing a chapter for Kalen Delaney's upcoming fourth book in the Inside SQL Server 2005 series: Query Tuning and Optimization.

I am hoping to resurrect my blog and to get things started, I'm posting a presentation that I recently delivered to the Pacific Northwest SQL Server User's Group. Several attendees asked whether they could have copies of this presentation, so here it is.

Parallel Hash Join

craigfr — Fri, 17 Nov 2006 07:08:00 GMT

SQL Server uses one of two different strategies to parallelize a hash join. The more common strategy uses hash partitioning. In some cases, we use broadcast partitioning; this strategy is often called a “broadcast hash join.”

Hash Partitioning

The more common strategy for parallelizing a hash join involves distributing the build rows (i.e., the rows from the first input) and the probe rows (i.e., the rows from the second input) among the individual hash join threads using hash partitioning. If a build and probe row share the same key value (i.e, they will join), they are guaranteed to hash to the same hash join thread. After the data has been hash partitioned among the threads, the hash join instances all run completely independently on their respective data sets. The absence of any inter-thread dependencies ensures that this strategy scales extremely well as we increase the degree of parallelism (i.e., the number of threads).

As with all of my parallelism examples, I am using a large table to induce the optimizer to choose a parallel plan. If you try these examples, it may take a few minutes to create these tables.

create table T1 (a int, b int, x char(200))

set nocount on

declare @i int

set @i = 0

while @i < 1000000

begin

insert T1 values(@i, @i, @i)

set @i = @i + 1

end

select * into T2 from T1

select * into T3 from T1

select * from T1 join T2 on T1.b = T2.a

|--Parallelism(Gather Streams)
       |--Hash Match(Inner Join, HASH:([T1].[b])=([T2].[a]), RESIDUAL:([T2].[a]=[T1].[b]))
            |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([T1].[b]))
            |    |--Table Scan(OBJECT:([T1]))
            |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([T2].[a]))
                 |--Table Scan(OBJECT:([T2]))

Note that unlike the parallel nested loops join, we have exchanges (i.e., parallelism operator) between both table scans (both the build and the probe inputs) and the hash join. These exchanges hash partition the data among the hash join threads.

Broadcast Partitioning

Consider what will happen if we try to parallelize a hash join using hash partitioning, but we have only a small number of rows on the build side of the hash join. If we have fewer rows than hash join threads, some threads might receive no rows at all. In this case, those threads would have no work to do during the probe phase of the join and would remain idle. Even if we have more rows than threads, due to the presence of duplicate key values and/or skew in the hash function, some threads might get many more rows than others.

To eliminate the risk of skew, when the optimizer estimates that the number of build rows is relatively small, it may choose to broadcast these rows to all of the hash join threads. Since all build rows are broadcast to all hash join threads, in a broadcast hash join, it does not matter where we send the probe rows. Each probe row can be sent to any thread and, if it can join with any build rows, it will.

Here is an example:

select * from T1 join T2 on T1.b = T2.a where T1.a = 0

|--Parallelism(Gather Streams)
       |--Hash Match(Inner Join, HASH:([T1].[b])=([T2].[a]), RESIDUAL:([T2].[a]=[T1].[b]))
            |--Parallelism(Distribute Streams, Broadcast Partitioning)
            |    |--Table Scan(OBJECT:([T1]), WHERE:([T1].[a]=(0)))
            |--Table Scan(OBJECT:([T2]))

Note that the exchange above the scan of T1 is now a broadcast exchange while we have completely eliminated the exchange above the scan of T2. We do not need an exchange above T2 because the parallel scan automatically distributes the pages and rows of T2 among the hash join threads. This is similar to how the parallel scan distributed rows among nested loops join threads for the parallel nested loops join. Similar to the parallel nested loops join, if we have a serial zone on the probe input of a broadcast hash join (e.g., due to a top operator), we may need a round robin exchange to redistribute the rows.

So, if broadcast hash joins are so great (they do reduce the risk of skew problems), why don’t we use them in all cases? The answer is that broadcast hash joins use more memory than their hash partitioned counterparts. Since we send every build row to every hash join thread, if we double the number of threads, we double the amount of memory that we need. With a hash partitioned parallel hash join, we need the same amount of memory regardless of the degree of parallelism.

Bitmap Filtering

select * from T1 join T2 on T1.b = T2.a where T1.a < 100000

Finally, suppose we have a fairly selective filter on the build input to a hash join:

|--Parallelism(Gather Streams)
       |--Hash Match(Inner Join, HASH:([T1].[b])=([T2].[a]), RESIDUAL:([T2].[a]=[T1].[b]))
            |--Bitmap(HASH:([T1].[b]), DEFINE:([Bitmap1008]))
            |    |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([T1].[b]))
            |         |--Table Scan(OBJECT:([T1]), WHERE:([T1].[a]<(100000)))
            |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([T2].[a]), WHERE:(PROBE([Bitmap1008])=TRUE))
                 |--Table Scan(OBJECT:([T2]))

What’s the bitmap operator? The predicate “T1.a < 100000” eliminates 90% of the build rows from T1. It also indirectly eliminates 90% of the rows from T2 because they no longer join with rows from T1. The bitmap operator provides an efficient way to apply the T1 filter directly to T2 without passing the rows all the way through the exchange to the join. As its name suggests, it builds a bitmap. Just like a hash join, we hash each row of T1 on the join key T1.b and set the corresponding bit in the bitmap. Once the scan of T1 and the hash join build is complete, we transfer the bitmap to the exchange above the scan of T2 where we use it as a filter. This time we hash each row of T2 on the join key T2.a and test the corresponding bit in the bitmap. If the bit is set, the row may join and we pass it along to the hash join. If the bit is not set, the row cannot join and we discard it. For more information on bitmaps see this post from the SQL Server Query Processing Team blog.

Parallel Nested Loops Join

craigfr — Thu, 09 Nov 2006 06:04:00 GMT

SQL Server parallelizes a nested loops join by distributing the outer rows (i.e., the rows from the first input) randomly among the nested loops threads. For example, if we have two threads running a nested loops join, we send about half of the rows to each thread. Each thread then runs the inner side (i.e., the second input) of the loop join for its set of rows as if it were running serially. That is, for each outer row assigned to it, the thread executes its inner input using that row as the source of any correlated parameters. In this way, the threads can run independently. SQL Server does not add exchanges to or parallelize the inner side of a nested loops join.

A simple example

Let’s consider a simple example. I am using a large table to induce the optimizer to choose a parallel plan. If you try these examples, it may take a few minutes to create these tables.

create table T1 (a int, b int, x char(200))

set nocount on

declare @i int

set @i = 0

while @i < 1000000

begin

insert T1 values(@i, @i, @i)

set @i = @i + 1

end

select * into T2 from T1

select * into T3 from T1

create unique clustered index T2a on T2(a)

create unique clustered index T3a on T3(a)

select * from T1 join T2 on T1.b = T2.a where T1.a < 100

Rows	Executes
100	1	\|--Parallelism(Gather Streams)
100	2	\|--Nested Loops(Inner Join, OUTER REFERENCES:([T1].[b], [Expr1007]) OPTIMIZED)
100	2	\|--Table Scan(OBJECT:([T1]), WHERE:([T1].[a]<(100)))
100	100	\|--Clustered Index Seek(OBJECT:([T2].[T2a]), SEEK:([T2].[a]=[T1].[b]) ORDERED FORWARD)

First note that there is only one exchange (i.e., parallelism operator) in this plan. Since this exchange is at the root of the query plan, all of the operators in this plan (the nested loops join, the table scan, and the clustered index seek) execute in each thread.

I deliberately did not create an index on T1. The lack of an index forces the table scan and the use of a residual predicate to evaluate “T1.a < 100”. Because there are one million rows in T1, the table scan is expensive and the optimizer chooses a parallel scan of T1.

The scan of T1 is not immediately below an exchange (i.e., a parallelism operator). In fact, it is on the outer side of the nested loops join and it is the nested loops join that is below the exchange. Nevertheless, because the scan is on the outer side of the join and because the join is below a start (i.e., a gather or redistribute) exchange, we use a parallel scan for T1.

If you recall my post from last week, a parallel scan assigns pages to threads dynamically. Thus, the scan distributes the T1 rows among the threads. It does not mater which rows it distributes to which threads.

Since I ran this query with degree of parallelism (DOP) two, we see that there are two executes each for the table scan and join (which are in the same thread). Moreover, the scan and join both return a total of 100 rows though we cannot tell from this output how many rows each thread returned. (We could determine this information using statistics XML output. I'll return to this question below.)

Next, the join executes its inner side (in this case the index seek on T2) for each of the 100 outer rows. Here is where things get a little tricky. Although each of the two threads contains an instance of the index seek and even though the index seek is below the join which is below the exchange, the index seek is on the inner side of the join and, thus, the seek does not use a parallel scan. Instead, the two seek instances execute independently of one another on two different outer rows and two different correlated parameters. Just as in a serial plan, we see 100 executes of the index seek: one for each row on the outer side of the join. No matter how complex the inner side of a nested loops join is, we always execute it as a serial plan just as in this simple example.

A more complex example

In the above example, SQL Server relies of the parallel scan to distribute rows uniformly among the threads. In some cases, this is not possible. In these cases, SQL Server may add a round robin exchange to distribute the rows. (A round robin exchange sends each subsequent packet of rows to the next consumer thread in a fixed sequence.) Here is one such example:

select *

from (select top 100 * from T1 order by a) T1top

join T2 on T1top.b = T2.a

|--Parallelism(Gather Streams)
       |--Nested Loops(Inner Join, OUTER REFERENCES:([T1].[b], [Expr1007]) WITH UNORDERED PREFETCH)
            |--Parallelism(Distribute Streams, RoundRobin Partitioning)
            |    |--Top(TOP EXPRESSION:((100)))
            |         |--Parallelism(Gather Streams, ORDER BY:([T1].[a] ASC))
            |              |--Sort(TOP 100, ORDER BY:([T1].[a] ASC))
            |                   |--Table Scan(OBJECT:([T1]))
            |--Clustered Index Seek(OBJECT:([T2].[T2a]), SEEK:([T2].[a]=[T1].[b]) ORDERED FORWARD)

The main difference between this plan and the plan from the original example is that this plan includes a top 100. The top 100 can only be correctly evaluated in a serial plan thread. (It cannot be split among multiple threads or we might end up with too few or too many rows.) Thus, we must have a stop (e.g., a distribute streams) exchange above the top and we cannot use the parallel scan to distribute the rows among the join threads. Instead we parallelize the join, by having this exchange use round robin partitioning distribute the rows among the join threads.

Miscellaneous issues

The parallel scan has one major advantage over the round robin exchange. A parallel scan automatically and dynamically balances the workload among the threads while a round robin exchange does not. As I demonstrated last week, if we have query where one thread is slower than the others, the parallel scan may help compensate.

Both the parallel scan and a round robin exchange may fail to keep all join threads busy if there are too many threads and too few pages and/or rows to be processed. Some threads may get no rows to process and end up idle. This problem can be more pronounced with a parallel scan since it doles out multiple pages at one time to each thread while the exchange distributes one packet (equivalent to one page) of rows at one time.

We can see this problem in the original (simple) example above by checking the statistics XML output:

</RunTimeInformation>

...

</RelOp>

All of the join’s output rows where processed by thread 1. Why? The table scan has a residual predicate “T1.a < 100”. This predicate is true for the first 100 rows in the table and false for the remaining rows. The (three) pages containing the first 100 rows are all assigned to the first thread.

In this example, this is not a big problem since the inner side of the join is fairly cheap and contributes a small percentage of the overall cost of the query (with the table scan itself representing a much larger percentage of the cost). However, this problem could be more significant if the inner side of the query were more expensive. The problem is especially notable with partitioned tables. I will write about partitioned tables in a future post. In the meantime, this post from the SQL Server Development Customer Advisory Team blog illustrates the problem.

Parallel Scan

craigfr — Thu, 02 Nov 2006 07:16:00 GMT

In this post, I’m going to take a look at how SQL Server parallelizes scans. The scan operator is one of the few operators that is parallel “aware.” Most operators neither need to know nor care whether they are executing in parallel; the scan is an exception.

How does parallel scan work?

The threads that compose a parallel scan work together to scan all of the rows in a table. There is no a-priori assignment or rows or pages to a particular thread. Instead, the storage engine dynamically hands out pages to threads. A parallel page supplier coordinates access to the pages of the table. The parallel page supplier ensures that each page is assigned to exactly one thread and, thus, is processed exactly once.

At the beginning of a parallel scan, each thread requests a set of pages from the parallel page supplier. The threads then begin processing these pages and returning rows. When a thread finishes with its assigned set of pages, it requests the next set of pages from the parallel page supplier.

This algorithm has a couple of advantages:

It is independent of the number of threads. We can add and remove threads from a parallel scan and it automatically adjusts. If we double the number of threads, each thread processes (approximately) half as many pages. And, if the I/O system can keep up, the scan runs twice as fast.
It is resilient to skew or load imbalances. If one thread runs slower than the other threads, that thread simply requests fewer pages while the other faster threads pick up the extra work. The total execution time degrades smoothly. (Compare this scenario to what would happen if we statically assigned pages to threads: the slow thread would dominate the total execution time.)

Example

Let’s begin with a simple example. To get a parallel plan, we’ll need a fairly big table; if the table is too small, the optimizer will conclude that a serial plan is perfectly adequate. The following script creates a table with 1,000,000 rows and (thanks to the fixed length char(200) column) about 27,000 pages. Warning: If you decide to run this example, it could a few minutes to populate this table.

create table T (a int, x char(200))

set nocount on

declare @i int

set @i = 0

while @i < 1000000

begin

insert T values(@i, @i)

set @i = @i + 1

end

Now, for the simplest possible query:

select * from T

|--Table Scan(OBJECT:([T]))

We get a serial plan! Why don’t we get a parallel plan? Parallelism is really about speeding up queries by applying more CPUs to the problem. The cost of this query is dominated by the cost of reading pages from disk (which is mitigated by read ahead rather than parallelism) and returning rows to the client. The query uses relatively few CPU cycles and, in fact, would probably run slower if we parallelized it.

If we add a fairly selective predicate to the query, we can get a parallel plan:

select * from T where a < 1000

|--Parallelism(Gather Streams)
|--Table Scan(OBJECT:([T]), WHERE:([T].[a]<CONVERT_IMPLICIT(int,[@1],0)))

By running this query in parallel, we can distribute the cost of evaluating this predicate across multiple CPUs. (In this case, the predicate is so cheap that it probably does not make much difference whether or not we run in parallel.)

Load Balancing

As I mentioned above, the parallel scan algorithm dynamically allocates pages to threads. We can see this in action. Consider this query which returns every row of the table:

select * from T where a % 2 = 0 or a % 2 = 1

The peculiar predicate confuses the optimizer which underestimates the cardinality and generates a parallel plan:

|--Parallelism(Gather Streams)
|--Table Scan(OBJECT:([T]), WHERE:([T].[a]%(2)=(0) OR [T].[a]%(2)=(1)))

On SQL Server 2005, using “SET STATISTICS XML ON” we can see exactly how many rows each thread processes. Here is an excerpt of the XML output on a two processor system:

</RunTimeInformation>

...

</RelOp>

We can see that both threads (threads 1 and 2) processed approximately half of the rows. (Thread 0 is the coordinator or main thread. It only executes the portion of the query plan above the topmost exchange. Thus, we do not expect it to process any rows for any parallel operators.)

Now let’s repeat the experiment, but let’s run an expensive serial query at the same time. This cross join query will run for a really long time (it needs to process one trillion rows) and use plenty of CPU cycles:

select min(T1.a + T2.a) from T T1 cross join T T2 option(maxdop 1)

This serial query will consume cycles from only one of the two CPUs. While it is running, let’s run the other query again:

select * from T where a % 2 = 0 or a % 2 = 1

</RunTimeInformation>

...

</RelOp>

This time thread 1 processed more than 90% of the rows while thread 2 which was busy executing the above serial plan processed far fewer rows. The parallel scan automatically balanced the work across the two threads. Since thread 1 had more free cycles (it wasn’t competing with the serial plan), it requested and scanned more pages.

If you try this experiment, don’t forget to kill the serial query when you are done! Otherwise, it will continue to run and waste cycles for a really long time.

The same load balancing that we just observed applies equally whether a thread is slowed down because of an external factor (such as the serial query in this example) or because of an internal factor. For example, if it costs more to process some rows than others, we will see the same behavior.

The Parallelism Operator (aka Exchange)

craigfr — Wed, 25 Oct 2006 22:40:00 GMT

As I noted in my Introduction to Parallel Query Execution post, the parallelism (or exchange) iterator actually implements parallelism in query execution. The optimizer places exchanges at the boundaries between threads; the exchange moves the rows between the threads.

The iterator that’s really two iterators

The exchange iterator is unique in that it is really two iterators: a producer and a consumer. We place the producer at the root of a query sub-tree (often called a branch). The producer reads input rows from its sub-tree, assembles the rows into packets, and routes these packets the appropriate consumers. We place the consumer at the leaf of the next query sub-tree. The consumer receives packets from its producer(s), removes the rows from these packets, and returns the rows to its parent iterator. For example, a repartition exchange running at degree of parallelism (DOP) two, consists of two producers and two consumers:

Note that while the data flow between most iterators is pull based (an iterator calls GetRow on its child when it is ready for another row), the data flow in between an exchange producer and consumer is push based. That is, the producer fills a packet with rows and “pushes” it to the consumer. This model allows the producer and consumer threads to execute independently. (We do have flow control to prevent a fast producer from flooding a slow consumer with excessive packets.)

How many different types of exchanges are there?

Exchanges can be classified in three different ways.

First, we can classify exchanges based on the number of producer and/or consumer threads:

Type	# producer threads	# consumer threads
Gather Streams	DOP	1
Repartition Streams	DOP	DOP
Distribute Streams	1	DOP

A gather streams exchange is often called a “start parallelism” exchange since the operators above it run serially while the operators below it run in parallel. The root exchange in any parallel plan is always a gather exchange since the results of any query plan must ultimately be funneled back to the single connection thread to be returned to the client. A distribute streams exchange is often called a “stop parallelism” exchange. It is the opposite of a gather streams exchange: the operators above a distribute steams exchange run in parallel while the operators below it run serially.

Second, we can classify exchanges based on how they route rows from the producer to the consumer. We refer to this property as the “partitioning type” of the exchange. Partitioning type only makes sense for a repartition or a distribute streams exchange since there is only one way to route rows in a gather exchange: to the single consumer thread. SQL Server supports the following partitioning types:

Partitioning Type	Description
Broadcast	Send all rows to all consumer threads.
Round Robin	Send each packet of rows to the next consumer thread in sequence.
Hash	Determine where to send each row by evaluating a hash function on one or more columns in the row.
Range	Determine where to send each row by evaluating a range function on one column in the row. (A range function splits the total possible set of values into a set of continguous ranges. This partition type is rare and is used only by certain parallel index build plans.)
Demand	Send the next row to the next consumer that asks. This partition type is the only type of exchange that uses a pull rather a push model for data flow and is used only in query plans with partitioned tables.

Third, we can classify exchanges as merging (or order preserving) and non-merging (or non-order preserving). The consumer in a merging exchange ensures that rows from multiple producers are returned in a sorted order. (The rows must already be in this sorted order at the producer; the merging exchange does not actually sort.) A merging exchange only make sense for a gather or a repartition streams exchange; with a distribute streams exchange, there is only one producer and, thus, only one stream of rows and nothing to merge at each consumer.

Showplan

SQL Server includes all of the above properties in showplan (graphical, text, and XML).

Speaking of showplan, in graphical showplan you can also tell at a glance which operators are running in parallel (i.e., which operators are between a start exchange and a stop exchange) by looking for a little parallelism symbol on the operator icons:

In my next post about parallelism, I’ll begin to explore some parallel query plans and demonstrate the different types of exchanges in action.

Introduction to Parallel Query Execution

craigfr — Wed, 11 Oct 2006 21:11:00 GMT

SQL Server has the ability to execute queries using multiple CPUs simultaneously. We refer to this capability as parallel query execution. Parallel query execution can be used to reduce the response time of (i.e., speed up) a large query. It can also be used to a run a bigger query (one that processes more data) in about the same amount of time as a smaller query (i.e., scale up) by increasing the number of CPUs used in processing the query. For most large queries SQL Server generally scales linearly or nearly linearly. For speed up, this means that if we double the number of CPUs, we see the response time drop in half. For scale up, it means that if we double the number of CPUs and the size of the query, we see the same response time.

When is parallelism useful?

As I noted above parallelism can be used to reduce the response time of a single query. However, parallelism comes with a cost: it increases the overhead associated with executing a query. While this overhead is relatively small, it does make parallelism inappropriate for small queries (especially for OLTP queries) where the overhead would dominate the total execution time. Moreover, while SQL Server does generally scale linearly, if we compare the same query running serially (i.e., without parallelism and on a single CPU) and in parallel on two CPUs, we will typically find that the parallel execution time is more than half of the serial execution time. Again, this effect is due to the parallelism overhead.

Parallelism is primarily useful on servers running a relatively small number of concurrent queries. On this type of server, parallelism can enable a small set of queries to keep many CPUs busy. On servers running many concurrent queries (such as an OLTP system), we do not need parallelism to keep the CPUs busy; the mere fact that we have so many queries to execute can keep the CPUs busy. Running these queries in parallel would just add overhead that would reduce the overall throughput of the system.

How does SQL Server parallelize queries?

SQL Server parallelizes queries by horizontally partitioning the input data into approximately equal sized sets, assigning one set to each CPU, and then performing the same operation (e.g., aggregate, join, etc.) on each set. For example, suppose that we want to use two CPUs to execute a hash aggregate that happens to be grouping on an integer column. We create two threads (one for each CPU). Each thread executes the same hash aggregate operator. We might partition the input data by sending rows where the group by column is odd to one thread and rows where the group by column is even to the other thread. As long as all rows that belong to one group are processed by one hash aggregate operator and one thread, we get the correct result.

This method of parallel query execution is both simple and scales well. In the above example, both hash aggregate threads execute independently. The two threads do not need to communicate or coordinate their work in any way. If we want to increase the degree of parallelism (DOP), we can simple add more threads and adjust our partitioning function. (In practice, we use a hash function to distribute rows for a hash aggregate. The hash function handles any data type, any number of group by columns, and any number of threads.)

The actual partitioning and movement of data between threads is handled by the parallelism (or exchange) iterator. Although it is unique in many respects, the parallelism iterator implements the same interfaces as any other iterator. Most of the other iterators do not need to be aware that they are executing in parallel. We simply place appropriate parallelism iterators in the plan and it runs in parallel.

Note that this method of parallelism is not the same as “pipeline” parallelism where multiple unrelated operators run concurrently in different threads. Although SQL Server frequently places different operators in different threads, the primary reason for doing so is to allow repartitioning of the data as it flows from one operator to the next. With pipeline parallelism the degree of parallelism and the total number of threads would be limited to the number of operators.

Who decides whether to parallelize a query?

The query optimizer decides whether we should execute a query in parallel. This decision, like most others, is cost based. A complex and expensive query that processes many rows is more likely to result in a parallel plan than a simple query that processes very few rows.

Who decides the degree of parallelism (DOP)?

The DOP is not part of the cached compiled plan and may change with each execution. We decide the DOP at the start of execution by considering the number of CPUs on the server, the “max degree of parallelism” and “max worker threads” sp_configure settings (only visible if the “show advanced options” setting is on), and the query MAXDOP hint if any. In short, we choose a DOP that maximizes parallelism while ensuring that we do not run out of worker threads. If you choose MAXDOP 1, we remove all parallelism iterators and run the query as a serial plan using a single thread.

Note that the number of threads used by a parallel query may exceed the DOP. If you check sys.sysprocesses while running a parallel query, you may see more threads than the DOP. As I noted above, if we need to repartition data between two operators, we place them in different threads. The DOP determines the number of threads per operator not the total number of threads per query plan. In SQL Server 2000 if the DOP was less than the number of CPUs, the extra threads could use the extra CPUs effectively defeating the MAXDOP settings. In SQL Server 2005, when we run a query with a given DOP, we also limit the number of schedulers to the selected DOP. That is, all threads used by the query are assigned to the same set of DOP schedulers and the query uses only DOP CPUs regardless of the total number of threads.