Craig Freedman's SQL Server Blog : Pivot and Unpivot

The UNPIVOT Operator

craigfr — Tue, 17 Jul 2007 20:36:00 GMT

The UNPIVOT operator is the opposite of the PIVOT operator. As I explained in my earlier post, the PIVOT operator takes a normalized table and transforms it into a new table where the columns of the new table are derived from the values in the original table. The UNPIVOT operator takes a pivoted table and transforms it back into a normalized form with one row per data point using the column names as values in the result. For example, suppose we have the following data:

CREATE TABLE PIVOT_Sales(EmpId INT, [2005] MONEY, [2006] MONEY, [2007] MONEY)
INSERT PIVOT_Sales VALUES(1, 12000, 18000, 25000)
INSERT PIVOT_Sales VALUES(2, 15000, 6000, NULL)
INSERT PIVOT_Sales VALUES(3, NULL, 20000, 24000)

This is the output of the PIVOT operation from my earlier posts. There is one row for each employee with up to three years of sales data per row. If there is no sales data for a particular employee for a particular year, we simply insert NULL. We can transform this table back to its original form with a single row per employee per year using the following UNPIVOT statement:

SELECT EmpId, CAST (Yr AS INT) AS Yr, Sales
FROM (SELECT EmpId, [2005], [2006], [2007] FROM PIVOT_Sales) AS p
UNPIVOT (Sales FOR Yr IN ([2005], [2006], [2007])) AS s

I've explicitly casted the Yr column to integer. The default type for the pivot column is NVARCHAR(128). This type is based on the maximum column name length of 128 characters.

The resulting output is:

EmpId       Yr          Sales
----------- ----------- ---------------------
1           2005        12000.00
1           2006        18000.00
1           2007        25000.00
2           2005        15000.00
2           2006        6000.00
3           2006        20000.00
3           2007        24000.00

Unlike PIVOT operations which may not be reversible, all UNPIVOT operations are reversible (so long as all of the input data is preserved). That is, we can always transform the output of an UNPIVOT operation back into the original table using an appropriate PIVOT operation. Unlike PIVOT operator, the UNPIVOT operator does not require or support aggregation functions.

Let's look at the plan for the above query:

|--Compute Scalar(DEFINE:([Expr1010]=CONVERT(int,[Expr1009],0)))
       |--Filter(WHERE:([Expr1008] IS NOT NULL))
            |--Nested Loops(Left Outer Join, OUTER REFERENCES:([PIVOT_Sales].[2005], [PIVOT_Sales].[2006], [PIVOT_Sales].[2007]))
                 |--Compute Scalar(DEFINE:([PIVOT_Sales].[EmpId]=[PIVOT_Sales].[EmpId]))
                 |    |--Table Scan(OBJECT:([PIVOT_Sales]))
                 |--Constant Scan(VALUES:((N'2005',[PIVOT_Sales].[2005]),(N'2006',[PIVOT_Sales].[2006]),(N'2007',[PIVOT_Sales].[2007])))

This query plan simply takes each row of the input table and joins it with a constant scan that generates three rows - one for each of the three columns listed in the UNPIVOT IN clause - for each input row. The plan then filters out any rows that have NULL data. (Note that [Expr1008] is the Sales column and [Expr1009] is the Yr column.) There are a couple of points worth noting about this query plan. First, the join must be a nested loops join because the constant scan operator uses the correlated parameters from the outer side of the join to generate rows. There is no way to generate these rows without these correlated parameters. Second, the join need not be a left outer join. The constant scan always produces exactly three rows and, thus, the outer rows always join and are never NULL extended. Nevertheless, the outer join is harmless in this context and behaves like an inner join.

Note that we can write the original query as:

SELECT p.EmpId, Yr, Sales
FROM PIVOT_Sales AS p CROSS APPLY
    (
    SELECT EmpId, 2005 AS Yr, [2005] AS Sales UNION ALL
    SELECT EmpId, 2006, [2006] UNION ALL
    SELECT EmpId, 2007, [2007]
    ) AS s
WHERE Sales IS NOT NULL

This query yields a nearly identical query plan. The UNION ALL syntax produces a similar result to the constant scan except that there are now three constant scans and a concatenation operator:

|--Filter(WHERE:([Union1007] IS NOT NULL))
       |--Nested Loops(Inner Join, OUTER REFERENCES:([p].[2005], [p].[2006], [p].[2007]))
            |--Table Scan(OBJECT:([tempdb].[dbo].[PIVOT_Sales] AS [p]))
            |--Concatenation
                 |--Constant Scan(VALUES:(((2005))))
                 |--Constant Scan(VALUES:(((2006))))
                 |--Constant Scan(VALUES:(((2007))))

In this plan, [Union1007] is the Sales column. We can actually see the definition of [Union1007] from the DefinedValues column of the concatenation operator in the SET SHOWPLAN_ALL ON output:

[Union1006] = ([Expr1003], [Expr1004], [Expr1005]), [Union1007] = ([p].[2005], [p].[2006], [p].[2007])

Notice that the values for [Union1007] are actually derived directly from the correlated parameters of the cross apply (from the PIVOT_Sales table) and not from the constant scans. [Union1006] is the Yr column and the values are derived from the constant scans.

PIVOT Query Plans

craigfr — Mon, 09 Jul 2007 21:05:00 GMT

In my last post, I gave an overview of the PIVOT operator. In this post, I'm going to take a look at the query plans generated by the PIVOT operator. As we'll see, SQL Server generates a surprisingly simple query plan that is essentially just a fancy aggregation query plan.

Let's use the same schema and queries from my previous post:

CREATE TABLE Sales (EmpId INT, Yr INT, Sales MONEY)
INSERT Sales VALUES(1, 2005, 12000)
INSERT Sales VALUES(1, 2006, 18000)
INSERT Sales VALUES(1, 2007, 25000)
INSERT Sales VALUES(2, 2005, 15000)
INSERT Sales VALUES(2, 2006, 6000)
INSERT Sales VALUES(3, 2006, 20000)
INSERT Sales VALUES(3, 2007, 24000)

SELECT [2005], [2006], [2007]
FROM (SELECT Yr, Sales FROM Sales) AS s
PIVOT (SUM(Sales) FOR Yr IN ([2005], [2006], [2007])) AS p

This query generates the following query plan:

  |--Compute Scalar(DEFINE:(
                 [Expr1006]=CASE WHEN [Expr1024]=(0) THEN NULL ELSE [Expr1025] END,
                 [Expr1007]=CASE WHEN [Expr1026]=(0) THEN NULL ELSE [Expr1027] END,
                 [Expr1008]=CASE WHEN [Expr1028]=(0) THEN NULL ELSE [Expr1029] END))
       |--Stream Aggregate(DEFINE:(
                      [Expr1024]=COUNT_BIG(CASE WHEN [Sales].[Yr]=(2005) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1025]=SUM(CASE WHEN [Sales].[Yr]=(2005) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1026]=COUNT_BIG(CASE WHEN [Sales].[Yr]=(2006) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1027]=SUM(CASE WHEN [Sales].[Yr]=(2006) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1028]=COUNT_BIG(CASE WHEN [Sales].[Yr]=(2007) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1029]=SUM(CASE WHEN [Sales].[Yr]=(2007) THEN [Sales].[Sales] ELSE NULL END)))
            |--Table Scan(OBJECT:([Sales]))

This is just a basic scalar aggregation query plan! It calculates one SUM aggregate for each year. Like any SUM aggregate, each aggregate actually computes both the count and the sum. If the count is zero, the query plan returns NULL else it returns the sum. (The compute scalar handles this logic.)

The only twist is that each SUM aggregate is actually computed over a CASE statement that filter for those rows that match the year for which it is summing sales. The CASE statement returns the value of the Sales column for those rows that match the year and NULLs for all other rows. To clarify what is happening, we can view the results of the CASE statements without the aggregation:

SELECT EmpId, Yr,
    CASE WHEN Yr = 2005 THEN Sales END AS [2005],
    CASE WHEN Yr = 2006 THEN Sales END AS [2006],
    CASE WHEN Yr = 2007 THEN Sales END AS [2007]
FROM Sales

EmpId       Yr          2005                  2006                  2007
----------- ----------- --------------------- --------------------- ---------------------
1           2005        12000.00              NULL                  NULL
1           2006        NULL                  18000.00              NULL
1           2007        NULL                  NULL                  25000.00
2           2005        15000.00              NULL                  NULL
2           2006        NULL                  6000.00               NULL
3           2006        NULL                  20000.00              NULL
3           2007        NULL                  NULL                  24000.00
2           2007        NULL                  NULL                  NULL

When computing the sums of each of the year columns, the query plan relies on the fact that aggregate functions discard NULLs; that is, the NULL values are not included in the results. Although this point may seem intuitive for a SUM aggregate, the significance is clearer for a COUNT aggregate:

CREATE TABLE T (A INT)
INSERT T VALUES(NULL)

-- Returns 1: the number rows in table T
SELECT COUNT(*) FROM T

-- Returns 0: the number of non-NULL values of column A
SELECT COUNT(A) FROM T

Note that we could just as easily have written the original query as:

SELECT
    SUM(CASE WHEN Yr = 2005 THEN Sales END) AS [2005],
    SUM(CASE WHEN Yr = 2006 THEN Sales END) AS [2006],
    SUM(CASE WHEN Yr = 2007 THEN Sales END) AS [2007]
FROM Sales

This query gets a nearly identical query plan. The only visible difference is the use of an extra compute scalar to evaluate the CASE statements.

  |--Compute Scalar(DEFINE:(
                 [Expr1004]=CASE WHEN [Expr1013]=(0) THEN NULL ELSE [Expr1014] END,
                 [Expr1005]=CASE WHEN [Expr1015]=(0) THEN NULL ELSE [Expr1016] END,
                 [Expr1006]=CASE WHEN [Expr1017]=(0) THEN NULL ELSE [Expr1018] END))
       |--Stream Aggregate(DEFINE:(
                      [Expr1013]=COUNT_BIG([Expr1007]), [Expr1014]=SUM([Expr1007]),
                      [Expr1015]=COUNT_BIG([Expr1008]), [Expr1016]=SUM([Expr1008]),
                      [Expr1017]=COUNT_BIG([Expr1009]), [Expr1018]=SUM([Expr1009])))
            |--Compute Scalar(DEFINE:(
                           [Expr1007]=CASE WHEN [Sales].[Yr]=(2005) THEN [Sales].[Sales] ELSE NULL END,
                           [Expr1008]=CASE WHEN [Sales].[Yr]=(2006) THEN [Sales].[Sales] ELSE NULL END,
                           [Expr1009]=CASE WHEN [Sales].[Yr]=(2007) THEN [Sales].[Sales] ELSE NULL END))
                 |--Table Scan(OBJECT:([Sales]))

The other big difference between the PIVOT syntax and query plan and the alternative syntax and query plan is that the PIVOT query suppresses the following warning about NULLs:

Warning: Null value is eliminated by an aggregate or other SET operation.

We could also suppress this warning by executing the following statement:

SET ANSI_WARNINGS OFF

At this point, the query plan for a multi-row PIVOT operation should not come as a surprise:

SELECT EmpId, [2005], [2006], [2007]
FROM (SELECT EmpId, Yr, Sales FROM Sales) AS s
PIVOT (SUM(Sales) FOR Yr IN ([2005], [2006], [2007])) AS p

The query plan for this PIVOT operation uses CASE statements to compute the same intermediate result that we saw above. Then, like any other GROUP BY query, it uses either stream or hash aggregate to group by EmpId and to compute the final result. In this case, the optimizer chooses a stream aggregate. Since we do not have an index to provide order, it must also introduce a sort.

  |--Compute Scalar(DEFINE:(
                 [Expr1007]=CASE WHEN [Expr1025]=(0) THEN NULL ELSE [Expr1026] END,
                 [Expr1008]=CASE WHEN [Expr1027]=(0) THEN NULL ELSE [Expr1028] END,
                 [Expr1009]=CASE WHEN [Expr1029]=(0) THEN NULL ELSE [Expr1030] END))
       |--Stream Aggregate(GROUP BY:([Sales].[EmpId]) DEFINE:(
                      [Expr1025]=COUNT_BIG(CASE WHEN [Sales].[Yr]=(2005) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1026]=SUM(CASE WHEN [Sales].[Yr]=(2005) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1027]=COUNT_BIG(CASE WHEN [Sales].[Yr]=(2006) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1028]=SUM(CASE WHEN [Sales].[Yr]=(2006) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1029]=COUNT_BIG(CASE WHEN [Sales].[Yr]=(2007) THEN [Sales].[Sales] ELSE NULL END),
                      [Expr1030]=SUM(CASE WHEN [Sales].[Yr]=(2007) THEN [Sales].[Sales] ELSE NULL END)))
            |--Compute Scalar(DEFINE:([Sales].[EmpId]=[Sales].[EmpId]))
                 |--Sort(ORDER BY:([Sales].[EmpId] ASC))
                      |--Table Scan(OBJECT:([Sales]))

To see how this query really is no different than any other GROUP BY query or to see some alternative query plans, try creating a clustered index on EmpId to eliminate the sort or using an OPTION(HASH GROUP) hint to force a hash aggregate.

The PIVOT Operator

craigfr — Tue, 03 Jul 2007 21:31:00 GMT

In my next few posts, I'm going to look at how SQL Server implements the PIVOT and UNPIVOT operators. Let's begin with the PIVOT operator. The PIVOT operator takes a normalized table and transforms it into a new table where the columns of the new table are derived from the values in the original table. For example, suppose we want to store annual sales data by employee. We might create a schema such as the following:

CREATE TABLE Sales (EmpId INT, Yr INT, Sales MONEY)
INSERT Sales VALUES(1, 2005, 12000)
INSERT Sales VALUES(1, 2006, 18000)
INSERT Sales VALUES(1, 2007, 25000)
INSERT Sales VALUES(2, 2005, 15000)
INSERT Sales VALUES(2, 2006, 6000)
INSERT Sales VALUES(3, 2006, 20000)
INSERT Sales VALUES(3, 2007, 24000)

Notice that this schema has one row per employee per year. Moreover, notice that in the sample data employees 2 and 3 only have sales data for two of the three years worth of data. Now suppose that we'd like to transform this data into a table that has one row per employee with all three years of sales data in each row. We can achieve this conversion very easily using PIVOT:

SELECT EmpId, [2005], [2006], [2007]
FROM (SELECT EmpId, Yr, Sales FROM Sales) AS s
PIVOT (SUM(Sales) FOR Yr IN ([2005], [2006], [2007])) AS p

I'm not going to delve into the PIVOT syntax which is already documented in Books Online. Suffice it to say that this statement sums up the sales for each employee for each of the specified years and outputs one row per employee. The resulting output is:

EmpId       2005                  2006                  2007
----------- --------------------- --------------------- ---------------------
1           12000.00              18000.00              25000.00
2           15000.00              6000.00               NULL
3           NULL                  20000.00              24000.00

Notice that SQL Server inserts NULLs for the missing sales data for employees 2 and 3.

The SUM keyword (or some other aggregate) is required. If the Sales table includes multiple rows for a particular employee for a particular year, PIVOT does aggregate them - in this case by summing them - into a single data point in the result. Of course, in this example, since the entry in each "cell" of the output table is the result of summing a single input row, we could just as easily have used another aggregate such as MIN or MAX. I've used SUM since it is more intuitive.

This PIVOT example is reversible. The information in the output table can be used to reconstruct the original input table using an UNPIVOT operation (which I will cover in a later post). However, not all PIVOT operations are reversible. To be reversible, a PIVOT operation must meet the following criteria:

All of the input data must be transformed. If we include a filter of any kind including on the IN clause, some data may be omitted from the PIVOT result. For example, if we altered the above example only to output sales for 2006 and 2007, clearly we could not reconstruct the 2005 sales data from the result.
Each cell in the output table must derive from a single input row. If multiple input rows are aggregated into a single cell, there is no way to reconstruct the original input rows.
The aggregate function must be an identity function (when used on a single input row). SUM, MIN, MAX, and AVG all return the single input value unchanged and, thus, can be reversed. COUNT does not return its input value unchanged and, thus, cannot be reversed.

Here is an example of a non-reversible PIVOT operation. This example, calculates the total sales for all employees for all three years. It does not itemize the output by employee.

SELECT [2005], [2006], [2007]
FROM (SELECT Yr, Sales FROM Sales) AS s
PIVOT (SUM(Sales) FOR Yr IN ([2005], [2006], [2007])) AS p

Here is the output. Each cell represents the sum of two or three rows from the input table.

2005                  2006                  2007
--------------------- --------------------- ---------------------
27000.00              44000.00              49000.00

In my next post, I'll look at some example PIVOT query plans.