Tips, Tricks, and Advice from the SQL Server Query Processing Team : Indexing

Index Build strategy in SQL Server - Part 4-2: Offline Serial/Parallel Partitioning (Non-aligned partitioned index build)

queryproc — Mon, 14 May 2007 07:15:00 GMT

Source Partitioned
While the table is partitioned, we may want to change the way it is partitioned when building the new index. For example, by using the same partition function and scheme, the new index can be partitioned on different columns than the original table.

Create table t (c1 int, c2 int) on ps(c2)

…….

Create clustered Index idx_t on t(c1) on ps(c1)

The serial plan looks like follows.
Index Insert
   |
Sort
   |
NL (Nested Loop)
/    \
CTS Scan
CTS is the Constant Table Scan. It scans each partition one by one and provides partition ID to the inner side (the lower part in graphics showplan) of NL. The inner side of NL scans the corresponding partition and sends data to the sort iterator. From there, it is exactly the same as source non-partitioned scenario. Not surprisingly, the memory and disk space requirement is the same too.

In the case of parallel plan, we have
      X (Distribute Streams)
       |
     Index Insert
       |
     Sort
       |
       X (Repartition Streams)
       |
      NL
     /   \
   X    Scan
/
CTS
The operator above CTS is a Gather Streams operator, meaning it has one producer and many consumers. There is no parallelism below this operator. Between the Gather Streams and the Repartition Streams, each worker is assigned (Number of Source's Partitions)/(Degree of Parallelism) number of source partitions. The source is only scanned once in total.

The Repartition Streams operator splits the query plan into two parallelism sections. Between the top-level Distribute Streams and the Repartition Streams, we have a different set of workers than the worker set below Repartition Streams. Each worker is assigned (Number of Target Index Partitions)/(Degree of Parallelism) target partitions. The target index partition information is pushed down to the Repartition Streams which redirects data to different sort tables based on target partition location. The rest, i.e. how the sort and index building works, is the same as the parallel plan in the source non-partitioned case. Again, the memory and disk space requirement is the same as in the source non-partitioned case.

Index Build strategy in SQL Server - Part 4-1: Offline Serial/Parallel Partitioning (Non-aligned partitioned index build)

queryproc — Tue, 08 May 2007 21:56:00 GMT

Recall that in the previous posts on index build, we defined "aligned" as the case when base object and in-build index use the same partition schema, and "non-aligned" to be the case when heap and index use different partition schemes, or the case when heap is not partitioned. In this post, we will talk about the two scenarios of non-aligned partitioned index build, source partitioned and source not partitioned.

Source Not Partitioned
Consider the following query.

Create Partition Function pf (int)

as range right for values (1, 100, 1000)

Create Partition Scheme ps as Partition pf

ALL TO ([PRIMARY])

Create table t (c1 int, c2 int) –the table is created on the primary filegroup by default

Create clustered Index idx_t on t(c1) on ps(c1) -- non-aligned index build

The serial plan is straightforward.

Index Insert (write data to the in-build index)
|
Sort (order by index key)
|
Scan (read data from source)

The sort iterator creates one sort table per target partition (there are four partitions in this example so we will construct four sort tables concurrently). By default, we use the user database for sort to spill data. As we mentioned before, we free each extent from sort table after all its rows are copied. By doing this, for each partition, we can reduce the disk space requirement from 3*partition size (source + sort table + b-tree) to just about 2.2*partition size. Therefore, each file group requires 2.2*(size of all partitions that belong to this file group) of disk space. If the users specify sort_in_tempdb, then all the sort tables reside in the tempdb. Therefore, we require 2.2*(Size of the whole index) of free space in tempdb.

Index insert iterator can start building index after the sort iterator finishes sorting all sort tables. Therefore, we will have as many sort tables as the number of partitions at the same time. Recall that each sort table requires at least 40 pages. So, the minimum required memory will be #PT*40pages.

When it comes to parallel plan, it looks like

X (Exchange)
   |
Index Insert
   |
Sort
   |
Scan

Each worker thread is assigned (Partition Count)/(Degree Of Parallelism) number of partitions (e.g. if we have 4 partitions and 4 worker threads each gets 1 partition), which can be skewed. The sort iterator creates one sort table per assigned partition. Each worker scans the source once and retrieves the rows that belong to its partition(s), the retrieved rows will be inserted into the corresponding sort table depending on which partition they belong to.

After all sort tables got populated, the index builder starts consuming rows from sort tables, it consumes one sort table after another, building b-tree(s) in each partition's file group. Currently workers do not share partitions. Therefore, it is possible for one worker to finish all assigned partitions and idle while another worker is busy inserting rows.

The disk space and memory requirements are exactly the same as the serial plan. This is because in both cases, we cannot start building the index until all the sort tables are populated.

Index Build strategy in SQL Server - Part 3: Offline Serial/Parallel Partitioning (Aligned partitioned parallel index build)

queryproc — Sat, 20 Jan 2007 04:20:00 GMT

Aligned partitioned parallel index build

In case of parallel build we scan and sort partitions in parallel and the actual number of sort tables existing at the same time will depends on the actual number of concurrent workers. Partitions are being chosen by workers one by one and when one worker completes with one partition it takes the next partition which is not yet taken by another worker. Each worker builds 0 - N partitions (we do not share one partition among multiple workers). Why can it be 0? If DOP > # of partitions, then we do not have enough partition to give out to all the workers(s). Which partition a worker is going to work on? This is non-deterministic per execution. First come first serve.

As we never share one partition among several workers, the biggest partition becomes a bottleneck. A situation could happen when all workers completed with their partitions and one worker still sorting the biggest one along. Which also means the resource being used by this query (such as memory and threads) will not be available for other queries.

There is no final stitch among workers for partitioned index. Each partition is being represented as a separated b-tree in storage engine.

How does it affect disc space requirements:

- In case of sorting in user’s database (default setting) the requirements are actually the same as in case of serial build as we are sorting in each filegroup for each corresponding partition we will need 2.2*(Size of partition) for each filegroup.

- In case of using Sort_in_tempdb (SORT_IN_TEMPDB = ON) we won’t have the same advantage as in case of serial build because we may have several sort tables at the same time and as long as we don’t know the actual distribution of data between the partitions we will still require the same 2.2*(Size of the whole index) of free space in tempdb.

Memory consideration:

We will have several sort tables at the same time (depends of #DOP and # of partitions) and we will need at least 40pages per each sort table to be able to start the index build operation. So, the minimum required memory will be #DOP*40pages.

Total memory = 40 * DOP + additional memory.

Note that additional memory does not change for serial or parallel plans, this is because the total number of rows we need to sort remain the same in both plans.

Index Build strategy in SQL Server - Part 3: Offline Serial/Parallel Partitioning

queryproc — Wed, 17 Jan 2007 02:27:00 GMT

There are two main categories of partitioned index build:

Aligned (when base object and in-build index use the same partition schema)
Not- Aligned (when heap and index use different partition schemas (including the case when base object is not partitioned at all and in-build index use partitions))

(see Index Build strategy in SQL Server - Introduction (II))

Aligned partitioned index build

Aligned partitioned serial index build

/ \

CTS Builder (write data to the in-build index)

[Sort] (order by index key) <-- optional

Scan (read data from source)

CTS: Constant Table Scan (the purpose of CTS is to provide partition IDs for index builder)

NL: Nested Loop

In case of aligned partition index build Constant Table Scan provides partition IDs to the inner side of Nested Loop so we can build one partition at a time – for each partition ID provided by CTS (outer side of the NL), inner side would build the index for that partition (not the entire index). Sort table gets created for each partition, but as we are doing it one by one and we are building final b-tree for each partition one by one we don’t need to keep this sort table for each partition at the same time. As a result we have only one sort table at any given moment.

How does it affect disc space requirements:

- In case of sorting in user’s database (default setting) we are actually sorting in each filegroup for each corresponding partition, which means we will need the same 2.2*(Size of the partition) for each filegroup. For example: let’s say we have 3 partitions located in filegroups FG1, FG2, FG3 and index data takes 1Gb, 2Gb, and 3Gb respectively. In this case we will need 2.2*1 = 2.2Gb of free space in FG1; 2.2*2 = 4.4Gb of free space in FG2, and 2.2*3 = 6.6Gb of free space in FG3. That means we will need total 9.9Gb of free space on the disk(s).

- In case of using Sort_in_tempdb (SORT_IN_TEMPDB = ON) option we can reuse the same space of tempdb for the sort table and as we are sorting partitions one by one we actually will need only 2.2*(Size of the biggest partition) of free space in tempdb. For example: let’s look at the case described above. We will need only 3.3Gb of free space in tempdb (this size of free space will let us build smaller partitions and then reuse this space for building the biggest partition).

Memory consideration:

We will have only one sort tables at the same time and we will need at least 40pages for the sort table to be able to start the index build operation. So, the minimum required memory will be 40pages.

Total memory = minimum required memory + additional memory*.

*additional memory calculated as row size multiplied by the estimated row count provided by Query Optimizer.

Posted by: Lyudmila Fokina

Index Build strategy in SQL Server - Part 2: Offline, Parallel, No Partitioning (Non stats plan (no histogram))

queryproc — Wed, 13 Dec 2006 22:55:00 GMT

Build (serial) (write data to the in-build index)

X (Merge exchange)

/ | \

Sort… Sort… Sort …(order by index key)

| | |

Scan… Scan… Scan…(read data from source)

When histogram is not available (for example when we building an index on a view) we can’t use the same methods as described in pervious post (For statistics gathering, it is possible only for ‘real’ object. We are building index on view, so index does not exist at this point and we are not able to gather sample stats on the view this is why we have a different plan here). So we will be using regular parallel scan which is not aware of data distribution at all.

How this works:

We will scan source data in parallel using parallel scan, but serial b-tree build.

Each worker scans a page from the heap the same way as in a previous parallel scan method. When scan is done sort table gets created and populated for each worker. Each worker maintains its own sort table - one sort table and the data from all workers will be merged (as these sort tables are not disjoint we can not just build separate b-tree’s and ‘stitch’ them; we have to merge sort tables) and produced to a final sorted output. After that, index build operation will be serial as we have one final output. The index builder consumes data from Merge Exchange and builds the final index.

Why is this plan relatively slow? We are building index in serial plus some extra overhead introduced by ‘merge exchange’.

Memory consideration:

In parallel index build, we are building multiple sort tables concurrently hence the basic memory requirement is higher and the calculation is a bit different. For memory calculation, we have 1) required memory, 2) additional memory. Each sort requires 40 pages of required memory. Let’s say we have DOP = 2 so we have 2 sort tables, we need 80 pages of required memory, but the total additional memory remains the same regardless DOP setting, this is because the total number of rows remain the same regardless DOP setting. For example, if serial plan needs 500 pages of additional memory, then parallel plan has the same request for additional memory, each worker will get 500/DOP pages of additional memory + 40 pages of required memory.

Index Build strategy in SQL Server - Part 2: Offline, Parallel, No Partitioning

queryproc — Tue, 12 Dec 2006 02:07:00 GMT

The type of parallel index build plan in SQL server depends on whether or not we have a histogram available with necessary statistics. Therefore, there are two broad categories of parallel index plans:

Histogram available:
No histogram

Histogram available (parallel sort and build):

X (Exchange)

| \ \

Builder… Build… Build… (write data to the in-build index)

| | |

Sort… Sort… Sort … (order by index key)

| / /

Scan (read data from source)

This type of Parallel Index build is getting chosen when we have statistics available (hence range partition information is available and can be used to identify data distribution).

How does scan happen in this case? We must have some statistics on the leading key column, so if we don’t have stats we will go ahead and generate sample statistics to determine whether and how to parallelize the index build operation. In some situations, however, we are not able to build sample stats, such as indexed view ("No stats plan"), and then different index build plan will be generated. Using the statistics and histogram we can identify data distribution (divide data in several buckets), so we can load balance the workload among workers in parallel plan, it also help us to make DOP (degree of parallelism) decision to achieve high utilization of system resource. Using the row count estimates from the histogram for each bucket in the distribution the workload is split into N ranges (N = DOP), one for each worker (this is an attempt to load balance the work among all workers).

Using range partition scan to scan data, each worker receives data belonging to its range and builds its own sort table and b-tree based on sort table, so each worker will have its own sort table and all the b-trees are disjoint. The coordinator thread will then stitch all the b-tree’s together at the end of index build operation and we build full statistics on the new b-tree and are done.

Parallel Index Build with histogram available can give us the best performance.

On the downside it is more memory consuming and will fail if there is not enough memory (because each worker creates #DOP sort tables). We can play with MAXDOP option to reduced to max number of DOP used in the index build and as a result – min memory required for the build. You can run sp_configure to figure out what is the default setting for ‘max degree of parallelism’ on the server. Max degree of parallelism = 0, means ‘uses the actual number of available CPUs depending on the current system workload’. You can explicitly limit the number of processors to use in parallel plan execution.

For example:

Create index idx_t on t(c1, c2)

WITH (MAXDOP = 2)

-- limit # of processor to use for index build to 2

//Next time - Non stats plan (no histogram) index build plan

Posted by: Lyudmila Fokina

Index Build strategy in SQL Server - Part 1: offline, serial, no partitioning

queryproc — Tue, 21 Nov 2006 00:19:00 GMT

Builder (write data to the in-build index)

Sort (order by index key)

Scan (read data from source)

In order to build the b-tree for the index we have to first sort the data from source. The flow is to scan the source, sort it (if possible - in memory*), then build the b-tree from the sort.

Why do we need to sort first before building the b-tree? In theory we don’t have to sort, we could use regular DML and directly insert data into the in-build index (no sort), but in this case we would be doing random inserts, random inserts in a b-tree require searching the b-tree for the correct leaf node first and then inserting the data. And while searching a b-tree is fairly fast, doing so before each insert is far from optimal. So for index build operation, we sort the data using the sort order for the new index, so when we insert data into the in-build index, it is not a random insert, it is actually an append operation and this is why the operation can be much faster than random inserts.

While inserting data between sort and index builder we free each extent from the sort table as soon as all of its rows are copied. In this way we reduce the overall disk space consumption from a possible 3*Index Size (source + sort table + b-tree) to just 2.2*Index Size (approximately).

*We do not guarantee in memory sort; the decision of whether we can do in memory sort or not depends on memory available and actual row count. ‘In-memory’ sort is, naturally, fast (also disk space requirements will be more relaxed in this case, because we don’t have to allocate space for the sort table on the disk), but it is not required; we can always spill data to disk, although the performance is much slower than in-memory sort.

For an index build operation, we use the user database (the database where we build the index) by default for sort to spill data, but if the user specifies sort_in_tempdb, then we use tempdb for spill.

Each sort table (even when we have very little data to sort) requires at least 40 pages (3200KB) to run (later we will see that in case of parallelism we can have several sort tables at the same time). When calculating memory for sort, we try to allocate enough memory to have an in-memory sort. For large index build operations it is not likely that we will be able to fit the entire sort in memory. If we can’t provide at least 40 pages for the Index Build operation, it will fail.

The last step of index build is to always build full statistics. Good statistics information helps the query optimizer to generate better plan, users can issue ‘create’ or ‘update’ stats commands to force SQL Server generate or refresh stats on a certain object. When we are building a new index, since we need to touch every row, we use this opportunity to build full stats at the same time as a side benefit.

Conclusion:

To be able to build a non partitioned Index offline with serial plan we will need free disk space (in user’s or in tempdb database) of approximately 2.2*IndexSize and at least 40 pages of memory available for the query executor to be able to start the process.

Read in next post: Index Build Scenario 2: Offline, Parallel, No Partitioning

Posted by: Lyudmila Fokina

Index Build strategy in SQL Server - Introduction (II)

queryproc — Fri, 10 Nov 2006 03:05:00 GMT

- Building Partitioned Index vs. Building non-Partitioned Index:

The data of partitioned tables and indexes is divided into units that can be spread across more than one filegroup in a database. The data is partitioned horizontally, so that groups of rows are mapped into individual partitions. The table or index is treated as a single logical entity when queries or updates are performed on the data. All partitions of a single index or table must reside in the same database.

Building Aligned Partitioned Index Build:

Although partitioned indexes can be implemented independently from their base tables, it generally makes sense to design a partitioned table and then create an index on the table. When you do this, SQL Server automatically partitions the index by using the same partition scheme and partitioning column as the table. As a result, the index is partitioned in essentially the same manner as the table. This makes the index aligned with the table.

An index does not have to participate in the same named partition function to be aligned with its base table. However, the partition function of the index and the base table must be essentially the same, in that

1) the arguments of the partition functions have the same data type,

2) they define the same number of partitions, and

3) they define the same boundary values for partitions.

Also If you build a non-clustered index on a partitioned base (partitioned heap or clustered index) without specifying partitioning function, then the non-clustered index will be aligned partitioned as well (see example below).

Example:

Create Partition Function pf (int)

as range right for values (NULL, 1, 100)

Create Partition Scheme ps

as Partition pf

TO ([PRIMARY], [FileGroup1], [FileGroup1], [FileGroup1])

Create table t (c1 int, c2 int)

on ps(c1)

Create Index idx_t on t(c1)

Building non-Aligned Partitioned Index Build:

SQL Server does not align the index with the table if you specify a different partition scheme or a separate filegroup on which to put the index at creation time.

You can turn a non-partitioned table into partitioned by building a partitioned clustered index – it will be non-aligned index build as well (see example below).

Example:

Create Partition Function pf (int)

as range right for values (NULL, 1, 100)

Create Partition Scheme ps

as Partition pf

TO ([PRIMARY], [FileGroup1], [FileGroup1], [FileGroup1])

Create table t (c1 int, c2 int)

Create clustered Index idx_t on t(c1)

on ps(c1)

Note: If you have a partitioned clustered index (as in the example above) and drop the clustered index, then the new heap will stay partitioned and will be located in the same partition scheme or filegroup as was defined for the clustered index unless you specify MOVE TO option when dropping the clustered index.

Example:

Drop Index idx_t on t

WITH(MOVE TO new_ps(c1))

In this example the base table is moved to a different partition scheme and the nonclustered indexes are not moved to coincide with the new location of the base table (heap). Therefore, even if the nonclustered indexes were previously aligned with the clustered index, they may no longer be aligned with the heap.

Read in next post: Index Build Scenario 1: Offline, Serial, No Partitioning

Posted by: Lyudmila Fokina

Index Build strategy in SQL Server - Introduction (I)

queryproc — Thu, 09 Nov 2006 02:05:00 GMT

Index Build strategy in SQL Server may vary depending on users needs. Each of these Index Build strategies may have different memory and disc space requirement. These different strategies will be discussed in the next several posts.

For the beginning let’s see what kind of Index Build types exist in SQL Server 2005:

- Online Index Build vs. Offline Index Build:

In SQL Server 2005, you can create, rebuild, or drop indexes online. The ONLINE option allows concurrent user access to the underlying table or clustered index data and any associated nonclustered indexes during these index operations. For example, while a clustered index is being rebuilt by one user, that user and others can continue to update and query the underlying data. When you perform DDL operations offline, such as building or rebuilding a clustered index; these operations hold exclusive locks on the underlying data and associated indexes. This prevents modifications and queries to the underlying data until the index operation is complete.

Example:

Create index idx_t on t(c1, c2)

WITH (ONLINE = ON)

- Serial Index Build vs. Parallel Index Build:

On multiprocessor computers index statements may use more processors to perform the scan, sort, and build operations associated with the index statement just like other queries do. The number of processors employed to run a single index statement is determined by the configuration option max degree of parallelism (set by sp_configure) (default value of 0 - uses all available processors), by MAXDOP index option (set in statements – see example below), by the current workload, and, in non-partitioned case, by data distribution of the first key column. The max degree of parallelism option limits the number of processors to use in parallel plan execution – in other words: it is setting the ceiling, meaning no more than this number but can be anything below it. If Database Engine detects that the system is busy, the degree of parallelism of the index operation is automatically reduced before statement execution starts.

Example:

Create index idx_t on t(c1, c2)

WITH (MAXDOP = 2)

-- limit # of processor to use for index build to 2

- Building Index storing the intermediate sort result in user’s database vs. storing in

tempdb database (SORT_IN_TEMPDB):

When you create or rebuild Index you can choose which database to use to store the intermediate

sort results, generated during index creation. It can be either user’s database (database where index

is being created) or tempdab database.

SORT_IN_TEMPDAB index option is used to set the desirable behavior. When set to ON, the

sort results are stored in tempdb. When OFF, the sort results are stored in the filegroup or partition

scheme in which the resulting index is stored.

Example:

Create clustered Index idx_t on t(c1)

WITH (SORT_IN_TEMPDB = ON)

Read in next post: Building partitioned vs. non-partitioned Indexes.

Posted by: Lyudmila Fokina