SQL Server Storage Engine : Bulk Import/Export

New update on minimal logging for SQL Server 2008

Sunil Agarwal — Fri, 24 Oct 2008 00:18:00 GMT

Based on the customer feedback, we have decided to make minimal logging functionality into a btree available to SQL 2008 customers. All you need to do is to enable TF-610 in RTM bits. As a cautionary note, we have seen some slowdown if you are loading data into a btrree using TF-610 on a slower IO subsystem. So please make sure you test it first.

Here are the two performance issues you need to be aware of. I will walk you through each of these with examples

1. Double writes

2. Log space reservation.

Example:

-- create the source table. This table stores 2 row per page

create table t_source (c1 int, c2 int, c3 char (3000))

-- I insert 10000 rows. With 2 rows/page, I will have 5000 pages so

-- the size of the table is 40MB

declare @i int

select @i = 1

while (@i < 10000)

begin

insert into t_source values (@i, @i+10000, 'hello')

select @i= @i + 1

end

-- create target table with clusttered index

create table t_ci (c1 int, c2 int, C3 CHAR(3000))

create clustered index ci on t_ci(c1)

-- I will now insert all the rows from the source table into the target

-- table and measure (a) logspace used (b)logspace reserved (c) number of writes

-- both with TF-610 ON and OFF.

-- Here is the INSERT statement

begin tran

insert into t_ci select * from t_source order by c1

rollback

	# of page writes(appx)	Logspace used (appx)	Logspace reserved (appx)
TF-610 OFF	5000	34MB	8MB
TF-610 ON	10000	3.5MB	82MB

What is interesting to note, is that without TF-610, we write the data page only once and it is fully logged. While with TF-610, we write the data page twice (i.e. double write) but we do get minimal logging (i.e. we took wrote only 3.5MB log). However, the log reserved space balloons to 82MB. This is done to allow for rollback of the transaction and this space is reserved assuming full-logging. You may wonder that it should atmost be 34MB, but it is not because of conservative log reservation algorithm.

For details, please refer to my blog entries

http://blogs.msdn.com/sqlserverstorageengine/archive/2008/03/23/minimal-logging-changes-in-sql-server-2008-part-3.aspx

http://blogs.msdn.com/sqlserverstorageengine/archive/2008/03/23/minimal-logging-changes-in-sql-server-2008-part-2.aspx

thanks

Minimal Logging changes in SQL Server 2008 (part-3)

Sunil Agarwal — Mon, 24 Mar 2008 00:08:00 GMT

So far we had discussed (http://blogs.msdn.com/sqlserverstorageengine/archive/2008/03/23/minimal-logging-changes-in-sql-server-2008-part-2.aspx ) how minimal logging changes impact when you are moving data from one table to another table. Now let us look at how does this change more conventional bulk import. As you will see, conventional bulk import takes advantage on these changes as well but these changes are only useful when importing into a btree as conventional bulk import already provides minimal logging for heaps. Here are the series of scenarios that I tried

(1) Inserting into an HEAP. No changes in this behavior

begin tran

-- this is optimized load

-- (1) show that individual rows are not logged

-- (2) show the BU lock

bulk insert t_heap

from 'C:\sql-server-test\minimal-logging\t_source-dat.dat'

with (TABLOCK)

Logging:: Minimal Logging

-- if we don't specify TABLOCK, it leads to full logging. This is same what we always had

-- (1) without TABLOCK, no minmal logging.

bulk insert t_heap

from 'C:\sql-server-test\minimal-logging\t_source-dat.dat'

LOGGING: Fully logged

(2) Insert into BTREE

a) Into empty BTREE

create table t_ci (c1 int, c2 int, c3 char(100), c4 char(1000))

create clustered index ci on t_ci(c1)

begin tran

-- this is optimized load

-- (1) show that individual rows are not logged

-- (2) Lock: X lock on the table

bulk insert t_ci

from 'C:\sql-server-test\minimal-logging\t_source-dat.dat'

with (TABLOCK, datafiletype = 'char')

-- Now, what if don't specify TABLOCK? This is a change from earlier versions. and this is also optimized load. You will need to use TF-610

-- this functionality

-- (1) we get minimal logging

-- (2) IX: at table level

-- (3) you can use this to do parallel load as long as you are

-- (4) importing into disjoint ranges

bulk insert t_ci

from 'C:\sql-server-test\minimal-logging\t_source-dat.dat'

with (datafiletype = 'char')

Here are the top 10 log records. Note, in this case, only the range lock is taken

b) Into non-empty btree (when inserting into an increasing range)

-- Inserting rows with disjoint range, we still get minimal logging

-- if you are inserting into a btree but the range you are inserting overlaps partially with the existing data,

-- the logging will go between minimal and full logging. The rule is that you will get minimal logging

-- only when you allocate a new page.

bulk insert t_ci

from 'C:\sql-server-test\minimal-logging\t_newrange-dat.dat'

with (datafiletype = 'char')

Here are the top 10 log records. Note, in this case, only the range lock is taken. Since the SQL Server does not take X lock on the table, you can do parallel bulk import with minimal logging into a btree.

c) Into a table with clustered and non-clustered indexes

create table t_ci_nci (c1 int, c2 int, c3 char(100), c4 char(1000))

create clustered index ci on t_ci_nci(c1)

create index nci on t_ci_nci(c2, c3)

begin tran

-- this is optimized load

-- (1) show that individual rows are not logged

-- (2) Lock: IX lock on the table

bulk insert t_ci_nci

from 'C:\sql-server-test\minimal-logging\t_source-dat.dat'

with (datafiletype = 'char')

Query Plan:: Notice NCI and CI data is being sorted

Logged Records:: For the clustered index

Logged Records:: For the non-clustered index. Note, that index key is atleast > 100 bytes, so clearly it is minimally logged

As you can see, with the minimal logging changes in SQL2008, you have lot more options to bulk import the data with minimal logging. These changes are available on SQL2008/RTM.

Thanks

Sunil

Minimal Logging changes in SQL Server 2008 (part-2)

Sunil Agarwal — Sun, 23 Mar 2008 21:54:00 GMT

In the previous example http://blogs.msdn.com/sqlserverstorageengine/archive/2008/03/06/minimal-logging-changes-in-sql-server-2008-part-1.aspx, I described minimal logging while moving data from a source table into a heap. You may recall that it requires a X table lock to get minimal logging. Now, I will show you what happens when you move data from a source table into a btree. In order to get minimal logging with btree, the only requirements are

Like minimal logging for heap, the database must be set to bulk-logged or simple recovery model
The input data must be sorted in the index key order. Note, it does not require X tablelock or the btree to be empty. SQL Server accomplishes this by acquiring a X range lock. So for example, if you have a btree with values 1, 100, 1001, 100 and now you want to insert 500 rows with the key range from 110 thru 610, the SQL Server does that by acquiring a X range lock between 100 and 1001 and accomplishes the load. The other transactions can access any data that is outside the range (100, 1001)
Enable TF-610

Here is one example:

Insert into empty BTREE without TABLOCK is minimally logged except for the rows inserted on the first page.

-- create the source table

create table t_source (c1 int, c2 int, c3 char (100), c4 char(1000))

declare @i int

select @i = 1

while (@i < 10000)

begin

insert into t_source values (@i, @i+10000, 'indexkey', 'hello')

select @i= @i + 1

end

-- create a table with clustered index

create table t_ci (c1 int, c2 int, c3 char(100), c4 char(1000))

create clustered index ci on t_ci(c1)

-- minimally logged except for the first page worth of rows

-- Order BY is not necessary

-- LOCK: IX at table level

insert into t_ci select * from t_source order by c1

Here is the query plan. Please note that SQL Server chose a SORT operator

Here are logged records. You will notice, that first 7 are fully logged but after that it is minimally logged. When inserting rows into the btree, the page where the first row is to be inserted may have some rows so for this reason each insert is fully logged. Only once it is done with this page, a new page needs to be allocated, it is minimally logged

When I insert data into new range into a non-empty btree from the previous example, I get minimal logging except for the last page. Again, this works with/without TABLOCK

-- create a table with an non-overlapping range

create table t_newrange (c1 int, c2 int, c3 char(100), c4 char(1000))

declare @i int

select @i = 10001

while (@i < 20000)

begin

insert into t_newrange values (@i, @i+10000, 'indexkey', 'hello')

select @i= @i + 1

end

-- minimally logged except for the rows inserted on to the last page.

-- Note, this inserting the data in a new data range.

-- LOCK: X lock on the table.

-- Same logging behavior with/without TABLOCK

insert into t_ci with (TABLOCK) select * from t_newrange order by c1

Here are the log records. You will notice that the first 2 inserts are fully logged. Again, there are fully logged because the two rows go to the existing page and from that point onwards, the rows go to new page and hence minimally logged

In fact, I can load data into this btree concurrently as follows

-- create a table with clustered index

create table t_ci (c1 int, c2 int, c3 char(100), c4 char(1000))

create clustered index ci on t_ci(c1)

Now execute the following in two different sessions

-- connection 1

insert into t_ci select * from t_source order by c1

-- connection 2

insert into t_ci select * from t_newrange order by c1

Both of these inserts are minimally logged and they will insert the data in concurrently.

Thanks

Sunil

Minimal Logging changes in SQL Server 2008

Sunil Agarwal — Thu, 06 Mar 2008 08:12:00 GMT

Please refer to the earlier post for the background information on minimal logging. Starting with SQL Server 2008, the minimal logging has been enhanced. These enhancements are available to regular TSQL Insert as well. One of the key customer scenario was to be able to transfer data from a staging table to the target table. The only choice the customers had was to use SELECT * INTO <target> from <staging-table> if they wanted minimal logging. The limitation of this solution was that customers had no control on DDL aspect of the target table. WIth this new enhancement, this restriction is now removed.

I used a flavor of the following query to find log records

select top 10 operation,context, [log record fixed length], [log record length], AllocUnitId, AllocUnitName

from fn_dblog(null, null)

where allocunitname='dbo.t_heap'

order by [Log Record Length] Desc

(1) Insert into a HEAP is minimally logged under TABLOCK but fully logged without TABLOCK. This is one of the features that customers have been asking for. By the way, the only drawback is that it holds X lock, unlike Bulk Insert which holds BU lock, so you cannot insert using multiple threads.

-- create the source table

create table t_source (c1 int, c2 int, c3 char (100), c4 char(1000))

declare @i int

select @i = 1

while (@i < 1000)

begin

insert into t_source values (@i, @i+10000, 'indexkey', 'hello')

select @i= @i + 1

end

-- create the target heap

create table t_heap (c1 int, c2 int, c3 char(100), c4 char(1000))

-- this is minimally logged.

-- LOCK: X lock on the table

-- this behavior is same even when ‘t_heap’ is not empty

begin tran

insert into t_heap with (TABLOCK) select * from t_source

-- here are the top-10 log records. You can see that it is minimally logged

-- this is fully logged

insert into t_heap select * from t_source

-- here are the top 10 log records. You can see that it is fully logged

Thanks

Sunil

Bulk Import Optimizations (Minimal Logging)

Sunil Agarwal — Tue, 05 Feb 2008 06:01:00 GMT

In the previous blog I had listed all bulk import optimizations. Now, I will describe each of these optimizations in detail in separate blog entries. Let me first start with minimal logging. By the way minimal logging is also called bulk-logging but the correct term is 'minimal logging'. Bulk logging refers more to the database recovery model that enables minimal logging. Note, that the minimal logging can also be enabled by SIMPLE recovery model as well. This BLOG decribes the behavior for minimal logging as it exists in SQL Server 2005.

Under minimal logging, the SQL Server does not log individual rows and logs only the page and extent allocations. So if each page can fit 100 rows, we are trading 100 log records for 1 log record, a significant improvement. It is useful here to step back and examine how SQL Server does ensures ACID properties even when individual rows are not logged or how does the log backup work.

When the bulk import transaction commits (i.e. the number of rows as specified in the batchsize), the SQL Server flushes all the data pages (a subset of these pages are likely to have been flushed as part of checkpoint) to disk to guarantee the atomocity of the transaction. This is not any more expensive than flushing the log records as the log records would contain the full image of all the rows. And if for some reason, the bulk import transaction needs to be rolled back, all SQL Server needs to do is to traverse the log records and mark the pages/extents to be free again. When the transaction log backup is taken, the SQL Server also backs up the data pages, including index pages when applicable, as part of the transaction log. This enables a database restore by restoring the database backup followed by zero or more transaction log backups. The only drawback of minimal logging is that it does not support point-in-time recovery. This is because during the transaction log backup, the state of the pages involved in the bulk import may have changed between the time bulk import was done and the transaction log back is initiated. One can argue that we could possibly do point-in-time recovery to just before the bulk import had started but it is not supported. If this is of concern, you can do transaction log backup before/after bulk operations. Please refer to BOL for details of point-in-time recovery and associated restrictions.

So far we discussed minimal logging at a conceptual level. Now let us consider a simple example where we will insert 5 rows into a table both with minimal logging and without it and examine the difference.

The table t_heap below has a row length of > 1000 bytes so the log record representing the an insert of the data row will also be > 1000 bytes so it will be easy to spot when we examie the log records

create table t_heap (c1 int, c2 int, c3 decimal (38,2), c4 char(1000))

Let us execute the following command to insert 5 rows through optimized Bulk Import

bulk insert t_heap

from 'C:\t_heap-c-small.dat' with (TABLOCK)

Now we can look at the top 5 log records by executing the following query

select 5 operation,context,[log record length]

from fn_dblog(null, null)

where allocunitname='dbo.t_heap'

order by [Log Record Length] Desc

Here are the top 5 log records returned. You will notice that all records are < 100 bytes. In other words, the individual row inserts were not logged. The log records below refer to PFS, an allocation meta-information related page that tracks free space in the allocated page.

Operation context [log record length]

--------- ------- -------------------

LOP_MODIFY_ROW LCX_PFS 92

Now let us take a look at the log records when we insert 5 rows through the normal insert code path as follows

declare @i int

select @i = 0

while (@i < 5)

begin

insert into t_heap values

(@i, @i + 5000, 10.2, replicate ('a', 60))

set @i = @i + 1

end

And here are the top 5 log records and you can see that in this case, each inserted row is fully logged.

operation context [log record length]

--------- ------- -------------------

LOP_INSERT_ROWS LCX_HEAP 1116

You can use fn_dblog() as a handy way to see if you are getting minimal logging or not.

Till now, we have discussed what does the minimal logging mean and how you can checks if you indeed are getting the minimal logging but we have not looked at the conditions that need to be met for minimal logging. The conditions for minimal logging are

· The database recovery model must be set to BULK_LOGGED or SIMPLE. Under FULL recovery model, as the name implies, all operations are fully logged.

· The target table should not have been marked for replication

· If database backup command is being run concurrently, the minimal logging is temporarily disabled but it does not stop bulk import. So you may see that part of bulk import is fully logged.

· Minimal logging depends on the indexes defined on the table and whether the table/index is empty or not as described in the table below. You will notice that TABLOCK is required to enable minimal logging.

Table Schema	Empty	TABLOCK	Logging

Heap	Yes	Yes	minimal-logging
Heap	Yes	No	full logging
Heap	No	Yes	minimal-ogging
Heap	No	No	full logging
Heap + non-clustered index	Yes	Yes	minimal-logging
heap + non-clustered index	Yes	No	full logging
Heap + non-clustered index	No	Yes	data- minimal-logging index-fulllogging
Heap + non-clustered index	No	No	full logging
clust Index	Yes	Yes	minimal-logging
clust Index	Yes	No	full logging
clust Index	No	Yes	full logging
clust Index	No	No	full logging

What are the Bulk Import Optimizations?

Sunil Agarwal — Mon, 04 Feb 2008 22:12:00 GMT

Bulk import provides an optimized insert path that minimizes this overhead with following optimizations

· Minimal Logging: under this logging mode, individual rows are not logged and only the changes to page allocation structures are logged. This reduces the amount of logging significantly.

· BU Locking: a special mode table level locking only available for bulk import path. Using this locking mode, each bulk import thread acquires a table level BU lock while still allowing concurrent bulk import threads (i.e. parallel bulk import) into the same target table.

· Batchsize: provides a way to break the bulk import into smaller transactions by committing the transaction after ‘batchsize’ worth of rows have been successfully imported.

· Option to enable/disable triggers and constraints

· Option to leverage the sort order in the input set. So for example if you are bulk importing into a table with a cluseterd index and your input data is already sorted on clustered key columns, the sort operation can be skiped.

· LOB orphan optimization which eliminates the redundant copy of the LOB data (e.g. text or image). For normal insert, the LOB data is first copied into tempdb, the data row is then constructed and finally inserted which involves copying the LOB data from tempdb into the filegroup associated with the target table. However, for optimized bulk import path, the LOB data is directly copied into its final destination there by saving extra copy for each LOB value.

Why Bulk Import?

Sunil Agarwal — Mon, 04 Feb 2008 21:50:00 GMT

Most IT shops using SQL Server need to load/import large amount of data obtained from external sources or from another Database Server into SQL Sever. It is typcially done using the optimized insert path provided by SQL Server through Bulk Insert and BCP commands. Customers often wonder what optimizations are available through Bulk Insert/BCP and under what conditions? In this series of Blog posts, I will walk you through all bulk import optimizations with examples.

To understand the optimized insert path better, let us first take a look at the normal insert path is used by the SQL Server to insert one or more rows. As you may recall, the SQL Server guarantees ACID (Atomicity, Consistency, Isolation, and Durability) properties for database transactions. Locking and Logging are some of building blocks used by SQL Server to provide ACID properties. The detailed discussion of how SQL Server implements ACID properties is beyond the scope of this paper. For the discussion, it suffices to say that when a data row is inserted, it needs to be locked for the duration of the transaciton to prevent concurrent transactions from accessing it. Similarly, the inserted data row needs to be logged so that the transaction can be rolled back or forward depending if the transaction needs to be undone or redone. At a high level, the steps to insert a row into a table are

· Lock the row. It is done by taking a row lock.

· Write a log record. This log record contains the full copy of the row being inserted besides other fields needed by the log record

· Row is inserted on the data page.

These steps are repeated for each row and for each index defined on the table. Now imagine that you are importing a large number of rows. In this case the overhead of each these steps can be significant. For example, let us consider importing 10 million rows into a table with an average rowsize of 300 bytes. To achieve this, SQL Server needs to acquire 10 million locks and write both data and log in the excess of 3 GB each. Note that each data row is written twice; first in the log record and then on the data page. Ideally you would like the data row to be written only once but clearly it is not the case here. Besides this, other overhead includes

· Acquiring/releasing 10 million locks takes precious memory and CPU. Yes, you can minimize this by giving a TABLE LOCK hint

· Writing 10 million log records adds to the contention in logging as there is only single point of insertion log records. Also, writing 10 million log records take the proportional space in the log and if you are inserting all the rows in one transacition, it can potentially lead to out of space in log because only the inactive portion of the log can be backed up. You can, of course, mitigate this issue by inserting 10 million rows in smaller groups say 10000 rows each.

· Constraint Checking: if the target table (i.e. the table you are importing the data into) has constraints defined, they need to be checked for each insert. You can optionally disable the constraints.

· Triggers: if the target table has a trigger defined on insert operation, the trigger needs to be executed for each insert. Like constraints, you can optionally disable triggers.

As you can see there is a fair amount of overhead for inserting a row. This is where the optimized bulk import comes to your rescue.

Skipping rows in the datafile that contains NULL for a column that does not allow NULLs

Sunil Agarwal — Fri, 29 Sep 2006 08:25:00 GMT

I recently ran into a customer who wanted to import data into a table that did not allow NULL for one column. However, his data file had some rows that had NULL for that column. He tried the following Bulk Insert command

bulk insert customer.dbo.test from 'c:\data.txt' with (maxerrors = 20) but this command failed with the following error

Msg 4869, Level 16, State 1, Line 1
The bulk load failed. Unexpected NULL value in data file row 2, column 1. The destination column (c1) is defined as NOT NULL

Maxerror does not help in this case because the NULL enforcement is done at much later stage and this constraint cannot be disabled. Interestingly, OPENROWSET can easily handle this as follows

insert into customer.dbo.test
select * from OPENROWSET (BULK 'c:\data.txt',FORMATFILE='c:\format.xml') as t1
where c1 is not NULL.

Have fun with OPENROWSET.

Importing UNIX data file

Sunil Agarwal — Thu, 08 Jun 2006 23:59:00 GMT

When bulk importing data, the data can come from any source including the files created in other operating systems like UNIX. Here are two ways that you can use to bulk import a data file created under UNIX.

· Without using a format file

You can use the following script to bulk load.

DECLARE @bulk_cmd varchar(1000)

SET @bulk_cmd = 'BULK INSERT AdventureWorks.Sales.SalesOrderDetail

FROM ''<drive>:\<path>\<filename>''

WITH (ROWTERMINATOR = '''+CHAR(10)+''')'

EXEC (@bulk_cmd)

However, the following will not work because '\n' automatically gets converted to '\r\n' since it's the typical EOF string for Windows text files. The reason that \n works like it does, because we want BCP commands from Unix world on Windows as well

bulk insert bcptest

from 'C:\unix\u.dat'

WITH (ROWTERMINATOR='\n')

· You can use explicit format file with the line terminator character as “\n”

Bulk Importing data with OPENROWSET in SQL2005

Sunil Agarwal — Tue, 30 May 2006 23:39:00 GMT

Before SQL2005, you could use Bulk Insert or BCP to import data into SQL Server. While both of these mechanisms have been popular and used widely, they don't provide any processing of input data before bulk importing into the target table. If you need to preprocess the data, you can use DTS (aka SSIS) or, now in SQL2005, you may be able to use OPENROWSET with BULK rowset provides to accompolish the same, but for simpler tasks using SELECT statement functionality.

Recently, I was asked how to Bulk Import data after removing whitespaces into a varchar column in the target table. The customer desribed the problem as "We're loading data from a text file with fixed column widths into a table in SQL Server 2005.All table fields are varchars of the same length as the source fields. The problem is that the BULK INSERT process doesn't truncate whitespace, and since the source fields can contain variable-length values or nothing at all we wind up with fields ending with or containing nothing but spaces. Our best plan at the moment is to RTRIM() every column and insert the data into another table, but since we'll be loading millions of rows every day we were looking for any more efficient methods"

You can easily accompolish this task using OPENROWSET. Here is one example

create table t_cust (c1 varchar(20), c2 varchar(20))

You can use the following command to remove the whitespaces and importing the data into the target table simultaneously. This Bulk Import follows the bulk optimizations rules (i.e. bulk logging) similar to the ones outlined for Bulk Insert in BOL.

INSERT into t_cust with (TABLOCK)

SELECT RTRIM(c1), RTRIM(C2)

from OPENROWSET (BULK <data-file>,<format-file> as t1

You will note that OPENROWSET reads the data from the datafile and then you can remove the trailing blanks by calling the function RTRIM for each value.

An example format file is

<?xml version="1.0" ?>

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>