WesleyB's Blog : Storage Engine

Data Compression and Heaps

wesback — Wed, 17 Jun 2009 11:14:00 GMT

I think we can all agree that Data Compression is a great new functionality in SQL Server 2008. I know many of you have been experimenting with it but there was one little catch I wanted you to be aware of.

When you apply page compression to a table the pages get compressed when they are full but there is an exception as you would have expected. When you have a heap the newly allocated pages are NOT compressed until you rebuild the table (or remove and reapply compression or add and remove a clustered index) unless you are using BULK INSERT.

I have added a test script so you can see it with your own eyes.

First the clustered table:

SET NOCOUNT ON
GO

IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[myClusteredTable]') AND type in (N'U'))
DROP TABLE [dbo].[myClusteredTable]
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[myClusteredTable](
[ID] [int] IDENTITY(1,1) NOT NULL,
[TheDate] [date] NOT NULL,
[TheTime] [time](7) NOT NULL,
[SomeFiller] [varchar](256) NOT NULL,
CONSTRAINT [PK_myClusteredTable] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

SET ANSI_PADDING OFF
GO

--Add some records to the table with varying length
INSERT INTO dbo.myClusteredTable
VALUES (SYSDATETIME(), SYSDATETIME(), REPLICATE('W', CEILING(RAND() * 100) + 1))

GO 15000

--Check the space used
sp_spaceused myClusteredTable

--Now apply compression
ALTER TABLE [dbo].[myClusteredTable] REBUILD PARTITION = ALL
WITH
(DATA_COMPRESSION = PAGE
)
GO

--Check the space used
sp_spaceused myClusteredTable
GO

--Add some more records to the table
INSERT INTO dbo.myClusteredTable (TheDate, TheTime, SomeFiller)
SELECT TheDate, TheTime, SomeFiller FROM myClusteredTable
GO

--Check the space used again
sp_spaceused myClusteredTable
GO

DROP TABLE myClusteredTable

As you can see the compression keeps going even when you insert new rows.
But what about the heap?

SET NOCOUNT ON
GO

IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[myHeap]') AND type in (N'U'))
DROP TABLE [dbo].[myHeap]
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[myHeap](
[ID] [int] IDENTITY(1,1) NOT NULL,
[TheDate] [date] NOT NULL,
[TheTime] [time](7) NOT NULL,
[SomeFiller] [varchar](256) NOT NULL,
CONSTRAINT [PK_myHeap] PRIMARY KEY NONCLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

SET ANSI_PADDING OFF
GO

--Add some records to the table with varying length
INSERT INTO dbo.myHeap
VALUES (SYSDATETIME(), SYSDATETIME(), REPLICATE('W', CEILING(RAND() * 100) + 1))

GO 15000

--Check the space used
sp_spaceused myHeap

--Now apply compression
ALTER TABLE [dbo].[myHeap] REBUILD PARTITION = ALL
WITH
(DATA_COMPRESSION = PAGE
)
GO

--Check the space used
sp_spaceused myHeap
GO

--Add some more records to the table
INSERT INTO dbo.myHeap (TheDate, TheTime, SomeFiller)
SELECT TheDate, TheTime, SomeFiller FROM myHeap
GO

--Check the space used again
sp_spaceused myHeap
GO

--Rebuild the heap?
ALTER TABLE myHeap REBUILD
GO

--Check the space used again
sp_spaceused myHeap
GO

--What about a bulk insert?
BULK INSERT TestDB.dbo.MyHeap
   FROM 'c:\temp\test.txt'
   WITH
      (
   FORMATFILE='c:\temp\myHeap.xml',
   KEEPIDENTITY,
   TABLOCK
      )
GO

--Check the space used again
sp_spaceused myHeap
GO

--Rebuild the heap?
ALTER TABLE myHeap REBUILD
GO

--Check the space used again
sp_spaceused myHeap
GO

--Seems compression was indeed applied with BULK INSERT, go ahead and drop the test table
DROP TABLE myHeap

This was tested on SQL Server 2008 SP1 + CU2 (build 10.0.2714).

The Data Loading Performance Guide

wesback — Fri, 13 Feb 2009 13:52:00 GMT

SQL Server has been gaining a lot of credibility in the data warehousing market the last couple of years. With that the challenge arises more and more to load massive amounts of data in SQL Server. Obviously this is a matter of good hardware in combination with the 'right' way of doing things. Lucky for us there is the SQLCAT team to do the hard work for us - learning the lessons the hard way that is - and they have published a very good whitepaper around data loading in SQL Server. This whitepaper is really a must read if you are thinking of loading data in SQL Server and just a very interesting read that helps you understand the concepts of massive loads, I/O and SQL Server behavior in general.

Check it out here.

What happens when I update my partitioning key?

wesback — Thu, 09 Oct 2008 12:00:00 GMT

This is a question someone asked me yesterday and if you think about it, this is really a good question.

The partitioning function defines on which partition the row will be mapped based on the partitioning column value. Imagine that you have a table on which you have defined partitioning to make all rows with RecordType 1 to go to the PRIMARY filegroup and those with RecordType 2 to the ACTIVEDATA filegroup. How do we handle updates that change the RecordType from 1 to 2? The record obviously physically needs to move to the ACTIVEDATA filegroup but we are not inserting new data, we are just updating an existing row which has already been placed on PRIMARY.

Internally we are actually doing this by using a DELETE-INSERT operation. The 'old' row is deleted and the row with updated values is inserted and thus using the partitioning function to decide the physical placement of the row.

This is actually the same as with clustered indexes, when the value of your clustering key changes it might well be possible that its location in the table needs to change too and this is done in exactly the same manner.

--Clustered key update
SET NOCOUNT ON
GO

IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[myClusteredTable]') AND type in (N'U'))
DROP TABLE [dbo].[myClusteredTable]
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[myClusteredTable](
[myClusteringKey] [int] NOT NULL,
[myFiller] [char](100) NOT NULL,
CONSTRAINT [PK_myClusteredTable] PRIMARY KEY CLUSTERED
(
[myClusteringKey] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO

SET ANSI_PADDING OFF
GO

INSERT INTO myClusteredTable VALUES (1, 'F')
GO

INSERT INTO myClusteredTable
SELECT TOP (1) myClusteringKey + 1, myFiller FROM myClusteredTable ORDER BY myClusteringKey DESC
GO 100

CHECKPOINT --If database is in SIMPLE recovery this will clear the transaction log
GO

UPDATE myClusteredTable
SET myClusteringKey = 102
WHERE myClusteringKey = 1

SELECT * FROM ::fn_dblog(null, null) --Look for LOP_DELETE_ROWS and LOP_INSERT_ROWS on the object
GO

--Partitioning key update
SET NOCOUNT ON
GO

IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[myPartitionedTable]') AND type in (N'U'))
DROP TABLE [dbo].[myPartitionedTable]
GO

IF EXISTS (SELECT * FROM sys.partition_schemes WHERE name = N'RecordType_PS')
DROP PARTITION SCHEME [RecordType_PS]
GO

IF EXISTS (SELECT * FROM sys.partition_functions WHERE name = N'RecordType_PF')
DROP PARTITION FUNCTION [RecordType_PF]
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE PARTITION FUNCTION [RecordType_PF](tinyint) AS RANGE LEFT FOR VALUES (N'1', N'2')
GO

CREATE PARTITION SCHEME [RecordType_PS] AS PARTITION [RecordType_PF] TO ([PRIMARY], [ACTIVEDATA], [PRIMARY])
GO

CREATE TABLE [dbo].[myPartitionedTable](
[ID] [int] IDENTITY(1,1) NOT NULL,
[RecordType] [tinyint] NOT NULL,
CONSTRAINT [PK_myPartitionedTable] PRIMARY KEY NONCLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
) ON RecordType_PS([RecordType])
GO

INSERT INTO myPartitionedTable VALUES(1)
INSERT INTO myPartitionedTable VALUES(2)
GO

CHECKPOINT --If database is in SIMPLE recovery this will clear the transaction log
GO

UPDATE myPartitionedTable
SET RecordType = 2
WHERE RecordType = 1
GO

SELECT * FROM ::fn_dblog(null, null) --Look for LOP_DELETE_ROWS and LOP_INSERT_ROWS on the object
WHERE AllocUnitName LIKE 'dbo.myPartitionedTable%'
GO