SQL Server Storage Engine : Data Compression

Changes to sp_estimate_data_compression_savings stored procedure in SQL2008R2

Sunil Agarwal — Wed, 16 Sep 2009 22:53:00 GMT

When you compress an object (index or table or partition), there are two components of space savings that you get. First component is fragmentaton (i.e. the original object might have been fragmented). The object gets degragmented as part of doing compression so you get some space savings. Second component is actual data compression savings. Many customers have asked the question 'Is there a way to know how much savings do I get from each of the two components?".

Interestingly, starting with SQL2008R2, you indeed have this capability. Here is one example where I create a table t1 and then simulate fragmentation by updating a column. Once the table is fragemented, I can use the sp_estimate_data_compression stored procedure to determine how much space savings I can get by defragmenting it.

drop table t1

-- Create the table

create table t1 (c1 int, c2 char(100), c3 varchar (300))

-- create a clustered index

create clustered index t1_ci on t1(c1)

-- load 10000 rows. Note, that for column C3, I am storing full-values that

-- take all 300 characters

declare @i int

select @i = 0

while (@i < 10000)

begin

insert into t1 values (@i, 'hello', replicate ('1', 300))

select @i = @i + 1

end

-- Now let us measure the average length of the row. It comes to 417 bytes.

-- This means that there are 19 rows per page

select avg_record_size_in_bytes

from sys.dm_db_index_physical_stats (db_id('TEMPDB'),

object_id('T1'),

null, null, 'SAMPLED')

where object_name (object_id) like 't1' AND index_id =1

-- check the space usage. This shows the space usage of 4296 KB

sp_spaceused t1

-- This update will cause fragmentation because I am changing the column C3 from 300

-- chars to 50 chars. With the result, the average row length

-- will now become 167 but the SQL Server will still keep 19 rows/page which

-- leaves more that 50% unused space on each page

update t1 set c3 = REPLICATE ('2', 50)

-- check the space usage. It shows that the allocated or reserved space

-- has not changed

sp_spaceused t1

-- estimate how much space you can save by defragmentation. Note

-- here I am executing the stored procedure on an 'uncompressed' table

-- with target compression setting is 'NONE',(i.e. no compression).

exec sp_estimate_data_compression_savings 'dbo',

'T1',1, NULL, 'none'

-- This stored proc returns

object_name schema_name index_id partition_number

------------ ----------- -------- ---------------

T1 dbo 1 1

size_with_current_compression_setting(KB) size_with_requested_compression_setting(KB)

---------------------------------------- ------------------------------------------

4480 1720

-- To verify this, I rebuild the index and see the space consumption

alter index t1_ci on t1 rebuild

-- check the space usage. It shows that the table now takes 1744KB

sp_spaceused t1

In SQL2008RTM, the stored procedure did not compute the space savings if the source and target compression settings were same. In otherwords, the stored proc would have returned the following instead

size_with_current_compression_setting(KB) size_with_requested_compression_setting(KB)

---------------------------------------- ------------------------------------------

4480 4480

Interestingly, you can use it to get tell-tale signs to know if ROW compressed table already takes advantage of unicode compression in SQL2008R2. Let us say customer was using ROW compression in SQL2008 and now upgrades to SQL10.5, s/he can use to the sp_estimate_data_compression_savings stored proc by specifying target compression to be ROW as well. Note, in this case, the compression savings will include the potential savings from data defragmentation as well so one can’t easily deduce that the savings are from Unicode compressions or due to both. However, customer can get some clues by referring to the sys.dm_db_index_physical_stats DMV to see if the object in question was fragmented or not.

Thanks

Sunil

A Unicode Compression example

Sunil Agarwal — Mon, 17 Aug 2009 06:37:00 GMT

Now that we have Unicode compression available in SQL Server 2008R2 as described in, let me take a simple example using AdventureWorksDW database to show you the additional compression that can be achieved on tables with one or more columns of type NCHAR or NVARCHAR.

use [AdventureWorksDW2008]

-- the table FactResellerSales is a FACT table with three columns of type

-- NVARCAHR types. Let us find the average length of each of these columns

select AVG( LEN(salesordernumber))

from FactResellerSales

This returns a value of 7. This represents number of character in the string and not the numnber of bytes.

select AVG( LEN(CarrierTrackingNUmber))

from FactResellerSales

This returns a value of 12. This represents number of character in the string and not the numnber of bytes.

select AVG( LEN(CustomerPONumber))

from FactResellerSales

This returns a value of 12. This represents number of character in the string and not the numnber of bytes.

Based on these computations, we have approximately 31 characters of NVARCHAR types. With Unicode compression, we will expect that the average length of the row will reduce further by 31 bytes (i.e. we will save 1 byte per UCS-2 representation). To measure this, we will attach the identical copy of AdventureWorks2008 database to two different instances of SQL Server, one instance of SQL 2008RTM and other instance of SQL 20088R2 and measure the average length of the row after applying the ROW compression. Before applying the ROW compression, let us measure the average length of the row in both instances using the following query

select avg_record_size_in_bytes

from sys.dm_db_index_physical_stats (db_id('AdventureWorksDW2008'),

object_id('dbo.factresellersales'),

null, null, 'SAMPLED')

where object_name (object_id) like '%ResellerSales%' AND index_id =1

In both instances, the average length of the row comes out to be around 191 bytes. This should not surprise you as there are no changes to the uncompressed data. Now, let us execute the command to enable ROW compression in each of the instances using the following command

alter table dbo.factresellersales rebuild

with (

data_compression = ROW)

And, now let us find the average row length in each of the instances. Since Unicode compression is available as part of ROW compression, we will expect that the average length of the row will be around 31 bytes less in SQL2008R2. Let us run the following query again to find the average row length after the row compression

select avg_record_size_in_bytes

from sys.dm_db_index_physical_stats (db_id('AdventureWorksDW2008'),

object_id('dbo.factresellersales'),

null, null, 'SAMPLED')

where object_name (object_id) like '%ResellerSales%' AND index_id =1

Here are the vallues returned

· Average Row length in SQL2008 after row compression = 132 (around 30% compression)

· Average Row length in SQL2008R2 after row compression = 100 (around 45% compression). This extra savings of 32 bytes is what we had expected.

As you can see that in this example,the unicode comrpession gave us extra 15% compression with ROW compression. We tried the Unicode compression with some in-house customer workloads and we observed the compression savings as shown in the table below. Of course, your savings will depend on your data and the schema but the key point to know is that space savings with Unicode compression can be significant if you consider tables in tera-byte range. In our testing we found that the additional impact on CPU is minimal.

ROW Compression	ROW with UNICODE
9%	43%
30%	53.2%
45%	64%
30%	45%

I enccourage you to download CTP2 verision (http://www.microsoft.com/sqlserver/2008/en/us/r2.aspx) of SQL2008R2 and see for yourself the kind of savings you can achieve with your workload.

Thanks

Sunil Agarwal

Unicode Compression in SQL Server 2008R2

Sunil Agarwal — Mon, 17 Aug 2009 04:00:00 GMT

In my previous blog, I had mentioned that unicode compression will be available as part of next SQL Server release named SQL 2008R2. You can down load the CTP2 version http://www.microsoft.com/sqlserver/2008/en/us/r2.aspx to play with it. This is a new exciting addition to the data compression offering.

Let me describe the problem and how it has been addressed in SQL Server.

Problem: As more and more businesses developing global customer base, applications are being developed/modified to use unicode based data types such as NCHAR and NVARCHAR. SQL Server uses UCS-2 encoding scheme that takes two bytes of storage regardless of the locale. For example, in ASCII character set when stored as NCHAR, each character only needs 1 byte of storage but it is stored using 2 bytes with the significant byte being 0. In fact most European languages need only 1 byte of storage. When an application is either converted or written to user unicode based data types, it can, depending on the size of strings, increase the storage requirements significantly.

Solution: SQL Server implements unicode compression using SCSU (Simple Compression Schecme for Unicode Data), a standard compression scheme for Unicode data (http://unicode.org/reports/tr6/). The following table shows the compression achieved with SCSU and compares it with UTF-8 as both DB2/Oracle have implemented UTF-8 encoding for unicode data. You will notice that for English (i.e. ASCII) we are compressing 50% or in other words, we will take 1 byte instead of 2. Same is true for German locale. On the other hand for Korean locale, SCSU scheme takes 2 bytes so there is no compression savings. The compression achieved using SCSU is very comparable to UTF-8.

Locale	SCSU	UTF-8
English	0.5	0.5
Japanese	.85	1.0
Korean	1.0	1.0
Turkish	.52	.53
German	.5	.5
Vietnamese	0.61	0.68
Hindi	0.5	1.0

The Unicode compression is enabled or disabled as part of ROW or PAGE compression. The implications are

1. The customers who are already using ROW or PAGE compression will get the benefit of unicode compression but it will require them to rebuild the index or table after the upgrade to SQL 2008R2. The interesting thing is that the Unicode compression will kick in when a new row inserted or an existing Unicode value is modified even if the corresponding compressed index or the table is not immediately rebuilt after the upgrade.

2. Existing scripts used for data compression can be used without modifications

If NCHAR or NVARCHAR string does provide any compression savings with unicode compression, like the example with korean locale, unicode compression is not applied. For this case, the compression overhead is paid at the time of compression to know if we can get unicode compression savings but not when the value is read. Like before no application changes are needed to take advantage of unicode compression.

At this time, the SQL Server will not support Unicode compression for NVARCHAR (MAX) type, including in-row values or for NTEXT. This is something we will consider in the future. In my next blog, I will provide an example using AdventureWorksDW database to show you the compression benefits achieved using new improved ROW compression.

Thanks

Sunil Agarwal

Customer feedback on Data Compression

Sunil Agarwal — Mon, 17 Aug 2009 03:44:00 GMT

As I described in my previous blog, the data compression feature has been very successful. We do appreciate all the feedback that we have received as this provides us a way to know how customers have been using the feature and the challenges they are facing. Based on the feedback, we plan to improve our data compression offering in future releases. Here is some of the feedack and the respective rationale. Please feel free to contact me for your suggestions on data compression.

(1) Provide Unicode compression: For the uninitiated, the SQL Server stores columns of type NCHAR, VARCHAR, NTEXT and NVARCHAR(MAX) in UCS-2 format which takes 2 bytes per character even when 1 byte will do (e.g. ASCII data), a rather common case for locale used with USA (en-USA) and other major european languages. This leads to a waste of 1 byte per Unicode character.

(2) Compress out of row LOB data: SQL Server compresses in-row LOB data but not if the LOB data is stored out of row. With the result many LOB heavy applications are unable to take full advantage of data compression. However, there are two workaround available. First, LOB data can be compressed at the application tier but this means (a) the application needs to be modified (b) the application cannot take advantage of search and partial update capabilities provided in SQL Engine. Second, use filestream feature to store LOB data on a compressed volume. This is our recommended solution where applicable. Please refer to Books-Online for details on filestream feature

(3) Improve column prefix compression: Currently PAGE compression provides column prefix compression but for some workloads the repeating words occur in the middle or towards the end of the column value which does not qualify for prefix compression. This reduces the data compression savings that can be achieved. It is not to say that these customers have not benefited from data compression but clearly they could have achieved even more compression savings.

(4) Enable estimate data compression savings computations in lower SKUs: Currently, you need to run sp_estimate_compression_savings stored procedure on EE or Dev edition. Many customers who are running on standard SKU would like to know what additional space savings they can get with data compression before they invest into a EE edition. Current work around is to use Dev edition to estimate the space savings which requires customers to install Dev Edition and restore the production database. It would be nice if we could just estimate the compression savings in the lower editions.

On this note, I am very pleased to announce that we have enhanced the data compression feature to include Unicode compression for NCHAR and NVARCHAR datatypes. The Unicode compression will be available as part of ROW compression in SQL Server 2008R2 release. You can down load the CTP2 version http://www.microsoft.com/sqlserver/2008/en/us/r2.aspx to play with it. In my next blogs, I will go into more details with Unicode compression.

Thanks

Sunil Agarwal

Update on data compression performance/space-savings and links to published white papers

Sunil Agarwal — Sun, 16 Aug 2009 23:11:00 GMT

It has been a while since I blogged about data compression so I thought it will be good to provide an update on data compression usage within SQL Server community. I am happy to say that the Data compression feature has been a tremendous success in SQL Server 2008 with customers. Many customers have been able to reduce the size of their database significantly leading to reduction in the cost of hardware and storage management. The table below shows the actual space savings that some of the customers have achieved with ROW and PAGE compression in their production environment. Each row in the table below represents a different customer. Couple of key observations; First, the space savings can be huge. For example, one customer has been able to achieve 81% space savings with PAGE compression. Second, not surprisingly, the PAGE compression provides better compression at the cost of additional CPU cycles. As I had emphasized in my earlier blogs, the space savings achieved will depend on the schema and the data distribution.

Data Compression Space Savings	Notes
70%	PAGE. Data Warehouse application.
40%	PAGE. OLTP Web application.
62%	PAGE. DW application.
38%, 21%	PAGE, ROW.
80%, 50%	PAGE, ROW.
52%	PAGE.
50%, 15%	PAGE, ROW.
81%	PAGE. ERP application.
35%	PAGE.

Now, this may all sound good, but one question that keep coming back to me is 'Well space savings is alright, but how does data compression impact the performance of the workload?'. I agree we can all create best/worst cases to make our point but the real answer is know how data compression has impacted the real production applications. The table below shows the performance impact of data compression (the green color shows the improvement in the performance while the red color shows the performance degradation). Each row in the table below represents a different customer.

Performance impact	Notes
5%	PAGE compression. OLTP Web application. Large volume of transactions.
40%-60%	PAGE compression. Large sequential range queries. A DW Application (IO intensive)
1%	PAGE compression. 500 users, 1500 Transactions / sec. OLTP with some reporting queries.
11%	PAGE compression. A lot of insert, update and delete activity which leads to increase in CPU usage. A better choice would have been to go with ROW compression.
2% - 3%	PAGE compression. OLTP Application.
3%	PAGE compression. ERP application – small transactions.

I am hoping that this has provided some excitement to customers on the sideline. I would like to encourage trying out the feature and seeing for yourself if this is right for your environment. I will also encourage you to read some of the white papers listed below

http://www.unisys.com/eprise/main/admin/corporate/doc/41371394.pdf

http://h71028.www7.hp.com/ERC/downloads/4AA1-8766ENW.pdf?jumpid=reg_R1002_USEN

http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000002956

http://www.microsoft.com/downloads/details.aspx?FamilyID=b5bb47a4-5ece-4a2a-a9b5-5435264f627d&DisplayLang=en

http://sqlcat.com/whitepapers/archive/2009/05/29/data-compression-strategy-capacity-planning-and-best-practices.aspx

Thanks

Sunil Agarwal

DML operations on a HEAP and compression

Sunil Agarwal — Mon, 22 Dec 2008 00:16:00 GMT

In my previous blog http://blogs.msdn.com/sqlserverstorageengine/archive/2008/12/20/enabling-compression-on-a-heap.aspx, I described how to enable compression on a HEAP. You can enable either ROW or PAGE compression on the heap. Let me now describe the implications of DML operations on a heap.

ROW compression: No issues. The updated/inserted data stays ROW compressed though out its life time.

PAGE Compression: There are two key points to note (a) when you enable PAGE compression on an index or a heap, SQL Server evaluates each page and only enable PAGE compression if the space savings crosses a minimal (internally defined) threshold. So it is possible that when you enable PAGE compression on a heap, not all pages will have PAGE compression structures. (b) The PAGE compression is only considered when the insert of a row causes a page split. Since pages are never split in a heap either for update (it uses row-forwarding) or insert operations, the DML operation on a heap offer special challenges with PAGE compression.

Let us start with the assumption that the heap has many pages worth of rows and all of them are PAGE compressed

· Update operations on existing rows will continue to use PAGE compression information (page dictionary and column prefix) as long as the updated row stays on the page and does not get forwarded.

· When the number of DML operations on a given page exceeds an internally defined threshold, the column prefix and page dictionary may get re-computed. Again, this is not any different than btree.

· For Insert operations, the PAGE compression may or may not get preserved. The behavior is as follows

o If a singleton Insert goes to an existing page with PAGE compression, SQL Server will see if it can take advantage column prefix and page dictionary structures. Note, this happens only on the page(s) that are PAGE compressed.

o If a singleton insert goes to an existing page that does not have PAGE compression structures, it is only ROW compressed. A singleton insert into a heap will never cause a conversion to PAGE compression for the reasons mentioned earlier.

o As a special case, the PAGE compression for new rows is only supported for the following cases.

§ Optimizd Bul lk Load into a heap

§ Insert into <heap> with (TABLOCK) select * from <source>. Note, that the TABLOCK is required and if you don't specify it, the rows that land on a new page will not be PAGE compressed but still be ROW compressed.

In these cases, when the data page does not have the space to accommodate the new row being inserted, it evaluates if PAGE compression will save space greater than the internally defined threshold and if the answer is yes, it enables PAGE compression on the page and the row is then inserted on that page.

Enabling compression on a HEAP

Sunil Agarwal — Sat, 20 Dec 2008 07:44:00 GMT

I often get a question how to do enable compression on a table that is a heap (i.e. it does not have a clustered index). Clearly, one could create the clustered index with compression option set to PAGE and then drop the clustered index. This is an expensive operation because

· Creating a clustered index requires a SORT

· When you drop the clustered index, internal allocation structure, namely the PFS (Page Free Space) needs to updated to reflect the free space on each page. Note, when a clustered index is dropped, the resultant heap retains the compression state.

Instead, in SQL Server 2008, there is a new REBUILD command available that can be used to rebuild a HEAP. You can use this command to enable compression on the HEAP. Also, you can use this command to defragment a HEAP. You may wonder what is there to defrag in a heap because the pages don’t need to be traversed in the key order? Well, it is possible the pages in a HEAP are not full so the defragmentation can be used to reclaim the free space. In any case, let us get back to compression. Let me start with the following example

create table t1_big (c1 int, c2 int, c3 char(1000))

- Load the data

declare @i int

select @i = 0

while (@i < 5000)

begin

insert into t1_big values (@i, @i + 5000,

replicate('a', 500))

set @i = @i + 1

end

- Look at the current row size

- The following query returns a length of 1015

select max_record_size_in_bytes

from sys.dm_db_index_physical_stats (db_id('compression'),

object_id('compression.t1_big'),

null, null, 'DETAILED')

where object_name (object_id) like '%t1_big%'

-- Now, you can use the following NEW command to enable

-- PAGE compression. You could have used this command to

-- enable ROW compression or even NONE compression. As

-- part of rebuilidng the HEAP, it is also defragmented.

-- This command can also be executed ONLINE

alter table t1_big rebuild

with (

data_compression = PAGE)

-- compression setting shows the PAGE compression

select object_name (object_id) as table_name, data_compression_desc

from sys.partitions

where object_name (object_id) like '%t1_big%'

-- check the length again and it returns a value 9

-- Note, the example I have used here is not realistic so it

-- show almost 99% compression but you get the idea.

select max_record_size_in_bytes

from sys.dm_db_index_physical_stats (db_id('compression'),

object_id('compression.t1_big'),

null, null, 'DETAILED')

where object_name (object_id) like '%t1_big%'

There are few restrictions on ALTER TABLE REBUILD command as follows

(1) If you use ONLINE option then HEAP rebuild is single threaded. Note, this restriction is not there when you are rebuilding an index which supports multi-threaded ONLINE build. However, in both cases, OFFLINE build does support multi-threading.

(2) When you rebuild a HEAP and specify compression option, the corresponding non clustered indexes are also rebuilt because RIDs will change but the compression state of the indexes will not change. If you want to alter compression option of existing nonclustered indexes, you will need to execute ‘Alter Index’ command on each such indexes. So effectively, you will need to rebuild the nonclustered index twice if you want to change the compression setting both for the HEAP and its associated nonclustered indexes. Here is the rationale for this

a. Most index rebuild options are not applicable to HEAP. For this reason, we did not add ‘ALL’ keyword like the one with Alter Index command.

b. Since indexes are typically much smaller compare to the table, we think most customers will not want to change compression state on indexes automatically as part of rebuilding the HEAP or clustered index for that matter.

In the next BLOG I will discuss DML operations on the HEAP and compression

Thanks

Sunil

Does data compression lead to more fragmentation?

Sunil Agarwal — Wed, 06 Aug 2008 07:40:00 GMT

Lately I have been asked how data compression impacts fragmentation (i.e. does it cause more or less fragmentation?). I believe this question is best answered by looking at how does fragmentation occur in the first place and then analyze each of these cases for compressed and uncompressed data.

Let us start off with create clustered index on a table so there is no fragmentation to begin with. Now the following operations will cause fragmentation

· Delete Operaton: This means the pages are not as full as they can be. The delete operation will cause fragmentation both with/without data compression but the impact will be lower with data compression. For example, if you assume 50% compression (i.e. compressed data row is ½ the size of the uncompressed row) for an average uncompressed rowsize of 200 bytes. Now if we delete 10 rows on a page, the uncompressed page will have 2000 bytes of free space (i.e. 25% additional fragmentation within the page) while the compressed page will have 1000 free bytes (i.e. 12.5% additional fragmentation within the page)

· Insert Operation: when a row is inserted, the SQL Server will find the appropriate page to insert the row. If there is space in the page, the row will be inserted without additional fragmentation. However, if there is not sufficient space on the page to store the row, it will cause a page split which will potentially lead to fragmentation. You should see similar fragmentation with/without data compression

· Update Operation: An update operation can cause a page split if the update is causes rowsize to increase, a common case when updating variable length columns. As you may recall, both with ROW and PAGE compression, we store fixed length data type as variable length. So if you are changing the value of a say an integer column from 1 to 100000, it will cause the rowsize to increase from 1 byte to 3 bytes. If there was no space on the page, it will lead to page split thereby causing potential fragmentation. However, if you had not compressed the data, then fixed length columns take the maximum space possible in the row so such updates will not cause any changes to the size of the row. So if your application changes fixed length column types, you can potentially see additional fragmentation with data compression.

As you can see from the discussion above, the impact of data compression on fragmentation will depend on the schema, data distribution and the application. You can minimize fragmentation by using a lower fill factor but then you table will take more space.

Compression Strategies

Sunil Agarwal — Sun, 27 Jan 2008 07:20:00 GMT

In the previous blogs, I had described the data compression in SQL Server 2008 and its implications on CPU and IO. One of the question that keeps coming up is ‘Should I compress my database? Or Why does the SQL Server not provide an option to compress the whole database’. I am not suggesting that you may never want to compress the whole database, but here are some basic guidelines that you should keep in mind.

First, why compressing the whole database blindly is not such a good idea? Well, to give you an extreme example, let us say that you have a heavily used table T with say 10 pages in a database with millions of pages, there is no benefit in compressing T in this case. Even if the SQL Server could compress 10 pages to 1 page, you hardly made a dent in the size of the database and it added some CPU overhead instead. Remember, the data needs to be uncompressed every time it is accessed. Granted that in real life workload, the choices are not this obvious but it does hi-light the point that you must evaluate the pros/cons before you compress the data. Our recommendation is that before you compress an object (e.g. table index or a partition), you must look at its size, usage and estimated compression savings using sp_estimate_data_compression_savings stored procedure. Let us look at each of these in some detail

· If the size of the object is much smaller than the overall size of the database, then it does NOT buy you much in space savings even if most of it can be compressed.

· If the object is used heavily both for DML and SELECT operations, then you will incur additional CPU overhead that can impact your workload especially if it makes it CPU bound. You can use sys.dm_db_index _operational_stats to find the usage pattern of objects to identify which tables, indexes and partitions are being hit the most. In this case, special care is needed to evaluate if to compress these objects or not.

· The compression savings are schema/data dependent and in fact for some objects the size after compression may be larger than before or the space savings is very insignificant. Cearly, if the compression saving is not much relative to the size of the object and/or of the database, you may not want to compress it.

Some common strategies: As you look at the strategies below, keep in mind that the index/data pages in buffer pool have compressed data and the SQL Server needs to uncompress the row/column, not the whole page, each time column/row(s) needs to be retrieved. The suggestions below should be taken as a guideline. You must test it with your workload and then decide what compression is right for you to minimize any surprises.

· How do you choose between ROW and PAGE compression? If you have lot of DML activity, typical of an OLTP workload, generally ROW compression will be a better option as the cost of DML operations with ROW compression is much lower compared to PAGE compression.

· If you have a table with multiple indexes, only compress indexes that are used occasionally.

· If you have an index that is used mainly for singleton lookup, it needs careful consideration for compression as the cost of decompressing data while traversing the index tree will be expensive. At minimum, SQL Server will to need uncompress key values for each level in the index tree. And if this index is a nonclustered index on a table with the clustered index, the clustered index tree will need to be traversed as well. Interestingly, when you compression an index with PAGE compression option, all internal index pages are compressed with ROW and only the leaf pages are compressed with PAGE compression. The reasons are (1) the number of non-leaf pages are quite small < 2% compared to leaf pages for a fanout of > 50; (2) it reduces the cost of uncompression for non-leaf pages. Yes, one can argue that by compressing the non leaf pages with PAGE compression, we can increase the fanout thereby reducing the height of the index tree which indirectly reduces the number of key comparisons. For now, the nonleaf pages are only ROW compressed.

· If you have a partitioned table where data in some partitions is accessed occasionally, you may want to compress those partitions and associated indexes with PAGE compression. This is a common scenario with partitioned tables where older partitions are referenced occasionally. One example can by sales data partitioned by quarters across many years. Commonly the queries are run on the current quarter while data from other quarters is not referenced as frequently. When the current quarter ends, you can change its compression setting.

In the next blog, I will describe the DDL to enable/disable compression

Details on page compression (page-dictionary)

Sunil Agarwal — Sat, 19 Jan 2008 00:02:00 GMT

In the previous blog, I described column-prefix compression that is done as part of PAGE compression. The other component of PAGE compression is the page-dictionary. As the name indicates, this dictionary is specific to a page and the page can be data page or a leaf-index page. The non-leaf pages of an index are ROW compressed even if you choose to compress the index with PAGE compression option. It was a design decision to reduce the CPU impact (it is more expensive to decompress a row with PAGE compression compared to ROW compression) of index tree traversal but on the flip side the height of the index tree could be smaller depending on the data in the index key. Now back to page-dictionary, the topic of this blog.

The dictionary stores common values (same ‘byte’ pattern) across all columns and rows on the page and then each column refers to the dictionary for the value. The entries in page dictionary are stored as an array with 0 based index and can be referenced as 0, 1…and so on. If you have many occurrences of repeating values, you can get proportional compression savings. You may wonder how it is different than column-prefix compression. The key differences are

· Unlike column-prefix, a dictionary value can be referenced by any column of any row on the page. For example, if you have a byte pattern ‘0x11223344 in col1 of row-1 and col2 of row-2, they both can refer to the same dictionary value. You may recall that in column-prefix we look for common prefix in the same column across all rows on the same page.

· A dictionary entry is created only if the value is repeats in its entirety in two or more columns. You may wonder

o What happens if we have the repeating value in the same column across multiple rows on the page? Don’t we create a column-prefix for that? The answer is most likely ‘yes’ unless there is another column-prefix value that provides more space savings. The rule is that we first apply column-prefix compression on the page and then apply page-dictionary compression in that order.

o How do we cleanup dictionary when one of those rows gets deleted? Or when one or more entries in page-dictionary are no longer referenced? Hold that thought for a future blog.

Now let us build on the example that we discussed in the previous blog in column-prefix compression. Here is how the page looked after column-prefix compression.

__________________________________
| HEADER |

|___________________________________
|___________________________________|
| 2BBB | 3BC | <null> |

|_____________|_____________|_______|

| 3BC | 0BBBB | <null> |

|_____________|_____________|_______|

| <null> | <null> | 0BBBB |

|_____________|_____________|_______|

You will notice that the (row1, col2) and (row2, col1) have the same value 3BC. Similarly, (row2, col2) and (ro23, col3) have the same value ‘0BBBB’. These are the only repeating values on the page. Note, we are not considering <null> as they don’t take any storage. The page dictionary is created and stored along with the anchor record and is jointly referred to as ‘Compression Information’ or CI in short but not to be confused with clustered index.

So the modified page after applying page-dictionary compression will look as follows:

__________________________________
| HEADER |

|3BC|0BBBB | | <<-----dictionary

|___________________________________
| |

|___________________________________|
| 2BBB | 0 | <null> |

|_____________|_____________|_______|

|0 | 1 | <null> |

|_____________|_____________|________|

| <null> | <null> | 1 |

|_____________|_____________|________|

(row2, col1) and (row1, col2) have a value 0, the index into the page-dictionary entry with value 3BC and (row2, col2) and (row3, col3) have a value 1, the index into the page-dictionary entry with value 0BBBB. When the value (row2, col1) is to be retrieved as part of say SELECT statement, the SQL Server will perform the following steps

· Access the column value as it is stored in the row2. It retrieves the value 0.

· Looking at the metadata in the row, SQL Server knows that 0 does not mean that the column has a value 0 but that it needs to access first entry in the dictionary. So it retrieves 3BC.

· By looking at the row metadata, SQL Server knows that the value 3 represents the number of bytes to pick from the column prefix in the anchor row, so it computes the value AAABC and returns it.

As you can see that with PAGE compression, we are able to reduce the redundant information on the page significantly but at the cost of extra CPU cycles. Note that the space savings achieved will depend on the data. In fact, two different pages of the same table, you may see different space savings and can even be negative (i.e. the size actually increased after compression). This brings up some interesting questions as follows

· Should I enable PAGE compression? The answer is to use the stored procedure sp_estimate_data_compression_savings to estimate the space savings. Clearly, if there is no or insignificant space savings with PAGE compression, there is no reason to enable it.

· PAGE compression does provide significant space savings on the table but a subset of pages have no data redundancy that can be exploited by PAGE compression. What does SQL Server do in this case? The answer is that SQL Server evaluates space savings with PAGE compression on each page and only-if the space savings is greater than an internally defined threshold, it is compressed. So a likely case is that your table with PAGE compression option will have a subset of pages that are PAGE compressed and others not.

In summary of this blog, let me emphasize that the space savings is data dependent and that the space savings come at the cost of additional CPU incurred when retrieving or manipulating compressed data. In the next blog, I will describe common scenarios for enabling compression to achieve a good balance between space savings and CPU overhead. As before, please feel free to send me your feedback/comments

Details on PAGE compression (column-prefix)

Sunil Agarwal — Fri, 18 Jan 2008 05:38:00 GMT

In the previous blog, I had mentioned that the PAGE compression is used to “minimize the data redundancy in columns in one or more rows on a given page”. You may wonder what exactly SQL Server does under the PAGE compression cover? With PAGE compression, the SQL Server eliminates two types of data redundancy known as column-prefix compression and dictionary compression. I will describe column-prefix compression in this blog followed by dictionary compression in the next.

Under column-prefix compression, the SQL Server looks for common ‘byte’ pattern at the beginning of a column across all rows on the page. If it finds at least two instances of the same columns with some common byte pattern, it stores that ‘byte’ pattern once on the page and refers to this byte pattern from those respective columns. This process is repeated for all columns in the row. Now, there are two things that need clarification here.

· We are referring to the ‘byte’ pattern. This means that under column-prefix compression, the SQL Server is only looking at bytes that are common without regards of the data type (i.e. it is type agnostic). It does not matter if the common bytes represent a part of integer value or float or character string.

· Common ‘byte’ pattern is matched starting from the beginning of the columns, hence the name column-prefix compression. So for example, if you have two values represented in hexadecimal representation as 0x020406FF and 0x0204CCFF. In this case, the common ‘byte’ pattern is ‘0x0204’ only and we don’t consider the value FF which is the 4^th byte because the 3^rd byte was different. Similarly, if you have two values as 0xFFAABBCCDDEE and 0x33AABBCCDDEE, there is no common ‘byte’ pattern or column prefix because the first byte is different even though the next 5 bytes are exactly the same. One can argue that we could potentially compress this too and you are obviously right. We decided not to compress it because our design goal was to find the balance between compression savings and CPU overhead of compressing/de-compressing the data.

You may wonder where and how SQL Server stores this common ‘byte’ pattern? SQL Server stores the column-prefix data in a new record, called the anchor-record. Anchor record has the same schema as the regular row (data or index) and is stored on the same page immediately following the page header. If a column does not have a column-prefix value, it is treated as empty or null just like any other empty or null value stored in a regular data or index row. If none of the columns have column-prefix, the anchor record is not created. During column-prefix compression, the SQL Server stores meta-information in the data or index record so that the full column value can be constructed. Let us consider an example to illustrate this.

The following figure shows a data page without column-prefix compression. For simplicity, this page shows three data rows with three columns of type CHAR. Like I had mentioned before, column-prefix compression is 'data type' agnostic so a similar example will hold for other data types as well.
______________________________________
| HEADER |

|_____________|_____________|_________|

| AAABC | BBBB | ABCD |

|_____________|_____________|_________|

| AAACCC | CCCDD | BBBB |

|_____________|_____________|_________|

Now, let us look at each column and find the common ‘byte’ pattern. For the first column, the common byte pattern is ‘AAA’, for the second column it is ‘CCC’ and for the third column it is ‘ABCD’. Using this information, we can create an anchor record. So the modified page will look as follows

__________________________________
| HEADER |

|_____________|_____________|_________|

| AAABC | BBBB | ABCD |

|_____________|_____________|_________|

| AAACCC | CCCDD | BBBB |

|_____________|_____________|_________|

I want to bring your attention to two points. First, the anchor record is just like any other data or index row but SQL Server knows to treat this record differently. So for example, you can never retrieve anchor record as part of SELECT query. Second, the anchor record not only stores the common byte pattern for each column but it also stores the value of the largest column containing the common byte pattern. So we store ‘AAACCC’ for the first column and similarly for the others. Now you may wonder where are we getting the space savings from? Well, we are not done with column-prefix yet. Once the anchor-record has been created, we need to refer to it from the respective columns.

So the ‘final’ modified page will look as follows. The first column in the first row has a value 2BBB which means: take the first 2 bytes from the corresponding column of the anchor record and then concatenate that value with ‘BBB’ so you will get the value ‘AABBB’. Another example is for the third column of row 2, the value <null> indicates to take the value of this column from the anchor record which evaluates to ‘ABCD’. Note, there is meta-information in the record that helps SQL Sever differentiate between this <null> and actual <null> value. You can see that the space savings will depend upon size of the column-prefix and the number of columns referencing it. Again, the SQL Server associates meta-information to differentiate between the ‘number’ of bytes to be fetched from the corresponding column in the anchor record and the actual data for the column (i.e. how do we know that the first byte is the length of prefix bytes or part of the column value). The value 0BBBB in the second column of the second row represents that pick 0 (zero) bytes from the corresponding column in anchor record. For this case, the real value of the column is ‘BBBB’. As you can see that constructing the real value of the column is more expensive than if the data was not compressed. Similarly, when a column value is modified or a new row is inserted, the SQL Server has to look at anchor-record and do the space optimization when applicable. So the cost of compression is paid both by SELECT and for DML operations.

__________________________________
| HEADER |

|_____________________________________
|_____________________________________|
| 2BBB | 3BC | <null> |

|_____________|_____________|_________|

| 3BC | 0BBBB | <null> |

|_____________|_____________|_________|

| <null> | <null> | 0BBBB |

|_____________|_____________|_________|

In the next blog, I will describe the dictionary compression.

Data Compression will be available in CTP-6

Sunil Agarwal — Tue, 20 Nov 2007 22:39:00 GMT

Thanks to all who have contacted me for their interest in the data compression feature.

Unfortunately, data compression feature is NOT available in the just released CTP-5 which is now available on the Microsoft Download Center as SQL Server 2008 Community Technology Preview (CTP) November 2007.

Data compression will be released with CTP-6. Also, Data Compression is an Enterprise Only feature.

Estimating the space savings with data compression

Sunil Agarwal — Tue, 13 Nov 2007 03:54:00 GMT

Like I had indicated in my previous blog, it is recommended that you estimate compression savings on the object of interest before actually enabling compression as enabling compression is an expensive operation. To show how to estimate data compression, I have created a very simplistic and definitely not realistic example to show case data compression estimates for the following table.

create table t1_big (c1 int, c2 int, c3 char(8000))

I will show you the compression estimates both for ROW and PAGE compression. You may recall that

· ROW compression removes unneeded bytes from the column values by storing them in variable length format.

· PAGE compression stores the repeating value only once per page and sets the pointer from the respective columns within the page.

Now load 6000 rows into this table. Since there is one row per page, the size of this table will be 48 MB (6000 * 8K = 48MB). Note that for the third column, we are inserting has same value for all the rows and that we are only inserting 60 characters even though the size of of the column is 8000.

declare @i int

select @i = 0

while (@i < 6000)

begin

insert into t1_big values (@i, @i + 6000, replicate (‘a’, 60))

set @i = @i + 1

end

-- find the current size of the uncompressed table

EXEC sp_spaceused N't1_big'

The output of this command is as follows showing that the size of the table is 48 MB as we expect.

Name Rows Reserved Data Index_size unused

t1_big 6000 48008 KB 48000 KB 8 KB 0 KB

Now, estimate the size of compression by executing the following stored procedure. When you execute this stored procedure, you are instructing SQL Server to find compression savings with ROW compression on the table t1_big. The parameter 3 and 4 refer to index-id and partition-id. You can use this stored procedure to estimate compression savings even for an index or for a specific partition(s).

exec sp_estimate_data_compression_savings

'dbo', 't1_big', NULL, NULL, 'ROW'

Here is the “re-arranged” output of the above command for better readability. There are few key points to note. First, the estimated space savings from ROW compression is around 99% for this example (the estimated size goes from 48MB to ½ MB). Though this kind of space savings are not realistic but it is easy to see why we got such a huge savings for the current schema. For example, in this case, the majority of space savings come from removing the trailing blanks from column C3. Second, when estimating compression savings, the SQL Server creates a sample table/index in tempdb and then compresses it with the requested compression mode to find the space savings. The stored procedure extrapolates the savings achieved on the sampled table to the original table.

object_name schema_name index_id partition_number

----------- ----------- ------- -----------------

t1_big dbo 0 1

size_with_current_compression_setting(KB)

-----------------------------------------

48008

size_with_requested_compression_setting(KB)

-------------------------------------------

472

sample_size_with_current_compression_setting(KB)

-----------------------------------------------

39648

sample_size_with_requested_compression_setting(KB)

--------------------------------------------------

392

Note, in the case of ROW compression, all SQL Server does is to remove the trailing blanks from the CHAR(8000) column type and remove extra bytes from integer values. Since we have the same value (60 characters) for all the rows for column C3, this value will need to be stored only once on the page with PAGE compression. So it is easy to see why PAGE compression will lead to huge space savings. Here is the estimate with PAGE compression.

-- estimate the PAGE compression

exec sp_estimate_data_compression_savings

'dbo', 't1_big', NULL, NULL, 'PAGE'

Here is the output that shows the estimated compression to be

object_name schema_name index_id partition_number

----------- ---------------- ---------- -----------------

t1_big dbo 0 1

size_with_current_compression_setting(KB)

-----------------------------------------

48008

size_with_requested_compression_setting(KB)

-------------------------------------------

sample_size_with_current_compression_setting(KB)

-----------------------------------------------

39960

sample_size_with_requested_compression_setting(KB)

--------------------------------------------------

In my next blog, I will show you the estimates for a more realistic example.

Types of data compression in SQL Server 2008

Sunil Agarwal — Mon, 12 Nov 2007 20:17:00 GMT

SQL Server deploys two strategies to compress the data

· First, it stores all fixed length data types in variable length format. If you recall, SQL Server 2005/SP2 provided a new variable length storage format decimal and numeric. Please refer to the series of blogs vardecimal-storage-format for details. SQL Server 2008 extends this to all fixed length data types. Some examples of fixed length data types are integer, char, and float data types. One important point to notes is that even though, the SQL Server stores these data types in variable length format, the semantics of the data type remains unchanged (i.e. it is still the fixed length data type from the perspective of the application). This means that you can avail the benefits for data compression without requiring any changes in your application(s).

Let us consider some examples to illustrate the space savings you can get by storing data in variable length storage format.

Example-1: Let us say you have a table that has an integer column along with many other columns. If the common integer value of this column can fit in 1 byte (i.e. the value is <= 255), then there is no reason to take 4 bytes to store it. With variable length storage format, SQL Server can shave off 3 bytes from most integer values for this case there by saving you 3 bytes off 4 byte value, a space saving of 75%.

Example-2: Let us say you have a table that has a CHAR(100) column. In fixed length format, this column will take 100 characters independent of the actual value stored in it. So if you store “Hello” or “This is a longer string”, both values will take space required to store 100 characters. However, when CHAR (100) is stored in variable length storage format, it will only take 5 characters for the first value (“hello”) and 23 characters for the second value (“This is a longer string”). You can see in this case, we are able to reduce the size of the first value by 95% and for the second value by 77%. A significant space saving.

Of course, if the character value in example-2 has many more characters or if the common integer values in example-1 take 2 or more bytes, the space savings will be lower. This means the space savings you can achieve will depend on the data distribution. Other point that is not so obvious is that the space savings achieved will also depend on the schema of the table as well. So for example, if the column was declared to be CHAR (150) instead of CHAR (100), you would get significantly more space (i.e. additional 50 characters) savings. Also, note as a special case, the 0s and NULL values require no storage.

Storing values in variable length format will typically get you some space savings, but there is a catch. When a fixed length value is stored in variable length format, the SQL Server needs to store its offset within the row. Note, this offset was not needed when the column value was stored as fixed length because its relative position in the row was always at the same offset. If you recall the record storage format in SQL Server record-structure-blog-post, it takes 2 bytes to store the offset of each variable length column. Given this, it makes little sense to create a column of type varchar(2) as the minimum overhead is 2 bytes already. Same is true for data type such as smallint or even for integer for that matter. SQL Server 2008 addresses this issue by introducing a new record format to be used for compressed data that minimizes the overhead of storing the variable length values. It uses only 4 bits of overhead to store the length of the variable length column that is less than or equal to 8 bytes. Note that the previous record format is fully supported in SQL Server 2008 and is the default record format.

This strategy is exposed as ROW compression externally through DDL. Also note that ROW compression is a superset of vardecimal storage format and that vardecimal storage format is fully supported in SQL Server 2008.

· Second, it minimizes the data redundancy in columns in one or more rows on a given page. It does that by storing the redundant data once on the page and then referencing it from the multiple columns. So for example, consider the following table

Table employee (name varchar(100),

status varchar (10) default ‘full time’)

Now, if you insert multiple rows into this table, it is possible that many rows will have the same value (i.e. the default) for status column. SQL Server can take advantage of this by storing ‘full time’ value once on the page and then referring to this value from all other occurrences of this value on the same page. As you can imagine, the space savings using this strategy will depend on the amount redundant data on the page.

This strategy is exposed as PAGE compression externally through DDL. A customer can choose to enable ROW or PAGE compression on a table or an index or even on an individual partition(s). PAGE compression includes ROW compression.

I will provide details on PAGE compression later but for now it will suffice to know that ROW compression means storing fixed length data in variable length storage format and PAGE compression means minimizing the data redundancy.

As you can imagine, enabling compression is an expensive operation because the uncompressed data needs to be converted to new record format and for PAGE compression, the data redundancy needs to be minimized. So before you enable compression, you may want to know how much space savings you can get. In the next blog, I will describe how to estimate compression savings.

Why not use compressed disk files or disk volumes?

Sunil Agarwal — Mon, 12 Nov 2007 07:19:00 GMT

When we think of compressing the database, one of the first question that pops up is why not used the compressed volume? You may know that Windows OS has supported compression of individual files, folders and the entire NTFS volumes since Windows 2000. Given this, you may wonder why don’t we just create databases on compressed volume and get the disk savings?

Well, there are few issues with it as follows

· Generally the compression methods are effective on large chunks of data. So you may need to compress many pages (think in mega bytes) together before you can get good compression. This runs counter to how data is actually accessed in the database. Most user queries access data “randomly” and in “small” parts. This means that accessing random data will force reading many pages on the compressed volume/file and this needs to be followed by decompression before the requested data can be retrieved. This becomes prohibitively more expensive.

· When data is read into the SQL Server memory (i.e. buffer pool), it is uncompressed. So there is no benefit from memory perspective either.

· The IO on compressed volume is synchronous which will impact the performance of your workload. Here is some information picked up from an excellent white paper written by Bob Dorr http://www.microsoft.com/technet/prodtechnol/sql/2000/maintain/sqlIObasics.mspx . “SQL Server relies heavily on asynchronous I/O capabilities to maximize resource usage. SQL Server Support has debugged problems with some filter drivers that do not allow the I/O request to be completed asynchronously. Instead, the filter driver requires the I/O to complete before returning control to SQL Server. This is easily observed by watching the disk queue lengths. When SQL Server is running, it commonly keeps more than one I/O posted. When I/Os become synchronous, the disk queue is often held at one outstanding I/O. This causes the SQL Server code to unnecessarily block. Because the disk sec/transfer time may not be a fair statistic, use it with caution. When less I/O is outstanding on a drive, the disk sec/transfer is often fast. The longer the disk queue length, the more variations that can occur on disk sec/transfer. However, because SQL Server understands the asynchronous nature of the I/O patterns, longer disk queue lengths, and somewhat longer disk sec/transfer can result in overall better throughput and resource utilization.”

In the next blog, I will describe how SQL Server compresses the data.