SQL Server Storage Engine : VarDecimal Storage Format

White Paper: Reducing Database Size by Using Vardecimal Storage Format

Sunil Agarwal — Wed, 30 May 2007 01:11:00 GMT

Hermann Daeubler and Sunil Agarwal have co-authored this white paper. Hermann is a Program Manager with the SQL team and is a SAP expert. Hermann was involved with vardecimal storage format team from the very beginning to guide us with the customer scenarios and the performance implications/trade-offs. This paper provides a general overview of vardecimal storage format usage scenarios, restrictions, database migration to SQL Server 2005/SP2 in the context of vardecimal, space savings and the impact on the performance. You can access the paper at

http://msdn2.microsoft.com/en-us/library/bb508963.aspx.

Data row before and after vardecimal storage format

Sunil Agarwal — Sun, 07 Jan 2007 06:22:00 GMT

Paul Randal in one of his earlier BLOGs described DBCC Page paul-tells-all

and the record layout. I thought it will be interesting to show how a row looks before and after the Vardecimal storage format is enabled. So here it is

Let us take a simple table

create table t_simple (c1 char (5), c2 decimal (38,2))

insert into t_simple values ('aaaaa', 1.0)

If you run the command DBCC Page with option 3, you will get the following output

….

Slot 0 Offset 0x60 Length 29

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP

Memory Dump @0x44D0C060

00000000: 10001a00 61616161 61016400 00000000 †....aaaaa.d.....

00000010: 00000000 00000000 00000200 fc††††††††.............

Key things to note here is that row length is 29 bytes computed as follows

Record Header = 4 bytes
Column C1 = 5 bytes
Null bit map and column count = 3 bytes
Fixed length decimal value = 17 bytes

Now, let us enable Vardecimal storage format on this table. The following shows the row in the new storage format

Slot 0 Offset 0x60 Length 18

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS

Memory Dump @0x44E8C060

00000000: 30000900 61616161 610200fc 01001200 †0...aaaaa.......

00000010: c019†††††††††††††††††††††††††††††††††..

Slot 0 Column 0 Offset 0x4 Length 5

c1 = aaaaa

Slot 0 Column 1 Offset 0x10 Length 2 [VarDecimal/VarNumeric]

c2 = 1.00

Note, now the row length is 18 bytes. So the size of the row is reduced from 29 bytes to 18 bytes representing a reduction in the size of the row of around 30%. Couple of other interesting points

Decimal value is now stored in variable length portion of the record. The value is represented as ‘c019’, which is just 2 bytes.
Since C2 now becomes the first variable length column, you see an overhead of 4 bytes for storing variable length column count (2 bytes) and offset array (2 bytes)

Boundary conditions for enabling vardecimal storage format

Sunil Agarwal — Fri, 05 Jan 2007 05:45:00 GMT

Have you ever tried updating a variable length column and fail? Well, it can happen if the modified row cannot fit on the page. One simple example of this as follows

create table boundary (c1 char(8000), c2 char(20),

c3 varchar(23), c4 decimal(38,2))

When you create this table, it gives you the following warning

Warning: The table "boundary" has been created, but its maximum row size exceeds the allowed maximum of 8060 bytes. INSERT or UPDATE to this table will fail if the resulting row exceeds the size limit.

-- this is the max allowed value

insert into boundary values ('a', 'b', replicate ('1', 12), 0.0)

-- this update fails with the error

Msg 511, Level 16, State 1, Line 1

Cannot create a row of size 8061 which is greater than the allowable maximum of 8060.The statement has been terminated.

update boundary set c3 = replicate('1', 13)

However, if you update a fixed length column value, you will never get the error 511 because the fixed length value, by definition, retains the same size regardless of the value. Most applications that update a fixed length column value don’t check for 511 error.

You may wonder what happens when we enable vardecimal storage format on a table? Since the numeric/decimal data is now stored using variable length storage, an update to decimal/numeric value can potentially fail just like it failed when updating varchar column in the example before. Since applications are not expecting updates to decimal/numeric column to fail (unless of course there are constraints defined on the decimal/numeric value), the application may encounter unexpected failure. To prevent this from happening, SQL Server only allows enabling vardecimal storage format on a table if and only if it can gurantee that updates to decimal/numeric value will nevel fail with error 511. So for the table in the above example, if I enable vardecimal storage format as follows

sp_tableoption 'boundary', 'vardecimal storage format', 1

You will get the following error.

Msg 1721, Level 16, State 2, Procedure sp_tableoption, Line 129

Altering table 'boundary' has failed because the row size using vardecimal storage format exceeds the maximum allowed table row size of 8060 bytes.

Reason:

For the insert above, since the decimal column as 0.0 value, it takes only 2 bytes (offset array) of storage in Vardecimal storage format. Had SQL Server allowed Vardecimal storage format on ‘boundary’ table, then the following update to decimal data can fail:

Steps:

(1) Update column c3 to ‘1234567890124567890’. When you do that, the row size becomes 8053 bytes.

(2) Now update the decimal column c4 to max digits (38) allowed which requires 18 additional bytes of storage but we have only space for (8060 - 8053) = 7 bytes

Please ignore the size computations, if you are not familiar with it but the point is that SQL Server guarantees that your update to decimal/numeric will not fail with error 511. You can read http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/23/644607.aspx for more details on row format.

Since tables defined in most application schemas, the row size is much less than 8060, we don’t need to concern ourselves with this issue. But it is reassuring to know that your application will not be caught off guard when Vardecimal storage format is enabled.

Besides this, SQL server allows enabling Vardecimal storage format on a table only if it can guarantee that you can always disable it.

Vardecimal Storage Format and its implications on Backup/Recovery

Sunil Agarwal — Fri, 29 Dec 2006 23:54:00 GMT

Has any one tried restoring or attaching a SQL Server 2005 database on SQL Server 2000? You will find that SQL Server 2000 will fail this restore or attach. The reason is simple. SQL Server 2000 does not understand the physical structure changes in SQL Server 2005 database. SQL Server detects this incompatibility using the database version stored in the bootpage of the database. You cannot attach/restore a database with higher database version on a SQL Server that does not support it. Note that it is always possible to attach/restore a SQL Server 2000 database to SQL Server 2005. In fact, this must be allowed as otherwise customers will not be able to upgrade their database to run with newer SQL Server versions. In this case, the SQL Server 2005 understands the physical structures of SQL Server 2000 database and converts them to SQL Server 2005 specific structures during the upgrade process.

You may wonder why I am telling you all this in the context of Vardecimal storage format? Well, the Vardecimal storage format is a new storage format, introduced in SQL Server SP2, to store decimal/numeric data. This new storage format is not understood by SQL Server 2005 or SQL Server 2005/SP1. Just like you cannot attach a SQL Server 2005 database to SQL Server 2000, attaching/restoring a SQL Server 2005/SP2 database that has been enabled for Vardecimal storage format to earlier versions of SQL Server 2005 will fail. SQL Sever implements this by incrementing the database version number when the database is enabled for Vardecimal storage format. When you disable Vardecimal storage format on a database, its database version is decremented so that the database can now be attached to earlier versions of SQL Server 2005. There is one interesting scenario/requirement that you need to be aware of when disabling Vardecimal storage format on the database.

Scenario:

· Do a full physical backup (DB) on SQL Server 2005/SP1. This backup will have the database version that is supported by SQL Server 2005/SP1.

· Attach the database to SQL Server 2005/SP2.

· Enable Vardecimal storage format on the database.

· Create a table and enable it for Vardecimal storage format

· Insert one row. The log records generated by this will have data in Vardecimal storage format

· Disable Vardecimal storage format on the database.

· Do the logs backup (L). Again, this log backup will have the database version that is supported by SQL Server 2005/SP1.

Now if you restore (DB + L) on SQL Server/SP1, it will not be able to detect that there were log records with Vardecimal storage format and will potentially fail unpredictably. To prevent this situation, SQL Server requires you to set database in SIMPLE recovery mode before you can disable Vardecimal storage format on the database. When you do that, the log chain is broken and the above scenario is prevented. Note that you are not required to set database in SIMPLE recovery mode when disabling Vardecimal storage format on individual table(s). It is ONLY needed when disabling it on the database.

Enabling vardecimal storage format

Sunil Agarwal — Mon, 13 Nov 2006 19:33:00 GMT

First, this feature is only available in EE and Dev SKU.

Enabling vardecimal storage format on a table is a two step process as follows:

First you need to enable database for Vardecimal storage format. This can be done using the stored procedure sp_db_vardecimal_storage_format. The exact command is as follows

exec sp_db_vardecimal_storage_format '<dbname>', 'ON'

When the above command is executed, SQL Server internally bumps the database version number but no tables are enabled for Vardecimal storage format. The database version needs to be bumped to indicate that the data in this database can potentially have a different storage format (i.e. the Vardecimal storage format). This is used to prevent attaching a Vardecimal enabled database to earlier versions of SQL Server 2005 as those versions don’t know how to interpret the new storage format. You can only enable Vardecimal storage format on user databases.

To find out which database(s) is enabled for Vardecimal storage format, you can use the following command

exec sp_db_vardecimal_storage_format

Once you have enabled the database for Vardecimal storage format, you can now choose to enable one or more tables (based on the potential disk savings using the tool described earlier) with this new storage format as follows

sp_tableoption '<table-name>', 'vardecimal storage format', 1

This command, potentially an expensive one (same order as creating an index), converts all the rows in the table containing columns of type decimal/numeric to Vardecimal storage format. During this conversion, the table is locked and is not available. If the table has no clustered index, then all non-clustered indexes are rebuilt because the RIDs of the rows will change due to storage format change. However, if you have clustered index on the table, then only the non-clustered indexes containing decimal numeric column as key or included column need to be rebuilt. Note, that you cannot enable vardecimal storage format on all tables. Before enabling vardecimal storage format, SQL Server needs to make sure that we can always revert back to static storage format for decimal data and that update of decimal/numeric data always succeeds. If these conditions are not satisfied, the conversion to vardecimal storage format is denied.

To disable Vardecimal storage format on the table, you can use the following command

sp_tableoption '<table-name>', 'vardecimal storage format', 0

SQL Server guarantees that you can always revert back to ‘static’ storage format unless you run out of disk space during conversion. Note, that the space overhead, to enable/disable Vardecimal storage format, is of the same order as building an index and it is not an online operation.

You can use the following command to find out which tables(s) has been enabled for Vardecimal storage format

select objectproperty(object_id('<table-name>’), 'TableHasVarDecimalStorageFormat')

select name, object_id, type_desc

from sys.objects

where objectproperty(object_id, N'TableHasVarDecimalStorageFormat') = 1

I will discuss the boundary conditions for enabling vardecimal storage format and the implications of changing database version on backup/recovery and mirroring in next blogs.

Estimating the space savings with vardecimal storage format

Sunil Agarwal — Mon, 13 Nov 2006 05:33:00 GMT

Before enabling Vardecimal storage format, you may want to know the potential reduction in the size of the table. Clearly, if the table has no decimal/numeric columns, there will be no savings. Note, that even if you have a table with decimal/numeric column types, there is no guarantee that you will be able to reduce the size of the table by enabling Vardecimal storage format. Again, this issue is similar to VARCHAR (17) vs. CHAR(17). If all the values in the column type has 17 characters, then average row length will be larger with VARCHAR(17) because it will be stored in the variable portion of the record structure. Recall, you need 2 bytes to store the offset of the variable length column. Also, if VARCAHR(17) is the only variable length column in the table, there is another overhead of 2 bytes to store number of variable length columns in the row. So in this case, the worst case, declaring column type as VARCAHR(17) may cost you 4 bytes more for each row than CHAR(17).

SQL Server 2005/SP2 provides you a tool, a stored procedure, to estimate the ‘reduction in row size’ with Vardecimal storage format. The following example illustrates the reduction in row size for two tables that have same scheme but different data, t_decimal being the best case and t_decimal2 being the worst case (where each decimal value has max 38 digits as allowed by the declared precision)

create table t_decimal (c1 int, c2 decimal(10,2), c3 decimal (38,2), c4 varchar(10))

create table t_decimal2 (c1 int, c2 decimal(10,2), c3 decimal (38,2), c4 varchar(10))

-- insert rows into these tables.

declare @i int

select @i = 0

while (@i < 1000)

begin

insert into t_decimal values (1, 0.0,0.0, 'hello')

insert into t_decimal2 values

(1,12345678.99,123456789012345678901234567890123499.99, 'hello')

set @i = @i + 1

end

-- Now let us find the potential space savings for each of these tables

-- This is the best case

exec sys.sp_estimated_rowsize_reduction_for_vardecimal 't_decimal'

Here is the output. Note, in this case, you can reduce the size of the row by almost 50%. Also, if you have more decimal/numeric columns, the savings will be proportionally larger.

avg_rowlen_fixed_format avg_rowlen_vardecimal_format row_count

--------------------------------------- --------------------------

46.00 24.00 1000

-- this is worst case. Note in this case, the average row length actually increases

-- with Vardecimal storage format.

exec sys.sp_estimated_rowsize_reduction_for_vardecimal 't_decimal2'

avg_rowlen_fixed_format avg_rowlen_vardecimal_format row_count

------------------------- ---------- -------------------- ------

46.00 48.00 1000

In the next blog

http://blogs.msdn.com/sqlserverstorageengine/archive/2006/11/13/enabling-vardecimal-storage-format.aspx , I will discuss how to enable vardecimal storage format on the table

Reducing the Size of your Database in SQL Server 2005/SP2

Sunil Agarwal — Fri, 10 Nov 2006 20:48:00 GMT

An exciting new feature in SQL Server 2005/SP2 is Vardecimal Storage Format. This storage format lets you reduce the size of your table significantly if the table has one of more columns of type decimal or numeric without requiring any changes to your application.

Up until now, the decimal and numeric types are stored as fixed length data in SQL Server. Both of these types are functionally equivalent and have a format of (p, s) where p is the precision (number of decimal digits) and s is the scale representing number of digits after the decimal. Depending on the precision (it can be declared in the range from 1 to 38), the decimal value can take anywhere from 5 bytes to 17 bytes. This can bloat the size of the table, especially when you have small decimal values for a column declared with high precision requirement. This issue is similar to char (17) vs. varchar(17). In this case, if most of your character data is 1 or 2 characters long but the max value is 17 characters long, you can reduce the size of the table by declaring the column to be of type varchar(17) instead of char(17).

The new vardecimal storage format stores the decimal/numeric values in a variable length storage format. It provides efficient storage of decimal/numeric data by eliminating the leading/trailing zeros and only storing the minimum required bytes. Using this format, you can get significant space savings (depending on your data distribution) in the space required to store decimal/numeric data. You can enable vardecimal storage format at a table level.

In our in-house testing, we have seen significant reduction in the size of the FACT table(s) that has large number of decimal columns. FACT tables are typically the largest table in a Data Warehouse. Here are some the numbers from our testing.

Best case reduction in the size of the table

57%

69%

51%

In my next post http://blogs.msdn.com/sqlserverstorageengine/archive/2006/11/13/estimating-the-space-savings-with-vardecimal-storage-format.aspx , I will describe when and how to enable Vardecimal storage format on one or more tables in your database.