Microsoft SQLCAT ISV Program Management Team

Asynchronous Population of a Cursor May Result in Performance Degradation

mssqlisv — Wed, 29 Jul 2009 19:22:00 GMT

SQL Server offers an option to asynchronously populate API created keyset and static server cursors via the sp_configure ‘cursor threshold’ parameter. When a cursor is asynchronously populated, a secondary thread is spawned to populate the cursor asynchronously in the background, while the original thread returns control to the client.

The default value for ‘cursor threshold’ is -1, which implies that the cursor will be populated synchronously. When ‘cursor threshold’ is set to a value equal to or greater than 0 (max = 2147483647), the SQL Server query optimizer estimates the number of rows that will be returned from the query execution and if the value is less than the ‘cursor threshold’ then synchronous population is used, else asynchronous population is used (refer to SQL Server Books Online for a detailed explanation). Asynchronous population for cursors is particularly useful for large result-sets where it may be preferable for the client application to not have to wait for the entire result set to be completely materialized before the data is accessible. Asynchronous population is not advisable for small result sets because the overhead of initiating a secondary thread for the cursor population often outweigh the benefits.

Recently we encountered a case where a customer accidentally changed the value of ‘cursor threshold’ from its default of -1 to 0 on their production database system for a large ERP (Enterprise Resource Planning) application. This rather small and what the DBA thought was an insignificant change caused a huge performance degradation, because ‘cursor threshold’ = 0 instructed SQL Server to populate all keyset and static cursors that had 0 or more rows to use a secondary thread for asynchronous population. To further complicate matters the database administrator did not remember exactly which parameter he had changed and what the original value was.

Debugging this problem was a bit challenging. We spent considerable time investigating the different attributes of the database server and finally managed to get to the root of the problem using data collected from two Data Management Views (DMVs): sys.dm_exec_cursors and sys.dm_os_tasks. The output of these DMVs showed that the keyset and static cursors were getting populated asynchronously which was not a recommended setting for the particular ERP application.

To present what we found by way of an example, we have executed a single keyset cursor query that populates a large result set. While the query is actively populating the result set of the keyset cursor we queried the two DMVs using the following T-SQL statements via a separate session.

SELECT session_id, cursor_id, properties, is_async_population FROM sys.dm_exec_cursors(0);

SELECT * FROM sys.dm_os_tasks

WHERE session_id = 52;

The screenshot shown in Figure (i) depicts the output of the DMVs when the ‘cursor threshold’ value is set to -1 (default).

Figure (i): DMV output corresponding to synchronous population of a keyset cursor.

As can be seen in the sys.dm_exec_cursors query output, session_id 51 which corresponded to the session through which we were executing the keyset cursor query has the is_async_population bit flag set to 0 during the cursor population phase signifying that the result set was being populated synchronously. In addition the output of the sys.dm_os_tasks query shows session_id 51 waiting in the RUNNABLE state during the cursor population phase.

Next, when we change the ‘cursor threshold’ value to 0 using the statement (sp_configure 'cursor threshold', 0) and the repeat the experiment. Figure (ii) below depicts the result of this experiment.

Figure (ii): DMV output corresponding to asynchronous population of a keyset cursor.

As can be seen in the sys.dm_exec_cursors query output, session_id 51 which again corresponded to the session via which we were executing the keyset cursor query now has the is_async_population bit flag set to 1 during the cursor population phase which signifies asynchronous population of the result set. Note: The is_async_population bit flag is only set to 1 during the actual population phase; once the population phase is complete the flag is set to 0 even if the cursor was populated asynchronously. The asynchronous population was further corroborated via the output of the sys.dm_os_tasks query in which we see two tasks associated with session_id 51 during the cursor population phase, one corresponding to the original query (SUSPENDED status) and a secondary one associated with the asynchronous population (RUNNING status).

So, our recommendation is to not change the instance-wide sp_configure ‘cursor threshold’ setting unless you are sure that the change will have a net positive impact on all applications running on that instance and have extensively tested the performance with the new value in a test environment.

Interesting issue with Filtered indexes.

mssqlisv — Mon, 29 Jun 2009 22:56:00 GMT

Recently, an ISV I work with ran into an interesting problem with Filtered Indexes.

The application does all SQL INSERT, DELETE, and UPDATE operations using individual stored procedures. To improve performance, they decided to create a Filtered Index (new in SQL 2008) to restrict the data such that it does not contain NULL values, in this case drastically reducing the number of rows in the index. See documentation here: http://msdn.microsoft.com/en-us/library/ms175049.aspx

This all worked fine, until months later, when the ‘UPDATE’ stored procedure was updated as part of a routine application upgrade. The stored proc was replaced with a newer version using a TSQL script. After adding the ‘new’ stored proc the application was tested and the following was observed;

· the SELECT statements accessing the table continued to use the filtered index

· the ‘INSERT’, and ‘DELETE’ stored procs continued to work

· However, the modified ‘UPDATE’ stored proc returned the following error:

InnerException: System.Data.SqlClient.SqlException: UPDATE failed because the following SET options have incorrect settings: 'ANSI_NULLS, QUOTED_IDENTIFIER'. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.

Note that the error message is very explicit and actually points us to the source of the problem. Can you guess what it is?

You are correct! The TSQL script used SET options that were set differently for the new ‘UPDATE’ stored proc than they were originally, and they don’t conform to the rules required to utilize Filtered Indexes. In this case, the SET ANSI_NULLS and QUOTED_IDENTIFIER were set to an invalid setting.

The solution to the problem was to recreate the stored proc using the ‘correct’ settings required to use Filtered indexes, documented here: http://msdn.microsoft.com/en-us/library/ms188783.aspx

The following simplified TSQL example shows the problem and the solution.

NOTE: the fact that Stored Procedures were used is important, because they ‘inherit’ the SET statement values they were created with, and NOT the values they are executed with.

USE USE master

CREATE DATABASE FI_Test

USE FI_Test

SET ANSI_NULLS ON

SET QUOTED_IDENTIFIER ON

CREATE TABLE [FactSalesQuota](

[SalesQuotaKey] [int] IDENTITY(1,1) NOT NULL,

[EmployeeKey] [int] NOT NULL,

[DateKey] [int] NOT NULL,

[CalendarYear] [smallint] NOT NULL,

[CalendarQuarter] [tinyint] NOT NULL,

[SalesAmountQuota] [money] NOT NULL,

CONSTRAINT [PK_FactSalesQuota_SalesQuotaKey] PRIMARY KEY CLUSTERED

([SalesQuotaKey] ASC))

CREATE NONCLUSTERED INDEX FI_FactSalesQuota

ON FactSalesQuota(Employeekey,CalendarQuarter)

WHERE CalendarYear = 2009 -- <---- This makes it a Filtered index

INSERT FactSalesQuota values(53,20090101,2009,4, 37000.00)

SELECT * FROM FactSalesQuota

--- this was the script to update the sp

use FI_Test

sp_rename UPDATE_FactSalesQuota , UPDATE_FactSalesQuota_V1

SET ANSI_NULLS ON

SET QUOTED_IDENTIFIER OFF -- <------ HERE is what caused the error

--SET QUOTED_IDENTIFIER ON -- <------ it will work if this is set correctly

CREATE PROCEDURE UPDATE_FactSalesQuota

@EmployeeKey int,

@CalendarYear smallint,

@CalendarQuarter tinyint,

@SalesAmountQuota money

UPDATE FactSalesQuota

SET SalesAmountQuota = @SalesAmountQuota,

DateKey = cast((CONVERT (char(8) ,getdate(), 112)) as int)

WHERE EmployeeKey = @EmployeeKey and

CalendarYear = @CalendarYear and

CalendarQuarter = @CalendarQuarter

--this fails

EXECUTE UPDATE_FactSalesQuota 53,2009,4,52000.00

SELECT * FROM FactSalesQuota

-- Now go back fix the script, and retry the sript and now it will work

Why did the size of my indexes expand when I rebuilt my indexes?

mssqlisv — Mon, 18 May 2009 20:57:00 GMT

Recently I worked with a partner who was seeing some interesting behavior. Upon rebuilding their indexes they noticed that the total space used by all indexes increased significantly. The table has no clustered index but does have a total of nine non-clustered indexes.

The sequence of events is as follows:

· Step 1: Approximately 12 million rows are inserted into an existing table via some batch loading of the data.

· Step 2: All the indexes on the table are rebuilt using:

”ALTER INDEX MyIndex ON MyTable WITH (SORT_IN_TEMPDB=ON, ONLINE=ON)”

The sp_spaceused procedure was used before and after each of the steps above to measure the amount of space used by the table and indexes. Here are the results:

BEFORE STEP 1:

sp_spaceused MyTable

Name Rows Reserved Data Index Size Unused

------ ------ ---------- ------- ---------- -----------

MyTable 1156563588 324009704 KB 88318384 KB 235511080 KB 180240 KB

AFTER STEP 1, BEFORE STEP 2 (Data added, nothing done to indexes):

sp_spaceused MyTable

Name Rows Reserved Data Index Size Unused

------ ------ ---------- ------- ---------- -----------

MyTable 1169556034 329729960 KB 89645944 KB 240051312 KB 32704 KB

AFTER STEP 2 (All indexes rebuilt):

sp_spaceused MyTable

Name Rows Reserved Data Index Size Unused

------ ------ ---------- ------- ---------- -----------

MyTable 1169595370 459848840 KB 89649160 KB 363548216 KB 6651464 KB

Notice the size of the index after adding the rows to the table and then after rebuilding the indexes (highlighted in red above). Before the rebuild the index size was approximately 240GB but after the rebuild it was nearly 365GB, an increase of nearly 50%.

It is also worth noting that in the above sample there were concurrent inserts against this table while the 12 million rows were added, as well as during the index rebuilds. This is why the indexes were rebuilt online and explains the differences in rowcount between each step.

Why did my index sizes increase so much?

The answer is related to the fact that:

Read Commited Snapshot Isolation (RCSI) was enabled on the database
the index was rebuilt ONLINE

When RCSI is enabled on a database there is an additional 14 bytes appended to each row as it is inserted, updated or deleted. This applies to the table as well as any index modified by the action. The purpose of this extra space is to maintain information about row versions that is needed for the RCSI functionality. This is described in greater depth in the following blog: http://blogs.msdn.com/sqlserverstorageengine/archive/2008/03/30/overhead-of-row-versioning.aspx.

When an index is rebuilt using the option ONLINE=ON SQL Server will append these 14 bytes to every row during the rebuild. However, when the index is rebuilt with the option ONLINE=OFF the 14 bytes are not appended but instead removed from any existing rows. The difference in behavior is by design.

Please note: When RCSI is not enabled, the above does not apply since there is no need to maintain row version information.

An existing index can be expected to increase in size after an online rebuild when either of the following is true.

1. An index with existing data has been rebuilt offline prior to the online rebuild. In this case the size of the index will increase because the 14 bytes removed during the offline rebuild are added to each row during the online rebuild.

2. The database had existing data prior to RCSI being enabled. In this case any online rebuild will add the additional 14 bytes to each row.

For this specific scenario the database had existing data prior to RCSI being enabled and these indexes were also rebuilt offline at some point in time before the online index rebuilds were performed.

How much can I expect my indexes to grow in size?

The size increase as a percentage of the original index sizes will be dependent on the size of the index keys for the indexes. This can be much higher in cases when the size of the key columns in bytes is relatively small.

The below illustrates the approximate size of the index keys and the increase in size as a percentage of the key size for two of the nine indexes. The Row Identifier below (RID) is added to each row since this table is a heap (no clustered index) and is used to identify the FILEID:PAGEID:ROWID for each index row.

CREATE UNIQUE NONCLUSTERED INDEX [NC_Idx_1] ON [MyTable]

(

[Column1] ASC, --[PersonID] [numeric](16, 0) (9 bytes)

[Column2] ASC, --[int] (4 bytes)

[Column3] ASC, --[varchar](20) (20 bytes max)

[Column4] ASC, --[varchar](20) (20 bytes max)

[Column5] ASC --[char](10) (10 bytes)

--9+4+20+20+10+8(RID)=71 (Maximum original key size)

-- +14 (RCSI – versioning information)

-- =85

-- (~20% increase in size, could be more if varchar columns have <20 bytes)

)

CREATE UNIQUE NONCLUSTERED INDEX [NC_Idx_2] ON [MyTable]

(

[Column2] ASC, --[int] (4 bytes)

[Column5] ASC --[char](10) (10 bytes)

--4+10+8(RID)=22 (Approx. original key size)

-- +14 (RCSI – versioning information)

-- =36

-- (~60% increase in size)

)

As shown above, the additional 14 bytes introduced as part of the rebuild is significantly more as a percentage of the total row size for NC_Idx_2 than for NC_Idx_1. For this particular example there were nine indexes on the existing table, five of which had relative small key sizes (similar to NC_Idx_2 above). This explains way there was such a large increase in size as a percentage of the original size. In addition to the sp_spaceused procedure, the DMV sys.dm_index_physical_stats exposes a column avg_record_size_in_bytes which can be used to measure the average row size within an index before and after index rebuilds. The avg_record_size_in_bytes includes the 14 bytes added by RCSI, if present, as part of the calculation.

Related to this behavior, there are some other interesting considerations.

1. After an index is rebuilt offline, workloads that do many updates or deletes may introduce fragmentation to the index. When an index is rebuilt offline any existing versioning information is removed from the row however any update or delete will add these 14 bytes back into to the row. When data pages are nearly full, as is the case after index rebuilds, the increase in row size as a result of the addition 14 bytes may result in page splits. This problem can be avoided by explicitly specifying a FILLFACTOR less than 100% when rebuilding an index. This will leave free space on the data/index pages and reduce the likelihood of splits. This is not a consideration if indexes are rebuilt online since the row versioning information will exist on each row already.

2. Data compression (in SQL Server 2008) performed on an index or table (either ROW or PAGE compression) using the ONLINE=ON option may result in less space saving than expected when RCSI is enabled due to the addition of the 14 bytes. These 14 bytes are initialized with a timestamp and place holder for the version record pointer and the information is not compressed by either ROW or PAGE compression. The stored procedure sp_estimate_data_compression_savings can be used as a method to measure the expected impact of data compression on the index.

It is possible that this behavior may change in a future release of SQL Server.

SQL Server 2005 / 2008 table partitioning : Important things to consider when switching-out partitions.

mssqlisv — Fri, 10 Apr 2009 23:43:00 GMT

An ISV recently found an anomoly in their implementation of a "drop table partition" function which could lead to unexpected partitions being switched out.

Typically, to do this "drop partition logic" SQL Server partitions are manipulated using the following operations:

· switch-out the partition which isn't needed anymore (or being archived) into an empty target table

· merge / reset the partition function ranges of the partitioned source table. For example; move some of the remaining data into a new partition / filegroup

· possibly drop the target table

The simple, standard way to switch-out a partition is to specify a partition number. The ISV implemented a select statement which used a boundary value of a partition range to return the corresponding partition number. Then the TSQL switch command was used to switch out this partition. This works fine as long as there is only one process doing the "switching".

But now let's assume that two processes would try this at the same time. Both processes will run the selects to get the partition numbers. Then the first one does the switch-out and the merge of the partition function. Unfortunately, the latter command will change the partition numbers as they are dynamically maintained by SQL Server. An activity like merge or splits of partitions will trigger a re-enumeration of all or parts of the partitions of a the table.

Afterwards the partition number which the second process got before the re-enumaration (tiggered by the first process) might not be accurate any more. A switch-out using the old partition number could result in switching out the wrong partition.

The solution is simple. We recommend using the $PARTITION function for the switch-out which allows you to specify a partition boundary value instead of a partition number.

The boundary values are not dynamic and therefore this issue won't come up.

Repro :

The repro script below does the following:

· create a partitioned test table with 5 boundary values: 1960, 1970, 1980, 1990, 2000

· insert 1 row into the 1970 range, 2 rows into the 1980 range and 5 rows into the 1990 range

· now switch out the 1970 range and the 1980 range by specifying the boundary values

· as expected, three ranges remain: 1960, 1990 with 5 rows and 2000

· now repeat the same test by specifying a partition number instead of the boundary value

· the output after creating the test table shows partition number 2 for boundary 1970 and 3 for boundary 1980

· using "hard-coded" partition numbers for the "switch partition" function simulates the select mentioned above

· however, now the results look different. Like before the boundaries 1960, 1990 and 2000 remain as expected. But the number of rows are not at all what we would expect!

· instead of 5 rows for boundary 1990 we see only 2 ! What happened ?

· well - the first process did the merge of the partition function. This changed the partition number of boundary 1990 from 4 to 3 and the one of boundary value 1980 from 3 to 2. The call of the "switch partition" function with partition number 3 will now switch-out the rows of boundary value 1990. Therfore the 5 rows are gone and the 2 rows of 1980 will be kept. The merge function would still use the correct boundary value. So the list of boundary values looks ok but the content is wrong !

The solution to use the $PARTITION function also works in case two processes interfere between switch-out and the partition function merge.

TSQL Repro:

if exists ( select * from sys.procedures where name = 'dp_reset_test' )

drop procedure dp_reset_test

if exists ( select * from sys.procedures where name = 'dp_list_partitions' )

drop procedure dp_list_partitions

if exists ( select * from sys.procedures where name = 'dp_switch_partition_via_boundary' )

drop procedure dp_switch_partition_via_boundary

if exists ( select * from sys.procedures where name = 'dp_switch_partition_via_partno' )

drop procedure dp_switch_partition_via_partno

set nocount on

-- stored procedure to create partitioned test table

create procedure dp_reset_test

begin

if exists ( select * from sys.objects where name = 'dp_test1' and type = 'U' )

drop table dp_test1

if exists ( select * from sys.objects where name = 'dp_test1_clone' and type = 'U' )

drop table dp_test1_clone

if exists ( select * from sys.partition_schemes where name = 'ps_year' )

drop partition scheme ps_year

if exists ( select * from sys.partition_functions where name = 'pf_year' )

drop partition function pf_year

CREATE PARTITION FUNCTION pf_year (int)

RANGE LEFT FOR VALUES ( 1960, 1970, 1980, 1990, 2000 )

CREATE PARTITION SCHEME ps_year AS PARTITION pf_year ALL TO ([PRIMARY])

create table dp_test1

( col1 int default 99 ,

col2 int,

col3 int primary key

) on ps_year(col3)

create table dp_test1_clone

( col1 int default 99 ,

col2 int,

col3 int primary key

)

insert into dp_test1 values ( 1966,1966,1966 )

insert into dp_test1 values ( 1971,1971,1971 )

insert into dp_test1 values ( 1972,1972,1972 )

insert into dp_test1 values ( 1984,1984,1984 )

insert into dp_test1 values ( 1985,1985,1985 )

insert into dp_test1 values ( 1986,1986,1986 )

insert into dp_test1 values ( 1987,1987,1987 )

insert into dp_test1 values ( 1988,1988,1988 )

print ''

end

-- stored procedure to print partitions info

create procedure dp_list_partitions ( @tabname char(20) )

begin

declare @p_number int

declare @p_rows int

declare @boundary_value int

declare p_details cursor for

select partition_number, rows, convert(int,sprv.value)

from sys.partitions sp,

sys.partition_functions spf,

sys.partition_range_values sprv

where object_id = OBJECT_ID(@tabname) and

spf.function_id = sprv.function_id and

sprv.boundary_id = sp.partition_number and

( sp.index_id = 1 or sp.index_id = 0 ) and

spf.name = 'pf_year'

order by partition_number

open p_details

FETCH NEXT FROM p_details

INTO @p_number, @p_rows, @boundary_value

print 'part no ' +

'# rows ' +

'boundary'

WHILE @@FETCH_STATUS = 0

begin

print convert(char(10), @p_number) +

' ' +

convert(char(10), @p_rows) +

' ' +

convert(char(10), @boundary_value)

FETCH NEXT FROM p_details

INTO @p_number, @p_rows, @boundary_value

end

close p_details

deallocate p_details

print ''

end

-- stored procedure to get rid of a partition by specifying the partion number

create procedure dp_switch_partition_via_partno ( @partno int, @boundary int )

begin

truncate table dp_test1_clone

alter table dp_test1 switch partition @partno to dp_test1_clone

alter partition function [pf_year]() merge range (@boundary)

end

-- stored procedure to get rid of a partition by specifying the boundary value

create procedure dp_switch_partition_via_boundary ( @boundary int, @merge_flag int )

begin

if( @merge_flag = 1 )

begin

truncate table dp_test1_clone

alter table dp_test1 switch partition $PARTITION.pf_year(@boundary) to dp_test1_clone

alter partition function [pf_year]() merge range (@boundary)

end

if( @merge_flag = 2 )

begin

truncate table dp_test1_clone

alter table dp_test1 switch partition $PARTITION.pf_year(@boundary) to dp_test1_clone

end

if( @merge_flag = 3 )

begin

alter partition function [pf_year]() merge range (@boundary)

end

-- test sample

-- create partitioned test table

execute dp_reset_test

print 'Test table with 5 boundary values : '

print ''

execute dp_list_partitions 'dp_test1'

print 'Switch partitions with boundary 1970,1980 via boundary value : '

print ''

execute dp_switch_partition_via_boundary 1970, 1

execute dp_switch_partition_via_boundary 1980, 1

execute dp_list_partitions 'dp_test1'

print 'as expected boundary values 1970 and 1980 are gone and 1990 remains with 5 rows'

print ''

-- reset test table

execute dp_reset_test

print 'Switch partitions with boundary 1970,1980 via partno : '

print ''

execute dp_switch_partition_via_partno 2, 1970

execute dp_switch_partition_via_partno 3, 1980

execute dp_list_partitions 'dp_test1'

print 'boundary values 1970 and 1980 are gone but 1990 remains with 2 rows which is unexpected !'

print ''

-- reset test table

execute dp_reset_test

print 'Switch partitions with boundary 1970,1980 via boundary value with '

print '"deferred merge of the partition function" : '

print ''

-- switch out only

execute dp_switch_partition_via_boundary 1970, 2

execute dp_switch_partition_via_boundary 1980, 2

-- merge partition function only

execute dp_switch_partition_via_boundary 1970, 3

execute dp_switch_partition_via_boundary 1980, 3

execute dp_list_partitions 'dp_test1'

print 'as expected boundary values 1970 and 1980 are gone and 1990 remains with 5 rows'

print ''

Avoid using JDK Date APIs to handle timezone sensitive date and time

mssqlisv — Fri, 27 Mar 2009 00:30:00 GMT

JDK APIs for Class “java.util.Date” and “java.sql.Timestamp” (subclass of Date) including getHours(), getMinutes(), getSeconds(), getTimestamp() allow you to retrieve date/time related information. However, the JVM (Java Virtual Machine) won’t handle timezone sensitive data properly using these APIs. As matter of fact, these APIs were deprecated starting JDK 1.1 (http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html). But I am still seeing ISV developers use these APIs in their JAVA applications, leading to incorrect results or behavior.

What happens is when date/time data is inserted into the SQL Server database, it’s stored correctly in SQL Server. However, when the date/time is read using the APIs mentioned above, the retrieved value is implicitly converted to “local time” depending on where the host of JVM (Java Virtual Machine) is. For example, when a java application reads “1/8/2009 3:30:00 AM” from SQL Server database, you would get different results depending on the location of the app.

SQL Server (in pacific time zone):

create table datetime_tbl (id INTEGER unique not null, dateTime_v DATETIME not null)

insert into datetime_tbl (id, dateTime_v) values (1, '2009-01-08 03:30:00')

Java application:

Statement s = connection.createStatement();

query = "select id, dateTime from datetime_tbl where id = " + 1;

s.execute(query);

ResultSet rs = s.getResultSet();

rs.next();

String localApptimezone = Calendar.getInstance().getTimeZone().getID();

// -8 is offset of GMT to read the time as pacific time.

java.util.TimeZone timeZoneP = new java.util.SimpleTimeZone(-8*3600000, "GMT-8 (Pacific)");

java.util.Calendar cal= java.util.Calendar.getInstance(timeZoneP);

Timestamp tStamp = rs.getTimestamp(2, cal);

System.out.println("local application timezone: " + localApptimezone);

System.out.println("time: " + tStamp.toString() + " in " + timeZoneP.getID());

Results (note: 1 hour difference for Arizona test):

Location of app/JVM

Result

Washington (Pacific time zone)

local application timezone: America/Los_Angeles

time: 2009-01-08 03:30:00.0 in GMT-8 (Pacific)

Arizona (Mountain time zone)

local application timezone: America/Phoenix

time: 2009-01-08 04:30:00.0 in GMT-8 (Pacific)

The recommended way of handling this type of scenario is to utilize DateFormat and avoid getTimestamp() and other aforementioned APIs. Alternatively, you can convert the date/time to character string directly.

Statement s = connection.createStatement();

query = "select id, dateTime from datetime_tbl where id = " + 1;

s.execute(query);

ResultSet rs = s.getResultSet();

rs.next();

String localApptimezone = Calendar.getInstance().getTimeZone().getID();

// -8 is offset of GMT to read the time as pacific time.

java.util.TimeZone timeZoneP = new java.util.SimpleTimeZone(-8*3600000, "GMT-8 (Pacific)");

java.util.Calendar cal= java.util.Calendar.getInstance(timeZoneP);

java.text.DateFormat dateFormat = java.text.DateFormat.getInstance();

dateFormat.setTimeZone(timeZoneP);

System.out.println("local application timezone: " + localApptimezone);

System.out.println("time: " + dateFormat.format(cal.getTime()) + " in " + timeZoneP.getID());

Results (note: consistent results for both region tests as expected):

Location of app/JVM

Result

Washington (Pacific time zone)

local application timezone: America/Los_Angeles

time: 1/8/09 3:30 AM in GMT-8 (Pacific)

Arizona (Mountain time zone)

local application timezone: America/Phoenix

time: 1/8/09 3:30 AM in GMT-8 (Pacific)

Zeroing in on blocking on seemingly unrelated tables

mssqlisv — Mon, 02 Feb 2009 21:20:00 GMT

In one of our recent lab tests we were surprised to see blocking occur on a table that did not participate in the transaction being reported as the cause of the blocking. From the sp_lock output we noticed an ‘X’ lock being held on the table, but we could guarantee that there was no insert, delete or update activity on the table. Needless to say, this was odd and baffled us for while and it was only when we analyzed the definitions of the tables that we could determine the source of the problem. Let’s take a look at this scenario via a simplified example and explain the cause of the blocking.

Consider the case where we have two tables, ORDERS and ORDER_LINE and a foreign-key relationship as shown below.

These tables are populated with the following 4 rows of data:

INSERT INTO ORDERS (ORDER_ID, CREATED_BY, DUE_DATE) VALUES

(100, 'Burzin', '11/24/2003'),

(101, 'Burzin', '11/28/2003');

INSERT INTO ORDER_LINE (ORDER_ID, ORDER_LINE_ID, ITEM, QUANTITY) VALUES

(100, 1, 1028, 12),

(101, 2, 1029, 24);

Furthermore, we had Read Committed Snapshot Isolation (RCSI) enabled on the database.

In our scenario we had two transactions executing the following two T-SQL statements via separate database connections (different SPIDs).

SPID-56

BEGIN TRAN

UPDATE ORDER_LINE SET ORDER_ID = 101 WHERE ORDER_ID = 100;

...

SPID-57

BEGIN TRAN

DELETE ORDERS WHERE ORDER_ID = 100;

...

(NOTE: both these transactions operate on different tables.)

When these transactions were executed, we observed that blocking occured on the ORDERS table. This was a bit non-intuitive and baffling at first. To get to the root of the problem we started by investigating the common causes but couldn’t find any reason for the two transactions operating on different tables to block each other. However, on further analysis we noticed that the ORDER_LINE table had a foreign-key relationship to the ORDERS table and because of this when the ORDER_LINE table was updated, the ORDERS table was referenced to ensure that the foreign-key relationship was being preserved. This was why the DELETE statement held a shared (‘S’) lock on the rows of child table, ORDER_LINE, even though it wass only deleting from parent table, ORDERS.

This solved a part of the mystery. Upon looking at the output of sp_lock we observed that there were two exclusive (X) Keylocks (see rows 11 and 12 in the screenshot below) acquired on the ORDERS table which were causing other transactions operating on the ORDERS table to block.

This once again was a bit confusing. If the ORDERS table was being accessed solely to verify the referential integrity of the data and preserve the foreign-key relationship, why were exclusive locks being acquired? Furthermore, since RCSI was enabled on the database we expected the read and write operations to not cause any blocking.

On digging deeper we determined that the database engine had to acquire an ‘X’ lock on the ORDERS table as soon as the second transaction tried to modify a row to prevent the possibility of the referential integrity being broken. One could imagine a pathological case where the first transaction updated the ORDER_LINE table with a value that qualified the foreign-key relationship, and then the second transaction DELETE the value from the ORDERS table leaving the referential integrity broken and the foreign-key pointing to a phantom value. In fact this is exactly what occurs in the example presented above and is depicted by the screenshot where ‘S’ lock on ORDER_LINE table requested by the DELETE statement is blocked by the UDATE statement. Therefore the blocking chain can be viewed as: Queries accessing the ORDERS table à blocked by DELETE statement à blocked by UPDATE statement. To prevent this situation from occurring, the database engine acquires an ‘X’ lock on a referenced table (ORDERS) as soon as it determines that there is a data modification transaction operating on it. This is by design and expected behavior as without this behavior there is a possibility of the referential integrity between the tables breaking.

If your application is encountering a similar issue you may want to try work around it by:

1. Modifying your application so that two transactions do not operate on the same or referenced objects concurrently, e.g. in the above example the update and the delete could have been performed via a single transaction

2. Removing the referential integrity. This assumes that your application does not need this, or can enforce it via some other means, e.g. within the business logic layer of the application.

Using SQL Server 2008 Management Data Warehouse for database monitoring in my application

mssqlisv — Thu, 29 Jan 2009 19:19:00 GMT

SQL Server 2008 introduced what we call the Management Data Warehouse. The Management Data Warehouse is a relational database that contains data that is collected from a server using the new SQL Server 2008 Data Collection mechanism. The Warehouse consists primarily of the following components:

· An extensible data collector: http://msdn.microsoft.com/en-us/library/bb677248.aspx

· A database schema which is indirectly extensible by additions in the data collection: http://msdn.microsoft.com/en-us/library/bb677306.aspx

· Stored procedures which allow the DBA to create their own data collection set and own the resultant data collection items: http://msdn.microsoft.com/en-us/library/bb630337.aspx

· Three Data Collections Sets which are delivered with SQL Server 2008 and which can be enabled at any time: http://msdn.microsoft.com/en-us/library/bb964725.aspx

· Standard reports delivered with SQL Server 2008 Management Studio display data collected by the three predefined Data Collection Sets. For DBA created Data Collections, reports need to be generated by the DBA or the data can be queried with normal T-SQL queries.

However, this DBA/management toolset can also be used by ISVs to collect and monitor performance data of SQL Server through their applications. The Management Data Warehouse as delivered with SQL Server can be used as a framework by an application as a basis for database monitoring functionality. Due to the high flexibility of the Management Warehouse this is actually a very easy task. Using the functionality of this new component, as a database monitoring framework, can make support much easier; it can make it simple to find answers to questions like: ‘Was the performance issue users complained about a few hours ago rooted on the database side, the storage back-end, or somewhere in the application?’

In the following sections I'll describe the primary steps to enable SQL Server Management Data Warehouse and Data Collections:

After the customer deploys the application, the customer would need to configure the Management Data Warehouse with SQL Server Management Studio (Object Explorer à Management à Data Collection). Thereby the customer needs to define which database the database schema of the Management Data Warehouse needs to be deployed in. The customer can now be instructed to create a database of a certain name and a certain database file locations or to use a database the application already created during the ISV application installation. The application can reference this database and can query the database after the DBA assigns the application users into the ‘dbo’ role of the Management Warehouse database. SQL Server now will create the following entities in the Management Warehouse database:

· A user named ‘mdw_check_operator_admin’.

· A schema named ‘core’ with a set of tables:
http://msdn.microsoft.com/en-us/library/bb677306.aspx

· A schema ‘snapshot’ which contains the tables needed for the System Data Collector Set which are delivered with SQL Server 2008 already: http://msdn.microsoft.com/en-us/library/bb677306.aspx

Other functionality required for making the whole warehouse operational is already deployed in SQL Server’s msdb database.

In order to start the collection of the 3 default collectors the customer needs to have SQLAgent started and needs to enable the 3 collection sets manually. However, these 3 collection sets only cover minimal aspects and often are not sufficient to detect whether the issue is on the database side or whether the issue is somewhere else. Additionally, if the performance problem is currently not present, but happened hours or days ago, it can be difficult to figure out what happened using the default collection mechanism.

Another possibility to enable the Management Warehouse and to configure it, can be done by T-SQL stored procedures as they are delivered in msdb.

Extending the Data Collections to fit our Need

The nice thing about the Management Data Warehouse is the fact that the data collector is a framework which allows the creation of user defined data collection sets and data collection items. The extension of the data collection or definition of an additional data collection also will extend the schema of the warehouse automatically. For an application vendor it means collecting what their typical support scenarios are. As a great example on how to extend such a collection set, one could script the 3 default collection sets into a Query Window. This can be done by marking one of the sets, click the right mouse button and select the options ‘Script Data Collection As’ à ‘Create to’ à ‘Query Window’. Unfortunately all the collections are ‘TSQL Query Collector Type’ and ‘Query Activity Collector Type based, whereas the SQL Server Data Collector also can be enhanced reading Windows Performance Counters and SQL Server Trace information. For different Collector types please see: http://msdn.microsoft.com/en-us/library/bb677328.aspx

In order to demonstrate how to extend the data collector we use one of the cases which keep us busy very often supporting ISV applications running on SQL Server and databases in general. ‘How was the I/O performance during the time the users were complaining of the application being so slow?’

Build a Data Collection which collects Query Performance Counters

Being faced with a support situation where hours or days ago users were complaining about performance of the application, it becomes extremely tricky to figure out what really happened. When everybody points to the database as source of the problems, it is hard to find any evidence at all unless the customer has a comprehensive set of Performance Counters recorded steadily. That however, is a rare case all too often. One cannot blame a customer not having a comprehensive Perfmon trace since the administration and archiving of such traces can be rather cumbersome. Therefore we want to show in our first case of extending the SQL Server 2008 Data Collector, a case where we collect Performance Monitor data. Opposite to the usage of the Perfmon on the OS side, the Data Collector and the Management Data Warehouse take responsibility of administering the data. First we’ll show the script to extend the Data Collector and then walk through it step by step.

So the script generating such a collection would look like:

use msdb;

Begin Transaction

Begin Try

Declare @collection_set_id_1 int

Declare @collection_set_uid_2 uniqueidentifier

EXEC [dbo].[sp_syscollector_create_collection_set]

@name=N'Disk Performance and SQL CPU',

@collection_mode=1,

@description=N'Collects logical disk performance counters and SQL Process CPU',

@target=N'',

@logging_level=0,

@days_until_expiration=7,

@proxy_name=N'',

@schedule_name=N'CollectorSchedule_Every_5min',

@collection_set_id=@collection_set_id_1 OUTPUT,

@collection_set_uid=@collection_set_uid_2 OUTPUT

Select @collection_set_id_1, @collection_set_uid_2

Declare @collector_type_uid_3 uniqueidentifier

Select @collector_type_uid_3 = collector_type_uid From [dbo].[syscollector_collector_types] Where name = N'Performance Counters Collector Type';

Declare @collection_item_id_4 int

EXEC [dbo].[sp_syscollector_create_collection_item]

@name=N'Logical Disk Collection and SQL Server CPU',

@parameters=N'<ns:PerformanceCountersCollector xmlns:ns="DataCollectorType">