Tips, Tricks, and Advice from the SQL Server Query Processing Team : Statistics Profile

Store Statistics XML in database tables using SQL Traces for further analysis.

queryproc — Fri, 01 Jun 2007 19:51:00 GMT

Since SQL Server 2005, query plan as well as statistics of query execution can be captured in XML format. Also, SQL Server 2005 has XQuery support to directly query XML document. By combining these two new features, users can analyze the query plans using queries.

However, in SQL Server, there is no easy way to capture the statitics XML into a table. Fortunately, there are SQL traces provided by SQL Server to capture the showplan XML and statistics XML information into trace files and loaded into tables.

Note: The only limitation is the 128 level of nesting levels supported by XML data type in SQL 2005. In that case, you have to write client code to parse the query plan, which going to be a very complex query plan.

Here is a small example using SQL traces to store the statistics XML and extract the estimated rows and actual rows.

/*Using Traces to Capture Statistics XML*/

declare @trace_id int

declare @trace_file nvarchar(200)

select @trace_file = 'c:\temp\test_stats_' + cast(newid() as varchar(100))

-- using trace table.

exec sp_trace_create @trace_id output,

@tracefile=@trace_file

-- capture statistics-xml, textdata, on

exec sp_trace_setevent @trace_id, 146, 1, 1

-- start

exec sp_trace_setstatus @trace_id, 1

-- test statement.

select * from sys.objects

-- stop

exec sp_trace_setstatus @trace_id, 0

-- close

exec sp_trace_setstatus @trace_id, 2

-- load trace files into table

if object_id('temp_trc') is not null

drop table temp_trc

select *

into temp_trc

from fn_trace_gettable(@trace_file + '.trc', default)

-- look at the captured stats xml

declare @plan xml

select @plan=cast(textdata as xml)

from temp_trc

where eventclass = 146;

-- collect the actual and also estimate stats.

with XMLNAMESPACES ('http://schemas.microsoft.com/sqlserver/2004/07/showplan' as sql)

select

ro.relop.value('@NodeId', 'int') NodeId,

ro.relop.value('@PhysicalOp', 'nvarchar(200)') PhysicalOp,

ro.relop.value('@LogicalOp', 'nvarchar(200)') LogicalOp,

(ro.relop.value('@EstimateRows', 'float')

* (ro.relop.value('@EstimateRewinds', 'float')

+ ro.relop.value('@EstimateRebinds', 'float')

+ 1.0)) EstimateRows,

case

when root_actual.ActualRows = 0

then null

else root_actual.ActualRows

end ActualRows,

cast(ro.relop.exist('*/sql:RelOp') as bit) IsNotLeaf

from @plan.nodes('//sql:RelOp') as ro(relop)

cross apply (

select sum(rti.info.value('@ActualRows', 'float')) ActualRows

from ro.relop.nodes('sql:RunTimeInformation/sql:RunTimeCountersPerThread') as rti(info)

) root_actual;

The output of the estimate rows and actual rows is given below:

NodeId	PhysicalOp	LogicalOp	EstimateRows	ActualRows	IsNotLeaf
0	Nested Loops	Left Outer Join	52	52	1
1	Nested Loops	Left Outer Join	52	52	1
2	Filter	Filter	52	52	1
3	Compute Scalar	Compute Scalar	52	NULL	1
4	Clustered Index Scan	Clustered Index Scan	52	52	0
13	Clustered Index Seek	Clustered Index Seek	52.000031	NULL	0
14	Clustered Index Seek	Clustered Index Seek	52	52	0

Statistics Profile Output Formatting

queryproc — Fri, 20 Oct 2006 23:57:00 GMT

Statistics profile output is an important tool when it comes to troubleshooting query plan issues. When enabled, it returns a textual representation of the query plan with a lot of detail about cost and cardinality estimates as well as actual counts.

When working in SQL Server management Studio (SSMS), it is advised to enable grid mode to output the results of statistics profile. This presents the results in a neatly formatted way. It also allows to easily copy and past the result in Excel for further manipulation (hiding columns, highlighting individual cells, adding columns with additional calculations, …). But what if grid output mode is not available? E.g., when running SQL Trace with an event to produce statistics profile output for each query, the result is always presented as flat text. For even mildly complex queries this output is hard to decipher.

The attached perl script helps to solve this formatting problem. It takes the flat text output of statistics profile and parses it to identify individual columns. It then generates tab delimited output which can be directly consumed by Excel.

Usage is straightforward: the script assumes one individual statistics profile in a regular text file (ANSI text only please, UNICODE output confuses the script). Just run

perl statspro.pl <input file> <output file>

E.g., lets assume I have my statistics profile output in a file called test.txt I’ll run

perl statspro.pl test.txt test.xls

The resulting xls file can be directly viewed with Excel.

Note that I’m not a perl wizard and there might be a more elegant/efficient way to achieve the same means. The script is provided “as is”. You are more than welcome to use and/or enhance the script.

Peter Zabback

Introduction to Showplan

queryproc — Sat, 30 Sep 2006 01:16:00 GMT

Showplan is a feature in SQL Server to display and read query plans. While some of you may already be very familiar with Showplan, it is one of the most important diagnostic tools that we use in the query processing team to locate and identify problems, and therefore deserves some extra exposure. Being able to collect, read, and understand Showplan data is a critical skill to have, and one which we plan to blog about a fair bit. Consider this an introductory posting to cover some of the basics which we will drill into more in further posts.

We generate two types of Showplans in SQL Server, one at query compilation time (when an optimized query plan is produced) and the second at query runtime (when the optimized query plan is executed). The former allows you to see the compiled query plan in its entirety, as well as a ton of useful information about each operator in the plan. The runtime plan, often referred to as query execution “statistics”, enables you to collect actual metrics about the query during its execution, such as its execution time and actual cost.

Showplan and statistics information can be extracted in two ways from the server, 1) using query SET options, 2) using Profiler Trace events. The various SET options available are:

· Legacy Showplan

o Compile Time: SET SHOWPLAN_ALL ON or SET SHOWPLAN_TEXT ON

o Runtime: SET STATISTICS PROFILE ON

· Showplan XML (Preferred)

o Compile Time: SET SHOWPLAN_XML ON

o Runtime: SET STATISTICS XML ON

SQL Server Management Studio (SSMS) uses Showplan to display a graphical representation of the query plan. A graphical view of the query plan for the following query on the ‘nwind’ database is shown in the picture below.

-- Query: select customer name and order date for orders placed by customers in Canada

use nwind

select ContactName, OrderDate

from Customers inner join Orders

on Orders.CustomerID = Customers.CustomerID

where Orders.ShipCountry = 'Canada'

(See the attached .bmp file for a picture of this showplan)

In SSMS, to view the graphical plan, click on the Execution Plan tab in the Results Pane. The graphical plan is read from top-to-bottom, right-to-left. When you select any operator in the query tree or simply hover the mouse on it, you will see a tooltip that describes the operator. It displays the Query Optimizer cost estimates (operator and subtree costs, number of rows, row size, etc.) and additional information like output columns, predicates, etc. The detailed operator information is shown in the Properties window (View --> Properties Window), which is usually displayed on the extreme right frame in SSMS.

To save the graphical showplan to a file you can right click on the Execution Plan and select 'Save Execution Plan As'. The query plan is saved with extension '.sqlplan' and can be reloaded into SSMS anytime, sent via email, etc.

Showplan output can be generated in text, grid or XML formats. The XML format was introduced in SQL Server 2005 which we refer to as “Showplan XML” and the other formats as “Legacy Showplan”. In the next posting we will describe how to generate and analyze Showplan XML.

- Steve and Gargi