Tips, Tricks, and Advice from the SQL Server Query Processing Team : Tracing

Store Statistics XML in database tables using SQL Traces for further analysis.

queryproc — Fri, 01 Jun 2007 19:51:00 GMT

Since SQL Server 2005, query plan as well as statistics of query execution can be captured in XML format. Also, SQL Server 2005 has XQuery support to directly query XML document. By combining these two new features, users can analyze the query plans using queries.

However, in SQL Server, there is no easy way to capture the statitics XML into a table. Fortunately, there are SQL traces provided by SQL Server to capture the showplan XML and statistics XML information into trace files and loaded into tables.

Note: The only limitation is the 128 level of nesting levels supported by XML data type in SQL 2005. In that case, you have to write client code to parse the query plan, which going to be a very complex query plan.

Here is a small example using SQL traces to store the statistics XML and extract the estimated rows and actual rows.

/*Using Traces to Capture Statistics XML*/

declare @trace_id int

declare @trace_file nvarchar(200)

select @trace_file = 'c:\temp\test_stats_' + cast(newid() as varchar(100))

-- using trace table.

exec sp_trace_create @trace_id output,

@tracefile=@trace_file

-- capture statistics-xml, textdata, on

exec sp_trace_setevent @trace_id, 146, 1, 1

-- start

exec sp_trace_setstatus @trace_id, 1

-- test statement.

select * from sys.objects

-- stop

exec sp_trace_setstatus @trace_id, 0

-- close

exec sp_trace_setstatus @trace_id, 2

-- load trace files into table

if object_id('temp_trc') is not null

drop table temp_trc

select *

into temp_trc

from fn_trace_gettable(@trace_file + '.trc', default)

-- look at the captured stats xml

declare @plan xml

select @plan=cast(textdata as xml)

from temp_trc

where eventclass = 146;

-- collect the actual and also estimate stats.

with XMLNAMESPACES ('http://schemas.microsoft.com/sqlserver/2004/07/showplan' as sql)

select

ro.relop.value('@NodeId', 'int') NodeId,

ro.relop.value('@PhysicalOp', 'nvarchar(200)') PhysicalOp,

ro.relop.value('@LogicalOp', 'nvarchar(200)') LogicalOp,

(ro.relop.value('@EstimateRows', 'float')

* (ro.relop.value('@EstimateRewinds', 'float')

+ ro.relop.value('@EstimateRebinds', 'float')

+ 1.0)) EstimateRows,

case

when root_actual.ActualRows = 0

then null

else root_actual.ActualRows

end ActualRows,

cast(ro.relop.exist('*/sql:RelOp') as bit) IsNotLeaf

from @plan.nodes('//sql:RelOp') as ro(relop)

cross apply (

select sum(rti.info.value('@ActualRows', 'float')) ActualRows

from ro.relop.nodes('sql:RunTimeInformation/sql:RunTimeCountersPerThread') as rti(info)

) root_actual;

The output of the estimate rows and actual rows is given below:

NodeId	PhysicalOp	LogicalOp	EstimateRows	ActualRows	IsNotLeaf
0	Nested Loops	Left Outer Join	52	52	1
1	Nested Loops	Left Outer Join	52	52	1
2	Filter	Filter	52	52	1
3	Compute Scalar	Compute Scalar	52	NULL	1
4	Clustered Index Scan	Clustered Index Scan	52	52	0
13	Clustered Index Seek	Clustered Index Seek	52.000031	NULL	0
14	Clustered Index Seek	Clustered Index Seek	52	52	0

Using ETW for SQL Server 2005

queryproc — Mon, 13 Nov 2006 05:32:00 GMT

ETW stands for “Event Tracing for Windows” and it is used by many Windows applications to provide debug trace functionality. This “wide” availability is a key point of using ETW because it can help to track certain activities from end to end. For example, you can literally track a request coming from IIS, passing through protocol layer, and then finally handled by database engine in single trace file. Unfortunately, not many people know that SQL Server 2005 provides a full ETW functionality which can output most of SQL Trace events available for SQL Server Profiler. In this article, I will explain how to use SQL ETW with examples.

First of all, how do I know if SQL ETW is really active on my machine? The following shows output on my server which has SQL Server 2005 Standard Edition as 2^nd instance (named Yukon).

C:\>logman query providers
Provider                                 GUID
-------------------------------------------------------------------------------
YUKON Trace                              {130A3BE1-85CC-4135-8EA7-5A724EE6CE2C}
.NET Common Language Runtime             {e13c0d23-ccbc-4e12-931b-d9cc2eee27e4}
ACPI Driver Trace Provider               {dab01d4d-2d48-477d-b1c3-daad0ce6f06b}
Active Directory: Kerberos               {bba3add2-c229-4cdb-ae2b-57eb6966b0c4}
IIS: Request Monitor                     {3b7b0b4b-4b01-44b4-a95e-3c755719aebf}
Local Security Authority (LSA)           {cc85922f-db41-11d2-9244-006008269001}

My other server has a default instance of SQL Server 2005 and it shows something like this:

MSSQLSERVER Trace {2373A92B-1C1C-4E71-B494-5CA97F96AA19}

Is there a connection between these 2 examples? Yes, the SQL ETW provider name appears to be a combination of “Instance name” and “Trace”, and its GUID is unique across instances. You can use either provider name or GUID to start and stop ETW session.

Please note this ETW functionality is not available for SQL Express, because, well…, you get what you paid for J. If you don’t see SQL ETW provider name, and you are sure you have a regular SQL Server 2005, then you may be able to fix it yourself. Here’s a short list of instructions.

Go to the directory where your SQL server binaries are. On my machine, it is on
“C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Binn”.
Look for “etwcls.mof”. If you cannot find it, something is definitely wrong with your installation. You probably need to seek help elsewhere.
Run “mofcomp etwcls.mof” in command window.
Try “logman query providers” again. You should see your SQL ETW provider.

Now I will describe how to configure SQL ETW to collect interesting traces. Go to the same directory described in the above instruction and find a file called “etwcnf.xml”. This is your configuration file and you can modify it with any text editor. There should several entries already defined in this file for your examples. See the following example picked out of etwcls.xml file.

This defines a tracing template called “TSQL replay” with template ID = 1. It contains a list of events with ID = 11, 13, and so forth. This event ID is matched to SQL Trace Event Class. By consulting BOL documentation on available SQL Trace events, you can find “RPC:Starting” event has ID of 11, and “SQL:BatchStarting” event has ID of 13. Yes, now you know how to change this template to suite your need!

Now let’s try to get some actual trace events. I’m using “logman.exe” and “tracerpt.exe” which should be available on most Windows platform.

First, create a text file as shown in the following example. You can activate multiple providers by listing them in separate lines. For simplicity, let’s enable only SQL ETW.

C:\etw>type prov.txt
"YUKON Trace" 1 0

As you have figured out already, the first column is SQL ETW provider name (you can use GUID also). The next number (1) is called “enable flag” and should match the template ID in etwcls.xml file. The 3^rd number (0) is called “enable level” and should be kept as zero for SQL ETW. The meaning of these 2 flags depends on providers and you should consult appropriate documentations for other providers.

Now you are ready to start your 1^st SQL ETW session! In my example, I decided to give a very creative/original name: “mytrace”. Here’s a screen dump from my machine.

C:\etw>logman start mytrace -pf prov.txt -ets
Name:                      mytrace
Age Limit:                 15
Buffer Size:               64
Buffers Written:           1
Clock Type:                System
Events Lost:               0
Flush Timer:               0
Buffers Free:              2
Buffers Lost:              0
File Mode:                 Sequential
File Name:                 C:\etw\mytrace.etl
Logger Id:                 3
Logger Thread Id:          1308
Maximum Buffers:           25
Maximum File Size:         0
Minimum Buffers:           3
Number of buffers:         3
Real Time Buffers Lost:    0

Provider                                  Flags                     Level
-------------------------------------------------------------------------------
* "YUKON Trace"                           0x00000001                0x00
{130A3BE1-85CC-4135-8EA7-5A724EE6CE2C} 0x00000001                0x00

To test if this ETW session can actually receive something from SQL Server, let’s issue “select * from sys.dm_exec_requests”. After that, you can stop the tracing as shown in the following example.

C:\etw>logman stop mytrace -ets
The command completed successfully.

Look for newly created file in the working directory (mytrace.etl). Unfortunately, this is a binary trace file and you will need some help to crack it open. “Tracerpt.exe” does some rudimentary job of converting this binary trace file into human-readable form.

C:\etw>tracerpt mytrace.etl
Input
----------------
File(s):
     mytrace.etl
100.00%
Output
---------------
Text (CSV):         dumpfile.csv
Summary:            summary.txt
The command completed successfully.

Note this creates 2 files: dumpfile.csv and summary.txt. Here’s a screen dump.

C:\etw>type dumpfile.csv
Event Name,       Type,        TID,           Clock-Time, Kernel(ms),   User(ms), User Data
EventTrace,     Header, 0x00000BF4,   128075886826813362,          0,         10,    65536, 33620485,     790, 1, 128075887366383792,   100144,        0, 0x00000000,        2,        1,        4,        0,     2785, 0x01402930, 001402940,                0,          3579545, 128075886826813362, 0x00000002,        0, 0, 0
SQL:BatchStarting,          0, 0x00000874,   128075886939625782,          0,         10, "select * from sys.dm_exec_reqests",       1,                0, "xxxx", "REDMOND", "JAYC8V1",     1580, "SQL Query Analyzer", "REDMOND\xxxx",       51, 28075598939600000,       13, "master", 010500000000000515000000A065CF7E784B9B5FE77C8770A9640000",        0,   0,      38,        0, "REDMOND\xxxx", 0, 0
C:\etw>

Yes, this looks pretty cryptic and you will probably need to look at etwcls.mof (and SQL Trace description) to decipher each column. From etwcls.mof file, we know 1^st column is “TextData”, 2^nd column is “DatabaseID”, 3^rd column is “TransactionID”, etc.

       [WmiDataId(1),
       Description("TextData"),
       format("w"),
       StringTermination("Counted"),
       read
       ]
       string TextData;
       [WmiDataId(2),
       Description("DatabaseID"),
       read
       ]
       sint32 DatabaseID;
       [WmiDataId(3),
       Description("TransactionID"),
       read
       ]
       sint64 TransactionID;

Now we know this ETW session received “SQL:BatchStarting” event correctly, and its TextData column contains correct TSQL statement.

OK, you know how to start and stop SQL ETW. But with very little UI support compared to the pretty looking SQL Server Profiler, why even bother to consider ETW if I don’t need end-to-end tracing? Well, one good thing about SQL ETW is that your ETW session survives SQL Server restart, unlike ordinary SQL Trace.

Jay Choe