Ken Henderson's WebLog : nexus

SQL Nexus is out!

khen1234 — Wed, 12 Dec 2007 22:40:00 GMT

In case you missed it in my last post, SQL Nexus, the SQL Server performance analysis platform built by Bart and me, has been released via CodePlex. Here's a post post where I talked about how Nexus works. And here's the link to its page on the CodePlex site.

While I'm at it, another tool you might find useful is SQLDiag. It ships as part of SQL Server 2005 and later. Read about it here and here.

For a high-speed, lean, command-line analysis tool for Profiler traces, check out the Retrace tool we released into the wild yesterday here.

A SQL Profiler trace Swiss Army Knife

khen1234 — Wed, 12 Dec 2007 13:25:00 GMT

Have you ever needed to find your most expensive queries and quickly grew weary of writing T-SQL against trace tables to try to ferret them out? Have you ever had to wade through gigabytes of trace data just to find one ill-behaving query? Have you ever struggled to decide what performance metrics really matter when analyzing Profiler traces: duration, reads, writes, etc? Today’s post is about a new tool Bart and I wrote to help you do all this and more.

Origins

The original concept behind Retrace was to create a simple tool to load a SQL Server Profiler trace file into a database ala the Relog tool that ships with Windows (and provides similar functionality for Perfmon logs). Retrace would “re-trace” a session of SQL Server activity as a server-side trace table. I often needed to query trace files using T-SQL and didn’t like resorting to fn_trace_gettable every time, so Retrace was born to automate that process for me. That was the original idea.

Of course, these kinds of things tend to take on a life of their own, and Retrace soon began to demand that I enhance it in all sorts of interesting ways. Around this same time, my friends in CSS got underway with getting the SQL Nexus performance analysis platform that Bart and I built released via CodePlex. (Great news: it’s out there now. More on that later.) One of the cooler features we built for Nexus was a facility we called “TraceBuster.” Nexus supports this notion of generic diagnostic data loaders that implement a common interface. Nexus interacts with these assemblies via this generic interface, and users can build their own loader assemblies so long as they adhere to the interface we expose. TraceBuster is a Nexus loader we built to process SQL Profiler trace files and load them into a data warehouse. It automatically parameterizes the queries it reads from each trace such that the performance of a query executed with different parameter sets can be tracked over time. Unlike SQL Profiler itself, TraceBuster is able to identify and aggregate performance statistics such as CPU use, duration, reads, writes, etc., for these query “templates” rather than treating multiple instances of the same query with different parameters as different queries. Lightweight, fast, and extremely simple from a coding standpoint, it does exactly what we wanted it to do and no more.

For a number of reasons, it was decided that TraceBuster would not be included in the CodePlex release of Nexus (though you can still get it here in binary form). That got Bart and I to thinking. How cool would it be if Retrace used TraceBuster and basically provided a nice command line tool to load SQL Profiler traces into a data warehouse? What if Retrace was merely a thin console app shell over the TraceBuster assembly originally designed for Nexus? Since Nexus interacted with TraceBuster and its other diagnostic loaders generically, there was no reason a small console app couldn’t do the same thing without requiring Nexus itself to be on the box. Rather than merely load the trace into a trace table on the server, what if we let TraceBuster do all of its cool aggregation and analysis for us, thus saving all kinds of work manually querying the trace files? And what if on top of that we used the client-side Reporting Services reports that Bart developed for TraceBuster’s data warehouse and had Retrace display them? Then we’d have a single tool that could blaze through our trace data, load it into a data warehouse for further perusal, and finally display useful reports over that data. How cool would that be?!

So, we spent a couple late nights and a weekend here or there and morphed Retrace into a “Swiss Army Knife” for Profiler traces. One of the things we found ourselves often doing in our tuning work within SQL Server development was running a SQLDiag collection immediately before doing something we wanted to measure on the server, then loading the Profiler traces it collected via Retrace. Given Retrace’s one-stop-shopping nature, we thought it made sense to build that functionality into Retrace, so we added a command line option to Retrace to allow it to call SQLDiag automatically before a load operation.

When it was all said and done, Retrace knew how to collect Profiler traces, to load those traces into a performance warehouse on the server, and to run reports over that warehouse. We had a simple little console app that automated many of the tasks we found ourselves doing over and over as we tuned this or that component in our day jobs.

Summary

Retrace knows how to do three basic things:

Collect a SQL Profiler trace (using the SQLDiag utility that ships with SQL Server 2005 and later)
Load Profiler traces into a data warehouse that resides in a SQL Server database. These can be traces it collects via SQLDiag or traces from other sources (e.g., SQL Profiler)
Show reports listing the top N most expensive queries from the trace files loaded into the warehouse

Examples

To load a trace file into a data warehouse on the default SQL Server instance using Windows authentication:

retrace -i mytrace.trc

To load a trace file into a data warehouse and display the GUI (Reporting Services-based) reports:

retrace -i mytrace.trc -r

To load a trace file into a data warehouse and display the top N queries report in HTML format:

retrace -i mytrace.trc -f html -r

To start a Profiler trace collection, then load the collected trace file(s) into a data warehouse:

retrace -c TRACE

To start a Profiler trace collection, load the collected trace file(s) into a data warehouse, and then display the GUI reports:

retrace -c TRACE -r

To load all trace files in a given folder into a data warehouse and display the GUI reports:

retrace -i myfolder\*.trc -r

To load a trace file into a data warehouse and write the top N queries report to an XML file:

retrace -i mytrace.trc -f xml_full -o myreport.xml

To display the GUI reports (following a previous load of trace data into the warehouse):

retrace -r

To load a trace file into a database other than [retrace] or to a non-default SQL Server instance:

retrace -d mydatabase -S myserver\myinstance -i mytrace.trc

As you can see, you can load one or more trace files from the command line. These can be ones you instructed the tool to collect via SQLDiag or ones you’ve collected through other means (e.g., SQL Profiler). You can then display reports showing the top N most expensive queries in a variety of formats (HTML, XML, PDF, Excel, JPEG, etc.) The default report format is a simple GUI report that makes use of client-side Reporting Services report files (you don’t need to have Reporting Services itself installed).

Customizing Retrace

You can customize Retrace’s output in a variety of ways. You can customize the RDLC files that make up the reports shown in the Retrace GUI if you like. And you can also modify the XML stylesheet (XSL file) that Retrace uses to translate its XML output into HTML when HTML output format is selected. This stylesheet is named retrace.xsl and resides in the XML subfolder. Here’s what the HTML output looks like by default:

You can customize the T-SQL script Retrace uses to create its data warehouse and reporting stored procedures by modifying the TraceAnalysis.sql script file in the Scripts subfolder. Be careful here, however, as modifying the objects in this script may prevent Retrace from being able to display meaningful analysis reports.

You can also customize the Retrace application itself. Retrace is a ScriptIt application. For those who don’t know what ScriptIt is, it’s a technology developed internally within Microsoft that allows you much of the flexibility you have with scripting tools such as Perl and VBScript with managed code applications. It combines the ease of deployment and quick modification you see in traditional scripting tools with being able to develop in your favorite managed code language. It takes advantage of the fact that every machine that has the .NET Framework 2.0 or later installed also has the full MSBuild environment (the toolset used by Visual Studio to compile and build application binaries). Using ScriptIt, you can modify the source of a managed code application, rebuild it, and redeploy using only the executable—you don’t need to store the source in separate files or keep track of solutions and projects in Visual Studio. You can edit your apps in Notepad if you like, then rebuild them on the spot without need of any other tools. The current plan is for ScriptIt to be released to the public next year, and you can contact Vance Morrison, its author, if you have any questions about it. Once ScriptIt is released, you’ll be able to edit Retrace’s source code and make whatever changes you need without resorting to full-blown application development.

SQL Nexus

I mentioned earlier that SQL Nexus has been released on CodePlex. This means that you can download the current binaries and source code today and extend/use Nexus in your own work. Read the license agreement over there for details. We will continue to keep a binary version of TraceBuster available here in case you’d like to use it with Nexus. If you get Nexus, you have a much more graphical, full-featured performance analysis tool than Retrace was intended to be. That said, if you like simple, fast command-line tools, you may find Retrace useful. I wrote a good chunk of both tools, and I think each has its place.

Conclusion

They say necessity is the mother of invention, and I guess necessity is just as responsible for Retrace as are Bart and I. Retrace was a tool we needed to do our jobs. It evolved into its current form through the necessity of our both needing a Swiss Army Knife-type Profiler trace tool and our belief that TraceBuster provided useful functionality that we should continue to leverage in our work on the product. Download it, use it to carve up a few of your own perf problems, and let us know what you think.

Random thoughts on my trip to the conference

khen1234 — Sun, 23 Sep 2007 22:38:00 GMT

I had a great time on my brief stint in Denver for the PASS Summit. We had a good session and a good crowd, and Kevin Kline did a marvelous job of running the whole thing. He had lots of great questions for me during the interview, and the crowd had many good ones, too. I hope those of you who attended felt it was worth your time.

One thing that came up during the talk was the status of SQL Nexus and what its future is. As I said when asked about it:

You can currently download the latest version of SQL Nexus from www.sqlnexus.net. For those who don’t know, SQL Nexus is a SQL Server performance analysis tool written by Bart Duncan and me. It loads the diagnostics collected by SQL Server’s SQLDiag facility (and similar tools) into a data warehouse (see the Books Online if you aren’t familiar with SQLDiag), then provides graphs and charts over that data. You can use it to find your most expensive queries, troubleshoot performance bottlenecks, investigate blocking, etc. It was originally released on the CD accompanying the book, SQL Server Practical Troubleshooting: The Database Engine.

Nexus is something Bart and I dreamed up nearly seven years ago before Microsoft had released any tools of this kind for SQL Server. Its influence is evident in similar tools that have appeared since then. For example, many of you are familiar with the ReadTrace tool. When Bart and I first proposed automated performance analysis to our management within Microsoft, ReadTrace had not yet been released to the public and was merely a trace file splitter used by SQL Server Support—it split Profiler trace files by SPID and wrote them out to separate files. This facilitated more easily seeing which users were running which queries rather than having to filter on the SPID column in Profiler. It did not do any analysis of the trace files it read back then—it was intended merely to simplify the job of seeing which users did what and play that work back as necessary. Some time after we’d made our proposal regarding automated performance analysis, some of these ideas found their way into the next release of ReadTrace. If you’ve ever wondered about the incongruity between the tool’s name and its analysis feature (which is probably its most useful feature from an end-user perspective), that’s why it’s there—ReadTrace didn’t originally do any analysis. Similarly, the Data Collector and Management Data Warehouse features in Katmai were influenced by our work with PSSDiag/SQLDiag and SQL Nexus. (Bart and I have been involved with the DC/MDW project from the start, and Bart recently joined the team fulltime.) We also know that various third party tools have built on the concepts we originally championed. From that one seed, lots of goodness has grown, and we couldn’t be happier about it.
Bart and I have given Microsoft’s Customer Support Services organization permission to release the source code to SQL Nexus via Microsoft’s CodePlex site. Current word is that this should be out internally by Thanksgiving (I don’t know when it will be available publicly on CodePlex, but I would expect it to shortly follow). CodePlex, for those of you who don’t know, is Microsoft’s shared source initiative, not unlike SourceForge and similar sites. Releasing the Nexus source will allow those of you in the user community who are so inclined to help develop and evolve Nexus further.

A concern we in SQL Server development had about releasing the Nexus source was that we might want to use parts of it in future products. To that end, we’ve asked CSS not to release the source code to a handful of Nexus components. One of these components is Nexus’ TraceBuster facility. TraceBuster, you may recall, is the Profiler trace loader in Nexus. It performs a similar function to the ReadTrace utility some of you may be familiar with, but differs in certain ways:

Like the rest of Nexus, TraceBuster is written in managed code. This means it integrates more seamlessly and more efficiently with the rest of the architecture.

Nexus offers a pluggable architecture wherein you can add new loaders to import diagnostic data into the data warehouse by merely editing an XML file or, for more involved work, by creating a simple managed code assembly and dropping it into a folder. You can easily implement new reports by designing them with the Reporting Services designer (in Visual Studio, for example) and dropping them into Nexus’ custom reports folder. TraceBuster reads Profiler trace files, normalizes the query text it finds in them by removing literals, and loads all this into the data warehouse. It also includes several reports for analyzing the most expensive queries in a workload, drilling into those queries, gauging their cost using a variety of metrics, etc.

While we also built a Nexus loader for ReadTrace, ReadTrace is not managed code and is a console app rather than an assembly, so the integration isn’t quite as efficient or seamless. The Nexus loader for ReadTrace must shell to a separate application rather than running it in the Nexus process itself as happens with TraceBuster.
TraceBuster does not require the code it analyzes to be fully parseable. This means that it’s more tolerant of the broken T-SQL sometimes seen in Profiler traces. Because of the vagaries of networks and SQL Server’s trace architecture, bad T-SQL in Profiler traces is more common than you might think. Missing quotes, truncated statements, missing pair members—these are not that uncommon in trace files. Utilities that analyze the T-SQL in trace files must be more tolerant of anomalies in it than, say, the database engine would be. TraceBuster didn’t begin as a full T-SQL parser that was then special-cased to handle all the eccentricities of traced T-SQL. It began as a no-frills “literal identifier” that merely replaces literals in query text with generic parameter markers so that you can easily compare instances of the same query with different filter values. It does not know and does not care whether the code it examines is perfectly intact. It does no more and no less than exactly what it needs to in order to provide meaningful query cost analysis. This makes it an order of magnitude simpler and more robust than full parser-based solutions.
TraceBuster uses SQL Server’s own facilities for reading Profiler trace files. It defaults to using SQL Server’s fn_trace_gettable function if the trace files you provide it are on the server machine (or accessible by it) and falls back on the SMO facility for reading trace files if fn_trace_gettable can’t get to them. It never directly accesses the trace files themselves. This means that TraceBuster is immune to file format changes across releases. It does not need (or want) intimate knowledge of the Profiler trace file format. The file format changed between SQL Server 2000 and SQL Server 2005, but the same version of TraceBuster can read trace files written by either release because it uses SQL Server’s built-in facilities for doing so.

Given all this and the fact that TraceBuster won’t be included in CSS’s CodePlex release of Nexus, you might be wondering whether you need TraceBuster and how you can get it after CSS releases the new version. You’ll have to decide for yourself whether you need TraceBuster. Tentatively, CSS plans to release a new version of ReadTrace that can read SQL Server 2005 Profiler trace files around the same time Nexus is released via CodePlex. ReadTrace and TraceBuster provide much of the same functionality, so it is unlikely that you would want to use both of them. And though they are similar, there are important differences between them, as well.

If you decide that you want to continue to use TraceBuster once it has been removed from Nexus, you can get the binaries (no source) from www.sqlnexus.net, the same site that will host Nexus itself until CSS releases it via CodePlex. The specific link is http://www.sqlnexus.net/download/tracebuster/TraceBuster.zip. If that link doesn't work, try this one: http://khenzden.googlepages.com/TraceBuster.zip.

I know all of this may be a bit confusing, so let me net it out:

The current version of SQL Nexus (including the TraceBuster component) is available today, without source code, via www.sqlnexus.net. It provides a rich performance analysis suite for SQL Server, including the type of Profiler trace analysis you may have done with ReadTrace in the past. It is functionally on par with the source-code version of Nexus that CSS plans to release via CodePlex. If you don’t plan to make use of the Nexus source code, there’s no reason to wait for the CodePlex release. You can get what you need today.
The next version, due to be released internally by CSS around Thanksgiving and sometime thereafter as source code via CodePlex, won’t include TraceBuster because CodePlex releases are necessarily source code-oriented, and the SQL Server development team has reserved the right to include pieces of TraceBuster in future products. It will, however, include support for ReadTrace (which should be released by CSS concurrently), a tool that performs much the same function, but that is not as well integrated with Nexus.
If you still want to use TraceBuster after the new version of Nexus has been released, it will be available here or here in binary form (no source), just as it is today. Once you’ve installed the CodePlex Nexus release, all you’ll need to do is copy the TraceBuster assembly and the included reports to the appropriate folders. There’s a readme file in the zip that provides specifics. When you then restart Nexus, you’ll see TraceBuster and its reports available in the GUI.

So, anyway, I had a good, if brief, PASS trip, and enjoyed meeting those of you who stopped by. Thanks for all the great questions during the interview, and I hope to run into you again in the future.

A funny thing that happened at the Avis counter at the airport:

Guy at the counter: Sorry, sir, the economy car you reserved isn’t available. We’ll have to upgrade you free of charge to a midsize. Is that okay?

Me: Okay, that’s fine. By the way, where’s the men’s room?

Guy: <funny look on his face> Is there something I could help you with?

Me: Uh, no. Just need the men’s room, thanks.

Guy: <a little peeved> Okay, just a moment. <then takes my paperwork and disappears through door #3 behind the counter>

Me: <a bemused look on my face, I’m wondering what the heck is up. I scan the place looking for hidden cameras>

It then occurs to me that the guy believes I’m not happy about the car switch and thinks I’ve requested a manager rather than a men’s room. Whew! I finally understand why he wanted to help :-)

I then tell the six women working the counter what the situation is. We’re all laughing hysterically by the time the guy returns with the manager. He is nonplussed.

Oh yeah, and the car I ended up with was the new, improved, quite sporty Dodge Charger of Dukes of Hazzard fame. Me likey—this one is a worthy heir to its muscle car forebears. The one thought that kept going through my mind as I zipped around Denver in the Charger was: I gotta get me one of these.