Carpe Datum

Quote of the Day – On Having a Purpose

2009-11-20T15:03:33Z

I try to live by this every day:

“No one is useless in this world who lightens the burden of another.” – Charles Dickens

Using Perfmon with SQL Server – Part Two

2009-11-19T15:17:50Z

Yesterday I posted a header from a script that shows you some links for how to use “Logman”, a Windows command-line utility that can automate your collection of Windows Performance Monitor counters. Towards the bottom, you can see the logman command I use, and you can see that it points to a counter file for the objects and counters I collect on a regular basis.

In the script that follows, you’ll see the counters I collect. There are a LOT of caveats to this script – For one, it assumes a Database Named PERF, that you have the same counters I do, and that you’ve collected them into a Comma-Separated Values (CSV) file in the C:\TEMP directory.

In fact, it’s best if you just read through this script, and understand what is going on rather than trying to run it. Once you understand it, you can create your own structure using the those concepts.

As always, this is on a test system, know what you’re doing, don’t run with scissors:

/*    Perfmon Example.sql
Purpose: Imports Perfmon Data into SQL Server
Author:   Buck Woody
Last Edited: 09/30/2009
Instructions: Proper Counters must be tracked. See table definition
    for that structure.
References:
You can also automate this: http://www.mssqltips.com/tip.asp?tip=1776
Another way: http://www.mssqltips.com/tip.asp?tip=1722
Counters to include: http://sqlblog.com/blogs/kevin_kline/archive/2008/03/28/the-most-important-performance-monitor-counters-for-sql-server.aspx
Interpretations: http://www.extremeexperts.com/SQL/Articles/SQLCounters.aspx
More Interpretations: http://technet.microsoft.com/en-us/magazine/2008.08.pulse.aspx
*/

/*
Logman automation - Comes with Windows OS:
logman /?
logman create /?
logman start /?
logman stop /?
logman start "SQL Server Counters"
*/

USE PERF;
GO

/* Clean up if desired
DROP TABLE PerfmonCounters
End Clean Up */

CREATE TABLE PerfmonCounters (
DateCollected VARCHAR (55)
, Processor_Total_Percent_Processor_Time varchar (100)
, PhysicalDisk_Total_Average_Disk_Bytes_Write varchar (100)
, PhysicalDisk_Total_Average_Disk_Bytes_Read varchar (100)
, PhysicalDisk_Total_Average_Disk_Queue_Length varchar (100)
, MSSQL_SQL2K8_Transactions_Longest_Transaction_Running_Time varchar (100)
, MSSQL_SQL2K8_SQL_Statistics_SQL_ReCompilations_sec varchar (100)
, MSSQL_SQL2K8_SQL_Statistics_SQL_Compilations_sec varchar (100)
, MSSQL_SQL2K8_Memory_Manager_Total_Server_Memory_KB varchar (100)
, MSSQL_SQL2K8_Memory_Manager_Target_Server_Memory_KB varchar (100)
, MSSQL_SQL2K8_General_Statistics_Logouts_sec varchar (100)
, MSSQL_SQL2K8_General_Statistics_Transactions varchar (100)
, MSSQL_SQL2K8_General_Statistics_User_Connections varchar (100)
, MSSQL_SQL2K8_General_Statistics_Active_Temp_Tables varchar (100)
, MSSQL_SQL2K8_General_Statistics_Logins_sec varchar (100)
, MSSQL_SQL2K8_General_Statistics_Processes_blocked varchar (100)
, MSSQL_SQL2K8_Databases_Total_Transactions_sec varchar (100)
, MSSQL_SQL2K8_Databases_Total_Data_File_Size_KB varchar (100)
, MSSQL_SQL2K8_Databases_Total_Backup_Restore_Throughput_sec varchar (100)
, MSSQL_SQL2K8_Databases_Total_Active_Transactions varchar (100)
, MSSQL_SQL2K8_Buffer_Manager_Buffer_cache_hit_ratio varchar (100)
, MSSQL_SQL2K8_Access_Methods_Page_Splits_sec varchar (100)
, MSSQL_SQL2K8_Access_Methods_Full_Scans_sec varchar (100)
, Memory_Percent_Committed_Bytes_In_Use varchar (100)
, Memory_Available_MBytes varchar (100)
, Memory_Cache_Faults_sec varchar (100)
)

/* Import */
BULK INSERT PerfmonCounters
FROM 'C:\TEMP\\ImportMe.csv'
WITH (FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n')
GO

/* Analysis */
SELECT CONVERT(DATETIME, [DateCollected]) AS 'DateAndTimeCollected'
      , CONVERT (NUMERIC(20,10), [Processor_Total_Percent_Processor_Time]) AS 'Processor_Total_Percent_Processor_Time'
      ,[PhysicalDisk_Total_Average_Disk_Bytes_Write]
      ,[PhysicalDisk_Total_Average_Disk_Bytes_Read]
      ,[PhysicalDisk_Total_Average_Disk_Queue_Length]
      ,[MSSQL_SQL2K8_Transactions_Longest_Transaction_Running_Time]
      ,[MSSQL_SQL2K8_SQL_Statistics_SQL_ReCompilations_sec]
      ,[MSSQL_SQL2K8_SQL_Statistics_SQL_Compilations_sec]
      ,[MSSQL_SQL2K8_Memory_Manager_Total_Server_Memory_KB]
      ,[MSSQL_SQL2K8_Memory_Manager_Target_Server_Memory_KB]
      ,[MSSQL_SQL2K8_General_Statistics_Logouts_sec]
      ,[MSSQL_SQL2K8_General_Statistics_Transactions]
      ,[MSSQL_SQL2K8_General_Statistics_User_Connections]
      ,[MSSQL_SQL2K8_General_Statistics_Active_Temp_Tables]
      ,[MSSQL_SQL2K8_General_Statistics_Logins_sec]
      ,[MSSQL_SQL2K8_General_Statistics_Processes_blocked]
      ,[MSSQL_SQL2K8_Databases_Total_Transactions_sec]
      ,[MSSQL_SQL2K8_Databases_Total_Data_File_Size_KB]
      ,[MSSQL_SQL2K8_Databases_Total_Backup_Restore_Throughput_sec]
      ,[MSSQL_SQL2K8_Databases_Total_Active_Transactions]
      ,[MSSQL_SQL2K8_Buffer_Manager_Buffer_cache_hit_ratio]
      ,[MSSQL_SQL2K8_Access_Methods_Page_Splits_sec]
      ,[MSSQL_SQL2K8_Access_Methods_Full_Scans_sec]
      ,[Memory_Percent_Committed_Bytes_In_Use]
      ,[Memory_Available_MBytes]
      ,[Memory_Cache_Faults_sec]
FROM [MDW].[dbo].[PerfmonCounters]
GO

-- Specific Analysis

SELECT
AVG(CONVERT (NUMERIC(20,10), [Processor_Total_Percent_Processor_Time])) AS 'Average'
, STDEV(CONVERT (NUMERIC(20,10), [Processor_Total_Percent_Processor_Time])) AS 'Standard_Deviation'
, MAX(CONVERT (NUMERIC(20,10), [Processor_Total_Percent_Processor_Time])) AS 'Maximum_Processor'
, MIN(CONVERT (NUMERIC(20,10), [Processor_Total_Percent_Processor_Time])) AS 'Minimum_Processor'
FROM [MDW].[dbo].[PerfmonCounters]
GO

/* End Perfmon Example.sql */

Using Perfmon with SQL Server – Part One

2009-11-18T16:34:00Z

I held a workshop on performance tuning in Portland, Oregon yesterday, and as part of that I mentioned a process I follow for automatically collecting Performance Monitor Counters for a system. I use this data to store a baseline and then subsequently to have a reference set of data to compare with.

I use the logman feature in the operating system, and I had some references there in my script, and I promised I would post the comments block at the top of the script.

Tomorrow I’ll post how I use the CSV file this collects and import it into SQL Server for analysis:

/*    Perfmon Example.sql
Purpose: Imports Perfmon Data into SQL Server
Author:   Buck Woody
Last Edited: 09/30/2009
Instructions: Proper Counters must be tracked. See table definition
                     for that structure.
References:
You can also automate this: http://www.mssqltips.com/tip.asp?tip=1776
Another way: http://www.mssqltips.com/tip.asp?tip=1722
Counters to include: http://sqlblog.com/blogs/kevin_kline/archive/2008/03/28/the-most-important-performance-monitor-counters-for-sql-server.aspx
Interpretations: http://www.extremeexperts.com/SQL/Articles/SQLCounters.aspx
More Interpretations: http://technet.microsoft.com/en-us/magazine/2008.08.pulse.aspx
*/

/*
Logman automation - Comes with Windows OS:
logman /?
logman create /?
logman start /?
logman stop /?

This one uses a file I have called “SQL Server Counters” – more on that later
logman start "SQL Server Counters"
*/

Channeling Bruce Lee: Tuning without Tuning

2009-11-17T15:03:34Z

I’m leading a Performance Tuning workshop here in Portland, Oregon today, and I normally start this kind of workshop with a segment on “pre-tuning” – or, the steps you take so that you don’t have to tune as much later.

When we set up a system is the greatest opportunity for optimal performance. From buying the right hardware (like using 64-bit architectures) to configuring the drive offset, stripe and block size, to installing the proper version and edition of the Operating System and SQL Server, you’ll never have a more impactful chance to make things go fast.

That goes for you, too, developers. Unit and Integration tests are the times to find out how your code is working, and there’s little pain in fixing it in the design compared to once it’s in production. You’ll find the fastest work is that which you don’t have to do – don’t use SELECT *, that kind of thing. Ditto for the database – proper normalization, data types and index choices are key.

So take Bruce Lee’s advice about fighting: “The best fighting strategy is not to fight.” The best tuning you can do is not to have to do it much at all.

Color Me Corrected – Will the REAL sysobjects please stand up?

2009-11-16T15:19:48Z

A few days ago I posted a way to look at the Dynamic Management Views (DMV’s) using a query, which I then copy and paste into the Help viewer’s Index tab to learn more.

Well, my good friend and Colleague, Cliff, pointed out that I was using the older sysobjects system table. Well, of course this is a deprecated feature, and not the correct way to reference system objects in newer versions like SQL Server 2005 and higher.

I did that because I know a lot of you out there are still on SQL Server 2000 (shame on you!), but when he mentioned it I realized I hadn’t included the “correct” way to do this, using sys.system_objects, which has other benefits. It’s also a more global view, so you don’t have to USE master first, making the code shorter – and shorter code is often better! So here’s the more modern way do that:

SELECT

'sys.' + name

FROM

sys.system_objects

WHERE

name LIKE 'dm%';

You can also toss in an ORDER BY name clause in there if you want to make the list sort.

Thanks Cliff -

Quote of the Day: Who Is Normal?

2009-11-13T15:24:59Z

No, this isn’t about third normal form or anything like that:

“Everyone seems normal until you get to know them.” - Unknown

Store XML Document as Binary, Read as XML

2009-11-12T15:12:19Z

I had an interesting question the other day, so I thought I would share both the question and the answer.

We had an older application that had stored an XML document in a table as a binary data type column. I know, I know – we’ve had an XML data type for some time, but not when this app was written. So of course the tables just moved along with the app, even though there was a better way to do it. The question I got was, can I read the binary field as XML, even though it isn’t?

The answer is: “It Depends”. No, that’s just the answer I always give. In this case, I was able to do it easily. If you want to try a test yourself, just create an XML document in a C:\temp directory (on a test system, your mileage may vary, don’t run with scissors) and then run this code:

/* Need a test table to work with */
CREATE TABLE TestTable(DocumentName varbinary(max)) ;
GO

/* I'll load it up with an XML document, but store that as binary */
INSERT INTO TestTable(DocumentName)
SELECT *
FROM OPENROWSET(BULK N'C:\temp\Test.xml', SINGLE_BLOB) AS Image;
GO

/* Let's see it in text */
SELECT CAST(TestTable.DocumentName AS varchar(max))
FROM TestTable;
GO

/* And in XML*/
SELECT CAST(TestTable.DocumentName AS XML)
FROM TestTable;
GO

There are other possible solutions as well. This worked for my situation.

How Microsoft Does IT - Updated

2009-11-11T15:49:55Z

I’ve pointed out this link before, and I’ll definitely bring it up again. Microsoft is one of the few companies I know, especially the big ones, that explains how they run their IT. We have a fantastic, I mean really fantastic site that you should definitely check out. Can save you lots of time, and is like free consulting.

You can find it here: http://technet.microsoft.com/en-us/library/bb687780.aspx

Visio Video (killed the radio star)

2009-11-10T15:18:40Z

I teach a class at the University of Washington, and in that class I have the students create a few Entity Relationship Diagrams (ERDs). I’m still looking for a decent tool to do that, since the only ones I know that do a good job are WAY too expensive. Visio is about the best I’ve found for them, and one of the students sent me a link he found for learning and using Visio – very nice. Thanks, Rod!

http://www.youtube.com/watch?v=1BYt3wmkgXE&feature=related

By the way, Visio Team – it’s a crime that I have to install all of Enterprise Architect just to get an ERD tool. Break it out of there into it’s own SKU!

Create an Excel Graph of your Big Tables – with PowerShell!

2009-11-09T14:54:43Z

I showed a demo of how to find the top ten tables in the database at the PASS Conference. Here’s that script – you’ll need to fix the server name, instance name, and database name. You can use this to display any numbers – and even more. The mind reels with the possibilities. This uses the PowerShell provider from SQL Server 2008 (sqlps.exe) but works against 2005 and even 2000 Instances.

Oh, you have to have Excel Installed, of course! All the usual caveats apply – use a test server, know what you’re doing and all that:

# Big Tables to Excel Chart

# Keep this next part on one line… This gets your objects to put in the chart

$BigTables= DIR SQLSERVER:\SQL\UNIVAC\DEFAULT\Databases\Adventureworks\Tables | sort-Object -Property RowCount -desc | select-Object -First 10

$excel = new-object -comobject excel.application

$excel.visible = $true

$chartType = "microsoft.office.interop.excel.xlChartType" -as [type]

$workbook = $excel.workbooks.add()

$workbook.WorkSheets.item(1).Name = "BigTables"

$sheet = $workbook.WorkSheets.Item("BigTables")

$x = 2

$sheet.cells.item(1,1) = "Schema Name"

$sheet.cells.item(1,2) = "Table Name"

$sheet.cells.item(1,3) = "RowCount"

Foreach($BigTable in $BigTables)

{

$sheet.cells.item($x,1) = $BigTable.Schema

$sheet.cells.item($x,2) = $BigTable.Name

$sheet.cells.item($x,3) = $BigTable.RowCount

$x++

}

$range = $sheet.usedRange

$range.EntireColumn.AutoFit()

$workbook.charts.add()

$workbook.ActiveChart.SetSourceData($range)

Quote of the Day – The Definition of an Expert

2009-11-06T14:50:42Z

I’ve been at the PASS conference this week, and I’ve been around a lot of really smart Experts – and this definition definitely fits:

An expert is someone who knows some of the worst mistakes that can be made in their subject, and how to avoid them. - Werner Karl Heisenberg

Learning New Tech on SQL Server Virtualization – From Vendors

2009-11-05T17:58:19Z

I’m at PASS this year, and one of the things I’ve learned this time is that I can learn from – the vendors! Most of the “big” names are here, like Idera, Redgate, Quest, and more, but I ran into some folks yesterday I hadn’t spoken with before – XKoto.

I’m actually surprised I hadn’t run into them earlier. I talk with folks all the time about virtualizing, which should really be a conversation around consolidation, and the XKoto folks have an incredible product that handles scale-out in a new way. Check them out: http://www.xkoto.com/

Looks like even and old dog can learn some new tricks…

Find and learn DMVs

2009-11-04T16:04:57Z

This morning I showed how I find DMVs, and find out how to use them. First, I just run this query:

USE MASTER;

SELECT ‘sys.’ + name

FROM sys.sysobjects

WHERE name LIKE ‘dm%’

Then I copy and paste a result into the “Index” panel of Books Online. I also use a web search to learn more.

I also use the new Activity Monitor, and just hover over a column header in that tool – Microsoft tells you what DMV they are using (if they are using one) and even a little about it. How cool is that!

DBA? No – Data Professionals

2009-11-03T16:24:29Z

I’m here at the Professional Association for SQL Server (PASS) this week, and I’m here in the keynote. A few days ago I ranted about the fact that DBA should be a Data Professional – not an administrator. I’ve heard a lot of response about that blog – and all were in support.

And now here at PASS, what is the theme? Taking a strategic approach, and being professional. w00t! Have we turned the corner? Are other folks realizing that you hold the keys to the kingdom when you’re the DBA? I think so. We’re getting there…

Win7 – Install SQL Server Native or go Virtual PC?

2009-11-02T14:28:00Z

I have Windows 7 on my laptop, and I also teach, demo and use SQL Server 2005 and SQL server 2008. Should I install this “native” on my outside operating system or use the Virtual PC (VPC) software that comes with Windows 7? Well, there are arguments for each.

It’s far simpler to start up SQL Server Management Studio (SSMS) or Business Information Development Studio (BIDS) without having to do anything else on my system. Point: outside operating system. And I can install the developer edition of SQL Server 2005 and 2008 with no trouble on Windows 7. Point: outside operating system.

Using a VPC images lets me install a server-level operating system, SQL Server 2000, 2005 and 2008. That lets me more closely replicate my true production environment. I can also install other things I want on the image and not “pollute” my outside operating system. I can copy the VPC file elsewhere, set recovery points to try something and then “roll it back” to a previous state (very helpful with service packs!) and I can upgrade the VPC to Hyper-V. Point(s): VPC.

So here’s what I do: I have all of the client tools on my PC, along with an Instance of SQL Server Express. But for the most part I use a VPC, for all of the reasons above. It’s the best of both worlds.