Carpe Datum

Read a Web Page from PowerShell, Make a Web Page from a Database Query From PowerShell

2009-07-09T15:46:45Z

I presented at the Pacific Northwest SQL Server User Group here in the Seattle area last night, and I was asked at the break about reading a web page in PowerShell, and being able to do something with the data it has. There are actually a few ways to do that, but you should probably start with the net.webclient call - there's a quick overview of that here: http://www.johndcook.com/PowerShellCookbook.html#a20 and more info on it here: http://msdn.microsoft.com/en-us/library/system.net.webclient(VS.80).aspx.

Once you have the web page object, you'll need to parse it using string methods to find the data you want to get, store those as variables, and then use standard database insert calls to put the data back. Here's an article on inserting data, and there are others as well: http://www.scriptinganswers.com/forum2/forum_posts.asp?TID=2316

You can also hit a web service even easier, and then store the results directly into a variable: http://thepowershellguy.com/blogs/posh/archive/2009/05/15/powershell-v2-get-weather-function-using-a-web-service.aspx

And I've written an article that shows how to go the other way and make a simple web page by using PowerShell to extract data from a database. You can find that here: http://www.informit.com/guides/content.aspx?g=sqlserver&seqNum=272.

ODBC Drivers Are Updated in Windows 7

2009-07-08T15:14:29Z

One of the first exposures I had to an ODBC driver was NOT a good one - the particular behavior for ANSI NULLS ended up causing some issues in a database application that I had inherited, so I tend to keep my eye on these things. Mind you, it was back in the SQL Server 6.5 days, and I haven't had an issue since then, but I keep an eye on it nonetheless.

Windows 7, which is set to release soon, includes a new ODBC driver - 3.80. It gives you several new enhancements which you can read about here, but keep this in mind when you're deploying a new operating system - this is exactly the situation that bit me. The point is, as always, whenever you make a change, test, test, test.

SQL Server 2008 Books Online Update is Released

2009-07-07T15:12:04Z

I try not to post things that others have already discussed, but I make a few exceptions. I'll (re)tell you about security issues and service packs, and I'll also mention when Books Online is updated. I run into so many issues that are already covered in BOL, so I think it's important to track when new content is added or altered.

You can find the On-Line version (which is always the latest) here: http://msdn.microsoft.com/en-us/library/ms130214.aspx

You can find what has changed for this version here: http://msdn.microsoft.com/en-us/library/dd776386.aspx

You'll get the latest version in your Microsoft Update Services for the installed BOL. How can you tell what version of BOL you have installed? Look in the title bar.

Consolidation Strategies: Profiling Your Systems

2009-07-06T15:00:37Z

Many shops are considering consolidating their SQL Server Instances onto fewer boxes, as I've blogged about before. There are lots of ways to think about consolidation, from placing more databases on one Instance (Database Stacking), putting multiple Instances on one server (Instance Stacking), and using a Hypervisor (Virtualization). But before you do any of that, you should profile the servers and Instances you currently have so that you know what applications are candidates for consolidation.

So how do you do that? Well, you could learn lots of counters, objects, dynamic management views and so on, or you could just download the latest Microsoft Assessment and Planning Tool and run that. Here's the link for that, and here's an article I did for the product.

Quote of the Day: On Keeping Ease in its Place

2009-07-03T15:07:13Z

I admire people who have overcome difficulties to achieve. I find so few people today that can stand any adversity - people whine and complain at the slightest inconvenience. In the past character seemed to matter more, and Helen Keller was one such person of character. She said:

"Character cannot be developed in ease and quiet. Only through experiences of trial and suffering can the soul be strengthened, vision cleared, ambition inspired and success achieved." - Helen Keller

PowerShell and SQL Server: Script all Tables

2009-07-02T14:27:52Z

This is a script that I found/put together/re-arranged that will script out all of the tables from a database - in this case, Adventureworks2008. You need to change the BWOODY1 part to the name of your server, and the SQL2K8 part to your Instance name. You can change the database name as well, of course, and a better method would be to make a function out of this and feed those variables in.

There are other ways to do this, but this script illustrates setting some of the scripting options. You can look up the SMO model for all of them.

As always, test this script out thoroughly, and understand what you're doing before you put this on a production system. Also, don't run with scissors, and don't open attachments from people you do not know:

# Script all tables in the AdventureWorks2008 database

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.SMO") | out-null

$s = new-object ('Microsoft.SqlServer.Management.Smo.Server') 'BWOODY1\SQL2K8'

$db = $s.Databases['AdventureWorks2008']

$scrp = new-object ('Microsoft.SqlServer.Management.Smo.Scripter') ($s)

$scrp.Options.AppendToFile = $True

$scrp.Options.ClusteredIndexes = $True

$scrp.Options.DriAll = $True

$scrp.Options.ScriptDrops = $False

$scrp.Options.IncludeHeaders = $True

$scrp.Options.ToFileOnly = $True

$scrp.Options.Indexes = $True

$scrp.Options.WithDependencies = $True

$scrp.Options.FileName = 'C:\TEMP\AdventureWorks2008Tables.SQL'

$scrp.Script($db.Tables)

SQL Server Best Practices: Auto-Shrink Should Be Off

2009-07-01T14:08:46Z

SQL Server is one of the easiest databases to maintain because of all of the automatic settings it has, but as I mentioned with Auto-Close, some of them should be left off. The Auto-Shrink setting is another.

That might surprise a few people. You might think from the name of this operation, that you would want your databases to automatically reclaim any space they take whenever they can. And if SQL Server will handle that for you, all the more wonderful. But in fact, shrinking a database can cause your Indexes to be fragmented, especially if it happens a lot. I never shrink my databases unless I have a huge deletion of data, and I know that the data won't come back. That's a pretty rare event, and when it does happen I run the shrink operation manually and rebuild my indexes after.

My friend Paul Randall has a great explanation of why this happens, and another way to reclaim that space without running shrink at all: http://www.sqlskills.com/BLOGS/PAUL/post/Why-you-should-not-shrink-your-data-files.aspx

What's a Microsoft "Fast Track" for Data Warehousing?

2009-06-30T16:10:31Z

Microsoft is a platform company, meaning that we don't often sell pre-packaged solutions, but rather tools for you to make them. In other words, we sell Microsoft Word, we don't type the letters for you.

But in some cases you might like to have a little more guidance than just "Here's a database product. Go nuts." You might want to know what OS goes with which edition, and you might even want to know what hardware configuration to buy.

To help you with that, we came out with this concept of a "Fast Track". It's basically a manifest of hardware and software to buy and a list of how to set it all up, that ends up with a solution you can use for Data Warehousing. I really like the idea, and I hope we extend it to the database platform, although I've not heard anything to that effect. At any rate, you can check it out here: http://www.microsoft.com/sqlserver/2008/en/us/fasttrack.aspx

Grid Painting and Database Projects

2009-06-29T15:08:27Z

I'm not very good at drawing. My drawings of people and objects with depth usually looks like what your 5th grader does, but not the talented 5th grader. But I found a trick that helps me draw not only depth, but complex things as well.

It involves taking a picture you want to duplicate and laying out a grid of lines over it. Then you just draw the same grid on another sheet of paper, and instead of trying to draw the entire person, landscape or horse, you focus on one little box in the original grid and draw the squiggly lines you see there. Then move one box to the right and do that again, and again, and again, and eventually you've drawn the entire scene. Carefully erase the grid lines that are left, and you've drawn better than you really can by freehand!

I've been asked by some DBA's how to approach a complex project, and I think the trick is to do the same thing. I lay out an imaginary "grid" or work to do. The first grid I lay out is really large:

Design the database solution
Implement the solution
Set up maintenance (both database and code-related) for the solution
Set up High Availability (HA), Disaster Recovery (DR), and Security for the solution

Then I set up a finer "grid" on each of these - Design breaks down into:

Define the business requirements
Look for nouns and verbs
Turn those into tables and relationships
Tease out the columns for the tables
Normalize the tables
Set up the Referential Integrity...

And so on. I do the same for each of the main "grid" segments, and then off I go. Sure, this is just turning big problems into smaller ones, but I find that it really helps - it removes the stress, focuses the attention and allows me to create a fairly accurate timeline for my part of the project.

Quote of the Day: Aesop On Learning

2009-06-26T15:21:46Z

It is truly remarkable how many things we learn from the ancients - and how much we don't:

"Good judgment comes from experience, and experience--well that comes from poor judgment." - Aesop

Use PowerShell to Backup All User Databases

2009-06-25T15:16:46Z

This script will back up all user databases - you need to change the BWOODY1\SQL2K8 part to your server\instance name, and of course, you should only run this on a test system until you completely understand it.

Unfortunately, the SQL Server PowerShell Provider doesn't make this very easy - and there may be a better way to do this once I research it more.

Also - this script was adapted from one given to me earlier, and unfortunately I don't recall the source. If this is partly your work, please add a comment here for attribution:

# Performs a Full backup followed by a transaction log backup on all user databases

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.SMO")  | out-null

$s = new-object ('Microsoft.SqlServer.Management.Smo.Server') 'BWOODY1\SQL2K8'

$bkdir = $s.Settings.BackupDirectory

$dbs = $s.Databases

$dbs | foreach-object {

    $db = $_

    if ($db.IsSystemObject -eq $False -and $db.IsMirroringEnabled -eq $False) {

        $dbname = $db.Name

        $dt = get-date -format yyyyMMddHHmmss

        $dbbk = new-object ('Microsoft.SqlServer.Management.Smo.Backup')

        $dbbk.Action = 'Database'

        $dbbk.BackupSetDescription = "Full backup of " + $dbname

        $dbbk.BackupSetName = $dbname + " Backup"

        $dbbk.Database = $dbname

        $dbbk.MediaDescription = "Disk"

        $dbbk.Devices.AddDevice($bkdir + "\" + $dbname + "_db_" + $dt + ".bak", 'File')

        $dbbk.SqlBackup($s)

        if ($db.DatabaseOptions.RecoveryModel -ne 'Simple') {

            $dt = get-date -format yyyyMMddHHmmss

            $dbtrn = new-object ('Microsoft.SqlServer.Management.Smo.Backup')

            $dbtrn.Action = 'Log'

            $dbtrn.BackupSetDescription = "Trans Log backup of " + $dbname

            $dbtrn.BackupSetName = $dbname + " Backup"

            $dbtrn.Database = $dbname

            $dbtrn.MediaDescription = "Disk"

            $dbtrn.Devices.AddDevice($bkdir + "\" + $dbname + "_tlog_" + $dt + ".trn", 'File')

            $dbtrn.SqlBackup($s)

SQL Server Best Practices: AutoClose Should be Off

2009-06-24T15:50:21Z

When SQL Server "opens" a database, resources are assigned to maintaining that state. Memory for locks, buffers, security tokens and so on is assigned, and there is associated CPU and even a little I/O. When connections are made to the database, these resources are ready and waiting. The AutoClose setting dictates how these resources are handled. If this setting is ON, then when the last user connection that disconnects, SQL Server releases all of those resources back to the server. That might sound like a good thing, but if you think about the pattern of connections, the "last" connection doesn't mean that someone might not connect one tenth of a second later. In that case, SQL Server would shut all of the resources down, and then have to spin them right back up again. As a result, the server makes the connection wait, which makes the system seem slower.

So the best practice is to leave this setting OFF. In fact, SQL Server 2008 Books Online states that this setting is going away anyway, so you should have them turned off (http://msdn.microsoft.com/en-us/library/ms135094(SQL.90).aspx).

Is Your Infrastructure Crumbling?

2009-06-23T14:20:56Z

I watched a documentary (of sorts) last night called "Crumbling America" that talked about America's infrastructure issues. They detailed how roads, bridges, electrical grids and water systems are having serious issues and are basically falling apart. I think they were sensationalizing a bit, but it was interesting to note that America spends only 4% of it's income on infrastructure now, down from a high over over three times that amount. Even the "stimulus" plan containing almost a trillion dollars has less than 9% of the money earmarked for infrastructure.

I found all this interesting because it has corollaries to our IT infrastructure. While I can't control our federal infrastructure, but I can affect my local IT infrastructure, especially the database. I pulled out three things from the television program that I thought was applicable to IT. They mentioned three main issues with infrastructure that I think we should evaluate in our own systems:

1. The system had a design flaw.

The first thing that they pointed out about a road or bridge that failed was that in some cases the engineers decided to go with a lighter steel plate, or a different kind of surface that turned out not to be strong enough or last long enough. This holds true in IT as well. I've seen many systems that just had a fundamental design flaw that caused an issue later. What can we do about this? Two things: educate yourself about good design patterns and implement them in your own code, and evaluate any "canned" solutions your company buys and ask the hard questions about the design of the software. After all, you have to live with it when it is gone.

2. The system can't handle the load or speeds required in a modern environment.

In the case of a water system or electrical grid, the initial design or even the overall arrangement of the components cannot handle how much work required from the users. This of course has obvious parallels to IT. The thing we can do as DBA's and Developers is to do a good job of projecting load, and constantly monitoring the systems to make sure we know when we're reaching a limit.

3. The system upgrades keep getting deferred.

Keeping a road, bridge or other system maintained and upgraded is very expensive and disruptive. Politicians take the easy way out and defer that maintenance over and over. This leads to eventual disaster. It's the same in IT. As you monitor your systems, make sure you push for upgrades along the way. I run into firms every day that are still using SQL Server 2000 - even though it is out of maintenance, has lower security, and does not perform as well as the newer versions. I'm told "my vendor won't upgrade" or "it's working just fine" and it reminds me of the inspectors on the bridges that fail.

There are more parallels, but I think you have the idea. You can take almost any system that handles "traffic" or load and learn from it. The question is, will you?

Help Me Help You

2009-06-22T14:53:31Z

Before I start this post, let me say that it is not pointed at any one person - if you've asked me for help recently, this isn't about you!

The programming and administration computer disciplines are fairly unique, in my mind, in one interesting way: we help each other a lot. Since the beginning of the DARPANET, which eventually became the Internet, administrators have used e-mail, newsgroups and other methods to chat with each other - "hey, I'm having this issue....anyone know how to handle that?" It was a form of necessity, since everyone had to come up to speed quickly on a technology that wasn't documented yet because it didn't exist yet.

And that tradition continues today. Those of us that work at Microsoft, and of course tons of people who don't, participate in forums, e-mail exchanges, newsgroups and other forms of user-to-user assistance. But many times I'm not able (or very willing, at least) to help someone because they get in their own way. So I thought I would create a blog entry that gives you some ways to get help from those willing to give it. If you'll follow these simple steps, you'll probably find more of your posts answered:

1. State your question clearly.

Instead of "I tried this terrible software and it doesn't work right - how do I fix that?" be very specific: "When I implement the Maintenance Plan Wizard, on step three of the Wizard I get error 980234 from SQL Server Management Studio. Has anyone else encountered that, and where should I look to start fixing this?" The more precise you are the better the odds that someone can help you.

2. State what you've done to fix the problem so far.

Explain all of the steps you've taken to research or correct the issue so far: "...so I looked up error 980234 in Books Online, but it doesn't seem to apply because I'm not on a cluster. I searched on "non-clustered" and "980234" and I didn't come up with any hits." If you haven't even read the manual or looked at any one of the thousands of pages of documentation on the subject, don't ask the question. Do your homework first, put some effort into finding the answer.

On this note, don't expect others to re-write your code or do your work for you. There's a very big line between helping you find the answer to a question and doing your job - that's why they pay you, not me.

3. Be professional.

No one cares how much you hate a vendor or software product. No one cares how much time you've spent on this or how frustrated you are. None of that matters - what matters is that you need your problem fixed. If you ask a vague question, and insult Microsoft (or any vendor, actually), I just move on to another question. And don't demand and answer: "I need someone to help me fix this now!" If you can't be nice, why should someone else?

Quote of the Day: On Taking Chances

2009-06-19T14:24:27Z

Sometimes when things are tight people take fewer chances - but I think that's the very time you should go after the big things:

"Why not go out on a limb? Isn't that where the fruit is?" - Frank Scully