Ken Henderson's WebLog

How to Beat Halo 3 in 20 Minutes

khen1234 — Mon, 21 Jan 2008 11:29:00 GMT

(Or, How to Beat Its Longest Level, The Covenant, in 20 Minutes and Get the Full 3x Time Bonus)

(Or, What I Did On the Day after Christmas)

(Or, How to Set a New World Record for Speed Play in Halo 3)

Over the recent holiday break I decided to play some games with my kids. My teenage son loves Halo 3, so we played some of that. For those who aren’t Halo warriors, some back story: you can play against other people via Xbox Live or you can play against computer-controlled bad guys (The Campaign) and can optionally team up with other players against these computer baddies. My son and I used to play Halo 1 competitively over XBConnect years ago, but I lost interest when Halo 2 came out, Xbox Live debuted, and you couldn’t decide what maps, game variants, etc. you wanted to play—you had to play whatever the Powers That Be decided. I didn’t like that and basically retired from competitive Halo play. Despite my son and I (and occasionally Bart) having played, I’d estimate, hundreds of games against other teams and never having lost, I decided it was time to hang up the cyborg armor and shoulder holster. My son played on and continues to.

Anyway, we played some campaign over the holidays, and one of the new intriguing features of H3 is campaign scoring and time bonuses. You can opt to have the game record individual or team scores while battling the computer-controlled bad guys and get a time bonus for finishing a level within a specified time limit. For the full bonus, you must finish a level in less than 20 minutes. You get a 3x bonus (i.e., your score is multiplied by 3) for this.

This works out nicely for short- to medium-length levels, but not so well for really long ones. The longest level in Halo 3, The Covenant, takes nearly 20 minutes just to navigate through—forget killing bad guys. This level is, I think, the longest in any version of Halo so far, and really ought to be two or even three separate levels. Everyone I talked to thought it was simply impossible to get the full bonus for this level on Normal difficulty. The current record for completing The Covenant was 11-12 minutes (as recorded at http://www.highspeedhalo.net/?times), but it was done on Easy difficulty, which few veteran players would really consider valid. The game practically plays itself on Easy, and you play differently when there’s virtually no chance that the bad guys can kill you. You try tactics and take shortcuts that simply would not work on Normal difficulty or one of the higher ones. For me, the record for Normal difficulty or above was all I was interested in, and, not surprisingly, there were no records for Normal or above—no one had yet done it—not officially, anyway. I decided to find out whether I could. I haven’t set many records in my time, but this was one I thought might be within my grasp.

You might be surprised to learn that there’s an entire community oriented around high-speed Halo play, but that’s exactly what the fine folks at HighSpeedHalo are about. They’re a thriving group of expert players dedicated to nothing but completing the various levels of the three versions of the game as quickly as possible. Some of the times posted for some of the shorter levels are simply insane. Me, I like to play a little more deliberately, so the mammoth level The Covenant was right up my alley. I spent the day after Christmas figuring out how to beat it as quickly as humanly possible and, in so doing, earn the 3x time bonus. I learned a number of interesting details about the game and came to respect it on a whole new level. When you feel certain about something--when you're literally inspired by it--you ought to say so and throw away all the qualifiers and hedge words people like to work into everyday speech. For me, Halo 3 is, quite simply, the best video game ever published. I've played a lot of video games in my time, but it towers above them all--plain and simple. The folks at Bungie should be very proud of the final installment in their trilogy. Pondering the sheer amount of work that must have gone into making it the masterpiece that it is simply boggles the mind—even for an old programmer like me.

So, below is a little piece I wrote about my journey through The Covenant in under 20 minutes. It was originally linked on the Bungie forums, but I thought you might enjoy reading it. It includes step-by-step instructions in case you want to do it yourself (this didn’t take as long as it might appear—I wrote it mostly from memory). The piece also links video of me playing the game in case you want to see the record actually being set. My first record time was 19:45. Subsequently, I was able to get it down to 14:59, the current record. I think that could go still lower. As you might expect, my current techniques have evolved a bit from the first time I broke the 20-minute barrier. You can view all the current records here.

I’m back retired from Halo—I just don’t have time for it. But it’s a wonderful game, and I’ll return to it someday after some of my projects are done and it looks like there’s another mountain to climb.

How to Beat Halo 3 in 20 Minutes

I haven’t played much Halo since the original version. I used to play online via XBConnect on a regular basis, but I didn’t like some of the limitations of play on Xbox Live when Halo 2 came out, and I gradually lost interest. I was partial to Capture the Flag on Blood Gulch and didn’t like having to play umpteen other game variants and maps just to get a combination I actually enjoyed. I also didn’t like the prospect of having to play all the time just to be as competitive as I had been. I have a day job and a family and some semblance of a life. So, as fun as it is being schooled by someone who’s played a map a hundred times more than I have and can recite where every weapon and power-up is in his sleep, I haven’t played much H3 multiplayer. I don’t like playing with noobs (which is, of course, where you start in the Xbox Live ranking system when you haven’t played much), and I also don’t like getting my butt kicked by people who may not be great players in general but who simply have more time on their hands than I do (what you might call “educated noobs”—a gaming variant of the “educated fool”).

I’m also not a big fan of twitch-centric games. I like killing baddies as much as the next guy, but I prefer something with some strategy to it, some teamwork, some planning. Playing merely by reflex can be fun, but I got my fill of that back in the days of Doom and deathmatch—I want something with a little more substance to it now. That said, I don’t want a mere strategy game—the FPS still is and probably will always be my favorite kind of game. And I certainly enjoy racking up kills. It’s just that I want it to be for something more than getting the kill title for the game or bumping my experience points. I couldn’t care less about player ranking, armor types, or any of that other nonsense. I need an objective of some type, and I want it to involve teammates and strategy.

Playing the H3 campaign with others over Xbox Live for points and/or speed became a nice substitute for me. You have definite objectives and you have the team play and strategizing that I find so much fun. Turn on some skulls and play on Legendary—or play for speed—and you actually have a challenging game, something approximating the fun I had on XBConnect. Until I can create custom games that I can then advertise publicly and play random teams in (like I did with XBConnect), this is probably the only way I’ll play.

I got interested in playing H3’s longest level, The Covenant, for speed, and was surprised to learn that no one had recorded an official time on Normal difficulty or above for it. It appeared no one had been able to beat it in less than 20 minutes and earned the full time bonus. My teammates were pretty sure that getting the full bonus wasn’t even possible on Normal or above, and I did some online research that seemed to confirm that. No videos of anyone ever doing it, no step-by-step guide on how to do it, no discussion of it whatsoever. Most people seemed resigned to getting only a partial time bonus and making up for it with all the additional kills one could get thanks to the level’s inordinate length.

I had hoped that Bungie had play-tested each level before deciding on the 20 minute cap. Seemed reasonable to assume that if they were going to have a uniform time limit for all the levels, they’d start with what was a reasonably brisk pace for the longest level and use it for all the others. Doing otherwise could leave you with a time bonus that was simply unattainable. So, I wondered whether there might have been something we’d missed, a way perhaps to finish this humongous level in under 20 mins that we hadn’t yet discovered.

The day after Christmas I finally sat down and figured it out. It took many tries, but was embarrassingly simple once I finally got my act together. So, without further ado, here’s a step-by-step guide for beating Halo 3’s seventh level, The Covenant, in under 20 minutes. If you can beat this level in less than 20 minutes, you should be able to beat any H3 level in less than 20 minutes. The YouTube video of my first record-setting game is located here: Part 1, Part 2, and Part 3. The game film for my latest record is in Slot 1 under gamertag Ryanator if you have an ID on Bungie.net.

A few general recommendations:

· If you get killed, it’s very difficult to hit the full time bonus on this one. For that reason, I’ve been known to turn on the Iron Skull when I play this level for time. There’s no point in continuing to play for the full time bonus if it becomes impossible because you got killed and burned a bunch of time. Beating the level is not hard—on any difficulty. Beating it in under 20 mins is. Turning on the Iron Skull automatically resets you to the level start (if you’re playing solo), so it’s a time saver while you’re first learning how to quickly beat it.

· When playing strictly for time, you have to think differently than you normally do. For example, you’ll have to resist the urge to kill every enemy you come across—you won’t have time. You’ll also have to avoid doing things that leave you waiting for the game, such as waiting for your computer-controlled helpers or equipping them with desirable weapons. For example, when playing for points or with a bunch of skulls enabled, I frequently equip my passenger in the Warthog and the passenger on the Mongoose with a Fuel Rod and a Spartan Laser, respectively, before heading for the first tower because they will have unlimited ammo and are a great help in taking down the Banshees, Ghosts, and other things you come across. When playing for time, I don’t do this. I don’t care about racking up points, and every second counts (I actually missed the 20 minute mark once by one second—yes, one second—and I was not happy).

· My approach here (and the only one that works, as far as I know) is to skip as much of the level as possible. I’m fairly sure this is the only way to finish it in less than 20 minutes (without cheating by getting out of the map, that is—search YouTube for videos demonstrating that). This begs the question as to why Bungie would make the limit so low that you have to not play parts of it in order to get the bonus (kind of defeats the purpose, doesn’t it?), but I don’t see any other way to do it. I originally thought I might be able to hijack one of the Banshees that harass you after you kill the AA Wraith at the beginning of the level, but not so. You can down them with a charged Plasma Pistol burst, but you cannot then board them—Bungie has evidently disabled this in order to prevent you doing what many of us did on H1’s Assault on the Control Room level: grabbing a Banshee before you were supposed to have it and bypassing/laying waste to much of the level.

We’ll first begin by bypassing the forces in and around the Wraith hill in the interest of time. Note: on Normal difficulty, it is also possible to clear these forces quickly enough that doing so and driving (rather than walking) to the tower is actually faster. The instructions I provide below work on any level of difficulty and should also work regardless of the skulls you have enabled.

1. When you exit the Pelican, head toward the right of the boulder in front of you, toward the waterfall. There will be 2-3 Grunts in front of you (at about 2 o’clock). Take them out and head for the waterfall.

2. Continue on up the pathway on the right edge of the map immediately adjacent to the waterfall. Hug the wall as you walk, being careful to situate yourself behind the trees, rocks, and other cover that Bungie’s level designers helpfully placed in order to protect you from attack by the baddies on your left.

3. This will bring you to the far right of the downed Pelican and the hill with the AA Wraith. Just keep walking, hugging the wall as you do. If you come across Grunts behind the downed craft that are shooting at you, quickly take them out and keep walking.

4. Continue hugging the wall and walking toward the entrance into the canyon forest that eventually leads to the first tower. Be sure to hug the wall until you are out of range of the Brute Chieftain on the Wraith hill. There is a spot during the walk before you round the corner that is out in the open a bit and exposes you to the Chieftan for a brief time. I sometimes bunny hop as I go through this to make it more challenging for him. I’ve not been shot yet. If you manage to get around the corner without being shot, you did fine. If you get killed by a Fuel Rod blast, you didn’t do so well.

5. You will have to walk to the first tower. This is probably the most tedious part of this approach, but it saves an enormous amount of time, especially on Heroric and Legendary difficulty.

6. As you walk, you’ll notice that the turret, bad guys, and Ghosts that are usually waiting for you here have not yet been deployed. Keep walking until you reach the clearing where you can look down on the Wraith immediately in front of the first tower. As you walk, you’ll hear the audio clip play re: the AA Wraith and related forces being “neutralized.” The game has killed them for you, and rather than waiting for the Warthog and Mongoose to be dropped back at the beach, you are almost to the tower when this happens.

I’m certain Bungie designed the map such that the “battle for the Wraith hill” could be bypassed in this way. That’s why the path on the right edge of the area exists in the first place and provides adequate cover to keep you alive. And that’s why the AA Wraith is eventually taken out, anyway, and the rest of the level plays normally once you reach the tower.

Note: if you cleared the AA Wraith hill yourself and drove to the tower (hopefully in a Warthog), you can jump immediately off the cliff in the Warthog and head straight for the tower entrance. Bypass the Wraith, the Prowlers, and the Ghosts, and ram the other occupants of the corridor. If you do this, skip steps 7-15.

7. As you approach the clearing, switch to your Battle Rifle and zoom it in on the area to the right of the Ghost near the ammo containers. You’ll see a Grunt about to board the Ghost. Normally, you come rushing in in a vehicle and can catch him before he takes off with the Ghost. Since you’re on foot, you’ll have to snipe him with the BR before he does so. Kill him before he takes the Ghost as you will need it very shortly.

8. After you kill the Grunt, go to the edge of the plateau overlooking the canyon where the Wraith hovers. Take it out with your Spartan Laser. If it’s facing away from you, one shot to its rear should do it; otherwise, you’ll likely need two shots.

9. Get in the Ghost and move it to be directly adjacent to the ammo container holding the Fuel Rod gun (but don’t get the Fuel Rod yet). This is very important. You’ll hear audio instructing you to take out the tower. This will trigger the three Ghosts in the canyon to come into firing range in the general vicinity of the downed Wraith. Normally, you probably encounter these guys near the waterfall or near where you currently are. But taking out the tank and boarding the empty Ghost triggers them to head into the canyon and wait to ambush you.

10. Using at most one shot each from your Spartan Laser, take out the three Ghosts. If you were able to take out the Wraith with a single shot, you can afford one miss; otherwise, you can’t. This is important because you’ll be rushing the tower in just a moment, and you don’t want the little dudes in the Ghosts firing on you as you do. Like a lot of things when playing for time, it’s important to be as quick as possible without missing.

11. Once the Ghosts are down, grab the Fuel Rod gun (drop the laser) and jump in the Ghost. Head down toward the waterfall and around the bend into the canyon. You are going to rush the tunnel leading into the tower in your Ghost. Why shouldn’t you just jump off the canyon wall and save time? Why go through the water at all? Because you need the checkpoint, and, more importantly, the subsequent exit of the tunnel by the Prowlers to occur, before you rush it. If you jump off the canyon wall and head straight for the tower, they don’t come out on their own and will happily shoot and ram you when you try to enter the tunnel.

12. As you approach the tunnel entrance, watch out for the turret up on the plateau to your left. I will sometimes go ahead and take this out with the laser before I board the Ghost if I have a round left over after dispatching the Wraith and the three Ghosts. If the turret is still active, just strafe horizontally a bit to the right of the tunnel entrance. What you’re waiting on here is for the two Prowlers to exit the tunnel and come after you. Don’t hover immediately in front of the tunnel, as that would allow them and the scads of others in the tunnel to shoot you. But get close enough so that when they come out, you can immediately boost in behind them.

13. Once the Prowlers come out after you, boost in behind them and ram the minions in the tunnel. Be careful because doing so often throws the lightweight Ghost off course. Ram into the turret in the tunnel and destroy it and splatter as many baddies as you can. Be careful not to let one of the Brutes hijack your Ghost. That will result in almost certain death and cost you time.

14. You will probably see your shields depleted from this little maneuver. If you do, boost over to the rocky cave-like structure immediately past the door so that you can recover before making a run for the door itself. This is another of those map features that was obviously put in for people who charge the door like this. I had never used it before playing this level for time. I generally commandeer the Wraith and destroy everything in the tunnel before even entering it.

15. Once you’ve healed, make a run for the door, picking up the Active Camo power-up and the two plasma grenades (as necessary) as you go. Shoot anyone you have to, but don’t worry about clearing the tunnel entrance. Your job is to get inside asap.

16. Once inside, kill the two or three baddies guarding the entrance to the first room you come to. Using your Battle Rifle, you should be able to dispatch them in no time.

17. Next approach the room entrance and enable Active Cammo, then run through the middle of the room to the door behind it. If possible, take out the Brute as you go. If he’s facing away from you, simply melee him. If he’s facing you, knock down his shield with your Fuel Rod gun, then shoot him once in the head with the BR to kill him. Regardless, keep moving as your invisibility will run out before you reach the door, and the Grunts and assorted baddies in the room won’t take kindly to your presence.

Note: alternatively, you can take the left corridor and jump from the stacked Covenant ammo cans up through the window and on to the second story of the building. This will let you immediately proceed to the elevator and bypass all the bad guys in between. However, keep in mind that this means that you’ll face these very same bad guys when exiting the tower. Jumping through the window is usually faster, but on Heroic and Legendary, it can then be difficult to get out of the tower alive, so the difference is usually a wash.

18. Run through the door and take out anyone immediately inside. Often, there are three Grunts immediately inside the door. Three headshots, and you should be done with them.

19. Run up the ramp and shoot the exploding canisters in the center of the room. This will kill most of the baddies in the room.

20. Continue on to the ramp on the far side (your left) of the room. Kill anyone you find there and enter the antechamber at the top of the ramp.

21. Repeating what you did in the earlier room, take out the Brute. Either Fuel Rod once followed by a BR to the head (a variation on the old plasma pistol, BR combo) or melee. The reason you want to take out these Brutes now is that you’ll have to face them when exiting the tower if you don’t.

22. As you run through the room and take out the Brute, continue on through to the elevator. Keep moving as the Grunts and assorted other baddies will be shooting at you. Engage the elevator, and, as you ascend, take out as many of the bad guys you’re leaving behind as you can. You’ll face the remaining ones when exiting the tower.

23. Up in the tower, the Fuel Rod gun is your friend. Unless you’ve enabled skulls, two shots on any guy in the room will take him out, but don’t waste your ammo. You’ll need a good 4-5 rounds for the next tower, so don’t waste them. You’ve got 15 rounds, so don’t burn more than 10 in this room. Remember that once a Brute’s shields are down, a single shot to the head with the BR will take him out. Also, don’t forget you have grenades.

24. You will immediately be charged by two Brutes. I like to stick them with spike grenades, myself (there are two in the right corner of the room on the floor), but use whatever works for you. Quickly take them out and proceed to the room proper.

25. Inside the room, begin by taking out the Chieftain if you can. A good Fuel Rod blast followed by a BR shot to the head shoot do it. You will have two other Brutes shooting Brute Shots at you while you move through the room, so be careful. Dispatch everyone in the room as quickly as possible, then hit the switch as the audio clip instructs.

26. Taking out the Chieftain early has the advantage of yielding an Invincible power-up pretty often. This is very nice to have when exiting the next tower, so pick it up if he drops it. I’ve noticed that you have a better chance of this if he has not yet enabled invincibility himself (glows all white). Also grab the Gravity Hammer (trade out your Battle Rifle—you won’t need it for awhile).

27. Engage the elevator and head back downstairs. The main thing to look out for as you exit the tower is being stuck by a grenade. If the Grunts massed near the door are near enough to each other to be worth it, jump in the middle of them and hit one with your Gravity Hammer to kill a bunch of them at once. Style points are always nice to have, even when playing for speed.

28. Outside, grab the Mongoose, the fastest vehicle in the game, and don’t wait for any passengers. Head back out to the beach as instructed. Drive full throttle the whole way and avoid obstacles. As you near the beach, you’ll see some of the bad guys deployed that weren’t there earlier when you walked to the tower. Just drive right past them—don’t stop for anything.

The next section involves getting from the beach to the third tower (you skip the second one because the Arbiter and friends took it down). The key here is to move as quickly as possible. The Hornet’s purpose here is not to help you rack up kills but rather to get you safely to the next tower as quickly as possible.

29. Grab the first Hornet you come to and don’t wait for anyone to join you. You’re not concerned about racking up kills with the Hornet—it’s just your transportation to the next tower.

30. Fly to the next tower without engaging the Banshees or Phantom as you go. If one of them gets directly in front of you to where your reticle turns red without requiring your to deviate in order to shoot them, go ahead and squeeze off a missile or two. There’s nothing wrong with scoring more points, but you don’t want it to cost you time or draw you into a protracted air battle just to stay alive.

31. Since I’m not varying my course horizontally much, I usually vary my altitude a bit in order to keep from being an easy target for an enterprising Banshee. This keeps me on course without burning a lot of time off the clock.

32. As you near the third tower, you’ll pass a mountainous, rocky outcropping to your right that stretches high into the sky. It has a notch missing from the top of it just large enough for your Hornet to pass through. Fly through this notch, then hug the wall as you head toward the tower. This is a natural shield that will protect you from the Banshees behind you. They will have to follow you to the tower in order to shoot at you, and most won’t.

33. At the tower, land the Hornet directly in front of the right tunnel entrance. Turn the Hornet sideways to your right before landing if possible—this will allow it to serve as a shield against the guys behind you. Don’t worry about taking out the Wraiths or anyone else, just get landed and quickly head down the tunnel.

34. Immediately in front of the door leading into the tunnel, you’ll encounter several Brutes guarding it. Quickly dispatch them with a combination of Fuel Rod blasts and grenades. Don’t take too long or the bad guys behind you will begin attacking. Dispatch the Brutes and get inside. Try to reserve a Fuel Rod round or two for use inside the tower.

35. Once inside the tower, head immediately to the corridor on the left (or right—doesn’t matter) of the room. The Drones will follow you and mass directly in front of you. Either run into the middle of them, then leap up and hit one of them with the Gravity Hammer or shoot a Fuel Rod blast right into the mass of them. Done correctly, most if not all of them will drop. A second Hammer swing or Fuel Rod blast will take out the remaining mass if necessary.

36. Hurry to the door immediately behind the room. If the Hunters are blocking you, retreat back into the corridor, enter the center room through one of the windows between the corridor and the room, and, again, head for the door. Use this tactic (being a weenie) rather than directly engaging the Hunters. You don’t have time to die, and you don’t have time to waste taking them out in the usual way.

37. Once you’re through the door, you’ll see a pack of Brutes enter the foyer in front of you from the far (to your right) side. Use a combination of grenades and Fuel Rod blasts to kill most of them, then jump through the opening beyond the ramp in the center of the foyer that allows you to bypass the corridor and antechamber where they’ve massed.

38. Avoid engaging them and head straight for the elevator.

39. Your two problems at this point are the remaining Brutes behind you and the Drones you will encounter on the elevator itself. Dispatch the Drones by leaping into the mass of them above your head and hitting one with the Gravity Hammer. This will kill most of them. Repeat as necessary.

40. If you didn’t get the Invincibility power-up in the first tower but have another one, now would be a good time to use it. If you have the Invincibility power-up, save it for when you exit this tower. If you have a Bubble Shield or Portable Cover, deploy it to avoid the Brutes and their Maulers. If you’ve got a Regenerator, you might deploy it here to keep the Brutes (and possibly Hunters if they followed you) from killing you as easily. Also, remember that you can take cover behind the elevator mechanism itself. But the main thing to do here in order to stay alive is to engage the elevator as quickly as possible. If you do, you’ll soon be out of range of the guys in the base of the tower—at least for now.

41. Once you’ve ascended to the top of the tower, you’ll encounter four invisible Brutes as well as a Chieftain manning a Plasma Cannon. If you have ammo in your Fuel Rod, take out as many as possible with it. When you run out, trade it out for the Needler in the corner ammo container. A combination of grenades and Needler rounds should dispatch the invisible guys very quickly. Try to take them out such that at least some of the incendiary grenades they each drop are not detonated by the Needler explosions you’re ripping them apart with. These come in handy for taking out the Chieftain manning the Plasma Cannon and the Flood you’ll encounter when you exit the room.

42. Take out the Chieftain and hit the switch as instructed by the audio clip. I’ve found that a single incendiary grenade can take him out on Normal difficulty. On Heroic and Legendary, you’ll need to do more.

43. Once you flip the switch, the Flood descends on you. Head through the center of the room back to the elevator. Don’t waste time killing any of them unless you have to. Jump over/run under them as necessary until you reach the elevator and can engage it. If you must take them out, use your Gravity Hammer as it will inflict the most damage in as little time as possible.

44. Once the elevator has finished descending into the base of the tower, you will encounter the Flood and anyone you left behind before you ascended. It’s possible that the Hunters will be there waiting for you, and it’s also possible that a stray Drone or two will be there. These leftover enemies will likely be behind you, happily shooting you in the back as you exit the elevator. The Flood will be in front of you, between you and the door, and also to the side and behind you.

45. Run for the door, taking out any Flood you come across as necessary.

46. As you approach the door, pause for a split second to let it open. Often, there will be a phalanx of Brutes immediately in the threshold as it opens. You don’t want to run into them and let them immediately begin beating on you and stealing your health. Pause for a split sec as you approach the door (it will open on its own), then rush them (as necessary) and take them all out with your Gravity Hammer.

47. Next, head up the right side corridor toward the green Phantom firing on the bad guys. Remember that Invincibility power-up you (hopefully) grabbed earlier? Engage it here if necessary to get out of the tower alive. You’ll likely encounter both Flood and Brutes as you go. Kill as many as you have to on your way out of the tower. If possible, trade out your Hammer for a Battle Rifle on the floor of the right corridor as you run.

48. As you head toward the green Phantom, you’ll happen upon a small ramp leading out of the tower structure down toward left. There you’ll see a Gauss Warthog. Jump in it. If you successfully grabbed a Battle Rifle and you’re playing on Normal (or Easy), take off. If not, you’ll need to do a couple of other things before leaving. Jumping in the Warthog will trigger the marines up on the tower base to follow you. Soon, you’ll be joined by a passenger and a gunner. After they get in, get out of the Warthog and run back toward the ramp, into the mass of marines coming toward you. You are looking for one with a Battle Rifle. There will be at least one. Trade your Needler for his BR—you’ll need it in the last section of the level. Get back in the Warthog and head out. Do this quickly as you don’t want an NPC taking your hog. (And don’t grab the Scorpion—it’s too slow—remember: you’re playing for time).

This next section involves navigating around a snowy mountain to a valley outside the scene of the final battle in the level—the citadel—where you must destroy two Scarabs. The key here is to move as quickly as possible and don’t let any of the other vehicles ram you or flip you over.

49. Rush through the snowy level toward the area where you take on the Scarabs. There are no shortcuts—just blaze through there and let your gunner take out the bad guys you encounter. In the YouTube video above, you’ll see that I actually get killed by my gunner because the idiot shoots a Ghost that has just rammed us en route. The Ghost is immediately in front of us, and when my gunner Gausses him, it blows up the Warthog, killing me. I wasn’t happy, but was so far ahead on time, I was pretty sure I could still hit the bonus.

50. When your reach the Scarab valley, jump in a Hornet and fly straight to the back of one of the Scarabs (I usually take the one on the left, but it doesn’t really matter which). Ignore everything else as much as possible and simply gun for the light above the Scarab’s tail section until you’ve blown off its back cover, then shoot the exposed reactor core until you destroy the Scarab. Don’t wait for it to actually detonate—head for the other Scarab immediately and repeat the procedure. Once you’ve taken out the second scarab, head for the top of the ramp leading into the citadel and land the Hornet.

51. You’ll soon be joined by the Arbiter and 343 Guilty Spark. When the Arbiter starts to tell Spark to enable the bridge, immediately fly the Hornet to the door leading into the citadel and wait for the bridge to materialize beneath you. When the bridge materializes, get out of the Hornet and go inside. This is faster than walking over the bridge and will save you some time.

This last section is where you face the Covenant one last time, then fight the Flood to get out of the citadel. You go through three sections of the citadel, then backtrack through them to finish the level. The key here is to move as quickly as possible—time will likely be of the essence—and avoid getting killed. As with most of Halo, strategically wielding the right weapons at the right time can make the difference between success and death.

52. Continue walking toward the inner door inside the citadel. Eventually, the Arbiter will be teleported forward to join you. You’ll then have to sit through a Cortana clip (God, I hate these), followed by a cut scene. Skip through the cut scene and get ready to battle the Covenant one last time.

53. Inside the next room, on either side of the end of the causeway, two Grunts will be positioned with Fuel Rod guns. They will continue to rain down terror on you and your Flood allies until you take them out, so start by doing that. That’s why you needed to trade for the Battle Rifle earlier. You should be able to easily snipe them both, then proceed to take out the remaining Covenant forces as your proceed down the causeway. You’ll encounter a Brute at the end of the room, near the door. Stick him with a grenade or nail him with your Gravity Hammer and proceed to the area near the door where the two Fuel Rod Grunts were. Pick up the first Fuel Rod gun, quickly reload it, then run over the other one to pick up as much ammo as possible. If you encountered and killed a third Fuel Rod Grunt (sometimes they come through the door as you approach it), obviously you’ll want to grab that ammo, too. Proceed through the door to the next room.

54. You’ll run into Brutes with jetpacks in the next room. Take them out with a combination of grenades and Fuel Rod blasts and proceed on through the room. For a couple of reasons, it’s helpful to hang back a bit at first and let the Flood take the brunt of the Brutes’ onslaught. One, they like to fly over and get behind you, effectively flanking you, and attack you from front and back simultaneously. Two, many of them are carrying Covenant Carbines, and these can drain your shields/health quickly. Hang back, let them attack the Flood, then take them out with Fuel Rod and grenade blasts.

55. You’ll encounter a Chieftain with a Gravity Hammer of his own near the door. Take him out, preferably before he engages invincibility himself and preferably via a headshot. If he drops an Invincibility power-up, pick it up.

56. Proceed on through the next room, hit the switch on the bridge at the end, then turn around and backtrack through the citadel to battle your former Flood allies.

57. Jump over as many of these guys as you can, engaging as few of them as possible. You will likely run out of Fuel Rod ammo during this run, so trade it out for a Needler as you go and use that as necessary as you backtrack.

58. You’ll have no choice but to engage the Flood at the doors between the rooms, as they like to mass there. Use your Gravity Hammer to clear the doors and let you proceed through. I also like to toss grenades up on the doors themselves as I approach in order to provide a nice surprise to the Flood baddies who like to pour out when I get there.

59. Engage your Invincibility power-up if you have it and if your health is getting perilously low as you head for the end of the level. You’ll be getting shot in the back a lot, so this will probably be necessary. Wait as long as possible before engaging it, though, as it may run out before you reach the end of the level, and, take it from someone who knows, it’s a real drag to get killed this close to getting the bonus.

60. Proceed on back through the level until you reach the chute where you jump down to finish the level. If all goes well, you should beat the 20-minute mark.

Last Week

khen1234 — Mon, 17 Dec 2007 21:21:00 GMT

I was in Redmond last week on business and had a lot of fun this time around. The trip began and ended with a visit to Key Arena in downtown Seattle. Monday night it was the Van Halen concert. I’d already seen Van Halen in Kansas City back in October, but I thought, “What the heck, it was a great show, let’s go see it again.” I had floor seats this time and could easily see everything that was happening on stage just a short distance away. The person I took with me said they’d never been to a better concert, and I came away thinking it was even better than the Kansas City show.

Seattle set records for rainfall Monday night, but I was thankfully out of most of it. My guest and I had a nice dinner prior to the show at some restaurant near the venue, and we met up with some other friends after the show as well. I saw some light rain, but nothing like the torrents the news was reporting.

During the week, Kalen Delaney came by to see me at the office. You may have heard: she has left Solid Quality Learning and is back doing her own thing now. Contact her here for details. Kalen is one of those people I always love to meet up with and feel fortunate to know. She’s got such an enthusiasm for technology and is such a great communicator—it’s always a joy to sit down and shoot the breeze with her. We talked about all kinds of things during her visit and even reminisced a little about the old days.

I went out with friends every evening throughout the week. I often don’t do that—I’m usually too busy—but I made it a point to get out a little bit this time. It’s Christmastime, and I felt like socializing, I guess J.

Friday night, I ended up taking in a Seattle Supersonics NBA game back at Key Arena courtesy of some free tickets scored by my boss (his daughter is a dancer for the team). They played the Milwaukee Bucks, the teams were evenly matched, and it was a great game. I got to meet my boss’s daughter and the rest of the dancers during the game and had a great time in general. The person I went to the game with hadn’t seen many basketball games, but said they thoroughly enjoyed this one.

Saturday morning came early as my flight lifted off at 6:30 a.m. My driver showed up on time, but didn’t call my hotel room like he usually does, so I finally came downstairs and found him waiting. Fortunately, there wasn’t much traffic on the way to SEATAC so early in the morning, and I was on my way back home in no time.

I was home by 1 or 2 in the afternoon and surrounded by the usual sounds of kids, pets, and the occasional Tarzan yell. It was a quick week, but a productive one. I hadn’t been in awhile, and I really enjoyed myself this time around.

SQL Nexus is out!

khen1234 — Wed, 12 Dec 2007 19:40:00 GMT

In case you missed it in my last post, SQL Nexus, the SQL Server performance analysis platform built by Bart and me, has been released via CodePlex. Here's a post post where I talked about how Nexus works. And here's the link to its page on the CodePlex site.

While I'm at it, another tool you might find useful is SQLDiag. It ships as part of SQL Server 2005 and later. Read about it here and here.

For a high-speed, lean, command-line analysis tool for Profiler traces, check out the Retrace tool we released into the wild yesterday here.

A SQL Profiler trace Swiss Army Knife

khen1234 — Wed, 12 Dec 2007 10:25:00 GMT

Have you ever needed to find your most expensive queries and quickly grew weary of writing T-SQL against trace tables to try to ferret them out? Have you ever had to wade through gigabytes of trace data just to find one ill-behaving query? Have you ever struggled to decide what performance metrics really matter when analyzing Profiler traces: duration, reads, writes, etc? Today’s post is about a new tool Bart and I wrote to help you do all this and more.

Origins

The original concept behind Retrace was to create a simple tool to load a SQL Server Profiler trace file into a database ala the Relog tool that ships with Windows (and provides similar functionality for Perfmon logs). Retrace would “re-trace” a session of SQL Server activity as a server-side trace table. I often needed to query trace files using T-SQL and didn’t like resorting to fn_trace_gettable every time, so Retrace was born to automate that process for me. That was the original idea.

Of course, these kinds of things tend to take on a life of their own, and Retrace soon began to demand that I enhance it in all sorts of interesting ways. Around this same time, my friends in CSS got underway with getting the SQL Nexus performance analysis platform that Bart and I built released via CodePlex. (Great news: it’s out there now. More on that later.) One of the cooler features we built for Nexus was a facility we called “TraceBuster.” Nexus supports this notion of generic diagnostic data loaders that implement a common interface. Nexus interacts with these assemblies via this generic interface, and users can build their own loader assemblies so long as they adhere to the interface we expose. TraceBuster is a Nexus loader we built to process SQL Profiler trace files and load them into a data warehouse. It automatically parameterizes the queries it reads from each trace such that the performance of a query executed with different parameter sets can be tracked over time. Unlike SQL Profiler itself, TraceBuster is able to identify and aggregate performance statistics such as CPU use, duration, reads, writes, etc., for these query “templates” rather than treating multiple instances of the same query with different parameters as different queries. Lightweight, fast, and extremely simple from a coding standpoint, it does exactly what we wanted it to do and no more.

For a number of reasons, it was decided that TraceBuster would not be included in the CodePlex release of Nexus (though you can still get it here in binary form). That got Bart and I to thinking. How cool would it be if Retrace used TraceBuster and basically provided a nice command line tool to load SQL Profiler traces into a data warehouse? What if Retrace was merely a thin console app shell over the TraceBuster assembly originally designed for Nexus? Since Nexus interacted with TraceBuster and its other diagnostic loaders generically, there was no reason a small console app couldn’t do the same thing without requiring Nexus itself to be on the box. Rather than merely load the trace into a trace table on the server, what if we let TraceBuster do all of its cool aggregation and analysis for us, thus saving all kinds of work manually querying the trace files? And what if on top of that we used the client-side Reporting Services reports that Bart developed for TraceBuster’s data warehouse and had Retrace display them? Then we’d have a single tool that could blaze through our trace data, load it into a data warehouse for further perusal, and finally display useful reports over that data. How cool would that be?!

So, we spent a couple late nights and a weekend here or there and morphed Retrace into a “Swiss Army Knife” for Profiler traces. One of the things we found ourselves often doing in our tuning work within SQL Server development was running a SQLDiag collection immediately before doing something we wanted to measure on the server, then loading the Profiler traces it collected via Retrace. Given Retrace’s one-stop-shopping nature, we thought it made sense to build that functionality into Retrace, so we added a command line option to Retrace to allow it to call SQLDiag automatically before a load operation.

When it was all said and done, Retrace knew how to collect Profiler traces, to load those traces into a performance warehouse on the server, and to run reports over that warehouse. We had a simple little console app that automated many of the tasks we found ourselves doing over and over as we tuned this or that component in our day jobs.

Summary

Retrace knows how to do three basic things:

Collect a SQL Profiler trace (using the SQLDiag utility that ships with SQL Server 2005 and later)
Load Profiler traces into a data warehouse that resides in a SQL Server database. These can be traces it collects via SQLDiag or traces from other sources (e.g., SQL Profiler)
Show reports listing the top N most expensive queries from the trace files loaded into the warehouse

Examples

To load a trace file into a data warehouse on the default SQL Server instance using Windows authentication:

retrace -i mytrace.trc

To load a trace file into a data warehouse and display the GUI (Reporting Services-based) reports:

retrace -i mytrace.trc -r

To load a trace file into a data warehouse and display the top N queries report in HTML format:

retrace -i mytrace.trc -f html -r

To start a Profiler trace collection, then load the collected trace file(s) into a data warehouse:

retrace -c TRACE

To start a Profiler trace collection, load the collected trace file(s) into a data warehouse, and then display the GUI reports:

retrace -c TRACE -r

To load all trace files in a given folder into a data warehouse and display the GUI reports:

retrace -i myfolder\*.trc -r

To load a trace file into a data warehouse and write the top N queries report to an XML file:

retrace -i mytrace.trc -f xml_full -o myreport.xml

To display the GUI reports (following a previous load of trace data into the warehouse):

retrace -r

To load a trace file into a database other than [retrace] or to a non-default SQL Server instance:

retrace -d mydatabase -S myserver\myinstance -i mytrace.trc

As you can see, you can load one or more trace files from the command line. These can be ones you instructed the tool to collect via SQLDiag or ones you’ve collected through other means (e.g., SQL Profiler). You can then display reports showing the top N most expensive queries in a variety of formats (HTML, XML, PDF, Excel, JPEG, etc.) The default report format is a simple GUI report that makes use of client-side Reporting Services report files (you don’t need to have Reporting Services itself installed).

Customizing Retrace

You can customize Retrace’s output in a variety of ways. You can customize the RDLC files that make up the reports shown in the Retrace GUI if you like. And you can also modify the XML stylesheet (XSL file) that Retrace uses to translate its XML output into HTML when HTML output format is selected. This stylesheet is named retrace.xsl and resides in the XML subfolder. Here’s what the HTML output looks like by default:

You can customize the T-SQL script Retrace uses to create its data warehouse and reporting stored procedures by modifying the TraceAnalysis.sql script file in the Scripts subfolder. Be careful here, however, as modifying the objects in this script may prevent Retrace from being able to display meaningful analysis reports.

You can also customize the Retrace application itself. Retrace is a ScriptIt application. For those who don’t know what ScriptIt is, it’s a technology developed internally within Microsoft that allows you much of the flexibility you have with scripting tools such as Perl and VBScript with managed code applications. It combines the ease of deployment and quick modification you see in traditional scripting tools with being able to develop in your favorite managed code language. It takes advantage of the fact that every machine that has the .NET Framework 2.0 or later installed also has the full MSBuild environment (the toolset used by Visual Studio to compile and build application binaries). Using ScriptIt, you can modify the source of a managed code application, rebuild it, and redeploy using only the executable—you don’t need to store the source in separate files or keep track of solutions and projects in Visual Studio. You can edit your apps in Notepad if you like, then rebuild them on the spot without need of any other tools. The current plan is for ScriptIt to be released to the public next year, and you can contact Vance Morrison, its author, if you have any questions about it. Once ScriptIt is released, you’ll be able to edit Retrace’s source code and make whatever changes you need without resorting to full-blown application development.

SQL Nexus

I mentioned earlier that SQL Nexus has been released on CodePlex. This means that you can download the current binaries and source code today and extend/use Nexus in your own work. Read the license agreement over there for details. We will continue to keep a binary version of TraceBuster available here in case you’d like to use it with Nexus. If you get Nexus, you have a much more graphical, full-featured performance analysis tool than Retrace was intended to be. That said, if you like simple, fast command-line tools, you may find Retrace useful. I wrote a good chunk of both tools, and I think each has its place.

Conclusion

They say necessity is the mother of invention, and I guess necessity is just as responsible for Retrace as are Bart and I. Retrace was a tool we needed to do our jobs. It evolved into its current form through the necessity of our both needing a Swiss Army Knife-type Profiler trace tool and our belief that TraceBuster provided useful functionality that we should continue to leverage in our work on the product. Download it, use it to carve up a few of your own perf problems, and let us know what you think.

Random thoughts on my trip to the conference

khen1234 — Sun, 23 Sep 2007 19:38:00 GMT

I had a great time on my brief stint in Denver for the PASS Summit. We had a good session and a good crowd, and Kevin Kline did a marvelous job of running the whole thing. He had lots of great questions for me during the interview, and the crowd had many good ones, too. I hope those of you who attended felt it was worth your time.

One thing that came up during the talk was the status of SQL Nexus and what its future is. As I said when asked about it:

You can currently download the latest version of SQL Nexus from www.sqlnexus.net. For those who don’t know, SQL Nexus is a SQL Server performance analysis tool written by Bart Duncan and me. It loads the diagnostics collected by SQL Server’s SQLDiag facility (and similar tools) into a data warehouse (see the Books Online if you aren’t familiar with SQLDiag), then provides graphs and charts over that data. You can use it to find your most expensive queries, troubleshoot performance bottlenecks, investigate blocking, etc. It was originally released on the CD accompanying the book, SQL Server Practical Troubleshooting: The Database Engine.

Nexus is something Bart and I dreamed up nearly seven years ago before Microsoft had released any tools of this kind for SQL Server. Its influence is evident in similar tools that have appeared since then. For example, many of you are familiar with the ReadTrace tool. When Bart and I first proposed automated performance analysis to our management within Microsoft, ReadTrace had not yet been released to the public and was merely a trace file splitter used by SQL Server Support—it split Profiler trace files by SPID and wrote them out to separate files. This facilitated more easily seeing which users were running which queries rather than having to filter on the SPID column in Profiler. It did not do any analysis of the trace files it read back then—it was intended merely to simplify the job of seeing which users did what and play that work back as necessary. Some time after we’d made our proposal regarding automated performance analysis, some of these ideas found their way into the next release of ReadTrace. If you’ve ever wondered about the incongruity between the tool’s name and its analysis feature (which is probably its most useful feature from an end-user perspective), that’s why it’s there—ReadTrace didn’t originally do any analysis. Similarly, the Data Collector and Management Data Warehouse features in Katmai were influenced by our work with PSSDiag/SQLDiag and SQL Nexus. (Bart and I have been involved with the DC/MDW project from the start, and Bart recently joined the team fulltime.) We also know that various third party tools have built on the concepts we originally championed. From that one seed, lots of goodness has grown, and we couldn’t be happier about it.
Bart and I have given Microsoft’s Customer Support Services organization permission to release the source code to SQL Nexus via Microsoft’s CodePlex site. Current word is that this should be out internally by Thanksgiving (I don’t know when it will be available publicly on CodePlex, but I would expect it to shortly follow). CodePlex, for those of you who don’t know, is Microsoft’s shared source initiative, not unlike SourceForge and similar sites. Releasing the Nexus source will allow those of you in the user community who are so inclined to help develop and evolve Nexus further.

A concern we in SQL Server development had about releasing the Nexus source was that we might want to use parts of it in future products. To that end, we’ve asked CSS not to release the source code to a handful of Nexus components. One of these components is Nexus’ TraceBuster facility. TraceBuster, you may recall, is the Profiler trace loader in Nexus. It performs a similar function to the ReadTrace utility some of you may be familiar with, but differs in certain ways:

Like the rest of Nexus, TraceBuster is written in managed code. This means it integrates more seamlessly and more efficiently with the rest of the architecture.

Nexus offers a pluggable architecture wherein you can add new loaders to import diagnostic data into the data warehouse by merely editing an XML file or, for more involved work, by creating a simple managed code assembly and dropping it into a folder. You can easily implement new reports by designing them with the Reporting Services designer (in Visual Studio, for example) and dropping them into Nexus’ custom reports folder. TraceBuster reads Profiler trace files, normalizes the query text it finds in them by removing literals, and loads all this into the data warehouse. It also includes several reports for analyzing the most expensive queries in a workload, drilling into those queries, gauging their cost using a variety of metrics, etc.

While we also built a Nexus loader for ReadTrace, ReadTrace is not managed code and is a console app rather than an assembly, so the integration isn’t quite as efficient or seamless. The Nexus loader for ReadTrace must shell to a separate application rather than running it in the Nexus process itself as happens with TraceBuster.
TraceBuster does not require the code it analyzes to be fully parseable. This means that it’s more tolerant of the broken T-SQL sometimes seen in Profiler traces. Because of the vagaries of networks and SQL Server’s trace architecture, bad T-SQL in Profiler traces is more common than you might think. Missing quotes, truncated statements, missing pair members—these are not that uncommon in trace files. Utilities that analyze the T-SQL in trace files must be more tolerant of anomalies in it than, say, the database engine would be. TraceBuster didn’t begin as a full T-SQL parser that was then special-cased to handle all the eccentricities of traced T-SQL. It began as a no-frills “literal identifier” that merely replaces literals in query text with generic parameter markers so that you can easily compare instances of the same query with different filter values. It does not know and does not care whether the code it examines is perfectly intact. It does no more and no less than exactly what it needs to in order to provide meaningful query cost analysis. This makes it an order of magnitude simpler and more robust than full parser-based solutions.
TraceBuster uses SQL Server’s own facilities for reading Profiler trace files. It defaults to using SQL Server’s fn_trace_gettable function if the trace files you provide it are on the server machine (or accessible by it) and falls back on the SMO facility for reading trace files if fn_trace_gettable can’t get to them. It never directly accesses the trace files themselves. This means that TraceBuster is immune to file format changes across releases. It does not need (or want) intimate knowledge of the Profiler trace file format. The file format changed between SQL Server 2000 and SQL Server 2005, but the same version of TraceBuster can read trace files written by either release because it uses SQL Server’s built-in facilities for doing so.

Given all this and the fact that TraceBuster won’t be included in CSS’s CodePlex release of Nexus, you might be wondering whether you need TraceBuster and how you can get it after CSS releases the new version. You’ll have to decide for yourself whether you need TraceBuster. Tentatively, CSS plans to release a new version of ReadTrace that can read SQL Server 2005 Profiler trace files around the same time Nexus is released via CodePlex. ReadTrace and TraceBuster provide much of the same functionality, so it is unlikely that you would want to use both of them. And though they are similar, there are important differences between them, as well.

If you decide that you want to continue to use TraceBuster once it has been removed from Nexus, you can get the binaries (no source) from www.sqlnexus.net, the same site that will host Nexus itself until CSS releases it via CodePlex. The specific link is http://www.sqlnexus.net/download/tracebuster/TraceBuster.zip. If that link doesn't work, try this one: http://khenzden.googlepages.com/TraceBuster.zip.

I know all of this may be a bit confusing, so let me net it out:

The current version of SQL Nexus (including the TraceBuster component) is available today, without source code, via www.sqlnexus.net. It provides a rich performance analysis suite for SQL Server, including the type of Profiler trace analysis you may have done with ReadTrace in the past. It is functionally on par with the source-code version of Nexus that CSS plans to release via CodePlex. If you don’t plan to make use of the Nexus source code, there’s no reason to wait for the CodePlex release. You can get what you need today.
The next version, due to be released internally by CSS around Thanksgiving and sometime thereafter as source code via CodePlex, won’t include TraceBuster because CodePlex releases are necessarily source code-oriented, and the SQL Server development team has reserved the right to include pieces of TraceBuster in future products. It will, however, include support for ReadTrace (which should be released by CSS concurrently), a tool that performs much the same function, but that is not as well integrated with Nexus.
If you still want to use TraceBuster after the new version of Nexus has been released, it will be available here or here in binary form (no source), just as it is today. Once you’ve installed the CodePlex Nexus release, all you’ll need to do is copy the TraceBuster assembly and the included reports to the appropriate folders. There’s a readme file in the zip that provides specifics. When you then restart Nexus, you’ll see TraceBuster and its reports available in the GUI.

So, anyway, I had a good, if brief, PASS trip, and enjoyed meeting those of you who stopped by. Thanks for all the great questions during the interview, and I hope to run into you again in the future.

A funny thing that happened at the Avis counter at the airport:

Guy at the counter: Sorry, sir, the economy car you reserved isn’t available. We’ll have to upgrade you free of charge to a midsize. Is that okay?

Me: Okay, that’s fine. By the way, where’s the men’s room?

Guy: <funny look on his face> Is there something I could help you with?

Me: Uh, no. Just need the men’s room, thanks.

Guy: <a little peeved> Okay, just a moment. <then takes my paperwork and disappears through door #3 behind the counter>

Me: <a bemused look on my face, I’m wondering what the heck is up. I scan the place looking for hidden cameras>

It then occurs to me that the guy believes I’m not happy about the car switch and thinks I’ve requested a manager rather than a men’s room. Whew! I finally understand why he wanted to help :-)

I then tell the six women working the counter what the situation is. We’re all laughing hysterically by the time the guy returns with the manager. He is nonplussed.

Oh yeah, and the car I ended up with was the new, improved, quite sporty Dodge Charger of Dukes of Hazzard fame. Me likey—this one is a worthy heir to its muscle car forebears. The one thought that kept going through my mind as I zipped around Denver in the Charger was: I gotta get me one of these.

I'll be at the PASS Summit in Denver

khen1234 — Tue, 18 Sep 2007 21:34:00 GMT

I'm being interviewed on stage Friday afternoon from 2:30-3:45 by PASS president Kevin Kline (current session title is "SQL Server Q&A," but I think they'll be changing it). Topics we'll cover will include my background, where SQL Server is headed, what I've been working on, the status of the second editions of my Guru's Guide books, etc. As always, I'll have some new goodies that attendees can download after the talk.

I'd love to meet you, so stop by if you get a chance. If you want a book signed, feel free to bring it with you.

Today is a great day

khen1234 — Thu, 14 Jun 2007 02:55:00 GMT

...and in honor of it, I'm going to be especially cryptic and give you a little word puzzle--an anagram:

Catch Meek

The solution I have in mind for this one pretty much sums up how I feel about today. It's a great day :-)

A hint for those watching at home: 4nodcustody

Celko on Celko

khen1234 — Mon, 16 Apr 2007 03:28:00 GMT

I'm starting a little series here featuring interviews with industry luminaries. I figured people might like to hear what they have to say. My first subject is Joe Celko.

Q1: Tell us a little about yourself, in case somebody does not know you.

A1: I was a member of the ANSI X3H2 Database Standards Committee from 1987 to 1997 and helped write the ANSI/ISO SQL-89 and SQL-92 standards. I had been writing about Software Engineering in the trade press for years, so I just switched over to writing about the SQL standards as they emerged.

Two decades later, I have six books -- #7 is in the works now -- and around 800 columns in the computer trade and academic press, mostly dealing with data and databases.

Q2: What trends do you see in the Database Field?

A2: The obvious one is that databases are getting bigger. Just a few years ago, nobody used the prefix "Peta-" on anything in IT. A Terabit was huge! Today, you are just starting to hear about Exabytes (EB) and worse!

Why are databases getting bigger? It's not just more data; it is also more data sources. I have been teaching an intro RDBMS Design course on line for MySQL AB for the past few months. One of the Power Point slides shows some of the storage engines they can sit on top of. There are well over a dozen options from tiny embedded databases to federated systems.

Another cause is that more and more programmers are working with a database. There was a survey in the trade press this month that said about 60% of all developers are using a database today; it used to be about 40% in their last survey.

Q3: So how can we deal with all this extra data?

A3: The first thing is to get metadata in place. Without it, you cannot control all of the data sources pouring into your enterprise. In the old days, we did not have Google so finding industry standards was a lot of leg work and mountains of paper. Today, you can look up anything in less than a day. When I am teaching a college RDBMS class, I hand out 3x5 cards with the something that you might want to model in a database with a standardized scale of some kind -- shoe sizes, tires, whatever. Nobody has ever failed and they usually come back with several scales or standards.

Q4: What is the big news in hardware?

A4: Parallelism. We are looking at multicore processors being the normal state of affairs in the next decade. Intel has already announced 80 processors on a single chip. But the problem is that we do not have the software for that architecture. This month's DR. DOBBS JOURNAL has an article about writing a graph searching algorithm for the IBM Cell processor. The code is much larger than the traditional "mono-processor" solution but it runs orders of magnitude faster. The authors had to handle all of the processor and register assignments by hand. You need to have a compiler that will do that for you.

SQL is the only major language that has a history of parallelism because it is based on sets. You can partition a set, do quite a few things with each partition, then union the results back together to get an answer for the whole set. This is the basis for several VLDB products, such as Teradata, Kognitio and SAND engines.

Remember "Maxwell's Demon" from physics? It is an imaginary creature that only permits faster moving molecules to go past it and thus can keep things hot on its side of the room. Well, we can have "Celko's Demon" -- a processor devoted to a single row or column or partition in a table. When a query comes into the engine, each demon decides if he needs to ignore it or to pull up his data and pass it over to a higher level demon.

Q5: What about end users?

A5: There are more of them, too. Do you use a phone book to look up a telephone number or do you Google it? How many electronic newspaper and magazine editions do read every day? How do you shop now?

Your preferences and history are part of any serious commercial shopping site. Every one of us has gotten an email from Amazon.com telling us that people who bought a copy of some book also liked another book. It is the same with NetFlixs and every on-line company.

This kind of volume will have another hardware effect. Hard drives are going to die and be replaced by solid state devices. Right now I have a 512 Kbytes memory stick I use to hold my PowerPoint slides. The stick was a give-away at a trade show, not a personal purchase.

If I have tens or hundreds of thousands of end users trying to get information from my database, at pretty much the same time, then physically moving a read/write head across magnetic storage media is not going to work. I have to get access at the speed of electricity or light, not at electric motor speeds. If I cannot get a fast enough response, then the customer is not going to stay around and wait. In the old days, when we were using telephones with modems to process credit cards, the end users would hang up and re-dial if they did not get some signal in 3 seconds. One solution was to have red light or tone that went on in 1-2 seconds; it had nothing to do with the connection, but it made the end user feel good.

We are already hitting the limits today; that is why sites are mirrored. But at some point, you cannot keep throwing hardware at the problem. You need to throw a different kind of hardware at it. Solid state devices also have another advantage over hard drives; no moving parts to break. We have gotten very good at making reliable chips. Right now, solid state disk replacements are relatively small and expensive. Moore's Law will take care of the size problem and the need to have a fast response time will make any extra cost worth it.

Q6: You’re sometimes perceived as a bit of an ANSI SQL zealot and as being a bit acerbic toward people on newsgroups who don’t agree with you. Do you think this is a fair perception? Would you change it if you could? What do you have to say to your critics about it?

A6: It a fair perception and it is deliberate. My wife is a Soto Zen Monk who could beat you with a stick, so I am a pussy cat. I do it to get the attention of the poster.

Look at most of the posters in the newsgroups. They do not post DDL -- often because they have no idea what it is -- and invent their own pseudo-code. They have no idea what a spec is, but post requests for ways to implement an approach they have locked into -- and it is usually a non-relational approach at that.

What we have are some really lazy programmers who want to use the newsgroups to do their job or homework for them. Even worse, they want to get an instant college education which is not possible in a short reply. The questions they ask can most often be answered by (1) RTFM, well BOL (2) "Try it and see" (3) a quick Google search in the newsgroup to which they are posting.

What they get instead from most replies is a kludge to get rid of them. If this was a woodworking newsgroup and someone posted "What is the best kind of rocks to pound screws into fine furniture?" are you really helping them when you say "Granite! Use big hunks of granite!" I am the guy who replies with "Your question is bad. Don't you know about screwdrivers?" And I like to remind them that it takes six years to become a Journeyman Union Carpenter in New York State. Not Master, Journeyman.

I began my programming career in the Cold War, doing defense and medical systems. These are fields where bad programming kills the wrong people and you do not get a do-over. I did research on Software Engineering and wrote columns in the trade press on it for years before I became "the SQL Guy" for a living.

I am amazed at the amount of "cowboy coding" I see today. I have even had one guy email me about my "ANSI Standards hang up" telling me that I was just like those Structured Programming bastards that got rid of the GOTO statement . He was arguing for the return of spaghetti code!

I just made a few bucks inspecting a system that violated the principle of a tiered architecture. Their consultant decided it was faster to format some of the data in back end instead of the front end (leading zeros and UK versus US dates). The results were a very inconsistent database since his stored procedures might not quite match anyone else's, but he might have saved 15 minutes a year in computing time. Wow.

Does rebuilding a clustered index rebuild nonclustered indexes?

khen1234 — Tue, 27 Feb 2007 21:10:00 GMT

This came up yesterday with one of our support engineers, so I thought I’d post a brief entry on it. On SQL Server 2005, using ALTER INDEX…REBUILD to rebuild a clustered index does not rebuild its nonclustered indexes by default. If you think about it, it makes perfect sense—the clustered keys, which are the record locators in the NC indexes, aren’t changed merely because you rebuild the index. All rebuilding the index does is lay the pages back down. The rows may move around on the clustered index pages, but we don’t care because the NC indexes don’t use a RID to reference the data; they use the clustered key and have since SQL Server 7.0. Naturally, if you completely drop the clustered index, we have to rebuild the NC indexes because their record locators have to be changed to RIDs. And if you then recreate the clustered index, we have to reverse the process. So, being able to rebuild the clustered index in place is a real performance boon.

Not only can ALTER INDEX rebuild a clustered index in place, this is the default behavior. You have to specify ALL for the index name in order to rebuild all indexes.

Update to the latest book's CD

khen1234 — Sat, 10 Feb 2007 07:22:00 GMT

Many have noted that there are some files missing from the CD accompanying my latest book, SQL Server 2005 Practical Troubleshooting. My apologies for that. We’ve just posted an update to the book’s CD at http://www.khen.com/sspt.htm. Note: you’re welcome to grab the book’s CD regardless of whether you’ve actually purchased the book—we’re making it available to everyone.

Origins of khen and foo

khen1234 — Fri, 15 Dec 2006 22:02:00 GMT

For those wondering why I use the terms 'khen' and 'foo' so often, here's an article on the etymology of both:

http://www.youcancallmeal.com/type/archives/2006/03/origin_of_the_t.php

The new book is out

khen1234 — Fri, 15 Dec 2006 21:14:00 GMT

I've mentioned before that I had a new book coming out. It's finally available on Amazon. Again, I didn't write this one; I was just its editor. It was written by a "dream team" of developers from the SQL Server dev team and support engineers from Microsoft Customer Support Services. The focus of the book is on troubleshooting issues with the SQL Server database engine, and it's my hope that it will be the "go to" reference for people working through issues with the engine. You can find the book here.

SQL Server Support is at the PASS Summit again this year

khen1234 — Tue, 14 Nov 2006 00:11:00 GMT

My friends in Microsoft’s Product Support Services (PSS) group (aka SQL Server Support) are speaking again this year at the PASS Community Summit. They’re doing some main conference sessions, the PSS Boot camp, and putting on the PSS Service Center again (where attendees can work through hands-on labs designed by PSS that demonstrate how to troubleshoot SQL Server issues). I’d encourage you to make your way to their offerings if you possibly can. They’re always some of the best sessions and content at the conference.

They say the ultimate measure of success with an initiative is whether it survives the departure of the person or people who started it. Since its inception in 2003, I had coordinated the involvement of PSS in the PASS Summit. This year, I turned over the reigns to a group consisting of Todd DeDecker, a Group Manager within SQL Server Support, Bart Duncan, an Escalation Engineer within the support group and a good friend, and Haydn Richardson in SQL Server marketing. I’d long felt that PSS mgmt and the SQL Server marketing folks (who handle the product group’s involvement in other conferences) should work together to handle PSS’ presence at the conference. They had the charter and resources to do so, and I knew they could go to Bart for any questions they might have of a technical nature. Once the initiative progressed from toddlerhood to adolescence, I felt it was ready to survive without my involvement.

You might find some of the history behind this initiative fairly intriguing. I’ve not written much publicly about it in the past, so I’ll jot down my thoughts and memories before I forget them in case you find them interesting. I think it’s a good story of how fresh thinking and grassroots initiatives can succeed, even in a big company like Microsoft.

The idea of involving PSS in the Summit originally occurred to me shortly after I joined Microsoft and realized PSS was fairly out of touch with the user community. For a customer-facing org, this seemed awfully strange to me. Day in and day out, PSS spent more time with customers than the product team, but even the senior-most PSS folks couldn’t name more than one or two MVPs, had rarely if ever posted in the community newsgroups, and had never even been to the Summit, let alone spoken at it.

The same was true going the other direction. Many in the community knew people on the dev team and regularly spoke to them at conferences or exchanged messages over newsgroups and via email. There was and still is a nice synergy between SQL Server dev and the SQL Server user community. For the most part, that didn’t exist between PSS and the user community, and this really puzzled me.

So, I began to think about how to remedy it. I wanted to find a way to “hook up” PSS and PASS, to get them to know each other. It seemed obvious to me that there was mutual benefit for the support folk and for the user community in doing this: The support people could learn more about how MS customers actually used SQL Server in the wild (versus mostly seeing just the support issues related to it), and the community could learn more about many of the fine people in SQL Server Support, many of whom had been taking customer calls and helping the dev team find bugs in the product for several years. They could even learn to troubleshoot some of their own issues without having to call tech support in the first place.

My first thought was to somehow involve PSS in the PASS Summit. I’d spoken at the Summit for several years prior, and I thought it would be the perfect opportunity to let PSS and PASS get to know one another. Early in 2003, I approached PASS about this possibility, and we began talking about how to involve PSS in that year’s Summit in Seattle.

Initially, I had thought that a pre-con workshop would be the most appropriate forum for PSS to get to know the community better and to demonstrate its wealth of troubleshooting knowledge, but PASS felt that regular conference sessions would be more appropriate because they would allow the PSS involvement to be spread out more across the entire conference. More people would be exposed to PSS, and PSS would get to talk to more customers.

I also wanted a forum where customers could bring their support issues to PSS and receive hands-on help solving them. It occurred to me that this would also be a perfect opportunity to show people how to troubleshoot their own problems—along the lines of the “teach a man to fish instead of give him a fish” philosophy. I’d long felt that too much of the knowledge related to the SQL Server troubleshooting domain was locked up in PSS; I felt that much of it belonged in the hands of customers. So, having a forum where the free exchange of this type of knowledge could occur was very attractive to me. From this, the idea of the PSS Service Center was born. The folks at PASS and I talked several times about all of this, and I got the green light to explore the possibilities with SQL Server Support.

I talked to several folks about possibly being involved. The first person I asked about it was Bart Duncan. Bart talked to his manager at the time who repeated the age-old refrain that they’d be generally supportive of him doing something like this, but funding might be a problem. Others I talked to were nervous about getting up in front of people, especially customers who might pummel them with support issues in a conference session free-for-all. One who was receptive to the idea was Bob Ward, an escalation engineer within the support group, and he got his management to agree to fund him to participate in the conference. So, we had at least one support engineer from PSS signed up, and he had funding to go, so all we needed was a few more people, and we’d have something we could reasonably call a PSS delegation. I was eventually able to enlist the help of a couple of other Seattle-area support engineers, and off PSS’ modest PASS Summit contingent went—setting sail on its first voyage into the uncharted waters of SQL Server user community involvement.

At the conference in Seattle, PASS generously set aside a room just for Service Center use and allotted lab machines for it. Though small by this year's standard, it was a start, and attendees loved the Service Center. PSS also did conference sessions and was involved in several mixers and other social events. All in all, it was fun to watch. Things had worked out as PASS and I had originally envisioned. We got to see PSS interact with customers on a one-on-one basis just as we’d hoped. Customers learned firsthand just how good some of the PSS folks were at their jobs. Watching the rapport that developed between them assured me that the hassles of getting it all going was worth it.

We did much the same thing the next year at the 2004 Summit in Orlando. PSS was able to involve a couple of new people, and the regular sessions and Service Center came off with aplomb. The relationship between PSS and the user community continued to grow.

The 2005 Summit was to be held in the Dallas area, and it occurred to me that we might be able to leverage the fact that Microsoft’s largest SQL Server support site was DFW-based in order to increase PSS’ participation in the conference even further. The PASS folks and I discussed this at length and eventually came up with two additional ways for PSS to be involved in the conference. The first was an intense pre-con session wherein the support folks could cover how they approached solving hard problems at length. This would be more in-depth and longer than anything they’d done in regular conference sessions in the past. The PASS folks loved this idea and named it the “PSS Boot Camp.” It was a big hit with attendees and has been a staple of three shows now—the 2005 Summit, the PASS European Summit in February of this year, and this year’s Summit in Seattle.

The second idea I had for furthering the PSS involvement in the 2005 PASS Summit was to have a seminar at the MS campus for the SQL Server MVPs. PASS chartered busses over to the campus one evening during the conference, and the MVPs got to spend some quality time with several members of the support team. I was proud of our people and happy to have been a part of making it happen. Having the conference in Dallas had presented some unique opportunities for those of us in the area, and I felt good about the fact that we’d tried to take advantage of those as best we could.

This year, they aren’t doing the MVP night function, but SQL Server Support is still involved in all the other ways I’ve mentioned. The boot camp has been expanded to two days, the Service Center is once again online, and the support folks have several main conference sessions. I’d encourage you to attend some of their sessions if you possibly can. I’m sure they’d like to meet you, and I’m sure you’ll find it’s time well-spent.

I think PSS' ongoing involvement in the Summit is a wonderful success story for the user community and especially for PASS. None of this would have been possible had the PASS folks not enthusiastically welcomed fresh ideas and worked hard to make the whole thing happen. None of it would have survived had the user community not embraced it, and PASS was a big part of that. I always considered the PASS folks my partner in helping connect PSS and the user community, and much of the credit for all of it coming together rightfully belongs to them.

Normalizing query text

khen1234 — Sun, 22 Oct 2006 17:56:00 GMT

A conversation with a friend over email got me to thinking that this info might be useful to a few of you.

It’s common for SQL Server application developers to want to “normalize” the query text returned in a Profiler trace such that executions of the same query with different parameters can be easily compared with one another and aggregated together. This allows the performance of a query to be more easily tracked over time and allows the consequences of using different parameters with it to be quantitatively measured.

Normalizing the text of a query amounts to replacing the literals in it with parameter markers. It amounts to flattening the parameter values for the query such that the statistics (e.g., duration, CPU used, reads, and writes) from different executions of the query can be aggregated together and compared against one another. So, take this query for example:

SELECT * FROM foo WHERE id=4

Here, the parameter is “4”, so, parameterized, the query would look something like this:

SELECT * FROM foo WHERE id=@p1

Then, every other execution of this query, regardless of the value on the right side of the equal sign, would reduce down to this same basic form. This is the same basic form you might see in the procedure cache if you viewed dm_exec_cached_plans and dm_exec_sql_text.

There are a couple of approaches commonly used to normalize queries. The first, and simplest, is to take a substring of the query in question and try to trim it down to its bare essence. I’d guess most people looking at this sort of thing take this approach. The problem with it is that parameters can appear virtually anywhere, especially when you consider subqueries, derived columns, and the like. There’s no set substring length that you can always take to accurately normalize a query. Thus, this approach is extremely error-prone and of limited usefulness in truly normalizing the queries from a trace.

Another approach is to parse the T-SQL text using your own parser. This solves the problems of the simplistic substring approach, but unfortunately introduces some of its own. First, parsers are notoriously difficult to write. Even with the help of tools like Lex and Yacc, you still have a fair job on your hands to create a language parser, especially for a language as unwieldy as Transact-SQL. Second, most SQL parsers available don’t handle the extensions Microsoft has made to Transact-SQL that well. There are many free parsers out there, as well as some available for minimal charge, and most, if not all, have problems with Transact-SQL’s extensions. Third, even for those that may fully support Transact-SQL at a particular point in time, they can quickly become outdated as the product evolves. They have to keep pace with the evolution of SQL Server in general and with the T-SQL language in particular, and they may not be able to do that. Consider, for example, the kind of monkey wrench supporting CTEs would throw into such a facility. Last, but not least, it takes time to parse code. With a trace of any size, even a highly efficient parser would need significant time to parse it. Full-blown parsing is a bit overkill, anyway. What you really need is a literal identifier.

Fortunately, SQL Server 2005 offers a facility that partially solves the problem. The sp_get_query_template stored proc takes the text of a query as a parameter, then returns the normalized text and the parameter list as output parameters. Although it was not designed for general query normalization, this gets you as close as you currently can get to invoking SQL Server’s own code to normalize your queries.

Unfortunately, the proc has several limitations: it doesn’t handle multi-statement batches, will throw a 207 error if your query text references a non-existent column in an existing table (just as a CREATE PROC will), and will throw an error when a statement doesn’t have any parameters (e.g., SELECT * FROM sys.objects). But, it’s part of the way there. For simple query parameterization, it’s probably the best way to go for now. In concert with some substring trickery, this might get you most of what you need to do some basic query performance aggregation.

You can use sp_get_query_template against the TextData column returned by fn_trace_gettable to generate some useful query performance aggregations over time. You can easily identify your N most expensive queries in terms of CPU used, reads, writes, duration, etc. And you can then drill into individual executions from the trace to see what sets of parameters may be more or less likely to be the cause of poor performance from the query. This can then inform your query and indexing design decisions.

Book sample chapter available for download

khen1234 — Fri, 20 Oct 2006 18:59:00 GMT

I mentioned in my last post here that we have a new book coming out in December that focuses on troubleshooting issues with broken SQL Servers. We've decided to release a sample chapter from the book (that has not yet been proofread) so that people can see what's coming. You can find that chapter here: http://www.khen.com/sspt04.pdf. Let me know what you think.