gpage's WebLog

Here's a question...

gpage — Fri, 24 Sep 2004 00:08:00 GMT

When someone says to me, "Clustering stinks. It never works reliably.", I often find myself wondering a few things (if it makes you feel more comfortable, you can insert your product, technology, pet here instead of cluster). I

I find myself asking "I know of three other customers who have this exact same certified hardware, yet they NEVER have problems". This can also be phrased, as someone said to me last night, regarding a mutual friend about the same age as me, "If you're working on your third divorce, have you considered that the women you're marrying might not be the problem?"

I think I know why some IT groups can make almost anything work and some can't make a toaster work, but I want to know what you think.

-G

P.S. In case your wondering, I'm 30, and before I turn 31, the aforementioned friend will be 30.

iSCSI is the stuff, so sayeth the Greg

gpage — Thu, 23 Sep 2004 16:57:00 GMT

I've been noticing recently that storage is becoming a problem. By that, I don't mean that storage itself is a problem, or acquiring storage is a problem. Rather, I mean that storage management is a problem.

Before we get into why it's a problem, let's backup just a bit an look at the evolution of storage over the past few years to see how suddenly, we find ourselves knee deep in volumes and switches and management applications still not having enough storage space.

In years past (and yes, this will be a gross over-simplification), you generally had a server with what you thought would be enough internal storage, 3 or 4 disks in a RAID set of some type. Quickly, after a few years, we realized that we needed a LOT more storage that would comfortably fit into the rack mount chassis of our x86 based server. Enter external SCSI storage as the solution. As a solution, it was pretty decent. Throw a bunch of SCSI disks in a cabinet, give the cabinet its own power supply and hook up some servers. Create some LUNs and away you go.

Unfortunately, this solution was (and is) neither very flexible nor very scalable. Enter SANs and fibre (yes, that's the correct way to spell "fibre" when referring to storage connectivity) storage. SAN solutions allow the use of switched to route traffic so that many machines (servers) can see individual LUNs without worrying about which drives are mine, yours, etc. SANs give you the capability to connect dozens of servers to the same switch and setup zones defining which servers could see which disks, a major leap forward in technology that SCSI did not have. With technology moving forward, we also see in SANs the ability to create a raid set and carve multiple LUNs out of that, or to create disks using stripes of all disks, not just defined raid sets. Yes, SANs definitely were a leap forward in the technology area.

Now for the bad news. SANs are a total PITA to manage. I cannot tell you how many issues I've seen with zoning, presenting the wrong disks to the wrong server, presenting the same disk down multiple paths to the same server (or multiple servers) simultaneously, incompatible firmware between the HBA, the switch and storage processor….the list goes on and on. All of the flexibility and tuning ability came at the expense of management overhead, where one mistake (incorrect firmware on a switch) can toast all of your servers attached to the SAN.

Now, enter the beauty and the joy that is iSCSI. This is a major advancement in storage technology. iSCSI basically allows you to use the SCSI protocol over IP. On the surface, that doesn’t really sound like much, except a new transport media for the SCSI protocol (yes, SCSI is a protocol, defining disk communication as well as a hardware spec), but, it's much much much more than that.

In order to connect to SCSI storage, you no longer need switches, expensive HBAs (fibre HBA - $1200 for the latest model. Gigabit adapter, $110, both major brands). You now have the ability to freely move servers around without having to re-zone or worry about cabling. You can have the same server connected to multiple SCSI targets at the same time without multiple HBAs, switches or zoning (for instance, the XP machine I am writing this on is connected to a NetApp Filer and a Windows 2003 machine using the StringBean WinTarget (http://www.stringbeansoftware.com) software to share disks out as SCSI devices. Do I need to HBAs? Why no I don't. All I need is one network card and my Windows XP SP2 workstation is simultaneously connected to two separate SCSI targets. The NetApp filer is being used for a cluster tested for several of our clusters here. I have Virtual Server running on my desktop machine, loading the VMs I have stored on the 2003 server I am connected to as an iSCSI target, and the VMs are using iSCSI to connect to their cluster drives on the NetApp box (which, BTW, is a pretty darn cool little box. I haven't found anything I can't easily do. From start to finish, never having seen one EVER, it took me 7 hours to set it up and get it running the way we want). Basically, each machine gets a unique iSCSI initiator ID (part of that is a hash of the machine name) and you zone by creating a LUN and in a text window, simply popping in then initiator ID's that have access to that LUNs. It's even web based so you can cut-n-paste and manage it from anywhere on your network.

Speed? Is iSCSI fast? Check out these numbers from Equalogic, http://www.equallogic.com/docs/EqualLogic_ESG_Lab_Report_04.pdf. They are reporting numbers like "25 array Peer Storage system using Microsoft iSCSI Software initiator produced 1.2 million IOPs out of cache". Basically, it's fast, cheap, flexible and good. Especially if you used gigabit and MPIO.

So far, I haven't found ANYTHING that iSCSI can't do to meet my storage need. iSCSI is the stuff, I say. Please, download the iSCSI initiator, whitepapers, case studies, Kool-Aid, etc, at http://www.microsoft.com/windowsserversystem/storage/technologies/iscsi/default.mspx, because it's the stuff.

Vendor Dependence Bad. Beer Gooooood.

gpage — Thu, 24 Jun 2004 00:21:00 GMT

I just finished helping a customer with a cluster. Now, normally, that's not so unusual given that I do cluster support for a living. What is unusual is that in this particular case, the gentleman I was helping had an event log with an event entry that told him exactly what was wrong and what to do to fix the issue (expired account was preventing a service from starting using that service account). Nonetheless, he felt it was necessary to call Microsoft support and confirm with us that this was in fact the right thing to do. Why? Why, why, I ask you?

Vendor dependence, that's why. I'm not talking about having a support contract with a vendor and asking them for advice, I'm talking about being so dependent on your vendor(s) for support that you can't even reset an expired account without calling them and asking them if that's the right thing to do to get the account working.

Working in cluster support, I see a lot of other vendors and interact with them, primarily storage vendors. Overall, I have a good relationship with these guys and we chat a lot, and it seems that they've noticed this as well. It mystifies me. Bear in mind when I say that, I am not talking about my great grandmother trying to figure out how to do a simple sum in Excel and not bothering to read the help file. Heck, she's not even aware of the existence of the help file. She's barely away of the existence of the computer-thingy. I'm talking about multi-billion dollar corporations using very expensive and very mission critical servers (clusters, in my case), and not knowing *anything* about them.

Usually, I see this attitude in the storage area more than anywhere else. It's not an uncommon scenario to see a storage admin with several clusters attached have no idea how his zoning is setup (hold on, I'm going somewhere with this, knowledge of zoning not required). If everything is setup right, that's not a problem, right? (Right.) Fast forward 6 months. The vendor comes in and says “hey, you need to add that new server to the storage group, no problem, let me just make a zone change here...“. Zone change gets made, and in true Dave Chappelle style, BAM!, the clusters stop working.

Enter Microsoft support. We figure out the disks are busted and calmly inform the customer with the $10,000 per hour web transaction database now down that the hardware is busted somehow and we need to figure out what has changed or gone wrong. Of course, I know that some “changes" were made yesterday because the customer came told me the vendor came out and made some changes. That conversation usually goes something like this:

“Were any changes made? “

“Yes”

Pause…..

“What were they?”

“I’m not sure….”

”Why did they make them?”

“I dunno. They said something about a new server coming online”.

“Did they change the zoning?”

“You know, I don’t know.”

“Who was there with them? Maybe they know….”

“I was the one here”

Pause. Very, very pregnant pause…..

“Oh. Uh, well, then, I guess we should call the vendor….”

Now what mystifies me is this utter and complete dependence and trust that the vendor is going to know that customers environment well enough to send a field tech to make complicated changes. Secondly, if the environment is so important, why the heck doesn't someone who works for and is fiscally responsible to that company know *something* about how zoning works, how their storage should be setup, how to double check and make sure the vendor is doing the right thing?

This is unfortunately not limited to just clusters, but rather seems endemic to the industry, the complete vendor reliance. I don’t really have a good ending to put here, but it does seem silly and irresponsible, to take such critical systems and entrust them to the hands of someone else who will never EVER feel the same heat if something breaks. I’ve had to go onsite many times for issues and work three days without sleep to try to resolve issues caused by screwed zoning, but here’s the thing. When I come onsite and work for 3 days to fix the problem, I get to be the hero, but the storage admin who called us because he barely knew what zoning was….he’s may be looking for a job soon.

Remember that I am not advocating abandoning vendors or not utilizing them, but please, know how the stuff you depend on works. It makes all of us happy.

Oh yeah, and beer totally rules.

WOOT! I have a blog!

gpage — Mon, 29 Mar 2004 22:06:00 GMT

Oh yeah, I have my very first blog. I can just *feel* the crunchy-on-the-outside-but-rich-and-creamy-center blog goodness to follow. It's gonna be sooooo good.

........

Of course, now I can't think of anything to say. Except that I hate flying. Sort of. I'm in the midst of a love/hate relationship with work (not with MS, just read, you'll see what I mean), so right now I hate flying. I'm in the office, doing my thing (cluster support for MS, if you must now), after a bit, things start to get a little same old, same old, when along comes a customer who needs someone to come onsite for something, so I raise my hand. Going onsite lends a little excitement, gets you out of the office, get to see somewhere new, go onsite, fix something, be the hero, etc. It can be a big lure to get out of the office. Ususally, though, about two days into the trip, I realize I hate traveling but was once again lured into the trip by the chance to get out of the office and do something else for a few days. By the time I leave and am trudging back through the airport, I swear I will *NEVER EVER* travel again, even under pain of death and torture. Of course, two weeks later, I'm on a plane again. *sigh*.

So yeah, be happy for me. I have a blog. You can all look forward to more random thought goodness. My next (not-so) random thought will be “How to Call MS (or any vendor) And Help Us to Help You”. I do support. All day. Every day. Yeah yeah, look forward to it. It'll be tasty too.

-G