I just finished helping a customer with a cluster. Now, normally, that's not so unusual given that I do cluster support for a living. What is unusual is that in this particular case, the gentleman I was helping had an event log with an event entry that told him exactly what was wrong and what to do to fix the issue (expired account was preventing a service from starting using that service account). Nonetheless, he felt it was necessary to call Microsoft support and confirm with us that this was in fact the right thing to do. Why? Why, why, I ask you?

Vendor dependence, that's why. I'm not talking about having a support contract with a vendor and asking them for advice, I'm talking about being so dependent on your vendor(s) for support that you can't even reset an expired account without calling them and asking them if that's the right thing to do to get the account working.

Working in cluster support, I see a lot of other vendors and interact with them, primarily storage vendors. Overall, I have a good relationship with these guys and we chat a lot, and it seems that they've noticed this as well. It mystifies me. Bear in mind when I say that, I am not talking about my great grandmother trying to figure out how to do a simple sum in Excel and not bothering to read the help file. Heck, she's not even aware of the existence of the help file. She's barely away of the existence of the computer-thingy. I'm talking about multi-billion dollar corporations using very expensive and very mission critical servers (clusters, in my case), and not knowing *anything* about them.

Usually, I see this attitude in the storage area more than anywhere else. It's not an uncommon scenario to see a storage admin with several clusters attached have no idea how his zoning is setup (hold on, I'm going somewhere with this, knowledge of zoning not required). If everything is setup right, that's not a problem, right? (Right.) Fast forward 6 months. The vendor comes in and says “hey, you need to add that new server to the storage group, no problem, let me just make a zone change here...“. Zone change gets made, and in true Dave Chappelle style, BAM!, the clusters stop working.

Enter Microsoft support. We figure out the disks are busted and calmly inform the customer with the $10,000 per hour web transaction database now down that the hardware is busted somehow and we need to figure out what has changed or gone wrong. Of course, I know that some “changes" were made yesterday because the customer came told me the vendor came out and made some changes. That conversation usually goes something like this:

“Were any changes made? “

“Yes”

Pause…..

“What were they?”

“I’m not sure….”

”Why did they make them?”

“I dunno. They said something about a new server coming online”.

“Did they change the zoning?”

“You know, I don’t know.”

“Who was there with them? Maybe they know….”

“I was the one here”

Pause. Very, very pregnant pause…..

“Oh. Uh, well, then, I guess we should call the vendor….”

Now what mystifies me is this utter and complete dependence and trust that the vendor is going to know that customers environment well enough to send a field tech to make complicated changes. Secondly, if the environment is so important, why the heck doesn't someone who works for and is fiscally responsible to that company know *something* about how zoning works, how their storage should be setup, how to double check and make sure the vendor is doing the right thing?

This is unfortunately not limited to just clusters, but rather seems endemic to the industry, the complete vendor reliance. I don’t really have a good ending to put here, but it does seem silly and irresponsible, to take such critical systems and entrust them to the hands of someone else who will never EVER feel the same heat if something breaks. I’ve had to go onsite many times for issues and work three days without sleep to try to resolve issues caused by screwed zoning, but here’s the thing. When I come onsite and work for 3 days to fix the problem, I get to be the hero, but the storage admin who called us because he barely knew what zoning was….he’s may be looking for a job soon.

Remember that I am not advocating abandoning vendors or not utilizing them, but please, know how the stuff you depend on works. It makes all of us happy.

Oh yeah, and beer totally rules.