The New Year 2010 will be here tonight so like others my thoughts have been on the past … and the future. This happened because the other night I was sitting in bed scrolling through my smart phone emails when I saw a Computerworld email entitled “Y2K: The good, the bad and the crazy.” Boy did that ring true and bring back some stark memories about how this decade started, just 10 short years ago.

You see, in 1999, I was the CIO of the Edison Electric Institute and I was responsible for many of the organization’s efforts to understand the problem and lead the industry past the Y2K event. The experience certainly changed my career forever.

It started, as I recall, at the Nov. 1997 Edison Electric Institute/American Gas Association IT Conference in San Diego, as the Y2K bug began its way into our consciousness as a threat. During the main conference there was little mention of the Y2K bug but at the leadership meeting afterwards a number of conversations began to discuss the tip of the iceberg.

From that point on we (the IT leadership of the US investor owned utilities) built a community that began engaging in conversations, sharing best practices and developing remediation plans. It was imageprescient: shortly after that the skies opened up with worries. Not only did interest in the Y2K bug start to capture the national attention but the spotlight shifted intense focus on the electric power system as the most critical of critical infrastructures. Lose power and really bad things can start to happen really fast. Lose power for an extended time and modern society starts to grind to halt. That’s not news to anyone reading this blog, those in the utility industry who have witnessed the devastation of hurricanes and ice storms, as well as the short-lived but significant power blackouts that hit parts of North America, Europe and most recently Brazil during the past decade.

I can’t remember the date but it might have been early 1998 that the major U.S. trade electric associations (Edison Electric Institute, National Rural Electric Cooperative Association, American Public Power Association) and the North American Electric Reliability Council (it was a council rather than a corporation back then) were invited to attend a discussion at the U.S Department of Energy.   Remember, this was still very early in the game, when apprehension reigned. While Utilities were working on the issue, there wasn’t much visibility into or communications about the work that was being done. Governments, regulators and the public were understandably nervous. Plus, not much was known about the various interdependencies between various industries such as telecom, electric power, gas, railroads, etc. This was the meeting that set the wheels in motion for a continuous series of events that no one could have imagined.

NERC instituted and headed a formal process of monitoring and reporting. I worked closely with Gerry Cauley who led the NERC effort (incidentally, Gerry becomes NERC’s next president and CEO on Jan. 1, 2010).

The DOE also became very involved in the Y2K remediation efforts. I remember being at several events with then Energy Secretary Bill Richardson. Probably the most memorable was a meeting at the Pepco transmission control room in Maryland when Richardson proclaimed that the US had a third world grid. That comment didn’t really instill much public confidence in our most critical infrastructure. About this time the White House appointed John Koskinen as the Y2K Czar; he began a very successful effort of information coordination between industries and public awareness of the Y2K challenge.

From that point on Y2K and the electric power industry was the center point for governments, regulators, the press and the public. And guess who was given much of the responsibility for communicating about the situation, to assuage fears and convey facts about the industry’s preparations and remediation? You guessed it: Yours truly. image

The FOX Television Network set up cameras in my office for interviews. I spoke at the National Press Club and then was quoted in Time Magazine, the Wall Street Journal, and other newspapers all over the country, even Voice of America television broadcasts intended to help other countries deal with their Y2K problems. I remember doing the “black box room” interviews for FOX and MSNBC as well as many local TV and radio stations.

Ken Cohn (CIO of Pepco) and I remember all too well an event that occurred during a George Washington University/Washington Post panel on Y2K. Ken and I were describing the good work that the industry was doing overall and the specifics of Pepco’s work. Believe it or not, we got booed! Many in the audience didn’t believe (or didn’t want to believe) our positive news, that we were finding only minor problems across the infrastructure and saw no reason to believe that Jan 1. 2000 would be unlike any other day.

On another night I was speaking with the CIO of Montgomery County, Maryland on Y2K Readiness in front of 800 people at Fannie Mae. During the discussions a number of people shouted out that we must disconnect from the grid! I pointed out that it was the grid that gives us the reliability that we have today and this would be an option that would most likely be counterproductive to keeping the lights on. Of course the fear mongers were hard at work spreading the word that all of these microchips have calendar date dependencies and that they had deep penetration into the fabric of our society and there would be this great transfer of wealth on Jan. 1 as all the electric power, banking and financial systems would fail!

As we got closer to the date, the reporting and score carding of who was ready and who was not became more intense. I testified before Congress and gave numerous briefings to Senators and their staffers. I remember a private viewing of “Y2K The Movie” with Congressman Stephen Horn. Congressman Horn asked my opinion about the movie and I replied that the movie was more science friction than anything close to reality. Of course, we were all fearful that the movie would set off fear and panic, like yelling Fire in a theater, as it depicted collapsing air traffic control systems, nuclear power plant malfunctions and other disasters, if my memory serves me correctly. But I have to say that even many of the skeptics such as Ed Yardeni, Chief Global Economist and Investment Strategist of Deutsche Bank Securities in New York, who sounded the Y2K bug alarm early had started to back down from their earlier warnings of Y2K doom.

Then as we rolled toward the latter half of 1999, John Koskinen set up an operation call at the White House Information Coordination Center. The ICC funneled information between federal agencies and major industries and ran 24/7 starting a month before the rollover and expected to continue operations several months afterwards while the Y2K cleanup persisted.

I ran the Power Sector at the ICC and was lucky enough to man the 9 p.m.-9 a.m. shift. It was quite an operation run by retired Army Lt. Gen. Peter Kind. All the major industries and government agencies were co-located in this complex for information sharing, coordination and remediation activities after the rollover. There were intelligence operations monitoring for Y2K sabotage activities, weather events and even solar storms that might take out power systems during the rollover (this happened in 1989 in parts of Canada and the US).  But once again, electric power was dubbed the Achilles heel and all eyes were focused on us.  

We started the actual December 31 rollover monitoring with Pago Pago, which is the capital of American Samoa just across the International Date Line. From there we rolled through New Zealand then Australia and all eye and ears where focused on any signs of problems. At first we thought we had a problem in Australia as there were outages but it was quickly determined to be a result of high winds. From there we monitored Moscow, Paris, London – all passed through the rollover without an issue.  

As we approached the rollover on the east coast of the US we were allowed to roll up the blinds that concealed our operations and watch the fireworks over the Washington Moument and at the same time the ball fall at Times Square in NYC on the big plasma screens that were used to monitor the rollover around the world.  

As the date changed across the US we had some power outages, just like we have every New Year’s but they were minimal and only impacted a small number of customers. I recall a transformer in Louisiana being shot up with deer rifles as well as a low voltage transmission tower that toppled in the Pacific Northwest. That was thought to have been caused by sabotage but later it was determined to be improperly torqued bolts loosed by wind vibrations.

At about 2 a.m. Eastern time it had appeared that we had indeed squashed the Y2K bug. Hooray!

But our celebration was cut short as the next call to action came down: “What would have happened if we had not gone through all the remediation efforts?”  The reality for the power industry at the time (remember, this is pre-smart grid) is that by and large we were an industry composed of a bunch of metal wires and metal devices. Power system operations had few calendar date dependencies. Gerry Cauley stated this in one of the final NERC reports to DOE. But, it was work that had to be done because no one knew for sure and a catastrophic collapse of the electric power system requiring a total black start (grid completely down) could have had repercussions on the world economy that would have lasted months!

 At about 7 a.m. Eastern time I was asked “What about Hawaii?” Remember, NERC only covers North America and the extensive monitoring and reporting system that we had in place did not include Hawaii. I picked up the phone and called my friends at Hawaii Electric. They said they were “Hanging Ten” (It’s when a surfer has all ten toes on the nose of the board) and experiencing no significant issues, other than a minor problem with a GPS receiver that required a simple reset (known problem).

There were indeed problems with the corporate IT systems that needed to be resolved, particularly many of the old customer information systems and some utilities upgraded or replaced their ERP systems because of the Y2K issues. A few billing problems surfaced after Jan 1. Also the industry learned a lot about disaster preparedness, implementing plans and running drills that are much needed skills in today’s world.

As we rolled through January 1, 2000 – at that point I had been up for over 48 hours – there was a imagegiant sign of relief that all was well. But you could see the interest fading fast as my nightline interview with Ted Koppel was cancelled and many others in the media started to lose interest in the “non-event.” Yes, there were those that said this was just the tip of the iceberg (I remember one Computerworld columnist proclamations) but in the days that went by it was obvious that that the Y2K bug had been squashed and was not as ubiquitous in critical infrastructure as many had thought. After about a week John Koskinen began pulling up stakes at the Y2K command center – it was good news for me as I could get off the 9 p.m.-9 a.m. and start my transition back to a normal life. I was tired, like I was partying like it was 1999.

So when I reflect back on what I was doing 10 years ago this is what I see in my mind.  It’s probably the most significant  professional event of my lifetime. But you know what I remember the most? It was all the people working together and all the relationships that came out of Y2K. While Y2K the event was no more than a shooting star that is long forgotten, it was an event that has lasted much longer for me, and probably for many others in the industry as well, and it has been key to my growth professionally and spiritually.  

Happy New Year Everyone!

Jon C. Arnold

Managing Director, Worldwide Power & Utilities Industry

Microsoft Corporation