Fabulous Adventures In Coding
Eric Lippert is a principal developer on the C# compiler team. Learn more about Eric.
Welcome to 2011 everyone; I hope you all had as restful a time as I did over the winter break. We’ll get back into language and tools design next time; today’s post is dedicated to the memory of my friend Frank, an old and dear friend of my whole family who passed away shortly before Christmas.
As long-time readers of this space know, I take an annual pilgrimage to my family’s summer place on the beaver-shark infested shores of Lake Huron, Ontario’s vast inland sea. One Thursday in August several years ago I was relaxing on the beach with a few of my visiting Waterloo alum friends when there was an enormous explosion a couple of bays over. Four huge white clouds billowed into the clear blue sky.
This was less shocking than you might imagine. There had been rather a lot of loud bangs coming from the nuclear power plant near the cottage that year as they were dismantling the heavy water towers. CANDU nuclear power plants use heavy water as coolant; heavy water extraction requires large quantities of dangerously toxic heavier-than-air gasses. Since the utility had stockpiled enough heavy water to last for the expected lifetime of the facility, they were dismantling the apparatus and removing the toxic threat.
However, these four huge clouds did not look like the dust clouds you’d expect from a demolished tower; they looked like perfectly ordinary, if somewhat vertical, white fluffy clouds.
Well, whatever. We went back to our conversation.
A few minutes later one of my guests reported that the water was not working in the cottage. This also is a not entirely uncommon situation; cottage water systems can be finicky to say the least.
The first thing I checked was that all the water supplies were in fact not working; the toilet tank was not filling, the kitchen sink was not running, and so on. We get our water from a well; clearly the problem was with the pump.
I checked to make sure that the pump was switched on. It was switched on, but not running.
I checked the electrical service panel. None of the breakers were tripped.
I tried turning on a light.
Aha! We’ve had a power failure. That explains it.
And that was when I suddenly put two and two together and realized that a huge explosion at a nuclear power plant, and me not being able to run the water pump just might have a causal connection. This was when I started to get a bit more worried, and turned on a battery-powered satellite radio receiver.
As it turns out, the power wasn’t just off in my neighbourhood; for the last ten minutes there had been no power anywhere from the wilds of Northern Ontario to Manhattan Island; 55 million people were without power.
So I went and found my neighbour, Frank, who was relaxing on his deck with a cold drink. He’s a retired Ontario Hydro engineer, and therefore I held him personally responsible for the fact that I was going to have to barbecue the spaghetti I was planning on having for dinner. The first thing Frank pointed out to me was that in just a few minutes we were about to see a rare but entirely predictable event: a steady stream of vexed, bucket-carrying children streaming from their cottages to fetch lake water with which to flush their toilets. (Cheap manual labour is why you bring children to the beach.) And sure enough, within about two minutes there were half a dozen kids with buckets running up and down the sand.
The next thing he pointed out to me was that by all appearances the safety systems at the plant had done their job flawlessly. The interesting thing about electrical power is that every watt you produce has to get consumed immediately. There’s no storage capacity in the system whatsoever; we simply do not have the technology to quickly store excess power for later. Which means that when one power plant goes offline for some reason, power gets routed to the cities it was servicing from other providers. But the sudden increase in demand might cause dangerously high current on one of the other plant's lines, causing that plant to shut down, and now we’ve got two plants offline, but the city that is drawing that power is still trying to draw the power from somewhere. Perhaps the city distribution stations detect the overcurrent situation, trip their breakers and shut down, and suddenly you’ve got the opposite problem; now you have an entire city offline so there is now too much power being produced. And so it goes; as more and more power plants and cities go offline there are vast surges of power going back and forth across the grid in just a few seconds. Pretty soon there is nothing left connected to the grid except a few power plants all desperately trying to power something, anything. Huge backwashes of power come hurtling along the transmission lines back to the plants, which can't be good.
In that situation, what do you do? You have to stop producing power immediately, because that’s making it worse for everyone else. The plant near my cottage uses the heat produced by decaying uranium to make superheated heavy water steam. That is then passed through heat exchangers to make not-radioactive high pressure steam, which then runs conventional generator turbines. When those turbines need to turn themselves off in a hurry the safety system simply dumps the steam running through the turbines directly into the atmosphere, instantly liberating billions upon billions of joules into the atmosphere and making four really enormous fluffy clouds and a heck of a noise.
Upon digesting this explanation, I suggested that Frank go into his shed, find the crank that restarts the power plant, and head on over there to show them how it was done in his day, a suggestion which he politely declined. So we barbecued the spaghetti, drank lemonade, and sent small children to fetch water from the lake. One of the best things about being at the beach is that you don’t need electricity for anything.
Next time: Writing a "simple" refactoring isn't necessarily as simple as you might think.
Cool story. It is timely because my son is leaving tomorrow with with his class for winter camping at MacGregor Point. Their first order of business, however, is a tour at that same nuclear power plant.
Quite fine, though I hope he has a warm sleeping bag; I wouldn't want to be sleeping outdoors on MacGregor Point in January! The plant tour is quite interesting, but I am lucky in that I know a number of the plant engineers, so I've gotten to take the behind-the-scenes tour where I got to check out the simulator (they have a whole mock control room that simulates accidents for training purposes), the gravity-fed reactor poisoning system that shut it down in a hurry should the neutron flux get too high, the spent fuel swimming pools, the enormously thick wire where the power leaves the plant to go to the yard to be transformed to high voltage, etc. They have many fascinating systems in that plant. - Eric
"The interesting thing about electrical power is that every watt you produce has to get consumed immediately. There’s no storage capacity in the system whatsoever; we simply do not have the technology to quickly store excess power for later."
I keep telling people this in discussions about the problems of wind power, but nobody seems to believe me. I'm still looking for a clearly written reference to back up this fact.
The obvious solution is to hollow out the moon and transform it into a rechargeable battery. Of course, the tethering system might prove difficult.
Interesting story. Strange that you didn't mention the computer-related failures though, at least according to the Wikipedia article to which you linked:
A software bug known as a race condition existed in General Electric Energy's Unix-based XA/21 energy management system. Once triggered, the bug stalled FirstEnergy's control room alarm system for over an hour."
(And sorry to hear about your friend.)
I remembered learning about this in physics class, I thought it was the coolest thing: en.wikipedia.org/.../Pumped-storage_hydroelectricity
It's not the case that we don't have the technology to store/recover electricity - the UK does this pretty much every day using pumped storage - it's not designed to store an excess very quickly, because that's not nearly such a serious problem as covering a shortfall quickly, but it can certainly do the latter just fine. There are also fuel cell systems (there's one at Little Barford in England), but they've not commercially successful at the moment.
It's economics and geography that mean we don't currently store electricity all over the place to deal with specific transmission problems, but it's not written on tablets of stone that it's not a storable commodity. It's just, in general, cheaper to generate just-in-time.
Your story about the blackout is a lot cooler than mine. Incidentally, my friends and I also had spaghetti that night, although being in the city it was a bit different. We had to find a neighbor with a gas stove and worry about having water.
For those remarking about managed hydroelectic power: I think the operative word here is "quickly." It is not feasible to suddenly divert a few gigawatts to a pump. Believe me, if they could, they would - it is expensive to stop and restart a CANDU. (The neutron cross-section changes during these transitions in a way that promotes xenon flux - insulating bubbles - in the pressure tubes that interferes with heat exchange. This limits the speed at which the reactor can be restarted.)
Indeed, though would it not be more accurate to say that the fact that Xenon-135 is a powerful neutron absorber that really inhibits the speed at which the reactor can be restarted, not because it messes up the heat exchangers? It was my understanding that a failure to correctly and safely adjust for the rate at which "reactor poison" Xenon-135 gets converted to its Xenon-136 form that was the proximal cause of the Chornobyl disaster. However, I am not an expert in these things by any means. - Eric
Quite a few misconceptions in the comments I'm reading.
When Eric refers to the inability to store electric power, he's meaning exactly what he's saying: storing ELECTRIC power.
Pump storage is a completely different kind of storage called potential energy storage. It is basically the same as having a uranium depot or a fuel tank. You are most definitely not storing electric power.
Also it is important to note that pump storage is not even energy efficient. Its a great business but the net output is power consumption, in other words, it consumes more power than it is able to supply. This is due to the fact that the performance of pumps is worse than the performance of turbines. Utilities simply make a whole lot of money out of it because they pump at night when electric power is cheap and generate in peak hours when electric power is expensive. They do have the advantage that they provide power consumption in valley hours, typically nighttime, which avoids having to shut down thermal power plants (nuclear/coal and to some degree deisel) which is always expensive.
The inability to store electric power IN THE GRID means that there always has to be a fine balance between the power you are inputing into the grid and the power that's been consumed. If there is too much power input compared to how much is being consumed, grid frecquency goes up. On the other hand if there is too little power, frecquency goes down. In both cases, past a certain tolerance, power plants or consumption points will disconnect from the grid. If the power suddenly is big enough it can take down whole areas and theoretically if the transient is enough it can cause a national power grid black out.
Before uni I did some work in a gas turbine + steam turbine power station. One night a fan blade broke on one of the gas turbines and shot up straight through the top of the casing and hit the roof of the power station.
Thankfully I was well out of the danger / blame zone, at home in bed.
I remember that day well. I was sitting in a (large automotive manfacturer) assembly plant, working on a project completion report (...oh joy...) when the lights went out, my laptop switched to battery power (5% left!), I lost my hardwired LAN connection, the wireless backup LAN also failed, and everybody in the Engineering section I was sitting in started cursing and screaming. After 30 seconds, I heard the two MASSIVE desiel generators behind the plant start themselves up. The same generators I saw for the first time not 20 minutes eariler while the IT guy that normally mans this plant was giving me a brief tour. The last question I asked him before we parted ways, "How long can those things run without refueling?" Come to find out, not long enough!
Eric, so sorry for your loss. I've heard the story before, but it does not diminish in the retelling (you know what they say; awesome once, awesome every time).
Eric, thanks for sharing. Andy and I enjoyed reading the story - he always had such a quick wit along with his scientific explanations. Can't wait to see everyone in the summer. Take care, - Laurie