Fabulous Adventures In Coding : Threading

What is this thing you call "thread safe"?

Eric Lippert — Mon, 19 Oct 2009 13:50:00 GMT

Caveat: I am not an expert on multi-threading programming. In fact, I wouldn't even say that I am competent at it. My whole career, I've needed to write code to spin up a secondary worker thread probably less than half a dozen times. So take everything I say on the subject with some skepticism.

A question I'm frequently asked: "is this code thread safe?" To answer the question, clearly we need to know what "thread safe" means.

But before we get into that, there's something I want to clear up first. A question I am less frequently asked is "Eric, why does Michelle Pfeiffer always look so good in photographs?" To help answer this pressing question, I consulted Wikipedia:

"A photogenic subject is a subject that usually appears physically attractive or striking in photographs."

Why does Michelle Pfeiffer always look so good in photographs? Because she's photogenic. Obviously.

Well, I'm glad we've cleared up that mystery, but I seem to have wandered somehwat from the subject at hand. Wikipedia is just as helpful in defining thread safety:

"A piece of code is thread-safe if it functions correctly during simultaneous execution by multiple threads."

As with photogenicity, this is obvious question-begging. When we ask "is this code thread safe?" all we are really asking is "is this code correct when called in a particular manner?" So how do we determine if the code is correct? We haven't actually explained anything here.

Wikipedia goes on:

"In particular, it must satisfy the need for multiple threads to access the same shared data, ..."

This seems fair; this scenario is almost always what people mean when they talk about thread safety. But then:

"...and the need for a shared piece of data to be accessed by only one thread at any given time."

Now we're talking about techniques for creating thread safety, not defining what thread safety means. Locking data so that it can only be accessed by one thread at a time is just one possible technique for creating thread safety; it is not itself the definition of thread safety.

My point is not that the definition is wrong; as informal definitions of thread safety go, this one is not terrible. Rather, my point is that the definition indicates that the concept itself is completely vague and essentially means nothing more than "behaves correctly in some situations". Therefore, when I'm asked "is this code thread safe?" I always have to push back and ask "what are the exact threading scenarios you are concerned about?" and "exactly what is correct behaviour of the object in every one of those scenarios?"

Communication problems arise when people with different answers to those questions try to communicate about thread safety. For example, suppose I told you that I have a "threadsafe mutable queue" that you can use in your program. You then cheerfully write the following code that runs on one thread while another thread is busy adding and removing items from the mutable queue:

if (!queue.IsEmpty) Console.WriteLine(queue.Peek());

Your code then crashes when the Peek throws a QueueEmptyException. What is going on here? I said this thing was thread safe, and yet your code is crashing in a multi-threaded scenario.

When I said "the queue is threadsafe" I meant that the queue maintains its internal state consistently no matter what sequence of individual operations are happening on other threads. But I did not mean that you can use my queue in any scenario that requires logical consistency maintained across multiple operations in a sequence. In short, my opinion of "correct behaviour" and your opinion of the same differed because what we thought of as the relevant scenario was completely different. I care only about not crashing, but you care about being able to reason logically about the information returned from each method call.

In this example, you and I are probably talking about different kinds of thread safety. Thread safety of mutable data structures is usually all about ensuring that the operations on the shared data always operate on the most up-to-date state of the shared data as it mutates, even if that means that a particular combination of operations appears to be logically inconsistent, as in our example above. Thread safety of immutable data structures is all about ensuring that use of the data across all operations is logically consistent, at the expense of the fact that you're looking at an immutable snapshot that might be out-of-date.

The problem here is that the choice about whether to access the first element or not is based on "stale" data. Designing a truly thread-safe mutable data structure in a world where nothing is allowed to be stale can be very difficult. Consider what you'd have to do in order to make the "Peek" operation above actually threadsafe. You'd need a new method:

if (!queue.Peek(out first)) Console.WriteLine(first);

Is this "thread safe"? It certainly seems better. But what if after the Peek, a different thread dequeues the queue? Now you're not crashing, but you've changed the behaviour of the previous program considerably. In the previous program, if, after the test there was a dequeue on another thread that changed what the first element was, then you'd either crash or print out the up-to-date first element in the queue. Now you're printing out a stale first element. Is that correct? Not if we always want to operate on up-to-date data!

But wait a moment -- actually, the previous version of the code had this problem as well. What if the dequeue on the other thread happened after the call to Peek succeeded but before the Console.WriteLine call executed? Again, you could be printing out stale data.

What if you want to ensure that you are always printing out up-to-date data? What you really need to make this threadsafe is:

queue.DoSomethingToHead(first=>{Console.WriteLine(first);});

Now the queue author and the queue user agree on what the relevant scenarios are, so this is truly threadsafe. Right?

Except... there could be something super-complicated in that delegate. What if whatever is in the delegate happens to cause an event that triggers code to run on another thread, which in turn causes some queue operation to run, which in turn blocks in such a manner that we've produced a deadlock? Is a deadlock "correct behaviour"? And if not, is this method truly "safe"?

Yuck.

By now you take my point I'm sure. As I pointed out earlier, it is unhelpful to say that a building or a hunk of code is "secure" without somehow communicating which threats the utilized security mechanism are and are not proof against. Similarly, it is unhelpful to say that code is "thread safe" without somehow communicating what undesirable behaviors the utilized thread safety mechanisms do and do not prevent. "Thread safety" is nothing more nor less than a code contract, like any other code contract. You agree to talk to an object in a particular manner, and it agrees to give you correct results if you do so; working out exactly what that manner is, and what the correct responses are, is a potentially tough problem.

************

(*) Yes, I'm aware that if I think something on Wikipedia is wrong, I can change it. There are two reasons why I should not do so. First, as I've already stated I'm not an expert in this area; I leave it to the experts to sort out amongst themselves what the right thing to say here is. And second, my point is not that the Wikipedia page is wrong, but rather that it illustrates that the term itself is vague by nature.

Events and Races

Eric Lippert — Wed, 29 Apr 2009 15:00:00 GMT

Here’s a question similar to one I saw on stackoverflow the other day. Suppose you have an event:

public event Action Foo;

The standard pattern for firing this event is:

Action temp = Foo;
if (temp != null)
temp();

What the heck is up with that? Why not just call “Foo()” ?

First off, this pattern ensures that the thing that is invoked is not the null delegate reference, which would cause a null reference exception to be thrown. But if that’s what we want, then surely we could have skipped the temporary variable – just “if (Foo != null) Foo();” would do. Why the temp?

The temporary variable ensures that this is “thread safe”. If the event’s handlers are being modified on another thread it is possible for Foo to be non-null at the point of the check, then set to null on a different thread, and then the invocation throws.

Using the temporary variable effectively makes a copy of the current set of event handlers. Remember, multi-cast delegates are immutable; when you add or remove a handler, you replace the existing multi-cast delegate object with a different delegate object that has different behaviour. You do not modify an existing object, you modify the variable that stores the event handler. Therefore, stashing away the current reference stored in that variable into a temporary variable effectively makes a copy of the current state.

Is that clear? Make sure it is, because now things start to get really confusing.

A common criticism of this pattern is that it trades one race condition for another. Let’s consider that scenario again more carefully:

The event handler contains a single handler, H. Thread Alpha makes a copy of the delegate to H in temp and determines that it is not null. Thread Beta decides that H must no longer be called when the event is fired, so it sets the handler to null. Thread Beta then assumes that H will never be called, and destroys a bunch of state that H needs to execute successfully. Thread Alpha then regains control and executes H, which behaves crazily since its necessary state has been destroyed. Thread Beta’s attempt to ensure that H is not called has been defeated by the race condition.

A frequently-stated principle of good software design is that code which calls attention to bugs by throwing exceptions is better than code which hides bugs by muddling on through and doing the wrong thing. This code has a race condition that results in incorrect behaviour. Perhaps an application of that principle is to stop using a temporary and instead crash if it races. That is to say, this code has a failure mode; surely it is better to highlight that failure with a crisp exception than to turn it into crazy wrong behaviour.

That seems plausibly argued, I agree, but the conclusion is wrong.

Suppose we remove the temporary variable but keep the null check. Does that solve the problem? No! We still have the same race conditions. Suppose the event handler delegate contains a reference to H. Thread Alpha checks to see whether it is null; it is not. Thread Alpha pushes the object to invoke on the runtime stack. Between the push of the delegate value and the call to invoke it, thread Beta sets the event handler to null. And once again, H will be invoked after thread Beta has removed the handler.

Suppose we remove the temporary and the null check and just invoke the delegate directly. Does that help? No, we still have the same race condition. The contents of the event handling variable can still be changed between the push of the delegate object onto the stack and the invocation of the delegate.

Removing the code around the call site does not decrease the number of race conditions in the code, and it does increase the number of crashes. Worse, doing so makes the race condition harder to detect by shrinking the window in which the race can occur without eliminating it.

Removing the null check and/or the temporary is a bad idea; the already-bad situation is only made worse.

Essentially there are two problems here that are being conflated. The two problems are:

1) The event handler delegate can be null at any time.
2) A “stale” handler can be invoked even after it has been removed.

These two problems are actually orthogonal and have different solutions. The onus for solving the first problem is laid upon the code which does the invocation; it is required to ensure that it does not dereference null. The store-in-temporary-and-test pattern ensures that null is never dereferenced. (There are other ways to solve this problem; for example, initializing the handler to have an empty action that is never removed. But doing a null check is the standard pattern.)

The onus for solving the second problem is laid upon the code being invoked; event handlers are required to be robust in the face of being called even after the event has been unsubscribed. In the scenario I described, the bug is actually in H. It needs to be robust enough to check to see whether the state it needs is still there, and bail out cleanly if it is not. (Or, alternatively, some additional locking mechanism needs to be implemented which ensures that the code which fires the event cooperates with the code that changes the event handlers to ensure the desired behaviour.)

The point though is that the "null ref problem" and the "stale handler problem" are two separate problems that are easily confused because the symptoms of both arise at the exact same call site code. You've got to solve both problems if you want to do threadsafe events. How you do so is up to you; just don't confuse the solution of one problem for a solution of the other.

(Thanks to Microsofties Levi Broderick, Chris Burrows, Curt Hagenlocher and Wolf Logan; this article is based on a conversation about their analysis of this pattern.)

Locks and exceptions do not mix

Eric Lippert — Fri, 06 Mar 2009 19:41:00 GMT

A couple years ago I wrote a bit about how our codegen for the lock statement could sometimes lead to situations in which an unoptimized build had different potential deadlocks than an optimized build of the same source code. This is unfortunate, so we've fixed that for C# 4.0. However, all is still not rainbows, unicorns and Obama, as we'll see.

Recall that lock(obj){body} was a syntactic sugar for

var temp = obj;
Monitor.Enter(temp);
try { body }
finally { Monitor.Exit(temp); }

The problem here is that if the compiler generates a no-op instruction between the monitor enter and the try-protected region then it is possible for the runtime to throw a thread abort exception after the monitor enter but before the try. In that scenario, the finally never runs so the lock leaks, probably eventually deadlocking the program. It would be nice if this were impossible in unoptimized and optimized builds.

In C# 4.0 we've changed lock so that it now generates code as if it were

bool lockWasTaken = false;
var temp = obj;
try { Monitor.Enter(temp, ref lockWasTaken); { body } }
finally { if (lockWasTaken) Monitor.Exit(temp); }

The problem now becomes someone else's problem; the implementation of Monitor.Enter takes on responsibility for atomically setting the flag in a manner that is immune to thread abort exceptions messing it up.

So everything is good now, right?

Sadly, no. It's consistently bad now, which is better than being inconsistently bad. But there's an enormous problem with this approach. By choosing these semantics for "lock" we have made a potentially unwarranted and dangerous choice on your behalf; implicit in this codegen is the belief that a deadlocked program is the worst thing that can happen. That's not necessarily true! Sometimes deadlocking the program is the better thing to do -- the lesser of two evils.

The purpose of the lock statement is to help you protect the integrity of a mutable resource that is shared by multiple threads. But suppose an exception is thrown halfway through a mutation of the locked resource. Our implementation of lock does not magically roll back the mutation to its pristine state, and it does not complete the mutation. Rather, control immediately branches to the finally, releasing the lock and allowing every other thread that is patiently waiting to immediately view the messed-up partially mutated state! If that state has privacy, security, or human life and safety implications, the result could be very bad indeed. In that case it is possibly better to deadlock the program and protect the messed-up resource by denying access to it entirely. But that's obviously not good either.

Basically, we all have a difficult choice to make whenever doing multi-threaded programming. We can (1) automatically release locks upon exceptions, exposing inconsistent state and living with the resulting bugs (bad) (2) maintain locks upon exceptions, deadlocking the program (arguably often worse) or (3) carefully implement the bodies of locks that do mutations so that in the event of an exception, the mutated resource is rolled back to a pristine state before the lock is released. (Good, but hard.)

This is yet another reason why the body of a lock should do as little as possible. Usually the rationale for having small lock bodies is to get in and get out quickly, so that anyone waiting on the lock does not have to wait long. But an even better reason is because small, simple lock bodies minimize the chance that the thing in there is going to throw an exception. It's also easier to rewrite mutating lock bodies to have rollback behaviour if they don't do very much to begin with.

And of course, this is yet another reason why aborting a thread is pure evil. Try to never do so!

Arrays considered somewhat harmful

Eric Lippert — Mon, 22 Sep 2008 21:04:00 GMT

I got a moral question from an author of programming language textbooks the other day requesting my opinions on whether or not beginner programmers should be taught how to use arrays.

Rather than actually answer that question, I gave him a long list of my opinions about arrays, how I use arrays, how we expect arrays to be used in the future, and so on. This gets a bit long, but like Pascal, I didn't have time to make it shorter.

Let me start by saying when you definitely should not use arrays, and then wax more philosophical about the future of modern programming and the role of the array in the coming world.

You probably should not return an array as the value of a public method or property, particularly when the information content of the array is logically immutable. Let me give you an example of where we got that horridly wrong in a very visible way in the framework. If you take a look at the documentation for System.Type, you'll find that just looking at the method descriptions gives one a sense of existential dread. One sees a whole lot of sentences like "Returns an array of Type objects that represent the constraints on the current generic type parameter." Almost every method on System.Type returns an array it seems.

Now think about how that must be implemented. When you call, say, GetConstructors() on typeof(string), the implementation cannot possibly do this, as sensible as it seems.

public class Type {
   private ConstructorInfo[] ctorInfos;
   public ConstructorInfo[] GetConstructors()
   {
     if (ctorInfos == null) ctorInfos = GoGetConstructorInfosFromMetadata();
     return ctorInfos;
   }

Why? Because now the caller can take that array and replace the contents of it with whatever they please. Returning an array means that you have to make a fresh copy of the array every time you return it. You get called a hundred times, you’d better make a hundred array instances, no matter how large they are. It’s a performance nightmare – particularly if, like me, you are considering using reflection to build a compiler. Do you have any idea how many times a second I try to get type information out of reflection? Not nearly as many times as I could; every time I do it’s another freakin’ array allocation!

The frameworks designers were not foolish people; unfortunately, we did not have generic types in .NET 1.0. clearly the sensible thing now for GetConstructors() to return is IList<ConstructorInfo>. You can build yourself a nice read-only collection object once, and then just pass out references to it as much as you want.

What is the root cause of this malaise? It is simple to state: The caller is requesting values. The callee fulfills the request by handing back variables.

An array is a collection of variables. The caller doesn’t want variables, but it’ll take them if that’s the only way to get the values. But in this case, as in most cases, neither the callee nor the caller wants those variables to ever vary. Why on earth is the callee passing back variables then? Variables vary. Therefore, a fresh, different variable must be passed back every time, so that if it does vary, nothing bad happens to anyone else who has requested the same values.

If you are writing such an API, wrap the array in a ReadOnlyCollection<T> and return an IEnumerable<T> or an IList<T> or something, but not an array. (And of course, do not simply cast the array to IEnumerable<T> and think you’re done! That is still passing out variables; the caller can simply cast back to array! Only pass out an array if it is wrapped up by a read-only object.)

That’s the situation at present. What are the implications of array characteristics for the future of programming and programming languages?

Parallelism Problems

The physics aspects of Moore’s so-called “Law” are failing, as they eventually must. Clock speeds have stopped increasing, transistor density has stopped increasing. The laws of thermodynamics and the Uncertainty Principle are seeing to that. But manufacturing costs per chip are still falling, which means that our only hope of Moore’s "Law" continuing to hold over the coming decades is to cram more and more processors into each box.

We’re going to need programming languages that allow mere mortals to write code that is parallelizable to multiple cores.

Side-effecting change is the enemy of parallelization. Parallelizing in a world with observable side effects means locks, and locks means choosing between implementing lock ordering and dealing with random crashes or deadlocks. Lock ordering requires global knowledge of the program. Programs are becoming increasingly complex, to the point where one person cannot reasonably and confidently have global knowledge. Indeed, we prefer programming languages to have the property that programs in them can be understood by understanding one part at a time, not having to swallow the whole thing in one gulp.

Therefore we tools providers need to create ways for people to program effectively without causing observable side effects.

Of all the sort of “basic” types, arrays most strongly work against this goal. An array’s whole purpose is to be a mass of mutable state. Mutable state is hard for both humans and compilers to reason about. It will be hard for us to write compilers in the future that generate performant multi-core programs if developers use a lot of arrays.

Now, one might reasonably point out that List<T> is a mass of mutable state too. But at least one could create a threadsafe list class, or an immutable list class, or a list class that has transactional integrity, or uses some form of isolation or whatever. We have an extensibility model for lists because lists are classes. We have no ability to make an “immutable array”. Arrays are what they are and they’re never going to change.

Conceptual Problems

We want C# to be a language in which one can draw a line between code that implements a mechanism and code that implements a policy.

The “C” programming language is all about mechanisms. It lays bare almost exactly what the processor is actually doing, providing only the thinnest abstraction over the memory model. And though we want you to be able to write programs like that in C#, most of the time people should be writing code in the “policy” realm. That is, code that emphasizes what the code is supposed to do, not how it does it.

Coding which is more declarative than imperative, coding which avoids side effects, coding which emphasizes algorithms and purposes over mechanisms, that kind of coding is the future in a world of parallelism. (And you’ll note that LINQ is designed to be declarative, strongly abstract away from mechanisms, and be free of side effects.)

Arrays work against all of these factors. Arrays demand imperative code, arrays are all about side effects, arrays make you write code which emphasizes how the code works, not what the code is doing or why it is doing it. Arrays make optimizing for things like “swapping two values” easy, but destroy the larger ability to optimize for parallelism.

Practical Problems

And finally, given that arrays are mutable by design, the way an array restricts that mutability is deeply weird. All the contents of the collection are mutable, but the size is fixed. What is up with that? Does that solve a problem anyone actually has?

For this reason alone I do almost no programming with arrays anymore. Arrays simply do not model any problem that I have at all well – I rarely need a collection which has the rather contradictory properties of being completely mutable, and at the same time, fixed in size. If I want to mutate a collection it is almost always to add something to it or remove something from it, not to change what value an index maps to.

We have a class or interface for everything I need. If I need a sequence I’ll use IEnumerable<T>, if I need a mapping from contiguous numbers to data I’ll use a List<T>, if I need a mapping across arbitrary data I’ll use a Dictionary<K,V>, if I need a set I’ll use a HashSet<T>. I simply don’t need arrays for anything, so I almost never use them. They don’t solve a problem I have better than the other tools at my disposal.

Pedagogic Problems

It is important that beginning programmers understand arrays; it is an important and widely used concept. But it is also important to me that they understand the weaknesses and shortcomings of arrays. In almost every case, there is a better tool to use than an array.

The difficulty is, pedagogically, that it is hard to discuss the merits of those tools without already having down concepts like classes, interfaces, generics, asymptotic performance, query expressions, and so on. It’s a hard problem for the writer and for the teacher. Fortunately, for me, it's not a problem that I personally have to solve.

Subtleties of C# IL codegen

Eric Lippert — Fri, 17 Aug 2007 19:12:00 GMT

It must be CLR week over at The Old New Thing because it's been non-stop posts about C# lately. Raymond's last two technical posts have been about null checks and no-op instructions generated by the jitter when translating IL into machine code.

I'll comment on both posts here, but I want to get the no-op discussion done first, because there are some subtleties here I believe that Raymond's statement that the jitter does not generate no-ops when not debugging is not entirely correct. This is not a mere nitpick -- as we'll see, whether it does so or not actually has semantic relevance in rare stress cases.

Now, I'll slap a disclaimer of my own on here: I know way more about the compiler than the jitter/debugger interaction. This is my understanding of how it works. If someone who actually works on the jitter would like to confirm my and Raymond's interpretation of what we see going on here, I'd welcome that.

Before I get into the details, let me point out that in the C# compiler, "debug info emitting on/off" and "IL optimizations on/off" are orthogonal settings. One controls whether debug info is emitted, the other controls what IL the code generator spits out. It is sensible to set them as opposites but you certainly do not have to.

With optimizations off, the C# compiler emits no-op IL instructions all over the place. With debug info on and optimizations off, some of those no-ops will be there to be targets of breakpoints for statements or fragments of expressions which would otherwise be hard to put a breakpoint on.

The jitter then cheerfully turns IL no-ops into x86 no-ops. I suspect that it does so whether there is a debugger attached or not.

Furthermore, I have not heard that the jitter ever manufactures no-ops out of whole cloth for debugging purposes, as Raymond implies. I suspect -- but I have not verified -- that if you compile your C# program with debug info on AND optimizations on, then you'll see a lot fewer no-ops in the jitted code (and your debugging experience will be correspondingly worse). The jitter may of course generate no-ops for other purposes -- padding code out to word boundaries, etc.

Now we come to the important point: It is emphatically NOT the case that a no-op cannot affect the behaviour of a program, as many people incorrectly believe.

In C#, lock(expression) statement is a syntactic sugar for something like

temp = expression;
System.Threading.Monitor.Enter(temp);
try { statement } finally { System.Threading.Monitor.Exit(temp); }

The x86 jitter has the nice property that the code it generates guarantees that an exception is never thrown between the Enter and the try. This means that the finally always executes if the lock has been taken, which means that the locked resource is always unlocked.

That is, unless the C# compiler generates a no-op IL instruction between the Enter and the try! The jitter turns that into a no-op x86 instruction, and it is possible for another thread to cause a thread abort exception while the thread that just took the lock is in the no-op. This is a long-standing bug in C# which we will unfortunately not be fixing for C# 3.0.

If the scenario I've described happens then the finally will never be run, the lock will never be released and hey, now we're just begging for a deadlock.

That's the only situation I know of in which emitting a no-op can cause a serious semantic change in a program -- turning a working program into a deadlocking one. And that sucks.

I've been talking with some of the CLR jitter and threading guys about ways we can fix this more robustly than merely removing the no-op. I'm hoping we'll figure something out for some future version of the C# language.

As for the bit about emitting null checks: indeed, at the time of a call to an instance method, whether virtual or not, we guarantee that the object of the call is not null by throwing an exception if it is. The way this is implemented in IL is a little odd. There are two instructions we can emit: call, and callvirt. call does NOT do a null check and does a non-virtual call. callvirt does do a null check and does a virtual call if it is a virtual method, or a non-virtual call if it is not.

If you look at the IL generated for a non-virtual call on an instance method, you'll see that sometimes we generate a call, sometimes we generate a callvirt. Why? We generate the callvirt when we want to force the jitter to generate a null check. We generate a call when we know that no null check is necessary, thereby allowing the jitter to skip the null check and generate slightly faster and smaller code.

When do we know that the null check can be skipped? If you have something like (new Foo()).FooNonVirtualMethod() we know that the allocator never returns null, so we can skip the check. It's a nice, straightforward optimization, but the realization in the IL is a bit subtle.

Human sacrifice, dogs and cats living together, mass hysteria! and thread model errors!

Eric Lippert — Tue, 08 May 2007 02:22:00 GMT

Reader Shaka comments on my post about error messages that "catastrophic failure" really does take the cake as being a terrible error message.

I fondly remember the first time I saw "catastrophic failure" as an error message. I was an intern, running the build lab for Visual Basic for Applications, and I was trying to get the PowerPC cross-compiler working so that we could build PPC binaries on x86 hardware. (Note that this was in the early 1990's. The VS build lab is now an entire team of people who have dedicated circuits just for the cooling equipment for all their computers. We've come a long way from an intern, a closet, and six machines named after opponents of Godzilla (*).)

Anyway, the compiler had a bug which caused it to run out of stack while compiling a particular module and the error was "catastrophic failure". This amused me to no end at the time. I laughed out loud, literally. I mean, fatal, sure. That process is going down. But "catastrophic"? I expect catastrophes to at least result in my hard disk becoming unreadable. I want bridges collapsing and floods and mass hysteria!

I have therefore always been rather embarrassed that the error message you get when you call the script engine on the wrong thread is "catastrophic failure". That, unfortunately, is the somewhat breathless and overstated string associated with E_UNEXPECTED. Basically we want to say "you called us in a completely unexpected way which thoroughly violates the engine contract, please don't do that". But "catastrophic failure" is what you get. I wish now that we'd declared a new HRESULT just for "you've violated the engine contract". But it's far, far too late now.

(*) In the interests of total accuracy: the OLE Automation build machines were named after Godzilla's opponents: Mothra, Monster Zero, etc. The VBA build machines were named after species of marine mammals: sei, humpback, etc.

The Tragedy of Thread Happiness Disease

Eric Lippert — Sun, 15 Feb 2004 19:46:00 GMT

A JOS reader interested in developing server software asked recently

Is it possible to determine the number of concurrent threads a server can support from the server's specification?

Now, as I've said before, I'm no expert on performance tuning multi-threaded applications, but I have picked up a thing or two hanging around the real experts on the IIS team. In fact, the perf teams get questions in this vein all the time from customers, both internal and external.

The interesting thing about the question is that it is symptomatic of Thread Happiness, a peculiar disease which usually strikes programmers who are relatively new to large-scale multi-threaded software development.

How do I know that the questioner is getting Thread Happy? Because if they were writing an application where there were two threads or five threads or ten threads then they wouldn't be asking that question. They'd be asking "I want to write a server app with two/five/ten/whatever threads -- how buff does the server have to be?"

But they're asking the question "how many threads can I create on a given server?" so I can only assume that they are thinking of writing some application that is going to create a large and variable number of threads, as many as possible.

That's Thread Happiness right there. Don't do that! It's almost certainly a bad design. Massively multi-threaded applications cause more problems than they solve; they are very difficult to get correct, and even harder to get performant.

To actually answer the question, no, there's no way of telling that from the server spec. Why? Because there is nowhere near enough information in the "static" facts about the hardware. Server application threading performance depends on "dynamic" facts like:

What other processes are going to be running?
What are those threads going to be doing?
How much memory do you have free?
How much contention, locking, busy waiting, context switching, blah blah blah, is there going to be?
What are the performance metrics that will determine when things have gotten too bad.

So, in short, don't even go there. Come up with a design that uses a small number of threads and tune that.

Now, you might be wondering "What about real-world server applications which do spin up a variable number of threads then? How do they decide how many is too many?"

It's quite hard. These things used to be configurable in IIS, but a few years back the IIS team had the realization that (a) you can't expect a human being to figure out what the ideal thread limit is, and (b) the ideal thread limit changes dynamically because of all the factors I mentioned above.

IIS therefore keeps a relatively small thread pool, and continually monitors its own performance, tweaking the thread pool count, thread priorities, etc, as conditions change. It tries to keep the server well-tuned dynamically.

This was non-trivial code to write. Though I've done quite a bit of optimization of software that runs in-process with IIS, the details of the IIS thread timing algorithms are way, way beyond my skills. I wouldn't recommend such an approach unless you've got a lot of experience in the field. A huge performance lab with a wide range of server hardware and a dedicated staff of experienced performance testers doesn't hurt either!

Speeding Can Slow You Down

Eric Lippert — Tue, 02 Dec 2003 01:35:00 GMT

I hope all you readers living in the United States had a restful and enjoyable Thanksgiving holiday. I sure did.

I've been meaning to talk a bit about some of the performance issues you run into when tuning massively multi-threaded applications, like the ASP engine. I'd like to start off by saying that I am by no means an expert in this field, but I have picked up a thing or two from the real experts on the ASP perf team over the years.

One of the most intriguing things about tuning multi-threaded applications is that making the code faster can slow it down. How is that possible? It certainly seems counter-intuitive, doesn't it?

Let me give you an analogy.

Suppose you have an unusual road system in your town. You have a square grid of roads with stoplights at the intersections. But unlike the real world, these are perfect traffic lights -- they are only red if there actually is another car in the intersection. Unlike a normal road, each road goes only one way, and has at most one car on it at a time. Once a car reaches the end of the road, it disappears and a new car may appear at the start. Furthermore, there is a small number of drivers -- typically one or two, but maybe eight or sixteen, but probably not one for every car. The drivers drive a car for a while, then stop it and run to the next car! The drivers are pretty smart -- if their car is stopped at a red stoplight then they'll run to a stopped car that is not at a red stoplight (if one exists) and drive it for a while.

In our analogy each road represents a thread and each stoplight represents a mutex. A mutex is a "mutually exclusive" section of code, also known as a "critical section". Only one thread can be executing in that code at one time. The car represents the position of the instruction counter in this thread. When the car reaches the end of the road, the task is finished -- the page is served. The drivers represent the processors, which give attention to one thread at a time and then context switch to another thread. The time spent running from car to car is the time required to perform a thread context switch.

Now imagine that you want to maximize the throughput of this system -- the total number of cars that reach the end of their road per hour. How do you do it? There are some "obvious" ways to do so:

hire more drivers (use more processors)
eliminate some stoplights by building overpasses at intersections (eliminate critical sections)
buy faster cars (use faster processors)
make the roads shorter (write pages that require less code)

You'll note that each of these is either expensive or difficult. Perf isn't easy!

Now, I said that these are "obvious" ways to solve this problem, and those scare quotes were intentional. Imagine a complex grid of roads with lots of stoplights, a moderate number of cars on the road, and two drivers. It is quite unlikely that cars will spend a lot of time stopped at stoplights -- mostly they'll just breeze right on through. But what happens when you throw another six drivers into the mix? Sure, more cars are being driven at any one time, but that means that the likelihood of congestion at stoplights just went up. Even though there are four times as many drivers, the additional time spent at stoplights means that the perf improvement is less than a factor of four. We say that such systems are not scalable.

Or consider a moderately congested freeway system with a whole lot of cars, drivers and intersections. Now suppose that you keep the cars, drivers and intersections the same, but you shrink the whole system down to half its previous size. You make all the roads shorter, so that instead of having eight stoplights to the mile, now you've got twenty. Does total throughput get better or worse? In a real traffic system, that would probably make things worse, and it can in web servers as well. The cars spend all their time at intersections waiting for a driver to run over to them. Making code faster sometimes makes performance worse because thread congestion and context switches could be the problem, not total path length.

Similarly, making the cars faster often doesn't help. In the real world of Seattle traffic, upgrading from my 140-or-so horsepower Miata to a 300 HP BMW isn't going to get me home any faster. Getting a faster processor and shorter program only helps if the "freeway" is clear of traffic. Otherwise, you sit there in your souped-up ultimate driving machine going zero at stoplights like everyone else. Raw power does not scale in a world with lots of critical sections.

When perf tuning servers, use the performance monitor to keep a careful eye on not just pages served per second, but on thread switches per second, processor utilization and so on. If you cannot saturate the processor and the number of thread switches is extremely high, then what is likely happening is that the drivers are spending way too much time running from car to car and not enough time actually driving. Clearly that's not good for perf. Tracking down and eliminating critical sections is often the key to improving perf in these scenarios.

Arrrrr! Cap'n Eric be learnin' about threadin' the harrrrd way

Eric Lippert — Fri, 19 Sep 2003 20:00:00 GMT

Arr, it be National Talk Like A Pirate Day. Avast ye scurvy dogs!

A scurvy bilge rat commented on the preceding discussion about putting apartment threaded objects in Session scope:

back in the era of the NT4 Option Pack I wrote a lot of code that involved stashing Scripting.Dictionary objects in both session and application scope. [...] I forget now which version of the runtime changed the threading model they were registered with and broke everything for me.

Shiver me timbers! That be my fault. Sorry about that. When you create an ActiveX object, the COM runtime code checks the registry to see if the object is marked as participating in the Apartment, Free or Both threading models. (We'll go into the difference between Free and Both at another time.)

Now, when I was a young swabbie seven years ago I was given the task of implementing the Scripting.Dictionary object, and I didn't yet understand all the stuff I just told you maties about threading. In one build that was released to the public I accidentally marked the dictionary as Both, even though it is an apartment object. So when lubbers would put a dictionary into Session scope, it would be called by multiple threads at multiple times, in violation of the apartment contract. As long as there were only readers, it was generally OK, but as soon as there were readers and writers, it would usually crash and die.

And of course when we corrected the mistake, all those pages went from sometimes-crashing-but-fast to not-crashing-but-slow. That was my first majorly customer-impacting mistake, and probably the worst I ever personally made.

Speaking of mistakes, there was another interesting performance mistake in early releases of the Scripting.Dictionary object. It uses a very simple hash table for rapid lookup, but of course hash tables require that the hash function distribute hashes broadly given a narrow distribution of keys. I screwed up the hash algorithm, and one of the consequences was that hashing on a string consisting of five digits was likely to go to a very small number of hash buckets.

We discovered all this the day that MSN.COM decided to store EVERY ZIP CODE IN THE UNITED STATES in a Scripting.Dictionary object in Session scope! Perf of MSN.COM went way south, way fast.

The combination of the two mistakes above led the ASP team to write their own string table object, that really was Both threaded and blindingly fast.

Arr!

Why is it a bad idea to put script objects in Session scope?

Eric Lippert — Fri, 19 Sep 2003 05:35:00 GMT

Often a web site will have a series of related pages requested one after the other by the same user. As a convenience for the site developers, the ASP object model provides a Session object to store server-side state for a current user. It also has a global "Application" object which stores state for an entire virtual root.

Every so often some ASP coder out there tries to put a JScript or VBScript object into the Session (or Application) object. Things usually start going horribly wrong shortly thereafter -- either terrible performance ensues or things just break outright.

Well, like Groucho says, if it hurts when you do that, don't do that! Understanding why this is a bad idea will take us yet deeper into the land of threading models.

Marshaling

I mentioned earlier that when you have an apartment threaded object, you need to be in the right apartment (thread) if you want to talk to its occupants (object instances). But what if you are not? What if you are running some code in thread Alpha that really needs to call a method on an object running in thread Beta? Fortunately, COM provides a mechanism called cross-thread marshaling to achieve this.

The behind-the-scenes details are not particularly important to our discussion; suffice to say that windows messages are involved. The important thing to know is that when you marshal a call across threads the calling thread pauses until the called thread responds. That seems reasonable -- after all, when you call an ordinary function you sort of "pause" until the function returns.

But threads are usually busy doing something. If the called thread is not responding to messages because it is busy doing work of its own then the calling thread waits, and waits, and waits...

To continue with our previous apartment threading analogy, it is rather like each apartment has a mailbox. If you're in apartment Beta and you need someone in apartment Alpha to do something for you, you write up a request and hand it to the mailman who in turn sticks it in Alpha's mailslot. Alpha's occupants might be busy doing work and ignoring their mail, or they may have a huge stack of mail to get through, or they might get right on it. You, over in apartment Beta, can't do anything but wait for the mailman to deliver their reply.

And of course, even if the callee thread is completely responsive and ready to service your request, obviously calling across threads is orders of magnitude more expensive than calling in-thread. An in-thread call requires some arguments to be put on the stack and maybe stash a few values in registers. A cross-thread call gets the operating system involved in a comparatively deep and complex way.

Performance Woes

Now you have enough information to figure out why putting script objects in Session scope is a bad idea as far as performance is concerned.

Each ASP page is running on its own thread from the thread pool. The thread that reads the object from session state will likely not be the thread that put the object there, but the script object is apartment threaded. That means that essentially any page that accesses that session object must wait its turn for all other pages using that session object to finish up, because the marshaling code blocks the calling thread until the callee is available. You end up with a whole lot of blocking threads, and blocking threads are not fast.

Application scope is even worse -- if you put a script object in Application scope then every page in the entire vroot that accesses the Application object must wait its turn for the original thread to be free. You've effectively single-threaded your web server.

Script Engine Of The Living Dead

But it gets worse. Remember, when the page is done being served up, the engine is torn down. The compiled state remains, but the runtime state is thrown away. So suppose you have a JScript object sitting in the Session object, and the page that put it there was destroyed a few microseconds ago and the engine put back into the engine pool. Now on another page in the same session you try to fetch the JScript object and call a method on it. Where's the runtime state associated with that object? It's gone, dude.

When the script engine is torn down after the original page is served, the teardown code detects that there are existing objects that are owned by external callers. The script engine can't destroy these objects, otherwise the caller would crash when the caller tried to destroy them later. That's a fundamental rule of COM -- you can't destroy an object to which someone holds a reference.

But we know that the object is going to be useless, so what we do is tell the object "the engine you require for life is going away. Throw away everything you own and become a zombie."

These zombie objects look like script objects, but when you actually try to do something to them -- call a method, fetch a property, whatever -- they don't actually do anything. They can't -- all the infrastructure they need to live is gone, but they can't die. Basically they wander the night in ghostly torment until they are freed by whatever code is holding the reference.

So not only are those script objects sitting in Session state wrecking your performance, they're not even useful for anything once the original page goes away.

Incidentally, Windows Script Components each have their own engine which stays alive as long as they do, so WSC's are not affected by the zombie issue. They are still apartment threaded though.

Arrays Are Almost As Bad

JScript arrays are objects, so everything said above applies to them. VBScript arrays are not objects (more on the differences between these two kinds of arrays later) but even still, you shouldn't put them in Session scope either. Though they do not suffer from the threading problems or the lifetime problems mentioned above, arrays are stored into and passed out of Session scope using our old friend copy-in-copy-out semantics. That means that every single time you index into an array in Session scope, a copy of the array is made first. If that array is big, that's a big performance problem.

Why do we do copy-in-copy-out? Because arrays are not marshaled! We return to the fundamental problem: what if two pages tried to read and write the array in the Session object at the same time? The memory could be corrupted. We really don't have any good way to synchronize access to the array, so instead we simply make a complete copy of it every time someone tries to read or write it. This is extremely expensive, but it keeps the server heap from being corrupted. A corrupted web server heap can ruin your whole day.

My advice is to not even go there. Don’t put information into Session scope unless you absolutely have to. If you must, put in strings. There are lots of ways to store arrays or objects as strings and reconstitute them as needed.

How does Active Server Pages use the script engines?

Eric Lippert — Fri, 19 Sep 2003 00:32:00 GMT

It's always struck me as a little bit odd that Active Server Pages, a web server, encourages developers to use VBScript and JScript to write server-side scripts. I mean, the whole point of a web server is that it produces complex strings (web pages are just strings of HTML after all) as blindingly fast as possible, on demand. "Blindingly fast" and "script language" do not really go together, compared to, say, C.

My opinions -- and you'd better believe I have plenty of strong ones! -- on performance characteristics of the script engines will certainly come back in future posts. But for now I want to talk a little bit about what features we added to the script engines to make them work as well as they do for ASP.

One of the ways you make a server fast is by caching everything you possibly can cache. For example, suppose you need to do a job on a separate thread. You'd probably create a thread, do some work on the thread, and throw it away when you're done. OK, now suppose you need to do a million jobs but never more than, say, three at a time. Creating and destroying those million threads is potentially going to be a not-insignificant amount of the total processor time spent. You'd be better off creating three threads and re-using them as necessary.

This strategy is called “thread pooling”, and ASP certainly uses it.

(UPDATE: I've written a follow-up article on Thread Happiness Disease.)

ASP caches a lot more than that though. It also pools script engines and caches compiled state. Let me explain with an example.

Suppose a request comes in for this page, time.asp:

Suppose further that this is the first-ever request for this page since the server was started.

The thing above isn’t a legal script, so ASP parses out the non-script blocks and stores them in the Response object. It then generates a real script that looks something like

Response.WriteBlock 0
Response.Write Now()
Response.WriteBlock 1

ASP also has a pool of created-but-not-initialized script engines sitting around. I said in my last post that when the script engines are not initialized they can be called on any thread, so it does not matter that these engines were not created on whatever thread happened to be pulled out of the thread pool.

ASP initializes the engine on the current thread and passes in the script code above. The script engine compiles the script, runs it, and the ASP object model sends the resulting string off to the client.

The script engine then gets uninitialized, but does not throw away its compiled state. The compiled state – the proprietary bytecode form of the language – is maintained in memory because the second time someone asks for that page, ASP do not want to have to go through all the time and expense of translating the page to script and compiling the script to bytecode.

The thread and the now-uninitialized engine go back to their respective pools.

Now suppose a second request comes in for time.asp. This time ASP pulls a thread out of the thread pool, notices that it has an engine with the compiled state just sitting there, attaches the engine to the thread, and runs the script again. No time is spent creating the thread, creating the script engine, translating the page to code or compiling the script to bytecode.

Suppose further that in the few milliseconds that ASP is running this script, a third request comes in for this page. Then what happens? There are two obvious possibilities:

1) Wait for the current engine to finish, and start it right back up again when it is done.
2) Fetch a new engine from the engine pool and start over – translate the page to script, compile, run.

The trouble is, both of those are potentially slow. The first in particular could be extremely slow. We needed a third option.

In my previous post I discussed the sorts of trouble you can get into with multi-threading. Note that the problem I described happens because two threads tried to both read and write a particular value. If the value was read-only, then obviously there would not have been a problem. You can read a read-only value from as many threads as you want!

Hey, once the script engine compiles a script into bytecode, that bytecode is read-only. It’s never going to change again. This means that two script engines on two different threads can share bytecode!

I lied earlier when I said that InterruptScriptThread was the only exception to our weird threading model. We have another exception – at any time, you can call Clone from any thread. This takes an existing script engine and gives you a new script engine with exactly the same compiled state, but able to run on a different thread. Each engine maintains its own runtime state – variable values, etc – but the compiled state is shared.

So in this scenario, ASP clones the running engine onto another thread and then runs the new engine concurrently. This is somewhat expensive – it has to create a new engine, after all – but is much cheaper than compiling up all that state again.

Of course, ASP is much more complicated than that quick sketch. What I’m getting at here is that there actually is some rhyme and reason to our bizarre-seeming choices of threading model.

The script engine design was driven by the need to solve a very specific set of very disparate problems. Without knowing what the problems were, the resulting design looks kind of random. But once you know the history, well, I guess it looks a little less random.

UPDATEL: I've written a follow-up article on how the ASP compilation model is affected by having multiple languages per page.

What are "threading models", and what threading model do the script engines use?

Eric Lippert — Fri, 19 Sep 2003 00:28:00 GMT

I've got a few ideas for some future posts that depend on the reader understanding a little bit about COM threading. Since I myself understand only a little bit about COM threading, I'll just do a brain dump for you all right here.

I'm sure you all know about multi-threaded applications. The idea is that the operating system switches back and forth between threads within a process using a scheduling algorithm of some sort. When a thread is "frozen" all the context for that thread -- basically, the values of the registers in the processor -- is saved, and when it is "thawed" the state is restored and the thread continues like it was never interrupted.

That works great right up to the point where two threads try to access the same memory at the same time. Consider, for example, the standard implementation of IUnknown::Release():

ULONG MyClass::Release()
{

--this->m_cRef;

if (this->m_cRef == 0)

{

delete this;

return 0;

}

return this->m_cRef;

}

Now suppose the ref count is two and two threads try to each do a single release. That should work just fine, right?

Wrong. The problem is that though --this->m_cRef looks like a single "atomic" operation, the compiler actually spits out code that acts something vaguely like this pseudo-code:

// "this" is stored in Register1

1 copy address of "m_cRef" field of pointer stored in Register1 to Register2

2 copy contents of address stored in Register2 to Register3

3 decrease contents of Register3 by one

4 copy contents of Register3 back to address stored in Register2

5 compare contents of Register3 to zero, store Boolean result of comparison in Register4

6 if Register4 is false then return contents of Register3

7 ... etc -- do deletion, return zero

Notice that the compiler can be smart and re-use the contents of Register3 instead of fetching this->m_cRef three times. The compiler knows that no one has changed it since the decrease.

So suppose we have a red thread and a blue thread. Each has their own registers. Suppose the processor schedules them in this order:

1 copy address of "m_cRef" field of Register1 to Register2

2 copy contents of address stored in Register2 to Register3

CONTEXT SWITCH! Save Register1 = this, Register2 = &m_cRef, Register3 = 2

1 copy address of "m_cRef" field of Register 1 to Register 2

2 copy contents of address stored in Register 2 to Register 3

CONTEXT SWITCH! Save Register1 = this, Register2 = &m_cRef, Register3 = 2

CONTEXT SWITCH! Restore Register1 = this, Register2 = &m_cRef, Register3 = 2

3 decrease contents of Register3 by one

4 copy contents of Register3 to address stored in Register2

5 compare contents of Register3 to zero, store Boolean result of comparison in Register4

6 if Register4 is false then return contents of Register3

CONTEXT SWITCH! Restore Register1 = this, Register2 = &m_cRef, Register3 = 2

And now you see where this is going, I'm sure. Because the original value was stored in the red thread before the blue thread decremented it, we've lost a decrement. Both threads will return 1. This object's ref count will never go to zero, and its memory will leak. A similar problem plagues AddRef -- you can lose increment operations, which causes memory to be freed too soon.

How do we solve this problem? Basically there are two ways to do it:

1) Do the necessary work to ensure thread safety

2) Require your callers to behave in a manner such that you never get into this situation in the first place.

The operating system provides tools to make multi-threaded programming work. There are methods like InterlockedIncrement, which really do "atomically" bump up a counter. Signals and semaphores and critical sections and all the other tools you need to make multi-threaded programs are available. I'm not going to talk much about those.

Writing a truly free-threaded program is a lot of work. There are a lot of ways to get it wrong, and there are potential performance pitfalls as well. Fortunately, there is a middle ground between "only one thread allowed" and "any thread can call any method at any time".

The idea of the COM threading models is to provide a contract between callers and callees so that, as long as both sides follow the contract, situations like the one above never come to pass.

Suppose a caller has several instances of an object (the callee), and the caller has several threads going. The commonly used standard threading contracts are as follows:

* Single threaded -- all calls to all instances of the object must always be on the same thread. There are no synchronization issues because there is always only one thread no matter how many object instances there are. The caller is responsible for ensuring that all calls to all instances are on the same thread.

* Free threaded -- calls to the object can be on any thread at any time, including multiple threads at the same time. The object is responsible for all synchronization issues.

* Apartment threaded -- all calls to any given instance of the object must always be on the same thread, but different instances can be called on different threads at the same time. The caller is responsible for ensuring that given an instance, all calls to that instance happen on the same thread. The object is responsible for synchronizing access to global (that is, not-per-instance) data that it owns.

An analogy might help. Think of an apartment building where each apartment is a thread and each person is an object instance. You can put as many people into one apartment as you want, and you can put people into lots of different apartments, but once you've done so, you always have to go to a person's apartment if you want to talk to them. Why? Because they never move out once they're in an apartment, you have to wait for them to die before they ever leave. (Insert New Yorker joke here.)

Furthermore, you can't talk "through the walls" from one apartment to someone in another apartment. (Well, actually you can -- that's called "marshaling", and that's a subject for a future post.) And finally, if the people jointly own a shared resource -- say, a rooftop barbecue, to stretch this silly analogy to its limit -- then they must sort out amongst themselves how to synchronize access to the shared resource.

* Rental threaded -- calls to an object can be on any thread but the caller guarantees that only one thread is calling into the object at any time. Rental threading requires a different analogy: suppose the object instances are rented televisions and again threads are apartments. A television can be moved from apartment to apartment but can never be in more than one apartment at the same time. Multiple televisions can be in the same apartment, and multiple apartments can have multiple televisions. But if you want to watch a television, you have to go to the apartment where it is.

Whew, that was a long preamble. How does this pertain to the script engines?

Most COM objects -- almost all ActiveX objects, and all of the object models commonly used by script -- are apartment threaded objects. They expect that multiple instances of the object can be created on multiple threads, but once an instance is created on a thread, it will always be called on that thread. This gives us the best of both worlds -- the caller can be free threaded and can create multiple objects on multiple threads, but the callee does not have to synchronize access to any per-instance data.

But the script engines are free threaded objects. The script engines must ensure that they do not violate the apartment model contract.

So guess what? The script engines actually have a bizarre, custom contract that is a little more restrictive than free threading and less restrictive than apartment threading!

The script engine contract is as follows:

* When the script engine is in a state where it cannot possibly call an ActiveX object -- for instance, if it has just been created and has not started running code, or if it is just about to be shut down -- then the script engine really is free threaded, but who cares? It can't do much in this state.

* When the script engine is initialized -- when the script engine host has started the process of passing code and object model state to the engine -- the script engine morphs into an apartment threaded object. All calls to the script engine must be on the initializing thread until the script engine is shut down again.

* There are two exceptions to the previous rule -- the InterruptScriptThread and Clone methods can always be called from any thread. The former is the mechanism whereby the host can signal a long-running script that it is taking too long and needs to shut down. Obviously that has to be on a different thread than the one that is taking too long! The latter is how ASP starts a second request for a currently-executing page to run on a second thread without recompiling the script.

* If the script host (the browser, for instance) violates this contract, the script engine will usually return the E_UNEXPECTED error, which has the helpful error message "CATASTROPHIC FAILURE". (Not that we're trying to scare you or anything; it's just our way of saying "please don't do that".)

Anyway, that was a whole lot of not particularly germane information unless you are developing your own script host. But like I said, I have a few ideas for some future topics that will assume some background understanding of COM threading models.