Fabulous Adventures In Coding : SimpleScript

SimpleScript Part Seven: Binder Skeleton

Eric Lippert — Tue, 04 May 2004 17:51:00 GMT

In Part Five I was discussing modules: there is a "global module" and any number of additional modules. Each module is associated with a named item, and the only module which is associated with more than one named item is the global module. This means that each module is going to need its own name table to keep track of the functions, the variable names and values, etc, in it. We'll call such devices "binders" because they bind names to values.

For reasons which will become more clear in future episodes, it is convenient to have the binder implement the IDispatch interface. Normally if you had to implement IDispatch you'd build a type library, and then have your implementation of IDispatch call through to the ITypeInfo methods which look up dispatch ids, invoke methods, and so on. There's a good reason for that: rolling your own IDispatch code can be extremely tricky. But, we're pretty much stuck with it here -- we haven't got the convenience of a known vtable interface to wrap a type library around at compile time. The script engine name tables are going to be extremely dynamic, so we're going to have to roll our own IDispatch.

The code isn't nearly done yet, obviously, but I've got a good skeleton in place. Take a look at binder.cpp for the details. I've pretty much got the high-level semantics of the dispatch interface worked out, but the actual low-level implementation details all return E_NOTIMPL. What I need to build here is a special hash table which can efficiently look up values by both numeric id and name. That's not too challenging; what is going to be more difficult is implementing the SCRIPTITEM_GLOBALMEMBERS feature. We need to be able to add another arbitrary dispatch object to our binder and dynamically aggregate all of its methods, without causing any collision in dispatch ids!

Default property semantics are a little weird. In the wacky world of late bound objects, an object may have a "default" method. This feature was created so that you could call the Item method of a collection the same way you dereference an array in Visual Basic -- you make it look like a function call. That is, since this code works:

Dim arr(10)
arr(1) = 123

then so should this code

dict = CreateObject("Scripting.Dictionary")
dict(1) = 123

the last line is the same as

dict.item(1) = 123

Such is the wackiness of VB -- because the array dereference and function call syntax are conflated, they ended up making function calls on invisible methods be legal on the left hand side of an assignment statement. The mind fairly boggles. We'll be good citizens in SimpleScript. If we have an object in our binder and someone attempts to call it, we'll just pass on the arguments to its default method and let it sort out what the right thing to do is.

As you can see, there is a whole lot of parameter validation, and altogether about eleven separate cases that I cover in the implementation of Invoke. The actual implementation in VBScript and JScript is much, much more complex than this because they support IDispatchEx, garbage collection, property accessor functions, and numerous other features that complicate the code. But still, this is the longest function you're likely to see me write in this project; I'll be very surprised if I write any code more complicated than the dispatch logic.

Speaking of garbage collection, I'm not quite sure what kind of garbage collector I'm going to build into SimpleScript. Building a mark-and-sweep collector like JScript has would be instructive, but it would also take a lot of time and effort. What do you guys think? (If you look at invoke.cpp you'll see that I'm thinking ahead about issues we're going to run into in garbage collection. We're going to run into related issues in the script execution code; it is possible for ill-behaved hosts to shut down the script engine in a callback while it is executing! We need to be robust in the face of such behaviour.)

The next step is to get the actual guts of the binder working, and then build a list of code blocks to be executed when the script engine goes to started state. The clone semantics for the code blocks are a little tricky, so we'll get that straightened away, and then maybe, just maybe, actually write a language parser.

UPDATE: For your convenience, I've put a zip file containing all the sources at http://www.eric.lippert.com/simplescript.zip. I'll keep this zip file updated as I change the sources on the blog.

SimpleScript Part Six: Threading Technicalities

Eric Lippert — Tue, 27 Apr 2004 03:56:00 GMT

Refresher Course

Before you read this, you might want to take a quick refresher on my original posting on the script engine threading model. That was a somewhat simplified (!) description of the actual script engine contract. Let me just sum up:

free threaded objects can be called from any thread at any time; the object is responsible for synchronizing access to shared resources
apartment threaded objects can have multiple instances on multiple threads but once an object is on a thread, it can only be called from that thread. The caller is responsible for always calling an object on the thread on which it was created. The object is responsible for synchronizing access to resources shared across instances.
rental threaded objects can be called on any thread, but the caller is responsible for ensuring that the object is only called from one thread at a time.
an initialized engine is apartment threaded, with a couple exceptions -- InterruptScriptThread and Clone can both be called from any thread on an initialized engine.
an uninitialized engine is free-threaded

Things Get More Complicated

That's actually not quite right. It would be more accurate to say that an uninitialized engine is rental threaded. Why? Because otherwise it would be legal to do really dumb things like call Close on two different threads at the same time. If you look carefully at the code, you'll see that most of the methods are not robust in the face of full-on multithreading. It's the "the host isn't a bozo" threading model!

This is a ridiculously complex threading contract, I know. From COM's point of view, the script engine is free threaded -- the restrictions are so arcane that clearly the engine has got to be the one enforcing the rules, not COM, which is why you'll see lots of calls to check that the caller is on the right thread in my code.

If you take a look at the registration code, you'll see that I register the script engine as "Both", which means "either free threaded or apartment threaded, we'll sort it out". What the heck is up with that? Why not just call it "free threaded" and be done with it?

Because again, I oversimplified the description of an apartment in my original posting. A single-threaded-apartment (STA) object is created on a thread and is always called on that thread. Think of a person (an object) in a room (a thread) -- you want to talk to them (call a method), you go to their room. You can put as many people in a room as you'd like, and build as many rooms as you'd like, but you want to call a method, you do it from the thread where the callee lives. That's all fine and good.

But there is also a multi-threaded apartment! Imagine that you take some of the rooms and you knock holes in the walls. If you're in one room and you want to talk to someone in another room, you don't have to go there, you can just yell through the hole. The guy listening to you is responsible for synchronizing all the shouting going on, but at least he knows that no one is going to be trying to talk through a wall with no hole in it.

In any given process there is one "main" STA, possibly many more secondary STAs, and one MTA. You can't have two distinct sets of rooms that mutually communicate but don't talk to each other.

I briefly described in an earlier post the way that COM marshals calls "through the walls" from one apartment to another. Well, if we register the script engine as a free threaded object, COM is going to think that it lives in the default MTA, and that any STA objects created by the process live in their own STAs, and therefore, all calls between the two apartments are going to have to be marshaled by the Free Threaded Marshaler. That extra indirection really screws up your performance numbers, lemme tell ya. We register as "both" threaded, and COM says "OK, you sort it out then if you're so smart", which we do by requiring that the host call us on the right thread and give us objects that are on the right thread.

Even that is still a considerable oversimplification -- there's still the Neutral Threaded Apartment that I haven't talked about yet, and the interactions between the CLR threading model and the COM threading model, and how marshaling really works, but that's getting out of my depth. Go ask Christopher Brumme if you've got questions about that stuff.

Honouring Our End Of The Script Engine Contract

Let's take a look at the script engine contract through the example of one of its objects that we've already implemented -- the named item list. There are only four operations that can be performed on the named item list, and we know when they can be performed according to our contract and implementation:

Add can only be called on an initialized engine, and hence only from the engine thread. Add writes to the named item list.
Reset can only be called on an initialized engine as it is being moved to uninitialized state, hence only from the engine thread. Reset writes to the named item list by removing non-persisting entries.
Clear is only called when the engine is going to closed state. The engine might already be in uninitialized state and hence, this can be called on any thread if the engine is uninitialized. However if the engine is initialized then it can only be called from the main engine thread. Clear writes to the named item list by removing all entries.
Clone can be called on any thread, and reads from the named item list.

What then are the possible threading conflicts? The three writers, Add, Reset and Clear cannot be called at the same time on different threads by virtue of the fact that the engine must be initialized if Add or Reset are being called. There are a few cases which for completeness we should get right, but are in reality extremely unlikely. Why would any host be so dumb as to Clone an engine while in the middle of a call to Add? Or worse, Clone during Close? I won't assume that hosts won't pull shens like that, even though they are very unlikely.

What about two Clones at the same time on two different threads? On the one hand, they're only reading, so why should they block? On the other hand, boy, do I ever not want to implement single-writer-multi-reader mutexes just to make that extremely unlikely case marginally faster.

Therefore, we'll do it the easy way. The only thing we really need to worry about practically is one thread doing a Clone while another thread is doing a Reset, but we'll get it right for all the cases. The first thing we'll do when we enter any of those methods is enter a critical section, and the last thing we'll do before we leave is exit it. Rather than mess around with the operating system's somewhat gross critical section code in the object itself, I define a handy object to wrap it. See mutex.cpp for the implementation.

Something to note about this implementation is that it uses InitializeCriticalSectionAndSpinCount to initialize the critical section. The comments there and here are required reading if you need to make critical sections work on heavily loaded pre-Windows-XP boxes. Earlier versions throw exceptions rather than returning error codes, which means that every entry and exit to a critical section has to be protected with __try blocks and you then have to get the exception handling right! The VBScript and JScript engines have all kinds of totally gross code in them to handle the edge case where a heavily loaded server runs out of memory just as a critical section is about to be entered. (Yes, it happens. Every single out-of-memory case will eventually be exercised by a sufficiently loaded server, I know this from painful experience.)

I'm going to skip all that totally gross code here and assume that we all live in the happy world of Windows XP, where the operating system actually returns sensible errors.

I'm still trying to sort out how all this is going to work once code blocks are throw into the mix. I'll try out a few things and see how it goes. More bulletins as events warrant.

SimpleScript Part Five: Named Items and Modules

Eric Lippert — Fri, 23 Apr 2004 00:29:00 GMT

Named Items

"Named items" are what we call the "top level" objects of the host provided object model. WScript in WSH, window in Internet Explorer, Response in ASP, are all named items. A host tells an initialized script engine about named items via the aptly named AddNamedItem method on the IActiveScript interface.

HRESULT ScriptEngine::AddNamedItem(const WCHAR * pszName, DWORD flags)

A few things should immediately seem a little weird about this interface.

The First Four Flags

First off, what are the flags? There are six. The first four are pretty straightforward:

If SCRIPTITEM_ISVISIBLE is set then the name of the named item is added to the global namespace.

Uh, OK… why would you ever NOT want this set? Didn't I just say that named items were specifically for injecting named object model roots into the engine? What good is a named item if you can't see its name?

That brings us to the second flag; if SCRIPTITEM_GLOBALMEMBERS is set then all (immediate) children of the named item are treated as though they are themselves top-level objects/methods. That's how in Internet Explorer you can say

window.alert("hello");

alert("hello");

and they do the same thing. IE tells the script engine that window is a visible named item with global members.

Now it makes a little more sense why you might want to have an invisible named item. What if you had an object with lots of methods and properties that you wanted available in the global namespace, but the object itself didn’t have a sensible name? I can't think of any script host offhand that does that, but the capability is there if you need it.

There's a second reason why you might want a named item to be invisible, but we'll get to that in a minute.

The third flag is SCRIPTITEM_ISSOURCE. If that's set then we know that this object is an event source. If the language supports implicit event binding and the host moves the engine into connected state, we're going to need to know which named items to hook up to which events. It can be very expensive to do that hookup, so this provides a simple optimization. If the host knows that a particular named item does not source events, it can choose to not mark the named item as a source, and we therefore never spend any time trying to hook up event sinks to it.

The fourth flag is SCRIPTITEM_ISPERSISTENT. Recall that I said a while back that when the engine goes back to uninitialized state after being initialized, we throw away "some" named items, where "some" was to be defined later. Now you know -- the engine remembers named items marked as persistent. Information about those named items is not thrown away until the engine is closed. Also, cloning an engine is basically making a copy of the uninitialized state of an engine, so persistent named items get cloned when their engine gets cloned. As we'll see, this fact has implications for our implementation.

Modules

I'm sure you have a general idea of what I mean by a "module", though if you Google define:module you'll see that everyone has a slightly different definition. Modules in the script engine sense are philosophically fairly straightforward. I often want to have some way to say "this collection of functions can play with each other, but are isolated from this other collection of functions". I want to be able to resolve name collisions by having two methods with the same name coexist in different modules.

Well, I want that stuff in languages designed for "programming in the large". When using script languages, more often than not you simply don't need to chunk stuff into modules. But, bizarrely enough, the script engines support modules, and in a pretty goofy way at that. Of course, any language implementor can implement module semantics however they want, but I see no reason to mess around with the de facto standards. Here's how modules work in VBScript and JScript:

There is a "global" module. Procedures defined in the global module are callable from any module. This is where the "built in" methods in VBScript and JScript go.
Every named item is associated with a unique module, with some exceptions:

Named items with global members are associated with the global module.
Named items marked with SCRIPTITEM_NOCODE are not associated with any module

All visible named items are added to the global namespace, except for named items marked as SCRIPTITEM_CODEONLY. Those are just names of modules and are associated with no object.
Important distinction: though all visible named items are added to the global module's namespace, procedures in the named items' modules are not visible from the global module.

Let me try to make this a little more concrete. Let's consider a hypothetical declarative language that supports embedded imperative script. You might want to have something like this:

<application>
      <script>
            function toggle ( x )
               if (x == false) return true else return false
      </script>
      <form name="fred">
            <checkbox name="chuck" checked="false" />
            <button name="bob">
                  <caption>Hello world!</caption>
                  <click>
                        fred.checked = toggle(fred.checked)
                  </click>
                  <script>
                        function foo()
                        // button-specific code here

And so on. Suppose the host defined fred as a visible named item with global members, bob and chuck as named items each with their own module. Then code in bob's event handlers can access anything in the global application namespace and any member of fred. But if we now add script to chuck, chuck's module cannot see bob's code. chuck is free to define its own function foo, which will not collide.

What if bob wants to access code in chuck's module? We'll see in a later entry how chuck can do that -- there are many subtle issues involved and we'll need more infrastructure built into SimpleScript before we can tackle it.

Where's the Object?

One more weird thing -- where's the object? The method takes a name and some flags, but no object.

Let me answer that question with another question: suppose a persistent named item's object is thread affinitized, and you clone the engine and initialize the clone on a different thread? That thing can't use the object from the previous thread!

To solve this problem, we defer getting the actual object until we need it. The engine calls the site back and asks for the object. I haven't implemented that code yet; I'll talk more about that next time.

The Implementation

I've added a named item list object to SimpleScript in nameditemlist.cpp but I haven't quite figured out how I want to implement modules yet.

As you can see, I've rolled my own hash table rather than using hash_map. There was quite a long series of comments the other day about the pros and cons. I chose to roll my own rather than using an off-the-shelf solution for several reasons:

the implementation does not require a rocket-science hash table. It's very unlikely that we're going to get a huge number of named items in a script engine. Even very complex web pages seldom add more than a couple dozen named items. (In IE, every button, form, etc, is a named item.) Thus, I've implemented a fixed-number-of-buckets hashing-with-chaining table with a very simple string hashing algorithm.
I really have no clue how hash_map works, and I want to spend zero time debugging my way through acres of template code if there's a problem.
I realized that I was starting to succumb to Object Happiness as I was building a templatized hash table of my own. I'm going to need two or three hash tables in this project all told. And its not like templates actually make the generated code smaller or more efficient (though in C#, generics can -- but that's another story.) I'm going to keep it simple and avoid getting Happy.
hash_map requires that new throw exceptions. As I've mentioned before, that gives me the shakes. I don't like mixing C++ exception handling into what is fundamentally a COM program.
I think hash_map is threadsafe for the applications I'm going to use it in. I'm not sure. That makes me nervous. I am sure that my own code is going to be a lot more amenable to analysis with respect to the goofy script engine threading contract.
I ended up spending WAAAY more time investigating the template code than it took me to write the hash table from scratch. I'm doing this in my spare time here folks...

All those things totally outweigh the negligible benefits of re-using an existing generic map, no matter how bulletproof and performant it is.

That reminds me, I wanted to talk more about the script engine threading model. My earlier post on the subject glossed over some important details.

My team hit our Whidbey Beta One Zero Bug Bounce yesterday, so I'm off for a long weekend far, far away from computers. I'll cut it short right now; next week I'll pick it up again and talk a bit more about the threading model, the implications of engine cloning, and I'll kick around some ideas about how to implement modules and code blocks.

SimpleScript Part Four: Finite State Machines and Script Engines

Eric Lippert — Sun, 11 Apr 2004 00:18:00 GMT

Last time I said that I'd discuss finite state machines (also sometimes called finite state automata, same thing.) The FSM is a fundamental idea in theoretical computer science because it models computing machinery in a very simple, abstract and general way. Basically it goes like this: an FSM has a finite number of states (duh). Each state accepts a finite number of inputs. Each state has rules which describe the action of the machine for every input. An input may cause the machine to change state.

That's extremely general, I know. Let's consider a specific example -- a machine which has two inputs: a slot that can take in quarters, and a button. It has two outputs: quarters and gumballs. And it has two states: GOTQUARTER and NOQUARTER. We can chart the actions given a certain input:

Insert Quarter

Press Button

NOQUARTER

Change state to GOTQUARTER

Do nothing

GOTQUARTER

Output the quarter

Output gumball,

Change state to NOQUARTER

That's a pretty darn simple machine. Modern CPU's are also finite state machines -- after all, every bit in a machine can only be in two states, and there are only so many possible binary inputs from mice, keyboards, hard disks, and so on. The number of possible states a modern computer can be in is mind-bogglingly huge -- far more than the total number of particles in the universe -- but because the rules for the state transitions are very well defined it all works out nicely. (Because the number of states is so large, it is often more helpful to model machines as "Turing machines", which can have unlimited internal storage. But I'm digressing.)

The script engines use this idea of finite state machines explicitly. A script engine has six possible states, and it performs certain actions as it transitions between those states based on "inputs". In this case, the inputs are via calls to the various methods on the script engine. The six states are Uninitialized, Initialized, Started, Connected, Disconnected and Closed. A script engine is created by a host -- Windows Script Host, Internet Explorer, Active Server Pages, whatever. The host is responsible for changing the script engine state as it sees fit, within the rules I'll describe over the next few entries.

The script engine begins life Uninitialized. An uninitialized engine does not know anything about its host, and therefore cannot call any methods provided by the host. There's very little you can do to an uninitialized engine, but what little you can do, you can do from any thread, subject to the rental model. I'll talk more about threading issues later.

There's only one way to initialize an engine, and that's to pass a pointer to the host into SetScriptSite. Once an engine is Initialized it becomes apartment threaded, with a few exceptions that I'll talk about later. There is a lot more you can do with an initialized engine -- add named items (again, more on those in the future), add script source code, and so on. However, the code cannot actually run yet. At this point, the most commonly used input that affects the script engine state is the appropriately named SetScriptState method. More on that below.

A Started engine actually runs the script code, but does not hook up event handlers. (I discussed implicit event binding last time.) Moving the engine into Connected state hooks up the event handlers. If the host wants the event handlers to temporarily stop handling events, the hose can move the engine into Disconnected state.

That's the usual order in which things happen at startup. To tear down the engine, the IActiveScript::Close method tears down event handlers, throws away compiled state, throws away the reference to the host and moves the engine to Closed state.

The actions associated with SetScriptState are easiest to summarize in a table. (Except for illegal calls, this also sets the engine state to the new value.)

Old State	New State passed to SetScriptState
	Uninitialized	Initialized	Started	Connected	Disconnected	Closed
Uninitialized	Do nothing	Error -- use SetScriptSite	Error - can't start an uninitialized engine	Error	Error -- can't disconnect an unconnected engine	Error
Initialized	Discard some named items Throw away host information	Do nothing	Execute script code	Execute script code Bind events	Error	Discard all named items and script blocks Throw away host information
Started	Discard some script blocks, and as above	Discard some script blocks	Do nothing	Bind Events	Error	As above
Connected	Discard event bindings, and as above	Discard event bindings, and as above	Discard event bindings, execute code	Do nothing	Suspend event handling	Discard event bindings, and as above
Disconnected	As above	As above	As above	Resume event handling	Do nothing	As above
Closed	Error	Error -- use SetScriptSite	Error	Error	Error	Do nothing

As you can see, I've written methods that implement these state semantics in engine.cpp, though plenty of them are still stubbed out.

Next time, I want to talk more about the threading model, explain away a few oddities, and then we'll change gears for a bit and implement a simple language. That'll get us into language design, how to expose syntax errors, and so on.

SimpleScript Part Three: Engine Skeleton

Eric Lippert — Tue, 06 Apr 2004 00:32:00 GMT

Before I get into today's topic I want to quickly address a minor defect that Raymond Chen (who apparently spends his holiday in Sweden doing remote code-reviews, oddly enough) was good enough to point out to me. It is legal to unload a DLL if the only objects that depend on that DLL which are still outstanding are class factories. Therefore, creating the class factory shouldn't add a reference to the DLL. If the consumer of the class factory wants to explicitly lock the DLL in memory, they can use the locking mechanism. This code is wrong:

ClassFactory::ClassFactory()
{
m_cref = 1;
DLLAddRef();
}

Why is that the case? Why not just use the convention that class factory instances lock the DLL, and if you want the DLL locked, you keep the class factory alive? Because, though that logic makes perfect sense if the class factory is constructing in-process objects, it does not work if the factory is for a local (out of process) server. How's that? Because a local server creates class factories immediately when it starts up and destroys them only when it shuts down. If creating a class factory locked the local server in memory, then there's an obvious circular reference; the local server would never be shut down. Thus, the convention for all class factories is that they never lock the code in memory unless asked to, whether they're in-proc or not.

I've fixed the defect. Thanks again, Raymond.

I've uploaded two new files, engine.cpp and engine.h (and hooked it up to the class factory). Right now the engine code just knows how to create and destroy itself; no interfaces other than IUnknown are implemented. As you can see, over the next few entries I'm going to flesh out IActiveScript, IActiveScriptParse, IActiveScriptParseProcedure2 and IObjectSafety. I already covered the purpose of IObjectSafety in detail in my series "Script and IE Security", which you can read in full by going to my security archive. Part Two has a description of the interface. Why are there three script interfaces instead of one, and what's with that "2"?

When the original script team -- this is before my time! -- designed the interfaces they realized that there were some things that were going to be common to every possible engine. Every engine would need a way to communicate back to the host (SetScriptSite), start and stop the engine (SetScriptState) and perform various threading tasks (GetScriptThreadId, etc.) But how do the script engines do compilation-model dependent things such as compiling up new source code? Different engines might expose different interfaces to do these kinds of things, so the interface designers left one "base" interface, and if the host wants to use a particular compilation feature, the host and the engine can negotiate via QueryInterface.

Thus, IActiveScriptParse exposes two methods which expose compilation model features. ParseScriptText takes in raw source code, parses it, and adds the resulting code to the internal state of the script engine. If the code contains "global code" it might run, and if it contains procedures then they become available. This is what is called when you stick a chunk of script inside a plain vanilla <script> tag in Internet Explorer.

ParseScriptlet is very similar, but different -- it takes in source code for the body of an event handler procedure and adds a new event handler to the internal state. This is what happens when you have a <script for="… block.

ParseProcedureText, on the IActiveScriptParseProcedure2 interface is similar to ParseScriptlet in that it takes the body of a procedure. It not only adds the function to the global state, it returns an object which, when invoked, calls the function. In .NET-speak, it returns a delegate. This is handy in the IE event binding model where you bind events to an object by assigning a delegate to a property on the object.

The reason for the "2" is perfectly straightforward. When we first implemented IActiveScriptParseProcedure, we left the "name" argument off of the method. In a later release, we got a feature request to allow the host to control the name of the delegate, so we created a new interface and called it "2". That's all straightforward. But if you look in the activscp header file, you'll see IActiveScriptParseProcedure, IActiveScriptParseProcedure2 and IActiveScriptParseProcedureOld. What's up with that, and which ones should we implement?

The reason why we have IActiveScriptParseProcedureOld was going to be yet another entry in the Aargh! series, but I might as well just tell the tale of woe now.

A good design rule in COM is "never make an important decision based on the absence of an interface." Why? Because if you do, then you are risking that your program breaks if someone implements that interface in the future. I learned this rule the hard way because very early versions of Internet Explorer made exactly that mistake.

In the early days, VBScript did not support the "delegate" model of event binding at all; VBScript uses what we call the "implicit" event binding mechanism whereby we scan the VBScript code looking for procedures with names that match the pattern "objectname underbar eventname". When we find them, we construct the appropriate hookup code internally and everything is just fine.

However, doing that search has a potentially large performance cost. If the host decides that it doesn't need the implicit event binding model, the script engine exposes a way to prevent it from happening. We'll discuss the details later, but for now, I'll just say that if the host wants the feature they "connect" the engine and if they don't, they "start" the engine.

Also back in the early days, JScript did not support the "implicit" model. The IE developers therefore implemented the following (bad!) logic:

If the engine does not support "delegate" event binding, then "connect" the engine, otherwise "start" the engine.

That was the logic that shipped with IE4. One day the IE5 team came to me with a feature request -- they wanted to add "delegate" style event binding to VBScript in IE5. Piece of cake -- it was actually very easy to implement it in VBScript. A few lines of code, add the IActiveScriptParseProcedure2 interface implementation, fire it up, and… uh oh. Implicit event binding suddenly stopped working in VBScript. Aargh! It was drivin' me nuts! By making a decision to turn a feature off based on the absence of an interface, IE ensured that we could not implement that interface, ever!

The first thing that came to mind was "let's fix that in IE5", which, of course, we did -- IE5 used a much more sensible algorithm to determine whether to connect the engine or not. Problem solved? No. That wasn't good enough, because at the time it was a supported, legal, by-design scenario to install new script engines without installing a new browser. That meant that if an IE4 customer got the new VBScript engine, their implicit event handlers would stop working!

What we needed to do was to retire the old interfaces and ensure that no one would ever use them again. Normally we'd create IActiveScriptParseProcedure3, but I wanted to ensure that no one accidentally use the old interface, so I renamed IActiveScriptParseProcedure2 to IActiveScriptParseProcedureOld, and created a new, different IActiveScriptParseProcedure2. Only JScript implements IActiveScriptParseProcedure and IActiveScriptParseProcedureOld. If you implement those interfaces then IE3 and IE4 will not connect your engine. But given that IE3 and IE4 are a pretty small percentage of the browser market these days, we will dispense with them entirely and only use the new IActiveScriptParseProcedure2.

I've seen this same mistake in persistence code. (In fact, I think I may have made this same mistake myself in the VSTO2 persistence code. I need to double check that...) It's very easy to say “If this object doesn't support such and such an interface, reconstitute it as follows“ and then have the reconstitution code broken when a later version of the object adds that interface.

Coming up: soon I'll start to flesh out the engine a little more, and add some multi-threading support. I'll also discuss the theory and practice of the "state machine" style of programming and how the host initializes the engine.

SimpleScript Part Two: Class Factories Are Also Boring

Eric Lippert — Fri, 02 Apr 2004 04:51:00 GMT

Before I get into it, a Lambda poster pointed me at the NullScript project, which is a very interesting illustration of how reverse engineering works. It's an implementation of a "null" script engine -- an engine with no language -- in ATL, which the intrepid developer created in order to try and understand how ASP works. Typical programmer! They could have just asked me, but where's the fun in that? J

I want to go well beyond the scope of the NullScript project, in several ways.

First off, someone asked why not use ATL? Aside from my general distaste for the ATL style of programming, the more fundamental reason is that ATL is all about hiding the details of how COM works from you. One of the points of this exercise is to show how it works. In ATL, you see

COM_INTERFACE_ENTRY(IDispatch)

And what does that tell you? It's just a black box, and when you open it up, it's full of weird macros that I don't understand. I'd much rather show you guys how this works at a much less abstract level.

Second, the point of NullScript was to have the smallest feature set that still worked, because it was being used as a probing tool, not a language tool. I want to actually talk about practical concerns here, like good language design, how to implement IDispatch correctly, etc, not just write a logging tool.

Today, more boilerplate code. I've added a class factory.

For those of you who don't know much about COM internals, you might wonder how instantiating a COM object works. Basically, it goes like this.

First, there's a progid -- the human-readable string that describes the object you want to create. As I mentioned yesterday, we create registry keys for the progid that map it to a class id…

HKCR\SimpleScript\CLSID\(Default) = "{...}"

… and then map that class id to a DLL…

HKCR\CLSID\{...}\InprocServer32\(Default) = "c:\simplescript.dll"

Thus, the registry has enough information to determine everything you need to know to actually get the code in memory -- the location of the code and the unique identifier for the class.

To create the object, COM loads up the DLL and calls DllGetClassObject in dllmain.cpp. It doesn't ask for an object instance, as you might expect. Instead, it asks for a class factory -- an object that creates objects. The class factory in classfac.cpp then knows how to create the actual object. Why this indirection? Because DllGetClassObject is fundamentally not extensible but COM objects are extensible -- you can add any interface on that you like -- thus the convention is to create a COM object that does the creation work rather than put more smarts into the entrypoint. For example, the object might implement IClassFactory2, which enables licensing semantics.

If you take a look at the implementation, you'll see that we have your basic class factory going here. There's nothing fancy.

You might wonder what the locking mechanism is for -- this is for the scenario where you know that you're going to be creating a whole bunch of objects and need to ensure that you're not unloading the DLL unnecessarily. That's an extremely unlikely scenario for us, but I've implemented the code anyway because its cheap, easy and the right thing to do.

As you can see, all of the objects are thread safe so far. We'll get into the threading model of the script engine in more detail later, but as a refresher, you might want to read this.

The class factory does everything but actually create the engine -- that's still E_NOTIMPL.

How far can we get now? If we compile up the code, register the DLL and run it through WSH:

<job>
<script language="SimpleScript">
Testing
</script>
</job>

Then we get the DLL loaded, the class factory created, and a call to create the engine, which fails:

Windows Script Host: An unimplemented function was called : SimpleScript

We're doing pretty well so far, but we're still a long way from computing 2 + 2 or writing "hello world".

Next time, I'll talk about the engine and site interfaces, engine state, and a skeleton of the engine interfaces, which I'll then flesh out over several entries.

SimpleScript Part One: DllMain is Boring

Eric Lippert — Thu, 01 Apr 2004 06:01:00 GMT

In talking with our support engineer it's just become more muddled. I'm pretty sure now actually that the customer does not want to build a script engine, but whether they want to build a script editor, a script host or a script debugger is unclear.

Before I go on, let me take this opportunity to say that anyone who wants to create a script debugger from scratch really is crazy. It took us many programmer-years to implement the Microsoft Script Debugger. There are dozens of interfaces that you need to get right. A debugger, even a simple, basic one like the MSD is a very complex piece of software. I don't understand how the MSD works, and therefore I'm not about to try and explain it to anyone else! Unless you have a lot of experience developing debugging technologies and understand it inside out, I'd recommend against going anywhere near the script debugger interfaces from the debugger side. From the engine and host sides, maybe, but not the debugger side.

I'm going to forge boldly ahead with my plan to develop a script engine from scratch. And if that works out, maybe we'll do a script host as well. This will take some time, be warned.

I'm going to post the code so far as "articles" so that they don't show up in the feed, and then call out any "interesting" parts of the code as posts. I've just dropped in the first few files. So far we can register and unregister the script engine, but not actually create the engine yet.

The code should compile against VC6 or above, and I'm only testing it on Windows XP.

I don't normally like to get all legal, but I think this would be a good time to point out that standard disclaimers apply. This code is provided as a public service. I'm writing it from scratch in my spare time, and I don't have a phalanx of Microsoft testers ensuring that everything works. Code is provided as-is, with no warranties expressed or implied, if you compile it and run it and the world ends, that's not my problem, etc, etc, etc. Aside from that, you're free to do whatever you want to this code.

OK, now that we've got that out of the way, let's take a look at what's going on here. So far this is just your basic Windows DLL. This is pretty boring, but hey, we're just getting going here.

In simplescript.def we've got some boilerplate export code that says that this DLL exports the four standard functions to start up, register, unregister and shut down the DLL.

In guids.h I define some useful guids. So far the only thing there is the class ID for our new engine, because we'll need that when we register it. Since the guid is a structure, it must both be declared as a symbol in the header, and somewhere actually turned into object code. The very short guids.cpp performs the latter task.

In headers.h we've got your straightforward "include the world" header file. It might be wise to turn this into a precompiled header when this thing starts getting huge, but we'll cross that bridge when we come to it.

I am paranoid about putting assertions into my code. The point of an assertion is to document in an active way what is known to be true about a program -- not what we hope is true, not what is true most of the time, but what must logically be true. Assertions are better than comments because assertions will tell you if they are ever violated -- comments will not! In assert.h you'll see that I define a macro that determines whether a given condition is true, and some special-purpose macros that check for things like "is this memory valid?" More on those in a minute.

I sense the outrage -- didn't I just say that I hated macros? Indeed I do, but in this case, I really like the ability to automatically determine the file and line number, and I want this stuff to be zero-impact in the retail build, so in this case they're worth it. Notice also that I am using macros in a very specific way. I'm not introducing new control flow primitives, etc. No one is ever going to use this thing in an expression. Also note that, where possible, the macro immediately calls a method which can be debugged.

If you take a look at those methods in assert.cpp you'll notice a few oddities. First off, you'll notice that I do not call IsBadWritePtr or IsBadReadPtr, because these guys are incredibly gross. Basically, the way that IsBadFooPtr works is that it simply attempts to do the action, wraps the attempt in a try-catch, and returns true if an exception is thrown. If the memory really is bad because you're at the end of the stack then this can potentially fault the stack in a bad way. If you're in a multi-threaded scenario then IsBadWritePtr can mess up your program due to race conditions.

Now, I realize that in this case we're talking about some debug-only code here -- if either of these ever return true then there's a major bug that's got to be fixed! But still, they give me the shivers. I would much rather have my assertions actually ask the operating system whether that's a good pointer than "try it an see" the way IsBadFooPtr does. (And in fact there is some code in the script engines that needs to check whether a pointer is bad, and it is a main-line common scenario that it will be -- in a not-debug-only scenario like that you cannot rely on the IsBad methods. Why we have to do that is another story, which I may blog about some time.)

And finally we come to the first bit of code that actually has semantic import -- the DLL startup, shutdown and registration code in dllmain.cpp.

The shutdown code is straightforward -- as you will see in the next episode, we will maintain a reference count on the DLL and only shut it down when there are no outstanding references. The startup code is even more straightforward -- we simply cache the module handle in a global variable.

It is extremely important that the startup and shutdown code be boring, boring, boring. If you don't understand why that is, or you feel tempted to put something cool in there, read this, this, this, this, this and this.

The registration code is the only code that's interesting at all here, and even it is pretty boring. First off, take a look at the coding style of DllRegisterServer. Except in some specific exceptions, the code I write in this sample is going to be brain-dead obvious insofar as the error code paths and object lifetimes are concerned. Almost every method I write will follow this pattern:

if internal method, assert arguments are good. If public method, check arguments for validity.
Initialize everything that needs to be freed later to NULL.
Initialize out parameters to NULL.
Do stuff. If you get an error, go to the bottom of the function
Fill in “out” parameters
Free everything

Glancing through the code now, I see that there are a few places where I forgot to assert that input strings were valid. I'll fix those up later.

Does this technique produce long, boring routines that emphasize the error handling and make it harder to see what the function is doing in the mainline case? Yes. But error handling and cleanup is sufficiently important that it is worth calling out -- I need to make sure that it is right, and the best way I know to do that is to make it obvious what everything is doing. And besides, I'm going to keep the routines short enough that it should be fairly easy to see what they're doing even if every line of “main line“ code has three lines of error handling to go with it.

The registration code creates these keys:

HKCR\SimpleScript\(Default) = "SimpleScript Language Engine"
HKCR\SimpleScript\OLEScript
HKCR\SimpleScript\CLSID\(Default) = "{...}"
HKCR\CLSID\{...}\(Default) = "SimpleScript Language Engine"
HKCR\CLSID\{...}\OLEScript
HKCR\CLSID\{...}\ProgID\(Default) = "SimpleScript"
HKCR\CLSID\{...}\InprocServer32\(Default) = "c:\simplescript.dll"
HKCR\CLSID\{...}\InprocServer32\ThreadingModel = "Both"

as well as some category keys that say "this thing is a script engine". The "OLEScript" tags are for backwards compatibility with VERY old script hosts, like in the Win16 timeframe -- strictly speaking I would imagine that they are no longer necessary, but it doesn't hurt.

The only thing interesting about the unregister code is that it fails silently. Actually writing code that checks the error conditions, determines whether the keys couldn't be deleted because (a) they're not there, (b) they are there but you can't see them because you don't have access or (c) they are there but you don't have access to delete them is a pain in the rear. It's not great that this fails silently when a non-admin tries to unregister the engine, but I'm not going to lose any sleep over it.

Next time, I'll build on this foundation a bit by adding a class factory and the skeleton of a script engine. Then we can get into the actual script interfaces.

SimpleScript, Part Zero

Eric Lippert — Mon, 29 Mar 2004 17:40:00 GMT

I'm going to embark upon something ambitious here. I got an email from Mike, our friendly support engineer on the east coast with the subject line "Need lots of info". Apparently he was contacted by a customer who wants to implement a new script engine that supports debugging, and, here's the part that you've got to really grit your teeth over: in C#.

My initial response was, of course, "They're crazy -- talk them out of it!" Not to put too fine a point on it, that's an immense amount of work in any language. But C#? The script interfaces were designed long, long before there was managed code. It's quite likely that it's going to be very hard to write a straightforward interop layer for those interfaces. "Could they use the managed IVsa interfaces instead?" I asked Mike.

I haven't heard back yet, but I got to thinking last night that maybe it would be interesting to develop a script engine from scratch, describing every interface as I implement it, giving some of the design history behind the interfaces, etc, etc, etc. I could start it in C++ and then see whether there were ways to make it work well in managed code, etc.

I could then blog a little bit of the code every few days. This would keep me in blog topics for weeks, certainly. Probably months. All of this stuff happens in my spare time, and I have precious little of that, particularly given the number of bugs I have to fix before the Whidbey beta. But I'll see what I can do.

I'm not sure that a piecemeal daily blog approach is even the right way to handle this kind of material, but I'm willing to make the experiment.

This is probably how it will go:

Phase One: Basic Infrastructure
-       dll entrypoints
-       registration
-       class factory
-       engine skeleton

Phase Two: Building the engine
-       script engines as state machines
-       threading concerns
-       named items
-       event handling

Phase Three: Processing the language
-       grammar
-       lexer
-       parser
-       code generator
-       interpreter
-       runtime library

Phase Four: Debugger support
-       breakpoints
-       evaluation
-       syntax colouring
-       context

Phase Five: Managed Code
- ?

That's just off the top of my head. We'll probably mix it up a bit as we go.