Fabulous Adventures In Coding : JScript

Why Doesn't C# Implement "Top Level" Methods?

Eric Lippert — Mon, 22 Jun 2009 18:39:00 GMT

C# requires that every method be in some class, even if it is a static method in a static class in the global namespace. Other languages allow "top level" functions. A recent stackoverflow post asks why that is.

I am asked "why doesn't C# implement feature X?" all the time. The answer is always the same: because no one ever designed, specified, implemented, tested, documented and shipped that feature. All six of those things are necessary to make a feature happen. All of them cost huge amounts of time, effort and money. Features are not cheap, and we try very hard to make sure that we are only shipping those features which give the best possible benefits to our users given our constrained time, effort and money budgets.

I understand that such a general answer probably does not address the specific question.

In this particular case, the clear user benefit was in the past not large enough to justify the complications to the language which would ensue. By restricting how different language entities nest inside each other we (1) restrict legal programs to be in a common, easily understood style, and (2) make it possible to define "identifier lookup" rules which are comprehensible, specifiable, implementable, testable and documentable.

By restricting method bodies to always be inside a struct or class, we make it easier to reason about the meaning of an unqualified identifier used in an invocation context; such a thing is always an invocable member of the current type (or a base type).

Now, JScript.NET has this feature. (And in fact, JScript.NET goes even further; you can have program statements "at the top level" too.) A reasonable question is "why is this feature good for JScript but bad for C#?"

First off, I reject the premise that the feature is "bad" for C#. The feature might well be good for C#, just not good enough compared to its costs (and to the opportunity cost of doing that feature instead of a more valuable feature.) The feature might become good enough for C# if its costs are lowered, or if the compelling benefit to customers becomes higher.

Second, the question assumes that the feature is good for JScript.NET. Why is it good for JScript.NET?

It's good for JScript.NET because JScript.NET was designed to be a "scripty" language as well as a "large-scale development" language. "JScript classic"'s original design as a scripting language requires that "a one-line program actually be one line". If your intention is to make a language that allows for rapid development of short, simple scripts by novice developers then you want to minimize the amount of "ritual incantations" that must happen in every program. In JScript you do not want to have to start with a bunch of using clauses and define a class and then put stuff in the class and have a Main routine and blah blah blah, all this ritual just to get Hello World running.

C# was designed to be a large-scale application development language geared towards pro devs from day one; it was never intended to be a scripting language. It's design therefore encourages enforcing the immediate organization of even small chunks of code into components. C# is a component-oriented language. We therefore want to encourage programming in a component-based style and discourage features that work against that style.

This is changing. "REPL" languages like F#, long popular in academia, are increasing in popularity in industry. There's a renewed interest in "scripty" application programmability via tools like Visual Studio Tools for Applications. These forces cause us to re-evaluate whether "a one line program is one line" is a sensible goal for hypothetical future versions of C#. Hitherto it has been an explicit non-goal of the language design.

(As always, whenever I discuss the hypothetical "next version of C#", keep in mind that we have not announced any next version, that it might never happen, and that it is utterly premature to think about feature sets or schedules. All speculation about future versions of unannounced products should be taken as "for entertainment purposes only" musings, not as promises about future offerings.)

We are therefore considering adding this feature to a hypothetical future version of C#, in order to better support "scripty" scenarios and REPL evaluation. When the existence of powerful new tools is predicated upon the existence of language features, that is points towards getting the language features done.

UPDATE: More thoughts on considerations motivating this potential change here.

Using undefined variables in JScript

Eric Lippert — Thu, 04 May 2006 19:28:00 GMT

I got a question the other day pointing out that in JScript, it is legal to assign a value to an undeclared variable, and the variable kind of gets implicitly declared for you, but it is not legal to read the value of an undeclared variable. The writer wanted to know if y = 1; for some undeclared variable has exactly the same semantics as var y = 1;.

Close, but not quite. Assigning to an undefined variable is not equivalent to replacing the assignment with a declaration because that might declare a local variable bound to a particular activation. Using a variable without declaring it in Jscript produces a new global variable, not a new local variable. You can clearly see that they are not equivalent because these two programs do different things:

function foo() {y = 1; }
foo();
print(y); // 1

function foo() {var y = 1; }
foo();
print(y); // error, y undefined

These behaviours of the Jscript engine are conformant to the ECMAScript specification, Revision 3, Section 8.7.

In VBScript, on the other hand, using an undefined variable does produce a new local variable, whether you're reading or writing.

The writer then went on to ask "how can I prevent JScript from creating a new variable in this case? I want this to be an error."

To explain, we should take a look at what exactly happens when we see y = 1; so that we can determine that we need to create a new variable. To do the assignment successfully, three things must happen: we must determine an address for y, we must determine the value, and we must store the value in that address. The way those things happen goes like this:

First we search the scope chain – with blocks, the activation frame, the globals – for any object that has a property y. We do not find one.
Next we search all the global host objects asking them if they have a property y. We do not find one.
Therefore we create a special "fake" address and put it on the script stack. This completes the first part of the binding.
Determining the value is trivial in this case.
When we do the store we detect that there's a fake address on the stack, allocate a new global variable in the script engine's global name table, and replace the fake address with this new real address.
Finally, we invoke the property put logic in the global name table to store the value to the correct address.

Notice that the only question we ask the host is "do you have this property?", and if the host says no, we create a variable. Therefore, if you want this to be an error, you have to write a script host that lies to the script engine. If you always say "yes, I have that property" even when you don't, then you can raise an exception when the store logic attempts to do the store. However, I've never heard of anyone trying that. There might be other negative consequences to consistently lying to the script engine about whether you have a property or not. You'll have to try it and see.

As an aside, you might be wondering why we go to the trouble of creating a fake address during the address lookup that gets turned into a real address during the store. Why not just create the real address in the first place?

Ponder that for a moment. Under what circumstances would we not want to create a real global variable until after the right hand side of the assignment was evaluated?

Well, you would not expect this program to produce a new global variable:

function foo() { try { y = eval("throw 1;"); } catch(e) { } }
foo();
print(y); // should be undefined!

The global variable creation must happen at the moment of actual assignment, not at the moment when the address is determined, because determining the value might prevent the store from ever happening.

Checking For Script Syntax Errors, This Time With Code

Eric Lippert — Wed, 12 Oct 2005 19:37:00 GMT

A number of people asked me to clarify yesterday's entry. Rather than try to talk you through it, I think the code is straightforward enough to speak for itself. Here's a little skeleton that I just whipped up.

#include <stdio.h>
#include <activscp.h>
#include <new>

const GUID CLSID_VBScript = {0xb54f3741, 0x5b07, 0x11cf, {0xa4, 0xb0, 0x00, 0xaa, 0x00, 0x4a, 0x55, 0xe8}};
const GUID CLSID_JScript = {0xf414c260, 0x6ac0, 0x11cf, {0xb6, 0xd1, 0x00, 0xaa, 0x00, 0xbb, 0xbb, 0x58}};

class MySite : public IActiveScriptSite {
private:

ULONG m_cref;
virtual ~MySite() {}

public:

  MySite() {
   this->m_cref = 1;
  }

STDMETHOD_(ULONG, AddRef)() {
return ++this->m_cref;
}

STDMETHOD_(ULONG,Release)() {
    --this->m_cref;
  if (this->m_cref == 0) {
     delete this;
      return 0;
  }
    return this->m_cref;
  }

  STDMETHOD(QueryInterface)(REFIID iid, void ** ppv) {
   if (ppv == NULL)
      return E_POINTER;
    if (IsEqualIID(iid, IID_IUnknown))
      *ppv = (IUnknown*)this;
    else if (IsEqualIID(iid, IID_IActiveScriptSite))
   *ppv = (IActiveScriptSite*)this;
    else {
      *ppv = NULL;
      return E_NOINTERFACE;
    }
    this->AddRef();
    return S_OK;
}

STDMETHOD(GetLCID)(LCID * plcid) {
return E_NOTIMPL;
}

  STDMETHOD(GetItemInfo)(
  LPCOLESTR pstrName,
   DWORD dwReturnMask,
    IUnknown ** ppunkItem,
    ITypeInfo ** ppti) {
    return E_NOTIMPL;
  }

STDMETHOD(GetDocVersionString)(BSTR * pbstrVersion) {
return E_NOTIMPL;
}

STDMETHOD(OnScriptTerminate)(
   const VARIANT * pvarResult,
   const EXCEPINFO * pexcepinfo) {
    return S_OK;
  }

STDMETHOD(OnStateChange)(SCRIPTSTATE state) {
return S_OK;
}

  STDMETHOD(OnEnterScript)() {
   return S_OK;
}

STDMETHOD(OnLeaveScript)() {
    return S_OK;
  }

STDMETHOD(OnScriptError)(IActiveScriptError * perror) {
    EXCEPINFO excepinfo;
    LONG column = 0;
    ULONG line = 0;
    DWORD context = 0;
    BSTR bstrLine = NULL;
    memset(&excepinfo, 0x00, sizeof excepinfo);
   perror->GetExceptionInfo(&excepinfo);
    if (excepinfo.pfnDeferredFillIn != NULL)
     excepinfo.pfnDeferredFillIn(&excepinfo);
    perror->GetSourceLineText(&bstrLine);
   perror->GetSourcePosition(&context, &line, &column);

   wprintf(L"Error on line %ld column %ld\n", line, column);
   if (bstrLine != NULL)
      wprintf(L"Line: %s\n", bstrLine);
    if (excepinfo.bstrDescription != NULL)
     wprintf(L"Description: %s\n", excepinfo.bstrDescription);
    if (excepinfo.bstrSource != NULL)
     wprintf(L"Source: %s\n", excepinfo.bstrSource);
    if (excepinfo.bstrHelpFile != NULL)
     wprintf(L"Help: %s\n", excepinfo.bstrHelpFile);

    SysFreeString(bstrLine);
    SysFreeString(excepinfo.bstrDescription);
    SysFreeString(excepinfo.bstrSource);
    SysFreeString(excepinfo.bstrHelpFile);

return S_OK;
}
};

void main() {
HRESULT hr = S_OK;
HRESULT hrInit;
IClassFactory * pfactory = NULL;
IActiveScript * pscript = NULL;
IActiveScriptParse * pparse = NULL;
IActiveScriptSite * psite = NULL;

hr = hrInit = OleInitialize(NULL);
if (FAILED(hr))
goto LError;

hr = CoGetClassObject(CLSID_VBScript, CLSCTX_SERVER, NULL,
IID_IClassFactory, (void**)&pfactory);
if (FAILED(hr))
goto LError;

hr = pfactory->CreateInstance(NULL, IID_IActiveScript, (void**)&pscript);
if (FAILED(hr))
goto LError;

psite = new(std::nothrow) MySite();
if (psite == NULL) {
hr = E_OUTOFMEMORY;
goto LError;
}

hr = pscript->SetScriptSite(psite);
if (FAILED(hr))
goto LError;

hr = pscript->QueryInterface(IID_IActiveScriptParse, (void**)&pparse);
if (FAILED(hr))
goto LError;

hr = pparse->ParseScriptText(L"Function Foo \n Foo = 123 \n End Funtcion \n",
NULL, NULL, NULL, 0, 1, 0, NULL, NULL);

if (FAILED(hr))
goto LError;

LError:

if (FAILED(hr))
printf("%0x\n", hr);

if (pparse != NULL)
pparse->Release();

if (psite != NULL)
psite->Release();

if (pscript != NULL) {
    pscript->Close();
   pscript->Release();
  }

if (pfactory != NULL)
pfactory->Release();

if (SUCCEEDED(hrInit))
OleUninitialize();
}

As you would expect, this program prints out the information about the error, and ParseScriptText returns SCRIPT_E_REPORTED to indicate that there was an error but it has already been reported. Had there been no error, the script would not have actually run; the engine is not started, just initialized.

Error on line 3 column 5
Line: End Funtcion
Description: Expected 'Function'
Source: Microsoft VBScript compilation error
80020101

Checking For Script Syntax Errors

Eric Lippert — Tue, 11 Oct 2005 20:35:00 GMT

A reader asked me recently whether there was a way to check a chunk of JScript or VBScript for syntax errors without actually running the code. I'm sure that there are many third-party tools which you could find that do this. If you have your own script host, you can do it yourself quite easily.

The trick is to initialize the engine by setting the script site, but do not do anything that would move the engine from initialized into started state. (Note that attempting to evaluate an expression moves the engine to started state.)

If the engine is initialzed but not started then calling ParseScriptText will check the text passed in for syntax errors but will not run the script. Rather, if the script compiles successfully, it is simply marked as "needs to run when the engine is started".

Unfortunately the script engines do not have an error-recovering parser, so they will detect only the first syntax error and then bail out. Still, better than nothing.

Local variables considered not very harmful

Eric Lippert — Tue, 04 Oct 2005 20:22:00 GMT

Like I said, that code from last time was just test code, not real production code. Though clearly it works, I'd never write code like that in a million years. I'm not thrilled with the way it uses the answer variable of the outer scope as an accumulator, and it is profoundly weird that the "do this function n times" function is a member of the Number prototype and not the Function prototype. Worse still, we have a function that is essentially used as a composition operator that only makes sense to use on functions that have side effects and no arguments or returns. That's a kind of crazy way to do composition operations.

I'd probably do something more like this:

First, define a self-composition factory on the Function prototype, so that it is of general use on all functions of one argument, not just functions of no arguments and no returns that have side effects. What we'll do is make a new member function for every function object that returns a new function which is equivalent to calling the input function n times:

Function.prototype.times = function(n) {
var f = this;
return function (x) {
    while (n > 0) {
      x = f(x);
      --n;
    }
    return x;
};
}

Obviously we don't need to compose functions of more than one argument because we want to feed the single output back into the input. Now we compose the function "multiply something by x" with itself n times. That gives us the function "multiply something by x, n times". Pass 1 to that function and we have a handy raiseTo function:

Number.prototype.raiseTo = function(power) {
var x = this;
return function(y) {
return x * y
}.times(power)(1);
}

No fuss, no muss, no accumulators, no side effects, no non-number-related functions on the number prototype. I like it!

Heck, if we wanted to get really crazy then we could eliminate the local variables entirely by introducing another function constructor in-line:

Function.prototype.times = function(n) {
return function(f) {
    return function (x) {
      while (n > 0) {
        x = f(x);
        --n;
      }
      return x;
    }
}(this);
}

Number.prototype.raiseTo = function(power) {
return function(x) {
   return function(y) {
      return x * y
    }
}(this).times(power)(1);
}

Local variables are actually an unnecessary syntactic sugar in Jscript. We can do everything that local variables do by cleverly nesting function scopes. However, it becomes somewhat less easy to read, so I would stick to using local variables if I were you!

Guru meditations on scope chains of closures

Eric Lippert — Fri, 30 Sep 2005 22:41:00 GMT

I just had a really interesting meeting with some of the scripting community MVPs. I may write up some notes on that meeting here next week, so watch this space.

In other news, someone asked me last night (and I quote)

Oh ECMA guru, can you provide a succinct explanation of why the following code works, but if I replace x with this it fails?

Number.prototype.times = function(f) {
for(var i = 0; i < this; i++)
f();
}
Number.prototype.raiseTo = function(n) {
var answer = 1;
var x = this;
n.times(function(){ answer = x * answer;}); // succeeds
//n.times(function(){ answer = this * answer;}); // fails!
return answer;
}
(3).raiseTo(2).raiseTo(5);

Well I'd hardly describe myself as a guru, but I take the questioner's point -- surely if you set x to this then x and this should be aliases for each other, right?

Often. But not in this case.

The inner function's execution context contains a scope chain that includes the variable object of the outer function, so the inner function can see x by looking up the scope chain. But this is not a variable, so it is not member of the outer function's variable object. Rather, it is a member of the outer function's execution context.

Therefore the inner function's execution context gets the global this, in accordance with the ECMAScript specification, revision 3, section 10.2.3:

The caller provides the this value. If the this value provided by the caller is not an object (including the case where it is null), then the this value is the global object.

The global object is not something that can be multiplied, so the alternate version of the program fails.

The code above was of course not real production code, it was just code that came up while testing an engine implementation. Obviously JScript already has a built-in power function that works much better than this crazy thing. But I'm in an expansive mood, so next time I'll talk a bit about some of the shortcomings of this programming style, and some ideas for improving it.

C# 3.0 is still statically typed, honest!

Eric Lippert — Tue, 27 Sep 2005 20:03:00 GMT

Since LINQ was announced I've seen a lot of really positive feedback and a lot of questions and concerns. Keep 'em coming! We need this feedback so that we can both correct misconceptions early and get the design right now.

One of the misconceptions that I've seen a lot over the last few days in forums, blog posts and private emails is a confusion about what the new "type inferencing" feature implies for the type safety of the language. Apparently we have not been sufficiently clear on this point: C# 3.0 will be statically typed, just like C# 1.0 and 2.0. The var declaration style does not introduce dynamic typing or duck typing to C#.

I think the confusion may arise from familiarity with other languages such as JScript. In JScript this is perfectly legal:

var foo = new Blah();
foo = 123;
foo = "hello";

JScript is a dynamically typed language. You can assign any value of any type to a var.

In C# 3.0, the var statement means "look at the type of the thing assigned to the variable, and act as though the variable was declared with that type." In other words, in C# the code above is just a syntactic sugar for

Blah foo = new Blah();
foo = 123;
foo = "hello";

which of course would produce a type error on the second and third lines.

If you take a look at section 26.1 of the C# 3.0 specification you'll see that the var statement has a lot of restrictions on it to ensure that the compiler always has enough information to make the correct type inference. Namely:

the declarator must include an initializer, so that we can infer the type of the variable from the type of the initializer
the initializer has to be something that we can figure out the type of – not null or a collection initializer

Compare this to JScript .NET, which has a much stronger type inference mechanism. JScript .NET does not require initializers in var statements; the compiler tracks all assignments to the variable and infers the best type. If, say, only strings are assigned to a variable then it will infer the string type. JScript .NET also infers return types of functions by a similar mechanism. But the goal of the JScript .NET type inference mechanism was to increase the performance of legacy dynamically typed code. If we can infer a type and thereby generate faster, smaller code, we do so. If not, we don't.

Then why introduce this syntactic sugar in C# 3.0? C# doesn't have a body of legacy dynamic code like JScript and already generates efficient code.

There are two reasons, one which exists today, one which will crop up in 3.0.

The first reason is that this code is incredibly ugly because of all the redundancy:

Dictionary<string, List<int>> mylists = new Dictionary<string, List<int>>();

And that's a simple example – I've written worse. Any time you're forced to type exactly the same thing twice, that's a redundancy that we can remove. Much nicer to write

var mylists = new Dictionary<string, List<int>>();

and let the compiler figure out what the type is based on the assignment.

Second, C# 3.0 introduces anonymous types. Since anonymous types by definition have no names, you need to be able to infer the type of the variable from the initializing expression if its type is anonymous.

We'll discuss the reasoning behind anonymous types in another post.

Implementing Event Handling, Part Two

Eric Lippert — Wed, 21 Sep 2005 21:51:00 GMT

It's been an insanely busy month for me, between having multiple out-of-town guests, throwing a party for the people who couldn't make it to the wedding, and oh yeah, getting up to speed on the C# compiler and trying to understand the implications that LINQ features are going to have on the current implementation. Not much time for blogging. Hopefully October will be a little bit more under control.

Anyway, I was talking about early-bound event binding. Basically the idea is that the source and the sink agree upon an interface that the source can call on the sink whenever the source wishes to fire an event. Before we get into that more, I want to talk briefly about early- and late-bound code.

At an implementation level, what is the difference between early- and late- bound code? In the early-bound world, a particular function on an interface gets called when the caller dereferences the callee's virtual function table and transfers control directly to the function by changing the instruction pointer on the chip. In the late-bound world, the caller calls IDispatch::Invoke and passes in a magic number that tells the callee what function it should be calling, and the callee is then responsible for dispatching the function appropriately. (Hence the name "IDispatch".)

The script engines use IDispatch all the time to call out to functions on objects. When calling an object late-bound, you don't know until you actually try the call whether or not it is going to succeed, because you do not know the interface the object implements. So you call GetIdsOfNames to get the magic number associated with a particular function, Invoke on that magic number, and let the callee sort it out. In this situation, the caller (the script engine) knows the name of the function it wants to call, and the callee (the object) can map the name to the appropriate identifier.

Now consider how this works if IDispatch is the interface over which the source (the object) is calling back the sink (the script host). This appears to be the same situation, but in fact it is completely different. It appears the same because from the perspective of how COM actually manages all the calls, its exactly the same. IDispatch is an interface like any other, and the source can call the sink's IDispatch to its heart's content if that's the interface that they agree to talk over.

But look at it from the point of view of the sink: the caller knows what the magic number means and the callee does not, but it's the callee who is being asked to do the dispatching! You're sitting there sinking events of who knows what object, and every now and then you get a call on IDispatch::Invoke with some unknown dispid. What the heck are you supposed to do with that? In the late-bound world, instead of getting a nice direct call on "Tick()... Tick()... Tick()..." you're getting "12... 12... 12..." and you have no idea what "12" means.

Let's tie this in to scripting. Suppose you're in Windows Script Host and you do something like:

Sub Timer_Tick()
'whatever
End Sub
Set Timer = CreateObject("mytimer")
WScript.ConnectObject Timer, "Timer_"
Timer.Start 10
WScript.Sleep 1000

There are many IDispatch objects here and we'll look at three of them. First, there's the source, that is, the timer object. Then there's the sink, owned by the host and created when the source is connected to it. Finally there is the script engine itself, upon which the host can dispatch calls to global functions such as the event handler.

ConnectObject is given a source object. It has no idea what early-bound outgoing interface is on that object, and it certainly doesn't have an implementation of such an interface even if there is one, so it is going to have to do a late bound sink. It creates an object to act as the sink, gets an IDispatch-enabled connection point from the source, and advises it.

Now the script engine calls Start (via IDispatch on the source) and goes to sleep.

Pretty soon the timer invokes the sink, passing in the dispatch identifier for Tick.

The sink needs to know which sub to call in the script engine.

Fortunately, it has an IDispatch pointer to the source, and a dispatch identifier.

Unfortunately, though IDispatch provides a function which maps from name to id, it provides no function that maps the other way.

Fortunately, IDispatch does provide a function that enables the host to obtain a type information structure which contains a list of all the methods and what their dispatch identifiers are. Therefore we can search all the methods in the type info and check to see which has the desired dispatch identifier. That then gives us the method name, so we know which event handler function to dispatch on the script engine.

Unfortunately there is a major design flaw in IDispatch -- it gives you the type info for the incoming interface -- the ITimer. But there is no way to take the ITypeInfo for ITimer and say "give me the type info for the class as a whole so that I can obtain the type info for the outgoing interface".

Fortunately, IProvideClassInfo was invented. IPCI gives you back the "root" type info for an object, from which you can obtain a type info for the default outgoing interface, from which you can map the dispatch identifier to the name.

Unfortunately, many objects do not implement IProvideClassInfo, and therefore cannot have their events hooked up by ConnectObject.

(I feel a little bit like the ending of Dr. Strangelove here. Fortunately, they stop General Ripper in time. Unfortunately, the secret code died with him. Fortunately they figure out the code. Unfortunately, the radio is broken. Fortunately, the bomb bay doors are stuck. Unfortunately, they fix them, and the Doomsday Device destroys the world. Bummer.)

The moral of this story is simple: for late bound events to work, at some point the class of the source must be known. And therefore:

"automagic event binding" in IE only works on "named items" present in the script engine. (The script engines require that the host provide a coclass type info for named items, and build the sinks very early on, before other code runs.)
WScript.CreateObject can always hook up events. The class type info is known because CreateObject had to create an instance of the class.
WScript.ConnectObject can hook up events only if the object implements IProvideClassInfo

I hope that clears up any confusion about what IProvideClassInfo is for!

Recursion, Part Six: Making CPS Work

Eric Lippert — Mon, 15 Aug 2005 13:00:00 GMT

JScript doesn't support CPS natively but we can write another dispatch engine that makes it work. There's only ever one active continuation, so lets have a new rule: JScript CPS functions are allowed to return, but the last thing that they do must be to tell our dispatch engine what the continuation was.

To keep things simple, we'll also say that every CPS-style function takes exactly one argument. Of course, that can be an object that contains multiple fields should the function require multiple arguments.

Let's go through our CPS treeDepth program and replace all the continuation calls with calls that tell the runtime engine what continuation to use next. We'll then return normally and let the runtime engine call the continuation for us.

The dispatch engine is very simple:

var continuation = null;
function cont(newfunc, newargs)
{
continuation = { func: newfunc, args : newargs };
}

function run()
{
while(continuation != null)
{
    var curfunc = continuation.func;
    var curargs = continuation.args;
    continuation = null;
    curfunc(curargs);
}
}

No stack at all this time. Just a single global variable saying what to call next. If there is nothing to call next, we're done. To determine the depth of a tree we simply tell the continuation engine what the next thing to do is, and start it running:

cont(treeDepth, {curtree: mytree, afterDepth: print});
run();

and hey presto, we've just implemented a recursive function with no stack that prints the depth of an arbitrary binary tree.

Nice trick! Of course, really what we've done is we've just hidden the call stack in another harder-to-see data structure. Where is it? (Hint: draw some diagrams that show what all the closure name table bindings are during the recursion.)

That's all I managed to get done here before my vacation. When I get back, maybe I'll pop the stack and pick up where we left off, or maybe I'll continue with something completely different and forget all about my current state. (Or maybe I'm actually interrupt driven, which seems most likely.) Who knows? Whatever happens though, it'll probably be another fabulous adventure in coding.

***********************************

Recursion, Part Five: More on CPS

Eric Lippert — Thu, 11 Aug 2005 13:00:00 GMT

Suppose we wanted to write this by-now-familiar little function in continuation passing style:

function treeDepth(curtree)
{
if (curtree == null)
    return 0;
else
{
    var leftDepth = treeDepth(curtree.left);
    var rightDepth = treeDepth(curtree.right);
    return 1 + Math.max(leftDepth, rightDepth);
}
}

Let's start by getting rid of the returns, since there are no returns in CPS. Rather, we "return" by passing the return value to the continuation. Remember, the continuation is "what are we supposed to do after this function is done?"

function treeDepth(curtree, afterDepth)
{
if (curtree == null)
    afterDepth(0);
else
{
    //UNDONE: var leftDepth = treeDepth(curtree.left);
    //UNDONE: var rightDepth = treeDepth(curtree.right);
    afterDepth(1 + Math.max(leftDepth, rightDepth));
}
}

We'll presume that the addition and maximum guys are not written in CPS, just to keep this simple.

OK, this is a start, but now we need to fix up those recursive calls. Consider the second recursive call. What is its continuation? That is, what needs to happen after the right tree depth is determined? Two things. First, the maximum of left and right must be calculated and incremented. Second, we've promised our caller that when we're done, we'll call its continuation. So let's write a closure that does those two things.

We're making progress. What do we need to do after we determine the depth of the left tree? Pass the left depth to a continuation that recurses on the right tree to determine the right depth, does the max, and calls the afterDepth continuation.

And we're done. We've got a CPS version of treeDepth which never returns and therefore we never need to keep track of what is on the stack, because we're never coming back.

Of course, like I said last time, in reality JScript does not detect that we're in this situation and optimize away the stack frames. This program still pushes interpreters on the system stack and still runs out of stack space if you have a 1500-deep branch in your tree.

But we can fake it out. Next time, we'll finish up with a CPS program that really doesn't run out of stack space, even in JScript.

(Eric is on vacation; this message was pre-recorded.)

Recursion, Part Four: Continuation Passing Style

Eric Lippert — Mon, 08 Aug 2005 13:00:00 GMT

We're getting hung up on the stack management aspects of recursive programming. Why do we need a stack at all? What purpose does it serve?

If you're like most developers, you probably learned about subroutines and functions at an early age. The idea is pretty straightforward. You stop what you're doing right now, go perform some other task, and then pick up where you left off, possibly using data computed by the other task. But this idea of "pick up where you left off" means that you have to have some mechanism for storing information about what you were doing and where to pick it up. A stack is a natural data structure for that, so we've been thinking a lot about stacks lately when we consider how to operate on recursive data structures.

But what if we simply denied the whole premise of functions? What if we said that no function was ever allowed to return? It either terminates the program, or itself calls another function.

You might be thinking that clearly the cure is worse than the disease. I mean, sure, now you don't have to keep track of what you were doing, but that's because you're never going to get back there! How is it possible to get any work done at all if every time you try to call a procedure for a little help, it takes over control and never comes back to you?

This sounds like crazy talk, but let's think about the consequences of removing all returns from the language. Three things are clear.

First, if you yourself write a function then calling another function has to be the last thing you do. Control is never coming back, so there's no point in having any code after a function call.

Second, suppose you have more work to do after you call a function foo. Foo is in the same position as you -- the last thing it does will either be a function call or terminating the program. Therefore, you need to put the work that needs to be done after foo into function bar, and make for darn sure that foo calls bar rather than terminating!

But if you didn't write foo, how do you know that foo is going to call bar when its done? You need to somehow tell foo to call bar, and hope that foo cooperates.

Third, if someone calls your function then they're in the same boat -- it's the last thing they're going to do, ever, because you're not coming back. You need to find some way to allow your caller to tell you what to do when you're done your work.

This idea of functions never returning, and callers passing their functions information about what needs to be done next, is a bizarre but powerful style of programming called Continuation Passing Style. The "information about what to do next" is called a "continuation", which is passed from function to function, hence the name.

Many languages support CPS natively -- Scheme, Ruby and Rhino, for example, all support CPS. To get it working in JScript will take some doing, but when we're done there will be no explicit stack at all in our formerly recursive program.

In the next episode I'm going to write our tree depth program in a mix of CPS and regular procedural programming. (To be purely CPS we'd write CPS versions of the addition operators, the Math.max function, and so on, but I'm not going to go there -- that's overkill.) But before we get into the tree depth program let's get a little more familiar with CPS.

Suppose we had this fragment of a JScript program:

function foo(x)
{
var y = bar();
blah(x,y);
}
function bar()
{
return 123;
}
function blah(a, b)
{
print(a+b);
}
foo(1);

I hope you agree that this is a very straightforward program. How would we do this in CPS? Well, first of all, every function would have to take an extra argument for the continuation. Our program calls foo and then terminates, so the continuation of the call to foo is "terminate the program". Let's presume that we have a magic function that does that.

function foo(x, cfoo)
{
// UNDONE: rewrite foo in CPS
}
foo(1, terminate);

Let's reason about foo from end to start. foo wants to ensure that three things happen in this order:

bar runs
blah runs
foo's continuation from its caller runs

But blah is never going to return. Therefore we'll make it blah's responsibility to call foo's continuation when it is done. We'll rewrite blah so that it does CPS, so that we can say:

function foo(x, cfoo)
{
var y = bar();
blah(x,y,cfoo);
}

OK, super, we've taken care of our responsibility to our caller by foisting it off onto blah. But now we have an additional problem. bar is never going to return, so we're never going to call blah! We've both failed in our responsibility to our caller and haven't done the work we need to do.

We have an additional problem. bar was going to return a value that we were going to use. But we've just removed returns from the language! Let's solve both of these problems. We'll say that bar takes a continuation and passes the value that it was going to return as an argument to its continuation. That's functionally the same thing, right? Before, you return the value to "whatever you were going to do next", now you pass the value to "whatever you were going to do next", same thing.

But what continuation should foo pass to bar? Well, what does foo want done when bar calls its continuation with its "return" value? That's easy. It wants to call blah with that value and foo's original arguments. We can write a helper function that does that.

foo(1, terminate);

foocontinuation is a closure, so it keeps track of what the values of x and afterfoo were when it was passed to bar. So we're all set here.

The global code passes 1, terminate to foo
foo passes foocontinuation to bar
bar passes 123 to foocontinuation
foocontinuation passes 1, 123, terminate to blah
blah adds together its arguments and passes 124, terminate to print
print presumably prints out 124 and calls terminate, and we're done

Not once did any function return in there, and we did everything in the right order.

Now of course in reality, JScript does not know that none of these functions are going to return. Nor is JScript smart enough to realize that even if they did return, none of these functions now does anything after the subroutine call, and therefore keeping track of the old frames on the stack is unnecessary, since they're never going to be read from again. If it did, we could totally write programs in this style and never worry about running out of stack space, but unfortunately, it doesn't.

But just as we helped JScript along by writing our own explicit frame stack, we can help it along by writing our own CPS system. Stay tuned!

(Eric is on vacation; this message was prerecorded.)

Recursion, Part Three: Building a Dispatch Engine

Eric Lippert — Thu, 04 Aug 2005 13:00:00 GMT

There's a particular technique that I like to use to solve problems -- it doesn't always work, but when it does, it can produce programs of surprising elegance and power. The technique is this: if you have a specific problem to solve, solve a more general problem, and then your problem is just a special case. It's like that joke someone mentioned during my series on high-dimensional arithmetic -- it's easy to imagine a 9-dimensional space, just imagine an n-dimensional space and set n=9.

That sounds like more work, doesn't it? Often it is, but paradoxically, sometimes it's less work.

Our solution last time for de-recursivizing a three-line recursive algorithm was an unreadable mess in part because the explicit stack logic was all mixed up with the bits of the function actually doing work. Rather than messing around trying to implement JScript's call logic in JScript itself, complete with frames and return values and activation objects, let's solve a more general problem. Let's build a little dispatch engine, and build some functions that use that dispatch engine to solve problems. Solving tree depth will then just be a special case.

Our dispatch engine will have two stacks: the functions-I-still-need-to-call stack, and the values-to-pass-to-them stack. But to keep it simple, the functions themselves will be responsible for taking the values on and off the value stack. With those design characteristics, our "dispatch engine" becomes very, very simple. It just pops a function off the function stack and calls it. When there are no more functions on the stack, we're done.

while(functionstack.length != 0)
functionstack.pop()();

We'll need to do a little stack magic to make it all work, but we can do it. We'll call the functions D, R and M:

D:
action: either computes the depth of a null node or pushes a program that "recursively" computes the depth of a node.
value stack precondition: value stack top contains tree node
value stack postcondition: if tree node on stack is null, replace it with zero. Otherwise push right and left trees
function stack postcondition: if tree node is not null, push MDRD onto function stack

R:
action: reorganizes the value stack so that the depth of the right tree can be determined
value stack precondition: top of stack is left tree depth, followed by tree node
value stack postcondition: top of stack is tree node, followed by left tree depth

M:
action: takes the max + 1 of two numbers
value stack precondition: two numbers on top of stack
value stack postcondition: one number on top of stack

Does that make sense? Let's think about how that would play out for a tree with root B, left child A, right child C. I'll use X to represent an empty (null) tree. We put D on the function stack and B on the value stack and start the engine:

functions values notes
D         B      B is not null, so we push its children on the value stack and MDRD on the function stack
MDRD      CA     Again, D is the top of the function stack, so we do it again, this time on A
MDRMDRD   CXX    Now D is at the top of the function stack and we have a null. Replace it with zero.
MDRMDR    CX0    Swap the top two
MDRMD     C0X    Replace null with zero
MDRM      C00    M pops both numbers, finds the larger, adds one, pushes it
MDR       C1     Swap the top two, replace the tree with its right child
MD        1C     and so on...
MMDRD     1XX
MMDR      1X0
MMD       10X
MM        100
M         11
          2

And if we've done it right, when we're done the value stack contains the depth of the tree that we started with, and the function stack is empty.

Looks promising. Let's write the code.

var functionstack = new Array();
var valuestack = new Array();

function treeDepth()
{
var curtree = valuestack.pop();
if (curtree == null)
    valuestack.push(0);
else
{
    functionstack.push(max);
    functionstack.push(treeDepth);
    functionstack.push(reorder);
    functionstack.push(treeDepth);
    valuestack.push(curtree.right);
    valuestack.push(curtree.left);
}
}

function reorder()
{
var leftdepth = valuestack.pop();
var righttree = valuestack.pop();
valuestack.push(leftdepth);
valuestack.push(righttree);
}

function max()
{
valuestack.push(1 + Math.max(valuestack.pop(), valuestack.pop()));
}

functionstack.push(treeDepth);
valuestack.push(mytree);
while(functionstack.length != 0)
functionstack.pop()();
print(valuestack.pop());

Each function is a lot cleaner now that the execution management has been pushed to an external engine. But we've paid a pretty high price in that it's now non-obvious that this program actually works on all trees. I mean, it sure looks like it works, but we haven't proven it. But at least its not directly recursive, so we're pretty sure that we don't have to worry about running out of system stack.

This time we attacked the problem of eliminating recursion by getting rid of the notion of explicit frames on the stack. Next time we'll take a very different tack and try to eliminate stacks altogether.

(Eric is on vacation; this message was pre-recorded.)

Recursion, Part Two: Unrolling a Recursive Function With an Explicit Stack

Eric Lippert — Mon, 01 Aug 2005 13:00:00 GMT

That recursive solution is pretty cool, to be sure. But there is one big problem with it. Consider this Jscript program that puts a few more nodes into our tree from last time:

function tree(value, left, right)
{
this.value = value;
  this.left = left;
  this.right = right;
}
for (var i = 1 ; i < 1500 ; ++i)
  mytree = new tree(i, mytree, null)
print(treeDepth(mytree));

This is not a very well-behaved binary tree – it’s actually mostly a linked list, hardly a tree at all. Recursing down 1500 branches to get to the bottom blows the stack.

Each time you enter a JScript function, that pushes a new JScript "activation" frame on the JScript stack. But the JScript stack is actually allocated off of the system heap, so this isn't the problem. Unfortunately, doing so also pushes a new copy of the interpreter on the system stack, which consumes a few hundred bytes of stack space. Since by default the system stack only has a million bytes available, this quickly consumes the entire system stack.

Recursive functions are sometimes the most elegant solution, but not always the most practical.

How are we going to solve this problem?

One way to do it is to not get into the situation in the first place. There are various algorithms for guarateeing that when you construct a binary tree that it is a "balanced" binary tree. That is, there are no long, deep paths. A perfectly balanced binary tree with 1500 nodes in it is guaranteed to require no more than 11 recursive steps, and doubling the number of nodes only adds on one more step. A perfect tree with a million nodes requires no more than 20 recursive steps, so odds are pretty good that you'll never blow the stack if you know you have a balanced tree. There are algorithms such as the famous red-black algorithm that guarantee that a binary tree is always within a factor of two of being perfectly balanced, so maybe a million-node tree is 40 deep, which seems doable.

Another approach is to keep track of how deep you are on the stack and bail out:

function treeDepth(curtree, stackdepth)
{
if (curtree == null)
    return 0;
if (stackdepth > 1000)
    throw "tree is too deep, bailing out!";
return 1 + Math.max(
    treeDepth(curtree.left, stackdepth + 1),
    treeDepth(curtree.right, stackdepth + 1));
}

But that's pretty hacked up. What if you're wrong about how much stack you have left? Maybe 1000 is too deep. What if you can't guarantee that the tree is balanced? What if you want to have an algorithm that works for any binary tree without danger of blowing the stack or getting a goofy exception?

Some languages, such as Scheme, have what’s called “tail recursion”, where they detect when they can optimize away pushing another frame on the stack, and therefore do not have this problem. But JScript does not have this optimization, and even if it did, this tree algorithm is a poor candidate for tail recursion. Tail recursion works really well when the recursive step is last, but this algorithm has two recursive steps, so one of them is going to be not last, obviously. (Though as we'll see in a few episodes, there are things we can do about that.)

Hmm.

I just said that the JScript stack is actually allocated off of the system heap -- the interpreter simulates a stack for the script engines rather than re-using the system stack. We can do the same thing with our recursive program: make the stack explicit rather than implicit.

That's going to be complicated. Think about what has to happen every time you call a function in JScript. Somehow the interpreter has to remember a whole bunch of things, like:

values of local variables in the current activation frame -- variables and arguments in the callee's frame must not change the variables in the caller's frame.
where to pick up execution when the callee returns
what to do with the return value of the callee
values of "anonymous locals" -- for example, when you have x = 1 + 1 + foo(); somehow the engine has to remember that there's a 2 sitting there in temporary storage waiting to be added when foo returns.

Let's take our recursive depth algorithm and refactor it a little bit so that the temporary variables are spelled out and each recursion happens on its own line:

function treeDepth(curtree)
{
if (curtree == null)
return 0;
var leftdepth = treeDepth(curtree.left);
var rightdepth = treeDepth(curtree.right);
return 1 + Math.max(leftdepth, rightdepth);
}

For each recursive step we'll need a frame that keeps track of what the three local variables are. Before each recursion we need to remember where we were, so that when we start executing, we know whether to pick up on the "var leftdepth..." line or "var rightdepth..." line or the "return 1 + ..." line. We'll remember that by keeping track of whether leftdepth and/or rightdepth are initialized on the frame. When we complete a calculation we'll pop off the current frame and fill in the "calling" frame's variables with the appropriate "return value" so that our execution loop knows where to go next.

Since there is no recursion here at all now, there's no danger of running out of stack. But good heavens, that's a freakin' mess compared to the three-line recursive solution.

This is probably more trouble than it's worth. Next time we'll look at another explicit-stack based approach that is a whole lot cleaner than this dog's breakfast.

(Eric is on vacation; this message was prerecorded.)

Recursion, Part One: Recursive Data Structures and Functions

Eric Lippert — Wed, 27 Jul 2005 13:00:00 GMT

The first thing to wrap your head around is recursively defined data structures. Let's start with something simple. Think about the abstract idea of “list”. Most people think of a “list” as an ordered collection of “items”, one after the other, with a beginning and an end. That’s a very non-recursive way to think about a list, but it’s the usual way of doing so.

The recursive way to think about a list emphasizes the similarity of a fragment of the list to the whole list. We might recursively define a list as either

the empty list with zero items on it, or
an item followed by a list.

This certainly appears to be a circular definition, but it’s really not. It’s more sort of a “spiral” definition. You start off with something extremely simple; it’s harder to get more simple than a list with no elements. Then you provide a rule for turning something simple into something very slightly more complicated. The recursive part of the definition means that you can take a list of any size, starting with zero, and construct a list one bigger than it, by sticking an item on to the beginning. It’s not a truly circular definition because there is always a “base case”. There is always some point from which you begin, and upon which you can build.

But the true power of the recursive definition comes from going the other way: if you have something that meets the definition, then there is usually a way to move back towards the base case, getting simpler and simpler as you go.

That’s what recursive programming is all about. All recursive functions follow the same basic pattern:

Am I in the base case? If so, return the easy solution.
Otherwise, figure out some way to make the problem closer to the base case and solve that problem.

That’s why you can get away with calling a function from itself – because you know that the recursion will eventually bottom out when you’ve found the base case.

Now, for list processing, a recursive solution is usually overkill. The iterative solution of “run down the list in a loop” is usually good enough. But many other problems are naturally solved by other recursively defined data structures. For example, consider the highly useful binary tree. A binary tree is either:

an empty tree, or
a value associated with two binary search trees, called the left and right child trees

For example, this is a binary tree:

          E
         / \
        /   \
       B     G
      / \     \
     A   C     H

The children of A, C and H, and the left child of G, are empty trees. (Sharp eyed readers will have noted that this particular example is a binary search tree, because every parent is between its children. We might discuss properties of search trees later.)

Pretty picture, but how are we going to represent this thing in code? Suppose you’ve got a binary tree: in JScript

var mytree =
{
left:
{
    left:
    {
      left: null,
      value : "A",
      right: null
    },
    value: "B",
    right:
    {
      left: null,
      value: "C",
      right: null
    }
},
value:"E",
right :
{
    left: null,
    value: "G",
    right:
    {
      left: null,
      value: "H",
      right: null
    }
}
};

Now we can use this recursive definition to write recursive functions that answer questions about this tree. For the rest of this series I'll use as my motivating example one of the simplest questions you can ask about a binary tree: what is the deepest you can go in a given tree?

We have a recursive definition, so come up with a recursive solution.

What’s the base case? An empty tree. Clearly the deepest you can go in an empty tree is zero steps.

What’s the recursive case? If you haven’t got an empty tree then you’ve got a value with two children. Each child is itself a tree. The deepest you can go from this point is one (counting yourself), plus the depth of the deepest child. And we've got a function that gives you the depth of a tree, so let's call it on each child:

function treeDepth(curtree) {
if (curtree == null)
return 0;
return 1 + Math.max(treeDepth(curtree.left), treeDepth(curtree.right));
}

It's magically delicious. It's like having n dominoes lined up. If you knock over the last one, and you know that every domino knocked over also knocks over the previous, then you know that they'll all go down eventually.

Formally, we could use complete induction to show that this works for all trees. We know that it works for trees of height zero. As an inductive hypothesis, assume that it works for trees of height <= k. Since a tree of height k+1 has one subtree of height k and another of height <=k, by hypothesis the recursion correctly calculates their heights and computes the max as k+1, and we're done. (Of course, how do we know that complete induction works? That question takes us into Peano's Axioms and other topics in the foundations of mathematical logic. Perhaps another time...)

Next time: Trouble in paradise: the out-of-stack error rears its ugly head.

Recursion, Part Zero

Eric Lippert — Mon, 25 Jul 2005 13:00:00 GMT

I’ve mentioned recursive programming techniques several times over the years in this blog (Topological Sort, How Not To Teach Recursion, Fibonacci Challenge, Recursion and Dynamic Programming, Sometimes Breadth Is Better Than Depth, and probably a few others). A lot of developers, particularly those without a formal education in computer science, are still pretty vague on this weird idea of functions that call themselves.

If you’re interviewing at Microsoft for a development position you can expect that at least one person will ask you a whiteboarding question that involves some kind of recursive problem. This isn’t because we’re a bunch of architecture astronauts or theory wonks, it’s because if you want to be a dev here, recursion is simply one of the tools that you have to have in your toolbox. We use recursive data structures and algorithms quite a bit, and people are expected to know how they work.

I’d therefore like to take this opportunity to do a series on some of the theory and practice of writing recursive and recursive-like functions. I want to show that (a) they're not as mysterious as they seem, but (b) that they are potentially dangerous, and (c) there are some absolutely crazy-but-fun techniques for turning a recursive program into a nonrecursive program. Hopefully by the time we're done here your head will hurt, but hey, it builds character.

I'm going to get as much of this done as possible and queued up, because I'm going to go dark shortly. I'm getting married in less than two weeks! Between getting ready for that and finishing up stuff at work, I'm going to be swamped. (And you'd better believe I'm not blogging about recursive programming techniques from my honeymoon.) I'll be back in late August and things should pick up then.