jaredpar's WebLog : Closures

Why is LINQ absent from debugger windows?

Jared Parsons — Wed, 26 Aug 2009 15:00:00 GMT

As the owner of the VB.Net portion of the overall debugging experience, I frequently hear the request from customers to add LINQ support into the Watch / Immediate and Locals window. Virtually every other type of expression is available in the debugger windows so why leave one of the most popular ones out?

Quick Diversion: the specifics of this article are written from the point of view of the VB.Net expression evaluator. However, the limitations blocking LINQ support (in both the architecture and overall design) are very similar between VB.Net and C#.

As usual, the primary issue is cost and the cost for LINQ in the debugger windows is very high. To understand why the cost is so high though, we must start by getting a better understanding how a language service interacts with the debugging services of Visual Studio and the general philosophy around compiler features in the debugger.

Languages in Visual Studio typically provide the following major components to support the debugging experience.

Expression Evaluator (EE): This is the language specific component which provides all of the data used in the watch, locals and immediate window, data tips, conditional breakpoints and several other components. It’s primary input is an expression in string form which is converted to a value (typically an ICorDebugValue instance) and outputs a COM object capable of inspecting that value to the core debugger [1]. Everything typed into the debugger windows goes through the EE.

This component lives in the MTA of Visual Studio and has almost no interaction with the UI / STA thread.
ENC Service: This is the component which is the work horse of ENC operations. It provides rude edit detection, metadata differencing and metadata + IL generation .

This component lives in the main STA of Visual Studio

The important thing to understand about the expression evaluator is that it’s purpose is primarily to provide an expression evaluation and data inspection service. How expression evaluation works is an article in itself but suffice it to say that it converts the string to a very low level AST then walks the nodes bottom up and evaluating the expressions using the ICorDebug APIs. The EE component has no UI and is simply a data provider for the core debugger services.

The design philosophy for Both VB.Net and C# is to have the highest level of fidelity between expressions evaluated in the EE and the actual running program. To do otherwise would lead to extremely confusing results for users. When spec’ing feature support in the VB.Net EE we start from the point of 100% fidelity, determine the problems with this design (if any) and then start the difficult process of making compromises.

LINQ expressions are very different than any other expression previously allowed in the EE window because of the features interaction with metadata. All LINQ expressions require the generation or manipulation of metadata to support the underlying lambda and/or closure. Adding support for this is one of the biggest hurdles to getting LINQ (and other features) into the EE.

Evaluating a LINQ expression is actually much closer to an ENC operation than a traditional EE one.

Currently the EE’s have no capacity for generating metadata, only interpreting it. Operations which mutate or generate metadata have traditionally only been allowed via the ENC service. Getting LINQ to work in the EE with true fidelity would require at least a minimal amount of ENC feature work.

Without getting into too much detail, lets enumerate the new major features necessary to evaluate a LINQ expression in the EE with true fidelity to the running program. To simplify things, we’ll start by assuming there is no other LINQ expression used in the current method and the method is only being executed at most once at any given time in the process.

A metadata generation service to support the backing for closures and lambda expressions
Convert expressions typed in the EE into IL [2]
ENC support for metadata to push the new metadata for closures and lambdas into the currently executing assembly
ENC support for method body IL to remove lifted variables from the current method and redirect the references inside the closure

Issues #1 and #2 are for the most part internal architecture issues and can be solved via normal processes of code base refactoring and adding new features to an existing component. I don’t mean to imply these problems are cheap (in fact they are relatively expensive) But fixing these is somewhat of an understood quantity.

Issues #3 and #4 are where the problems start. As implemented EE’s do not have capability to create or modify metadata in the running process (that is the job of the ENC service). EE’s do have access to the underlying CLR ENC APIs so it is possible to implement ENC operations in the EE. It’s just simply not been done yet.

Wait! Why not reuse our ENC implementation in the EE? Unfortunately the ENC service is currently tied heavily to our in memory IDE compiler and many other IDE / STA features. It in fact lives in a completely separate DLL, separate COM apartment and works on a different symbol table than the EE.

More fundamental though is that it’s designed for a completely different purpose. ENC is designed to track edits in live code, determining the differences and applying them to the running program. The hypothetical EE feature would be tracking expressions that modify a running DLL for which code is not guaranteed (and not likely) to be available and applying the difference to the running program. There are some similarities but the differences are significant enough to make code reuse have limited value.

Some people may wonder why it’s necessary to implement #4. Couldn’t we just avoid removing the variable from the current method and make the feature cheaper? This is possible but it would cause a significant fidelity difference in the feature. Any mutations of the local variable within the LINQ expression would not be visible on the stack frame as it would if the LINQ expression was present in the original program.

It’s easy to think of this ENC in the EE as just the same type of cost as #1 and #2 (in many ways it is). However taking advantage of the CLR ENC APIs in the EE also means that we inherit it’s limitations as well. ENC as as implemented in the CLR has many limitations which fly in the face of LINQ. In particular the following ENC limitations present major problems (ENC limitations reference).

Cannot add members to a value type. This prevents evaluating LINQ when stopped in a value type method
Cannot remove locals from a function. This is an implementation of LINQ but we could work around this problem by changing the IL to simply no longer accessing the local variables.
Cannot change anything in a generic type. This prevents evaluating LINQ when stopped in a generic type.
Cannot specify an initial value for newly added fields.
Allowable ENC operations differ between the top of the stack and operations elsewhere within the stack
Modification of a non-top stack frame cannot significantly edit the current function call. So if a LINQ expression captures a variable used in that call (think ref passing) it would likely be unable to be evaluated.

So even if we implemented everything possible on the language service side the resulting feature would be limited in several impactful ways. Additionally these limitations are somewhat orthogonal to the current limitations EE’s face and would require a bit of user education. And this is only for the most basic LINQ feature under unrealistic scenarios.

Lets now consider the problems of evaluating a LINQ expression when the current method already contains a LINQ expression that lifts at least 1 variable. Evaluating a new expression at the same scope would require the modification of the current closure to maintain fidelity as opposed to generating a new one. This brings along with it a couple more problems.

A lambda expression which uses 2 variables, 1 of which is captured in an existing lambda expression. To maintain fidelity we would have to modify an existing closure signature to contain the new variable.

Often times generated closures are generic so we would run straight into ENC limitation #3. Additionally we would need the newly added field to have the same value as it has in the current method which runs us into ENC limitation #4. Now also consider that a closure instance can live much longer than the method in which it was created. For those closures no stack value is available so what value should we give fields in that instance? Ideally it should have the value of the variable at the point the method exited but realistically that’s not possible.
A lambda expression which uses 1 variable in a method where an existing lambda expression captures that same variable and another. To maintain fidelity we would have to know about this and fake capture the second variable as well.

Now consider all of the problems above and consider the scenario where there are multiple threads and multiple instances of the current method active in the program. How to determine which closure instance belongs to which thread, and just as important which stack frame on which thread, with respect to initializing values?

None of the issues are an unsolvable problem but they do represent a significant cost to the feature. And once again even if we added all of these features to the EE, ENC limitations would significantly limit the usefulness of the resulting feature.

Does this mean that LINQ, or any future metadata requiring expression, will never be added to the debugger window? Absolutely not. We’ve just hit the difficult stage of making compromises on the level of fidelity in the feature. If you back off of true fidelity in a few small ways the resulting feature, while still very expensive, is significantly cheaper and removes many of the limitations imposed by ENC. This hypothetical feature will be discussed in my next article.

[1] For those interested the returned interface is IDebugProperty2

[2] Right now expressions are converted to a tree form slightly above IL and they are interpreted using the ICorDebug APIs. Converting all the way down to IL requires a lot more work

Script Blocks and Closures (or lack there of)

Jared Parsons — Thu, 08 Jan 2009 16:00:00 GMT

Script blocks are a concise way of representing an expression or statement group in Powershell. It’s the C#/F#/VB lambda equivalent for PowerShell.

One difference between C#/F#/VB lambda expressions and a scriptblock is the lack of lexical closures (otherwise known as variable capturing). This feature allows for a variable defined in an outer scope to be captured by the lambda in such a way that the value is maintained with the lambda expression. The details on how the variable is captured can vary from language to language but the basics are the same.

static void ClosureExample() {
    var name = "first";
    Func<string> captureName = () => name;
    name = "second";
    Console.WriteLine(captureName());   // prints: second
}

Due to the flexible nature of powershell it is possible for a scriptblock to appear to have captured a variable when in fact it’s just a quirk of variable name resolution. An important item to remember when considering how a scriptblock will execute is knowing that a script block is evaluated at the point of execution, not the point of definition.

PS) function example1() { $b = 42; { $b } }
PS) $b = 42
PS) $sb = example1
PS) & $sb
42

The above sample works because when $sb is evaluated there is a variable $b in scope and hence the expression binds to that value. Not the original one in “example1”.

This is a somewhat contrived example. But the problem can easily occur when scripts 1) contain the same variable name in multiple scopes/contexts, 2) uses one of those variables within a script block. I’ve run into this problem myself several times.

Here is a more complex sample that demonstrates the timing of the name resolution.

PS) function example2() { param ($p1) $v1 = "avalue"; & $p1 }
PS) example2 {$doesnotexist}
PS) example2 {$v1}
avalue

This behavior though can also be used as a feature. Part of the implicit contract of a scriptblock can be the existance of certain named variables in the scope where the script block is executed. Probably not the best code maintainability practice, but I think we can generate a few good samples in a future post.

Closures in VB Part 6: Limitations

Jared Parsons — Tue, 07 Aug 2007 01:36:40 GMT

For previous articles in this series please see ...

As powerful as closures are in the language they do have a few limitations. We worked hard in Orcas to put as few limitations in Orcas as possible. Below are the current limitations and some insight into why they exist this way.

1. Cannot use "ByRef" parameters in a closure

Example:

    Sub LiftAByRef(ByRef x As Integer)
        Dim f = Function() x
    End Sub

Message: error BC36639: 'ByRef' parameter 'x' cannot be used in a lambda expression.

The problem here is the expectation surrounding x. Any change in the value of "x" inside the method "LiftAByRef" should be reflected in the calling function. Normally for any lifted parameter we add a new field inside the closure and all read/writes are redirected into that value. For "ByRef" parameters we would additionally have to ensure that all writes are make to the parameter. Even in the presence of an exception. Not a trivial task.

2. Cannot use "Me" in a closure created inside a structure.

Example:

Structure S1
    Public F1 As Integer

    Public Sub M1()
        Dim f = Function() F1
    End Sub
End Structure

Message: error BC36638: Instance members and 'Me' cannot be used within a lambda expression in structures

Closures capture values by reference. It's not possible to capture the "Me" of a structure by reference in VB. The only other option is to capture them by value. If we did that then all changes to members of a structure inside a lambda would not affect the structure in which they were created; merely the value copy. This is very different behavior from every other place that closures are used. To avoid confusing behavior this is not a legal operation.

3. Cannot use a Restricted Type in a closure

Example:

    Sub LiftRestrictedType()
        Dim x As ArgIterator = Nothing
        Dim f = Function() x.GetNextArgType().GetModuleHandle()
    End Sub

Message: error BC36640: Instance of restricted type 'System.ArgIterator' cannot be used in a lambda expression.

This hopefully will not affect many users. There are several types in the CLR that are considered restricted because they have special semantics. Typically they are special cased by the CLR and as such we can't use them in a closure. Several of these cannot be used in VB at all. They are ...

System.TypedReference
System.ArgIterator
System.RuntimeArgumentHandle

4. Cannot Goto into scope that contains a closure

Example:

    Sub BadGoto()
        Dim x = 0

        GoTo Label1
        If x > 5 Then
Label1:
            Dim y = 5
            Dim f = Function() y
        End If
    End Sub

Message: error BC36597: 'GoTo Label1' is not valid because 'Label1' is inside a scope that defines a variable that is used in a lambda or query expression.

If you look back at Part 3 of this series you will see that a lot of work goes into initializing closures inside of a scope. Unfortunately allowing a user to jump into a block that contains a closure makes respecting these rules very difficult. In a even trivial example in makes the resulting code mostly unreadable. We decided to disable this in Orcas and reconsider it in a future release.

It is perfectly legal however to jump into any scope that is currently visible regardless of whether or not in contains a closure. Because jumping into a visible scope does not affect the creation of a variable lifetime (just the ending), it does not add any complications to the code.

5. Cannot mix "On Error Goto" and Closures

Message: error BC36595: Method cannot contain both an 'On Error GoTo' statement and a lambda or query expression.

Because of restriction #4 we must disable this scenario as well since it's very easy to hit this scenario with "On Error Goto".

Closures in VB Part 5: Looping

Jared Parsons — Thu, 26 Jul 2007 19:39:05 GMT

For previous articles in the series please see

Once again sorry for the long delay between posts.

Looping structures can cause unintended consequences when used with Lambda expressions. The problem occurs because lambda expressions do not execute when they are constructed but rather when they are invoked. For example take the following code.

    Sub LoopExample1()
        Dim list As New List(Of Func(Of Integer))
        For i = 0 To 3
            list.Add(Function() i)
        Next

        For Each cur In list
            Console.Write("{0} ", cur())
        Next
    End Sub

Many users are surprised to find out the above will print "4 4 4 4 ". The reason goes back to my previous 2 posts on variable lifetime and scope. All "For" and "For Each" blocks in Vb have 2 scopes.

Scope where iteration variables are defined
Body of the for loop

The first scope is entered only once no matter how many times the loop is executed. The second is entered once per iteration of the loop. Any iteration variables that are defined in a For/For Each loop are created in the first scope (in this case "i" and "cur"). Hence there is only one of those variables for every loop iteration and the lambda function lifts the single variable "i".

This has thrown off many users because the behavior works most of the time. For instance if I switched the code to run "Console.Write" inside the first loop, it would print out "0 1 2 3 " as expected.

To mitigate against this problem the above code will actually produce a warning in VB.

warning BC42324: Using the iteration variable in a lambda expression may have unexpected results. Instead, create a local variable within the loop and assign it the value of the iteration variable.

There are two ways to fix this problem depending on the behavior you want. If you see this warning and don't know if it affects you, the safest change is to do the following.

    Sub LoopExample2()
        Dim list As New List(Of Func(Of Integer))
        For iTemp = 0 To 3
            Dim i = iTemp
            list.Add(Function() i)
        Next

        For Each cur In list
            Console.Write("{0} ", cur())
        Next
    End Sub

This will cause "i" to be created in the second scope. Hence there will be a different value for every loop iteration and the code will print out "0 1 2 3" as expected.

If you do want the code to print out "4 4 4 4 " then add "Dim i = 0" before the start of the loop.

Closures in VB Part 4: Variable Lifetime

Jared Parsons — Fri, 15 Jun 2007 23:47:44 GMT

For previous articles in this series please see

Sorry for the long delay between posts here. We're getting Orcas out the door and getting this series completed takes a back door to shipping.

Originally I wanted to talk about looping structures next. However when I started writing that post I realized that I had to talk about lifetime before the looping structures would make sense.

Prior to Orcas the lifetime of a variable in VB was the entire function. This presented several problems from a closures perspective. Imagine you had a looping structure and the value was used in a lambda expression.

    Sub LifetimeExample()
        Dim list As New List(Of Func(Of Integer))
        For i = 0 To 5

            Dim x = i * 2
            If True Then
                list.Add(Function() x)
            End If
        Next

        For Each f In list
            Console.Write(f() & " ")
        Next
    End Sub

In this example if we left the lifetime rules unchanged, there would be a single variable "x" for the entire function. That means that we would end up printing out

10 10 10 10 10

This is somewhat unexpected and essentially means that VB could not support complex Lambda scenarios. To fix this we altered the lifetime of variables to be tied to the scope they were contained in. The end effect is that each iteration of the loop has a separate "x" since each iteration enters and leaves the scope of the "if" statement. As a result it will print out

0 2 4 6 8 10

We did make one backcompat adjustment for this change. The lifetime of variables in VB was visible if you tried to use an uninitialized variable in a loop/goto. For instance the following code will also print out 0 2 4 6 8 10 because it takes advantage of the fact that the variable "x" has a lifetime longer than the loop.

    Sub VisibleLifetime()
        For i = 0 To 5
            Dim x As Integer
            Console.WriteLine(x)
            x += 2
        Next
    End Sub

To make sure that we didn't break any existing code we had one little errata for the change. When a variable's scope is re-entered, and hence recreated, and it is not initialized to a value it will get the last value of the variable.

Closures in VB Part 3: Scope

Jared Parsons — Fri, 25 May 2007 23:59:45 GMT

For previous articles in this series please see

Thus far in the series we've only lifted variables that are declared in the same block/scope. What happens if we lift variables in different scope? The answer is that one closure class will be created for every unique scope where a lifted variable is declared and all of the variables in that scope that are lifted will be placed in that closure. Once again, examples speak best

    Sub Scope1()
        Dim x = 5
        Dim f1 = Function(ByVal z As Integer) x + z
        Console.WriteLine(f1(5))
        If x > 2 Then
            Dim y = 6
            Dim g = 7
            Dim f2 = Function(ByVal z As Integer) z + y + g
            Console.WriteLine(f2(4))
        End If
    End Sub

The code will end up looking like so ...

    Class Closure1
        Public x As Integer

        Function Lambda_f1(ByVal z As Integer)
            Return x + z
        End Function

    End Class

    Class Closure2
        Public y As Integer
        Public g As Integer

        Function Lambda_f2(ByVal z As Integer)
            Return y + z + g
        End Function
    End Class

    Sub Scope1()
        Dim c1 As New Closure1()
        c1.x = 5
        Console.WriteLine(c1.Lambda_f1(5))
        If c1.x > 2 Then
            Dim c2 As New Closure2()
            c2.y = 6
            c2.g = 7
            Console.WriteLine(c2.Lambda_f2(4))
        End If
    End Sub

There are a couple of items to take away from this example.

Only two closure classes were created even though three variables were lifted. The number of closures only depends on the number of scopes of all of the lifted declared variables.
The closures are created at the begining of the scope they are associated and not at the begining of the method. This will be more important in the next part of the series.
Each lambda instance is attached to the closure associated with the scope the lambda is declared in.

The next twist is what were to happen if the lambda "f2" were to also use the variable "x". As it's currently written there is no association between Closure1 and Closure2 therefore there is no way for it to access the lifted variable. The answer is two fold. Firstly to reduce clutter I pasted the closure classes as if they were separate entries. In fact Closure2 would appear as a nested class of Closure1 in the real generated code.

Secondly if x were used inside of "f2", the real use would be "c1.x". That's (almost) no different than "someOtherVar.x". Therefore the instance of c1 will be lifted into Closure2.

Dim f2 = Function(ByVal z As Integer) z + y + g + x

Woud result in the following definition of Closure2 ...

    Class Closure2
        Public y As Integer
        Public g As Integer
        Public c1 As Closure1

        Function Lambda_f2(ByVal z As Integer)
            Return y + z + g + c1.x
        End Function
    End Class

In deeply nested lambdas and scopes this type of lifting will continue recursively.

That's it for this entry, the next article will talk about looping structures and possibly variable lifetime.

Closures in VB Part 2: Method Calls

Jared Parsons — Fri, 04 May 2007 00:56:51 GMT

For previous articles in this series, please see

Part 1 - The basics

This part of the series will focus on how method calls are handled in closures. As stated in the previous article, the purpose of closures is to allow all operations inside a lambda or query expression that would normally be available inside the function or sub. To do this closures often need to capture (or lift) relevant variables from the function into the generated class.

There are 2 types of methods and method calls that closures have to handle.

Method calls to a shared method or methods on modules.
Method calls to instance members of a class

Scenario #1

Below is an example of a method call inside a lambda expression for scenario #1.

Module M1

    Function MyValue() As Integer
        Return 42
    End Function

    Sub Example1()
        Dim x = 5
        Dim f = Function() x + MyValue()
    End Sub

End Module

Here we are calling a module method inside a lambda. Module Methods or Shared methods can be called from anywhere because they require no specific variable for the call. This requires no special work from closures as the call can just be made naturally.

    Class Closure
        Private x As Integer

        Function Lambda_f() As Integer
            Return x + M1.MyValue
        End Function
    End Class

Scenario #2

Calling an instance method is more difficult than a shared method because it requires the referenc "Me". If you don't type this specifically in code the VB Compiler will add it for you under the hood. To make this work the closures code will also "lift" the variable "Me" in the same way that it lifts normal variables in a function.

Calling a instance method inside a lambda expression is little difference than calling a member method on a variable used in a lambda. The only difference is the variable is "Me". For example

Class C1
    Private m_myValue As Integer

    Function MyValue() As Integer
        Return m_myValue
    End Function

    Sub Example2()
        Dim x = 5
        Dim f = Function() x + MyValue()
    End Sub
End Class

In this case we need to access both "x" and "Me.MyValue()" from the closure. The generated code will create space for both of these variables and the transformed code in Example2 will store both of the values.

Class Closure
    Private x As Integer
    Private OriginalMe As C1

    Function Lambda_f()
        Return x + OriginalMe.MyValue()
    End Function
End Class

    Sub Example2()
        Dim c As New Closure
        c.x = 5
        c.OriginalMe = Me
        Dim f = New Func(Of Integer)(AddressOf c.Lambda_f)
    End Sub

As usual, the generated code is much uglier but this essentially what will be generated. That wraps it up for method calls. In the next part, I will discuss the variable liftetime and scoping issues that come into play with closures.

Closures in VB: Part 1

Jared Parsons — Sat, 28 Apr 2007 01:50:00 GMT

One of the features I implemented for VB 9.0 is lexical closure support. This a great addition to the VB language and I wanted to do a series of blog posts to describe this feature and how it will impact your code.

Lexical Closures (more often referred to as simply Closures) are the underpinnings for several new features in Visual Basic 9.0. The are part of the guts of Lambda and Query expressions. This will be a several part series on Closures in VB 9.0; how they work, their limitations, pitfalls surrounding their use.

To start off, let's get a basic summary of what a Closure is. Wikipedia defines it as "... a is a semantic concept referring to a function paired with an environment ...". I prefer to describe it as follows. A closure is a feature which allows users to seemlessly access an environment (locals, parameters and methods) from more than one function. Even better are samples :)

    Class C1
        Sub Test()
            Dim x = 5
            Dim f = Function(ByVal y As Integer) x + y
            Dim result = f(42)
        End Sub

    End Class

In this code we have a lambda expression which takes in a single parameter and adds it with a local variable. Lambda expressions are implemented as functions in VB (and C#). So now we have two functions, "Test" and "f", which are accessing a single local variable. This is where closures come into play. Closures are responsible for making the single variable "x" available to both functions in a process that is referred to as "lifting the variable".

To do this the compiler will take essentially 4 actions.

Create a class which will contain "x" in order to share it among both functions. Call it "Closure" for now
It will create a new function for the lambda expression in the class "Closure". Call it "f" for now
Create a new instance of the class "Closure" inside the sub "Test"
Rewrite all access of "x" into the member "x" of "Closure".

    Class Closure
        Public x As Integer

        Function f(ByVal y As Integer) As Integer
            Return x + y
        End Function
    End Class

    Class C1
        Sub Test()
            Dim c As New Closure()
            c.x = 5
            Dim f As Func(Of Integer, Integer) = AddressOf c.f
            Dim result = f(42)
        End Sub

    End Class

Now "x" is shared amongst both functions and the user didn't have to know anything about the code we generated. You can see from this simplified example just how much code Closures and all of the other new VB 9.0 features are saving you here (Type Inference, Lambda Expressions).

Note this is only a simulation of what is generated when you use a closure, the actual generated code is much uglier and involves lots of unbindable names "$Lambda_1", etc ...

In the next part of this article I'll dive into some more uses of closures (multiple variables, method access, terminology, etc...).