jaredpar's WebLog : DotNet

BclExtras Library

Jared Parsons — Fri, 23 Jan 2009 16:00:00 GMT

I published a .Net utility library on Code Gallery today called BclExtras. It’s a set of classes meant to be used in addition to the standard .Net base class libraries (BCL). The main focuses of the library are functional programming, multi-threading, LINQ extensions, unit testing and API’s designed to support type inference.

This project evolved from various classes and constructs I was using in personal projects. For the last year or so I’ve kept as a separate tested library. It started out with a lot of multi-threaded code constructs but lately is leaning to a lot of functional style API’s and collections.

The library includes source and binaries for .Net 2.0 and .Net 3.5. The .Net 2.0 version of the library includes many constructs added in 3.5 that don’t rely on any 2.0SP1 CLR features. Examples are Func<T>, Action<T>, the Extension attribute and a subset of the LINQ Enumerable class. It allows for most LINQ expressions in a 2.0 targeted application. These types are removed in the 3.5 version to avoid conflicts with types in System.Core.

I’ve previously released this library under the name RantPack. It originally started out as personal utility library of mine and hence ended up with the somewhat obscure name. But, besides to me, RantPack doesn’t really convey a useful meaning. So I decided to give it a more meaningful name for the general population.

Is IntPtr(long) truncating?

Jared Parsons — Tue, 28 Oct 2008 15:00:00 GMT

The short answer is: No, not when it matters

A colleague and I were discussing a particular scenario around IntPtr,PInvoke and 64 bit correctness. Eventually our discussion lead us to the IntPtr constructor which takes a long. To my surprise the code for the constructor is the following.

public unsafe IntPtr(long value) { this.m_value = (void*) ((int) value); }

The problem is long value is arbitrarily truncated to an int. This has the effect of essentially losing any address over the 4 GB range (in other words, no 64 bit addresses). This much to big of a hole to actually be the real behavior so I decided to see if it was a bug in the disassembler. I was using .Net Reflector so I switched to IL mode.

    L_0000: ldarg.0 
    L_0001: ldarg.1 
    L_0002: conv.ovf.i4 
    L_0003: conv.i 
    L_0004: stfld void* System.IntPtr::m_value
    L_0009: ret

This confirmed it is indeed truncating the value (and doing an overflow check to boot). But wait, mscorlib.dll is a processor specific DLL so perhaps this is just a 32 bit OS thing. I switched over to a 64 bit machine, fired up Reflctor and found to my dismay that it had the exact same code.

After a few minutes I thought to open up task manager and to my surprise reflector was running in a WoW64 bit process. This meant it was still loading up the 32 bit version of mscorlib.dll. Next I fired up ildasm, loaded up a 64 bit mscorlib and confirmed that the code will not truncate on 64 bit machines.

  IL_0000:  ldarg.0
  IL_0001:  ldarg.1
  IL_0002:  conv.u
  IL_0003:  stfld      void* System.IntPtr::m_value
  IL_0008:  ret

The conv.u code is a conversion to unsigned native platform int. On a 64 bit machine this will be an unsigned 8 byte number(see OpCodes.Conv_U for more details).

So what does this mean for the developer. Essentially IntPtr(long) will do the right thing independently of the platform a developer is using. On a 32 bit platform it will (correctly) throw exceptions if a non-4GB address is passed in. In 64 bit land it will essentially do nothing and rely on the programmer to give correct addresses.

RantPack - A utility library

Jared Parsons — Thu, 01 May 2008 06:47:00 GMT

I often post code examples, samples and snippets on this blog. Many of these samples are a part of a utility library I've been writing and maintaining for many years now. Essentially since I got involved in DotNet programming.

I write a lot of code for internal apps, demos and hobby projects. Having a utility library to tote around with me greatly increases my impact in a project because it gives me a place to put common code that I would otherwise have to repeatedly cut and paste into projects.

Whenever I work on something I think the community at large would benefit from I try to blog about it. Unfortunately for the more complex samples blogging is not always easy. Complex structures don't really fit on a blog and often utilities depend on utilities and it forces users to cut and paste between multiple blog entries. Not good.

Now with http://code.msdn.com I have a forum to fully share the items I've been working on. This is the perfect place to share code for the community and helps avoid the messiness of cutting and pasting blog posts. As such I've released my Library, RantPack, on Code Gallery. This includes both binaries and sources and is covered under the Microsoft Public License.

RantPack is available at http://code.msdn.microsoft.com/RantPack

A couple of notes on the sample. This is mainly for education and sharing purposes. The goal of this library is to make developing software faster and more reliable. It includes, but is not limited to

Functional Programming Patterns

Tuples
Immutable/Persistent collections

Threading

Futures
Active Objects
Various other primitives

General Patterns
General Utilities

Another item you will find browsing the sources are tests. Lots and lots of tests. At the time of this writing there are around 400 some odd unit tests on the library. There is nothing more frustrating than a utility library that doesn't work. As such I rigorously test the code I write.

Although I performance test much of my code and it performs great in the applications I use it for I'm sure there are a few cases I missed. Please feel free to point this out to me and I'll look into them.

Why the name RantPack? My blog is Rantings of a programmer and it's a pack of code :)

IEnumerable and IEnumerable(Of T) 2

Jared Parsons — Fri, 05 Oct 2007 18:43:06 GMT

Quick follow up to my earlier post. Fixing this issue in C# is even easier because of the existence of iterators.

        public static IEnumerable<object> Shim(System.Collections.IEnumerable enumerable)
        {
            foreach (var cur in enumerable)
            {
                yield return cur;
            }
        }

.Net Framework Source Code

Jared Parsons — Wed, 03 Oct 2007 21:58:04 GMT

If you haven't heard the news yet, you can read the full article on ScottGu's Blog.

In summary Microsoft has released the source code for the .Net Framework in such a way that you can step into the Framework while debugging. IMHO this is great for users because it allows us to get a lot of insight into parts of the API that are tricky or not well documented.

Take a look at his blog for the full details of hooking this up into the debugger.

Closures in VB Part 6: Limitations

Jared Parsons — Tue, 07 Aug 2007 01:36:40 GMT

For previous articles in this series please see ...

As powerful as closures are in the language they do have a few limitations. We worked hard in Orcas to put as few limitations in Orcas as possible. Below are the current limitations and some insight into why they exist this way.

1. Cannot use "ByRef" parameters in a closure

Example:

    Sub LiftAByRef(ByRef x As Integer)
        Dim f = Function() x
    End Sub

Message: error BC36639: 'ByRef' parameter 'x' cannot be used in a lambda expression.

The problem here is the expectation surrounding x. Any change in the value of "x" inside the method "LiftAByRef" should be reflected in the calling function. Normally for any lifted parameter we add a new field inside the closure and all read/writes are redirected into that value. For "ByRef" parameters we would additionally have to ensure that all writes are make to the parameter. Even in the presence of an exception. Not a trivial task.

2. Cannot use "Me" in a closure created inside a structure.

Example:

Structure S1
    Public F1 As Integer

    Public Sub M1()
        Dim f = Function() F1
    End Sub
End Structure

Message: error BC36638: Instance members and 'Me' cannot be used within a lambda expression in structures

Closures capture values by reference. It's not possible to capture the "Me" of a structure by reference in VB. The only other option is to capture them by value. If we did that then all changes to members of a structure inside a lambda would not affect the structure in which they were created; merely the value copy. This is very different behavior from every other place that closures are used. To avoid confusing behavior this is not a legal operation.

3. Cannot use a Restricted Type in a closure

Example:

    Sub LiftRestrictedType()
        Dim x As ArgIterator = Nothing
        Dim f = Function() x.GetNextArgType().GetModuleHandle()
    End Sub

Message: error BC36640: Instance of restricted type 'System.ArgIterator' cannot be used in a lambda expression.

This hopefully will not affect many users. There are several types in the CLR that are considered restricted because they have special semantics. Typically they are special cased by the CLR and as such we can't use them in a closure. Several of these cannot be used in VB at all. They are ...

System.TypedReference
System.ArgIterator
System.RuntimeArgumentHandle

4. Cannot Goto into scope that contains a closure

Example:

    Sub BadGoto()
        Dim x = 0

        GoTo Label1
        If x > 5 Then
Label1:
            Dim y = 5
            Dim f = Function() y
        End If
    End Sub

Message: error BC36597: 'GoTo Label1' is not valid because 'Label1' is inside a scope that defines a variable that is used in a lambda or query expression.

If you look back at Part 3 of this series you will see that a lot of work goes into initializing closures inside of a scope. Unfortunately allowing a user to jump into a block that contains a closure makes respecting these rules very difficult. In a even trivial example in makes the resulting code mostly unreadable. We decided to disable this in Orcas and reconsider it in a future release.

It is perfectly legal however to jump into any scope that is currently visible regardless of whether or not in contains a closure. Because jumping into a visible scope does not affect the creation of a variable lifetime (just the ending), it does not add any complications to the code.

5. Cannot mix "On Error Goto" and Closures

Message: error BC36595: Method cannot contain both an 'On Error GoTo' statement and a lambda or query expression.

Because of restriction #4 we must disable this scenario as well since it's very easy to hit this scenario with "On Error Goto".

Detecting if you are an Admin

Jared Parsons — Wed, 01 Aug 2007 18:56:02 GMT

This came up on an internal alias. A customer wanted to know how to determine if there were running as an admin in a tool. Below is a sample program that will print out whether or not you are the machine admin or a member of the machine administrators group.

This is essentially the same as the code in a previous post of mine but ported to VB. http://blogs.msdn.com/jaredpar/archive/2005/11/07/489942.aspx

Imports System.Security.Principal
Module Module1

    Function IsRunningAsLocalAdmin() As Boolean
        Dim cur As WindowsIdentity = WindowsIdentity.GetCurrent()
        For Each role As IdentityReference In cur.Groups
            If role.IsValidTargetType(GetType(SecurityIdentifier)) Then
                Dim sid As SecurityIdentifier = DirectCast(role.Translate(GetType(SecurityIdentifier)), SecurityIdentifier)
                If sid.IsWellKnown(WellKnownSidType.AccountAdministratorSid) OrElse sid.IsWellKnown(WellKnownSidType.BuiltinAdministratorsSid) Then
                    Return True
                End If

            End If
        Next

        Return False
    End Function

    Sub Main()
        Console.WriteLine("Is Admin {0}", IsRunningAsLocalAdmin())
    End Sub

End Module

Coding Quiz: Anonymous Type Types

Jared Parsons — Wed, 01 Aug 2007 18:34:37 GMT

Question: How can you create a variable in VB which is typed as an anonymous type but not actually create an instance of that type?

Answer in comments.

Note, doing this is not particularly useful it came about while I was playing around with a feature a few days ago.

IMetaDataImport::GetParamForMethodIndex

Jared Parsons — Mon, 30 Jul 2007 20:09:46 GMT

While investigating a recent bug I found about an interesting return for GetParamForMethodIndex. On a perfectly verifiable assembly, a call to GetParamForMethodIndex was returning a failure code. After some searching I found the return code was CLDB_E_RECORD_NOTFOUND. I was surprised at first because it's a verifiable assembly so how could the record for a parameter I knew existed not be there?

It turns out this is legal. GetParamForMethodIndex returns a mdParamDef token by which you can query for information about a parameter with GetParamProps. This will return the following information about a parameter.

Name of the parameter
Attributes about the param (ByRef, Marshal, etc ...)
Default value

In this particular case the assembly was generated without a parameter name. As it also had none of the information the parameter row was omitted from the metadata. The reason is adding an empty row takes up space and provides no data.

Note you can't reproduce this behavior with ILASM.exe. If you omit a parameter name, ILASM.exe will add one for you and hence generate a parameter row.

Closures in VB Part 5: Looping

Jared Parsons — Thu, 26 Jul 2007 19:39:05 GMT

For previous articles in the series please see

Once again sorry for the long delay between posts.

Looping structures can cause unintended consequences when used with Lambda expressions. The problem occurs because lambda expressions do not execute when they are constructed but rather when they are invoked. For example take the following code.

    Sub LoopExample1()
        Dim list As New List(Of Func(Of Integer))
        For i = 0 To 3
            list.Add(Function() i)
        Next

        For Each cur In list
            Console.Write("{0} ", cur())
        Next
    End Sub

Many users are surprised to find out the above will print "4 4 4 4 ". The reason goes back to my previous 2 posts on variable lifetime and scope. All "For" and "For Each" blocks in Vb have 2 scopes.

Scope where iteration variables are defined
Body of the for loop

The first scope is entered only once no matter how many times the loop is executed. The second is entered once per iteration of the loop. Any iteration variables that are defined in a For/For Each loop are created in the first scope (in this case "i" and "cur"). Hence there is only one of those variables for every loop iteration and the lambda function lifts the single variable "i".

This has thrown off many users because the behavior works most of the time. For instance if I switched the code to run "Console.Write" inside the first loop, it would print out "0 1 2 3 " as expected.

To mitigate against this problem the above code will actually produce a warning in VB.

warning BC42324: Using the iteration variable in a lambda expression may have unexpected results. Instead, create a local variable within the loop and assign it the value of the iteration variable.

There are two ways to fix this problem depending on the behavior you want. If you see this warning and don't know if it affects you, the safest change is to do the following.

    Sub LoopExample2()
        Dim list As New List(Of Func(Of Integer))
        For iTemp = 0 To 3
            Dim i = iTemp
            list.Add(Function() i)
        Next

        For Each cur In list
            Console.Write("{0} ", cur())
        Next
    End Sub

This will cause "i" to be created in the second scope. Hence there will be a different value for every loop iteration and the code will print out "0 1 2 3" as expected.

If you do want the code to print out "4 4 4 4 " then add "Dim i = 0" before the start of the loop.

Closures in VB Part 4: Variable Lifetime

Jared Parsons — Fri, 15 Jun 2007 23:47:44 GMT

For previous articles in this series please see

Sorry for the long delay between posts here. We're getting Orcas out the door and getting this series completed takes a back door to shipping.

Originally I wanted to talk about looping structures next. However when I started writing that post I realized that I had to talk about lifetime before the looping structures would make sense.

Prior to Orcas the lifetime of a variable in VB was the entire function. This presented several problems from a closures perspective. Imagine you had a looping structure and the value was used in a lambda expression.

    Sub LifetimeExample()
        Dim list As New List(Of Func(Of Integer))
        For i = 0 To 5

            Dim x = i * 2
            If True Then
                list.Add(Function() x)
            End If
        Next

        For Each f In list
            Console.Write(f() & " ")
        Next
    End Sub

In this example if we left the lifetime rules unchanged, there would be a single variable "x" for the entire function. That means that we would end up printing out

10 10 10 10 10

This is somewhat unexpected and essentially means that VB could not support complex Lambda scenarios. To fix this we altered the lifetime of variables to be tied to the scope they were contained in. The end effect is that each iteration of the loop has a separate "x" since each iteration enters and leaves the scope of the "if" statement. As a result it will print out

0 2 4 6 8 10

We did make one backcompat adjustment for this change. The lifetime of variables in VB was visible if you tried to use an uninitialized variable in a loop/goto. For instance the following code will also print out 0 2 4 6 8 10 because it takes advantage of the fact that the variable "x" has a lifetime longer than the loop.

    Sub VisibleLifetime()
        For i = 0 To 5
            Dim x As Integer
            Console.WriteLine(x)
            x += 2
        Next
    End Sub

To make sure that we didn't break any existing code we had one little errata for the change. When a variable's scope is re-entered, and hence recreated, and it is not initialized to a value it will get the last value of the variable.

Closures in VB Part 3: Scope

Jared Parsons — Fri, 25 May 2007 23:59:45 GMT

For previous articles in this series please see

Thus far in the series we've only lifted variables that are declared in the same block/scope. What happens if we lift variables in different scope? The answer is that one closure class will be created for every unique scope where a lifted variable is declared and all of the variables in that scope that are lifted will be placed in that closure. Once again, examples speak best

    Sub Scope1()
        Dim x = 5
        Dim f1 = Function(ByVal z As Integer) x + z
        Console.WriteLine(f1(5))
        If x > 2 Then
            Dim y = 6
            Dim g = 7
            Dim f2 = Function(ByVal z As Integer) z + y + g
            Console.WriteLine(f2(4))
        End If
    End Sub

The code will end up looking like so ...

    Class Closure1
        Public x As Integer

        Function Lambda_f1(ByVal z As Integer)
            Return x + z
        End Function

    End Class

    Class Closure2
        Public y As Integer
        Public g As Integer

        Function Lambda_f2(ByVal z As Integer)
            Return y + z + g
        End Function
    End Class

    Sub Scope1()
        Dim c1 As New Closure1()
        c1.x = 5
        Console.WriteLine(c1.Lambda_f1(5))
        If c1.x > 2 Then
            Dim c2 As New Closure2()
            c2.y = 6
            c2.g = 7
            Console.WriteLine(c2.Lambda_f2(4))
        End If
    End Sub

There are a couple of items to take away from this example.

Only two closure classes were created even though three variables were lifted. The number of closures only depends on the number of scopes of all of the lifted declared variables.
The closures are created at the begining of the scope they are associated and not at the begining of the method. This will be more important in the next part of the series.
Each lambda instance is attached to the closure associated with the scope the lambda is declared in.

The next twist is what were to happen if the lambda "f2" were to also use the variable "x". As it's currently written there is no association between Closure1 and Closure2 therefore there is no way for it to access the lifted variable. The answer is two fold. Firstly to reduce clutter I pasted the closure classes as if they were separate entries. In fact Closure2 would appear as a nested class of Closure1 in the real generated code.

Secondly if x were used inside of "f2", the real use would be "c1.x". That's (almost) no different than "someOtherVar.x". Therefore the instance of c1 will be lifted into Closure2.

Dim f2 = Function(ByVal z As Integer) z + y + g + x

Woud result in the following definition of Closure2 ...

    Class Closure2
        Public y As Integer
        Public g As Integer
        Public c1 As Closure1

        Function Lambda_f2(ByVal z As Integer)
            Return y + z + g + c1.x
        End Function
    End Class

In deeply nested lambdas and scopes this type of lifting will continue recursively.

That's it for this entry, the next article will talk about looping structures and possibly variable lifetime.

Closures in VB Part 2: Method Calls

Jared Parsons — Fri, 04 May 2007 00:56:51 GMT

For previous articles in this series, please see

Part 1 - The basics

This part of the series will focus on how method calls are handled in closures. As stated in the previous article, the purpose of closures is to allow all operations inside a lambda or query expression that would normally be available inside the function or sub. To do this closures often need to capture (or lift) relevant variables from the function into the generated class.

There are 2 types of methods and method calls that closures have to handle.

Method calls to a shared method or methods on modules.
Method calls to instance members of a class

Scenario #1

Below is an example of a method call inside a lambda expression for scenario #1.

Module M1

    Function MyValue() As Integer
        Return 42
    End Function

    Sub Example1()
        Dim x = 5
        Dim f = Function() x + MyValue()
    End Sub

End Module

Here we are calling a module method inside a lambda. Module Methods or Shared methods can be called from anywhere because they require no specific variable for the call. This requires no special work from closures as the call can just be made naturally.

    Class Closure
        Private x As Integer

        Function Lambda_f() As Integer
            Return x + M1.MyValue
        End Function
    End Class

Scenario #2

Calling an instance method is more difficult than a shared method because it requires the referenc "Me". If you don't type this specifically in code the VB Compiler will add it for you under the hood. To make this work the closures code will also "lift" the variable "Me" in the same way that it lifts normal variables in a function.

Calling a instance method inside a lambda expression is little difference than calling a member method on a variable used in a lambda. The only difference is the variable is "Me". For example

Class C1
    Private m_myValue As Integer

    Function MyValue() As Integer
        Return m_myValue
    End Function

    Sub Example2()
        Dim x = 5
        Dim f = Function() x + MyValue()
    End Sub
End Class

In this case we need to access both "x" and "Me.MyValue()" from the closure. The generated code will create space for both of these variables and the transformed code in Example2 will store both of the values.

Class Closure
    Private x As Integer
    Private OriginalMe As C1

    Function Lambda_f()
        Return x + OriginalMe.MyValue()
    End Function
End Class

    Sub Example2()
        Dim c As New Closure
        c.x = 5
        c.OriginalMe = Me
        Dim f = New Func(Of Integer)(AddressOf c.Lambda_f)
    End Sub

As usual, the generated code is much uglier but this essentially what will be generated. That wraps it up for method calls. In the next part, I will discuss the variable liftetime and scoping issues that come into play with closures.

Closures in VB: Part 1

Jared Parsons — Sat, 28 Apr 2007 01:50:00 GMT

One of the features I implemented for VB 9.0 is lexical closure support. This a great addition to the VB language and I wanted to do a series of blog posts to describe this feature and how it will impact your code.

Lexical Closures (more often referred to as simply Closures) are the underpinnings for several new features in Visual Basic 9.0. The are part of the guts of Lambda and Query expressions. This will be a several part series on Closures in VB 9.0; how they work, their limitations, pitfalls surrounding their use.

To start off, let's get a basic summary of what a Closure is. Wikipedia defines it as "... a is a semantic concept referring to a function paired with an environment ...". I prefer to describe it as follows. A closure is a feature which allows users to seemlessly access an environment (locals, parameters and methods) from more than one function. Even better are samples :)

    Class C1
        Sub Test()
            Dim x = 5
            Dim f = Function(ByVal y As Integer) x + y
            Dim result = f(42)
        End Sub

    End Class

In this code we have a lambda expression which takes in a single parameter and adds it with a local variable. Lambda expressions are implemented as functions in VB (and C#). So now we have two functions, "Test" and "f", which are accessing a single local variable. This is where closures come into play. Closures are responsible for making the single variable "x" available to both functions in a process that is referred to as "lifting the variable".

To do this the compiler will take essentially 4 actions.

Create a class which will contain "x" in order to share it among both functions. Call it "Closure" for now
It will create a new function for the lambda expression in the class "Closure". Call it "f" for now
Create a new instance of the class "Closure" inside the sub "Test"
Rewrite all access of "x" into the member "x" of "Closure".

    Class Closure
        Public x As Integer

        Function f(ByVal y As Integer) As Integer
            Return x + y
        End Function
    End Class

    Class C1
        Sub Test()
            Dim c As New Closure()
            c.x = 5
            Dim f As Func(Of Integer, Integer) = AddressOf c.f
            Dim result = f(42)
        End Sub

    End Class

Now "x" is shared amongst both functions and the user didn't have to know anything about the code we generated. You can see from this simplified example just how much code Closures and all of the other new VB 9.0 features are saving you here (Type Inference, Lambda Expressions).

Note this is only a simulation of what is generated when you use a closure, the actual generated code is much uglier and involves lots of unbindable names "$Lambda_1", etc ...

In the next part of this article I'll dive into some more uses of closures (multiple variables, method access, terminology, etc...).

CoSetProxyBlanket and Managed Code

Jared Parsons — Thu, 19 Apr 2007 19:15:39 GMT

When running FxCop on any managed code that uses CoSetProxyBlanket you will see an error message saying the method cannot be called reliably from managed code. I've hit that message before was frustrated by my attempts to find an explanation on the web. Part of the reason is I'm not the most efficient searcher on the web.

However and internal discussion revealed to me a great blog entry explaining exactly why this is unreliable and more importantly how to work around the issue.

http://blogs.msdn.com/mbend/archive/2007/04/18/cosetproxyblanket-not-supported-from-managed-code.aspx

Technorati tags: DotNet