Haibo Luo's weblog

ILVisualizer 2010 Solution

Haibo Luo - MSFT — Mon, 19 Apr 2010 17:38:00 GMT

The projects are set to be built against the .NET 4.0.

IronPython: System.Reflection.Assembly object

Haibo Luo - MSFT — Thu, 13 Mar 2008 06:12:00 GMT

IronPython offers a little bit more love to the Assembly object instance: we can directly access the assembly's top-level members (namespace, public type) via the dot operation. System.Reflection provides many ways to let you hold the assembly object, such as Assembly.Load method, Type.Assembly property, ...

For example, given assembly1.dll compiled from the C# file below, dir(a1) in the python below shows the class name "C2" and the top level namespace "NS1" in its' dictionary.

// assembly1.cs
namespace NS1 {
    namespace NS2 {
        public class C1 { }
    }
}
public class C2 {
    public class C3 { }
}
class C4 { }

# part 1 from file/test1.py
import System
a1 = System.Reflection.Assembly.Load("assembly1")

print dir(a1)  # ['C2', ..., 'NS1', ...]
print a1.NS1   # <module 'NS1' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print a1.C2    # <type 'C2'>

# access non-public type
print a1.C4    # AttributeError: 'Assembly' object has no attribute 'C4'

# access the down-level namespace, type with the dot operations
print a1.NS1.NS2.C1         # <type 'C1'>
# access the nested type
print a1.C2.C3              # <type 'C3'>

Normally I would use clr.AddReference to bring in the CLR assembly. The python code below gives the similar output.

# part 1 of file/test2.py
import clr
clr.AddReference("assembly1")
import NS1 
import C2

print NS1        # <module 'NS1' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print C2         # <type 'C2'>
print NS1.NS2    # <module 'NS2' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print NS1.NS2.C1 # <type 'C1'>
print C2.C3      # <type 'C3'>

However the first approach may help keep the global (and local) dictionary less "polluted" and sometimes can be used to avoid the namespace/type name collision. For example, suppose we need to load in another assembly built from the following C# code, and we use the clr.AddReference approach.

// assembly2.cs
namespace NS1 { 
    public class NS2 { } 
}

# part 2 of file/test2.py
clr.AddReference("assembly2")
print NS1.NS2    # <type 'NS2'>
print NS1.NS2.C1 # AttributeError: 'type' object has no attribute 'C1'

NS2 is a type now (no longer the namespace (or python module) NS1.NS2 from "assembly1"), and we lost the access to the type NS1.NS2.C1. If we use the assembly object approach, we can hold both the namespace NS1.NS2 from assembly1 and the type NS1.NS2 from assembly2; basically we can think the assembly variable name adds one more layer to prevent the name collision. Or in the other direction of thinking, what clr.AddReference does for you (related to implicit name merge/override) may not be what exactly you want.

# part 2 from file/test1.py
a2 = System.Reflection.Assembly.Load("assembly2")
print a2.NS1     # <module 'NS1' (CLS module from assembly2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print a2.NS1.NS2 # <type 'NS2'>
print a1.NS1     # <module 'NS1' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print a1.NS1.NS2 # <module 'NS2' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>

Note that such name collision should rarely happen for well-designed framework assemblies. Also IronPython merge types/down-level namespaces under the same namespaces, even if they are from different assemblies. One simple example is System.Int32 from mscorlib.dll and System.Uri from System.dll, both peacefully under the "System" module.

import System
print System            
# <module 'System' (CLS module, 2 assemblies loaded)>
print System.__file__   
# ['mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089', 
#  'System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089']
import clr
print clr.GetClrType(System.Int32).Assembly  
# <Assembly mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089>
print clr.GetClrType(System.Uri).Assembly    
# <Assembly System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089>

IronPython: System.Array

Haibo Luo - MSFT — Thu, 13 Mar 2008 06:04:13 GMT

When you start interop'ing with .NET in IronPython, sooner or later, you will find that you are in need of creating an array as argument. There are mainly 2 ways to create array objects:

Array with type indexing (for one-dimensional array object only), and
classic reflection API: Array.CreateInstance

System.Array indexing with a type creates a concrete array type, which can then take a collection of the element objects (or an enumerable) and form the array object. There are half dozen Array.CreateInstance overloads, which allow us to create one-dimensional empty array, also multi-dimensional/non-zero lower-bound arrays.

import System
array_int = System.Array[int]
print array_int               # <type 'Array[int]'>

# list
print array_int([1, 2])       # System.Int32[](1, 2)
# tuple
print array_int((3, 4, 5))    # System.Int32[](3, 4, 5)
# xrange
print array_int(xrange(6,10)) # System.Int32[](6, 7, 8, 9)
# CLR List
print array_int(System.Collections.Generic.List[int]()) # System.Int32[]()

# one-dimensional array 
a1 = System.Array.CreateInstance(int, 5)
for i in range(5): a1[i] = i * 10
print a1        # System.Int32[](0, 10, 20, 30, 40)

# two-dimensional array 
a2 = System.Array.CreateInstance(float, 2, 2)
a2[1, 1] = 3.14
print a2        # System.Double[,](
                # 0.0, 0.0
                # 0.0, 3.14)

IronPython also supports some python list-like operation to the CLR array objects, such as indexing with slice, +, *, in...

a1 = array_int(range(5))
a2 = array_int([100])

# slice
print a1[1:-1]      # System.Int32[](1, 2, 3)
# +
print a1 + a2       # System.Int32[](0, 1, 2, 3, 4, 100)
# *
print a1 * 2        # System.Int32[](0, 1, 2, 3, 4, 0, 1, 2, 3, 4)
# +, upcast the result to object[]
print a2 + System.Array[str]('py')  # System.Object[](100, 'p', 'y')

# slice with step
print a1[1::2]      # System.Int32[](1, 3)
# assignment 
a1[1::2] = [11, 13]
print a1            # System.Int32[](0, 11, 2, 13, 4)

# in/__contains__
print 11 in a1      # True

IronPython: ipyw.exe

Haibo Luo - MSFT — Thu, 13 Mar 2008 06:03:48 GMT

Each IronPython binary release ships two executable files: ipy.exe and ipyw.exe. Their (only) difference is, ipy.exe is a console application and ipyw.exe is a windows application. So given the following winform.py,

## winform.py
import clr
clr.AddReference("System.Windows.Forms")
from System.Windows.Forms import *

class SimpleForm(Form):
    def __init__(self):
        self.Text = 'Simple Winform'

Application.Run(SimpleForm())

Running it with ipy.exe will keep the associated console; however, if I launch it with ipyw.exe in a console window, I can continue working on other stuffs within that console.

I found that Shawn wrote some background about WINDOW_CUI/WINDOW_GUI subsystem here.

SignatureResolver (unfinished)

Haibo Luo - MSFT — Sun, 09 Mar 2008 01:44:00 GMT

While writing the ILVisualizer for dynamic method late 2005, I'd like to show the local variable information as well; so I started working on the managed signature parser, at least to parse LocalVarSig (Ecma-335 23.2.6). It was 2+ years ago, and never got to a finished state. Now perhaps I will never spend more time on it, so I refreshed the code a bit and you may find it useful for some scenarios.

Here is an example of how to use SignatureResolver to get LocalVarSig in IronPython. The static method SignatureResolver.GetLocalVarSig takes the byte array (the signature blob) and the enclosing method, and returns an array of LocalVar (which includes mainly the runtime type) inside that method.

# show local variables for every Array.CreateInstance overload methods
import clr
import System
clr.AddReference("ClrTest.Reflection.SignatureResolver")
from ClrTest.Reflection import *

for method in clr.GetClrType(System.Array).GetMember("CreateInstance"):
    body = method.GetMethodBody()   # C# won't allow this
    tok = body.LocalSignatureMetadataToken
    sig = method.DeclaringType.Module.ResolveSignature(tok)
    print "> %s, 0x%x" % (method, tok)
    for lv in SignatureResolver.GetLocalVarSig(sig, method):
        print lv

Next is part of the output showing one Array.CreateInstance method's local variable types. The result is, of course, same as what Reflector tells us in the below picture. "(p)" means the local variable is pinned.

> System.Array CreateInstance(System.Type, Int32[], Int32[]), 0x11000009
System.RuntimeType 
System.Int32 
System.Int32& (p)
System.Int32& (p)
System.Array 
System.Int32[] 
System.Int32[] 
System.RuntimeTypeHandle

Yes, I understand the MethodBody.LocalVariables property gives us the similar information; but the dynamic method can not. That was why I wanted to decode the bytes, and to use DynamicScopeTokenResolver to get a meaningful type string. I admit this version of code diverts from its original goal: it only works for RuntimeMethodInfo/RuntimeConstructorInfo, and won't work for dynamic method; but if one day, you are in need of parsing blob, the functions like GetToken/GetInt32Uncompressed could be handy. GetToken basically is the reverse of TypeDefOrRefEncoded (ecma-335: 23.2.8)

int GetToken() {
    int token = GetInt32Uncompressed();
    return s_mapForTokenType[token & 0x03] | (token >> 2);
}

int GetInt32Uncompressed() {
    byte first = _sig[_pos];

    if (first <= 0x7F) {
        return _sig[_pos++];
    } else if (first <= 0xBF) {
        byte[] bytes = new byte[2];
        bytes[1] = (byte)(_sig[_pos++] - 0x80);
        bytes[0] = _sig[_pos++];
        return BitConverter.ToInt16(bytes, 0);
    } else {
        byte[] bytes = new byte[4];
        bytes[3] = (byte)(_sig[_pos++] - 0xC0);
        bytes[2] = _sig[_pos++];
        bytes[1] = _sig[_pos++]; ;
        bytes[0] = _sig[_pos++];
        return BitConverter.ToInt32(bytes, 0);
    }
}

My SignatureResolver also provide another static method GetMethodRefSig to get MethodRefSig (Ecma-335 22.25 and 23.2.2).

If you are interested in this topic, you may download the attached file, unzip it, and take a look at the source. Also try to use VS2008 to build it.

If you are an IronPython user, you may try the accompanying tests (in IronPython). I compare the SignatureResolver.GetLocalVarSig result with the official reflection API MethodBody.LocalVariables: they are same for all method/constructors from all simple-LoadFrom-successful framework assemblies. If you have ClrTest.Reflection.ILReader.dll (or know how to get one), you can try test_methodrefsig.py, which tests GetMethodRefSig.

As stated in the title, this is *unfinished* and could be buggy.

Disclaimer: THE CODE IS PROVIDED "AS IS", WITH NO WARRANTIES INTENDED OR IMPLIED. USE AT YOUR OWN RISK.

ILVisualizer VS2008 solution

Haibo Luo - MSFT — Sat, 08 Mar 2008 03:11:00 GMT

I attached the ILVisualizer VS2008 solution in this post. There is no source code update; the only change is to upgrade the Microsoft.VisualStudio.DebuggerVisualizers.dll reference from version 8.0.0.0 to version 9.0.0.0.

Just for your convenience, so you can see a successful build after downloading in VS2008.

Thanks Julien for leaving the instruction here.

IronPython: Accessing the .NET Field

Haibo Luo - MSFT — Mon, 29 Oct 2007 06:10:00 GMT

The .NET libraries normally do not expose the public field, but we can still find its' presence for certain scenarios. IronPython treats the .NET fields like python attribute. For python attribute, the typical operations are set, get and delete. In general, IronPython throws TypeError when deleting members of CLR types and their objects (so for fields). To do set and get operation, the IronPython code looks similar to C# code. Updating (setting) values to the read-only or literal fields throws as we expect.

public class C {
public static int StaticField;
public int InstanceField;
}

>>> ...
>>> C.StaticField = 1          # set
>>> C.StaticField              # get
1
>>> c = C()
>>> c.InstanceField = 2        # set
>>> c.InstanceField            # get
2

Guess what you will get for C.InstanceField? It is "field descriptor", same as C.__dict__['InstanceField'], upon which we can call __set__/__get__ or two equivalent helper methods SetValue/GetValue (feel like repeating CLR reflection FieldInfo API here). Although not shown below, we can also set/get the value of the static field too by calling these methods on C.__dict__['StaticField'].

>>> C.InstanceField
<field# InstanceField on C>
>>> C.InstanceField is C.__dict__['InstanceField']
True
>>> C.InstanceField.__set__(c, 3)
>>> C.InstanceField.__get__(c)
3
>>> C.InstanceField.SetValue(c, 4)
>>> C.InstanceField.GetValue(c)
4

IronPython allows accessing static field on instance too. On the other hand, C# compiler requires type name to access static member (otherwise CS0176).

>>> c.StaticField = 5
>>> c.StaticField, C.StaticField
(5, 5)

Updating fields of value types is not allowed. If interested, you may go ahead read the design decision here. Until recently I did not know that as a remedy, the feature of setting fields/properties in the constructor call should be able to set field for value types while still keeping value types effectively immutable after the creation. Unfortunately we had no tests to cover this part (which never worked). This code below illustrates the expected behavior. I expect it to work soon.

public struct MyPoint {
public int X, Y;
public override string ToString() { return string.Format("X: {0}, Y: {1}", X, Y); }
}

>>> p = MyPoint(X=1, Y=2)
>>> p
<MyPoint object at 0x000000000000002C [X: 1, Y: 2]>
>>> p.X = 3
# one kind of exception still throws

VisualStudio as my IronPython editor

Haibo Luo - MSFT — Wed, 17 Oct 2007 08:00:00 GMT

The following steps are what I did to get Visual Studio ready as my IronPython (and IronRuby) editor.

Install the latest internal dogfood build of Visual Studio 2008.
- you may use Visual Studio 2005 or download the VS 2008 public beta2;
Download and install ASP.NET futures release (July 2007). This will give me the nice syntax coloring and (well... limited) intellisense for python code.
- you may download VSSDK 2008 CTP, and build the IronPython integration sample, which will give you the similar editing experience;
Download the attached add-in binary zip (poorly named as DlrToolsAddin); for my Vista box, I extract them under "%USERPROFILE%\Documents\Visual Studio 2008\Addins". I wrote this add-in basically to allow me to run the script inside the Visual Studio, and show the output in the output window.
- if you consider using it for VS 2005, you will need put it in "Visual Studio 2005\Addins".

Say, I am trying the example in my previous post, I create a new C# file (sample.cs) without creating solution/project, I can press Ctrl+1 (or the first button in the DlrTools toobar) to get it compiled to sample.dll in the local directory. You see the compile result at the bottom output window. Then I create a python file (method.py) in the same directory, type in some code (shown in color). Again, I can press Ctrl+1, the result are shown in the output window. I do not need leave VS and run these files in the cmd.exe window.

Sometimes I want to run the same code with different tools (for example, to check IronPython compatibility, I often run the same .py file against C-Python 2.5 too). That is the first combo box comes to play.

The second combo box is to set the working directory: "." means the current directory where the active file lives, ".." for the parent directory of the current file, or you may use the absolute path. Such support is needed to run C-Python regression tests. (For the add-in implementation side, I feel what I really want is a combo box of type vsCommandControlTypeDynamicCombo, which seems not available for add-in development).

The delete button is to remove your tool choice for those files you pressed "Run Script". By clicking the last button, an XML file (specifying which tools for which file extension) is opened so you may change it. You must update it with your tool paths, since the default setting is suited for myself.

You may also run part of the file by selecting those lines first (A temporary file is created and will be deleted after VS shutdown).

One bad thing I noticed of this add-in is that sometimes I press the "run script" button to start debugging, both icons have similar shapes.

To uninstall it, delete DlrToolsAddin.* files under "Addins"; and then run once "devenv.exe /resetaddin DlrToolsAddin.Connect".

The source code (VS2008 project) is also attached. Disclaimer: THE CODE IS PROVIDED "AS IS", WITH NO WARRANTIES INTENDED OR IMPLIED. USE AT YOUR OWN RISK. Hope you like Visual Studio as your IronPython editor.

IronPython: explicitly choose one method

Haibo Luo - MSFT — Fri, 12 Oct 2007 06:32:47 GMT

C# allows method overloading. Given an argument list, the compiler performs the overload resolution to select the best method to invoke. The chosen method token is baked into the assembly. IronPython does the method resolution at the IronPython run time; for most scenarios, it has no problem in picking up the most reasonable one based on the argument list. The rules are a bit complicated, and I will leave them for a future post. If you believe IronPython chooses the "unexpected" method for your scenario, please do let us know.

IronPython also provides the "Overloads" (python) attribute for the bound method, on which we can do index operation with parameter types to get the exact method we want. In the example below, the .NET reflection API Type.MakeByRefType and MakeArrayType are used in order to get those constructed types. Since these calls are only available for System.Type object, clr.GetClrType is called first on the python type to get the underlying CLR type. However, I am able to use the python type for the first 2 index calls (thanks to the type conversion).

public class ChooseOverload {
   public void M(int arg) { Console.WriteLine(1); }
   public void M(byte arg1, byte arg2) { Console.WriteLine(2); }
   public void M(ref int arg) { Console.WriteLine(3); arg = 10; }
   public void M(int[] arg) { Console.WriteLine(4); }
}

>>> ... # add the assembly
>>> import ChooseOverload
>>> o = ChooseOverload()
>>> o.M
<built-in method M of ChooseOverload object at 0x000000000000002B>
>>> o.M.Overloads
{'M(self, int arg)': <built-in function M>, 'M(self, Byte arg1, Byte arg2)': <built-in function M>, 'int M(self, int arg)': <built-in function M>, 'M(self, Array[int] arg)': <built-in function M>}
>>> o.M.Overloads[int](1)
1
>>> o.M.Overloads[System.Byte, System.Byte](1, 2)
2
>>> o.M.Overloads[clr.GetClrType(int).MakeByRefType()](1)
3
>>> o.M.Overloads[clr.GetClrType(int).MakeArrayType()](None)
4

When generics come to the picture, o.M could mean the generic method. Similar to the generic type, IronPython chooses the index operation to hold the instantiated generic method. Currently IronPython has no support to specify parameter types which are (or made from) generic type parameters (of either the generic method or type). Expect this will be supported one day.

public void M2<T>(T arg) { Console.WriteLine(5); }
public void M2<T>(byte arg) { Console.WriteLine(6); }

>>> o.M2[int](1) # bind to the first M2 given the argument "1"
5
>>> o.M2[int].Overloads[System.Byte](1) # in order to get the second M2
6
>>> o.M2[int].Overloads[???](1) # unable to pick the first M2 currently

IronPython: Provide (or not) Argument for by-ref Parameter

Haibo Luo - MSFT — Fri, 05 Oct 2007 08:21:00 GMT

Python function may return multiple objects as a tuple. The .NET method can only return one object as the result of a call; in order to return more than one objects, by-ref parameters are needed. They are decorated with the parameter modifier (ref, out) in C#.

Dictionary.TryGetValue(TKey key, out TValue value) has an output parameter "value". The API returns true if the dictionary contains an element with the specified key (the element value itself is returned to the parameter "value"), otherwise it returns false. When making calls to such methods in IronPython, we may not pass in argument for the output parameter. Instead, the result of such .NET method call in IronPython will likely be a tuple (unless the .NET method's return type is void and the method has only one by-ref parameter), which contains the value of the output parameter (see the example below).

import System
d = System.Collections.Generic.Dictionary[int, str]()
d[1], d[2] = 'One', 'Two'

print d.TryGetValue(1)         # (True, 'One')
print d.TryGetValue(2)[1]      # Two
print d.TryGetValue(3)         # (False, None)

We may also pass a clr.Reference object for the output parameter. The clr.Reference object has only one member "Value", which will carry the output parameter value after the call. "clr.Reference" object is of the type identical to System.Runtime.CompilerServices.StrongBox<T> in the new .NET Framework 3.5. When encountering this type of argument, IronPython codegen/runtime does something special to update the "Value".

x = clr.Reference[str]()             # like "new StrongBox<string>()" in C#
print d.TryGetValue(1, x), x.Value   # True One
print d.TryGetValue(3, x), x.Value   # False None

For reference parameter, we are required to pass in something. If the argument is not clr.Reference object, such reference argument value will also be part of the returned tuple; otherwise, the by-ref change is tracked inside clr.Reference. The following C# code is my twisted way to find the maximum number; the IronPython snippet shows the behaviors of calling that method not using and using clr.Reference.

public class C {
public bool M(int current, ref int max) {
    if (current > max) {
      max = current; return true;
    } else return false;
}
}

import C
o = C()

print o.M(10, 20)             # (False, 20)
print o.M(30, 20)             # (True, 30)

x = clr.Reference[int](20)
print o.M(10, x), x            # False 20
print o.M(30, x), x            # True 30

o.M(1)                        # TypeError: M() takes exactly 2 arguments (1 given)

Note if the .NET method contains more than one by-ref parameters, IronPython expects the user to provide proper clr.Reference objects for either ALL or NONE of them. A mix of clr.Reference object and normal argument/omission (for out parameter) will cause error (as illustrated below).

public int M2(ref int x, ref int y) {...}

o.M2(clr.Reference[int](1), 2) # TypeError

IronPython: Keyword Argument to Set .NET Property/Field

Haibo Luo - MSFT — Tue, 02 Oct 2007 22:16:00 GMT

Use keyword arguments to set properties or fields… you likely ask the question: where can we do this? IronPython allows it in the constructor call, the call against the .NET type. Such keyword should be the name of public field, or property (which is not read-only). After creating the object instance, each keyword argument value is then set to the corresponding field or property. Note keyword arguments can be used for the constructor parameters as usual.

The IronPython code below is to monitor any python file change under the directory "C:\temp". I used this special argument passing in the 2nd line.

import System
fsw = System.IO.FileSystemWatcher(r"C:\temp", Filter="*.py", IncludeSubdirectories = True, EnableRaisingEvents = True)
def on_changed(o,e): print e.ChangeType, e.FullPath
fsw.Changed += on_changed

System.Console.ReadLine()

FileSystemWatcher has three constructors (see below). The code uses the second one (not the third one)."c:\temp" is positional argument for "path"; the next 3 keywords arguments are transformed to the property (Filter/IncludeSubdirectories/…) assignments after the object creation.

public FileSystemWatcher();
public FileSystemWatcher(string path);
public FileSystemWatcher(string path, string filter);

I am not going to argue whether this feature is good or bad. You may find it handy in some places. The user could get confused when the constructor parameter name is same as the field/property name (Although it is unlikely as most libraries are following the .NET framework guideline: property name starts with the capital character, while the parameter name does opposite). One of our summer interns also proposed to extend this feature to event setting as keyword argument. It is currently not supported, not sure whether it will ever come in the future.

Note this code above actually doe not work in IronPython 2.0 alpha 4 (due to bugs). The upcoming 2.0 alpha5 will have the fix.

IronPython: Passing Arguments for a Call

Haibo Luo - MSFT — Tue, 02 Oct 2007 02:06:41 GMT

After getting a CLR type, we can play it like a python type. We can get the class/static methods using the dot operator ("attribute reference") on the type, or create an instance of it and then get the instance methods. In Python’s word, these methods are callable objects, and we can make "calls" on them using the call operator ("()"). I will write about the method call in the next couple of posts, this one is about passing arguments for calls in IronPython.

Python function definition allows several different parameter styles:

parameter which accepts excess positional arguments,
parameter with default value,
parameter which accepts excess keyword arguments.

Here are some examples showing the common parameter style, and each of 3 styles listed above.

def f1(x, y): print x, y
def f2(x, *y): print x, len(y)
def f3(x, y=5): print x, y
def f4(x, **y): print x, y

When making calls (on these functions), Python allows many different argument styles too. The following type (in C#) defines some methods, each with interesting parameter list.

// sample.cs
public class ArgumentList
public void M1(int x, int y) { Console.WriteLine("{0} {1}", x, y); }
public void M2(int x, params int[] y) { Console.WriteLine("{0} {1}", x, y.Length); }
public void M3(int x, [DefaultParameterValue(5)] int y) { Console.WriteLine("{0} {1}", x, y); }
}

In terms of the parameter style, we will see that M1 is equivalent as pure python function f1, same for M2/f2, M3/f3 (I was unable to find similar parameter kind in C# for f4). When calling these .NET methods, IronPython supports all argument styles as Python does.

M1 is the simplest in the parameter definition, but we can call it with different argument lists. Each call below prints "1, 2", same as if the python function "f1" is called.

import clr
clr.AddReference("sample")

import ArgumentList
o = ArgumentList()
f1 = o.M1

f1(1, 2)                 # positional arguments
f1(1, y = 2)             # positional & keyword arguments
f1(y = 2, x = 1)         # keyword arguments
f1(*(1, 2))              # unpack the sequence as positional args
f1(1, *(2,))             # positional & unpack
f1(**{'x': 1, 'y': 2})   # unpack the dictionary as keyword args

C# parameter array (parameter with "params", or exactly ParamArrayAttribute) is treated like "*y" in f2. The user are allowed to pass 0 or more arguments to the parameter "y".

f2 = o.M2
f2(1)              # print: 1 0
f2(1, 2)           #        1 1
f2(*(1, 2, 3))     #        1 2
f2(1, 2, 3, 4, 5) #        1 4

The C# language does not have built-in support for the default value argument (IL does, with the .param directive inside the method body); even if the parameter is decorated with System.Runtime.InteropServices.DefaultParameterValueAttribute, C# calls on such methods still require argument passed in for that parameter. However IronPython (like Python) honors such default value parameters.

f3 = o.M3
f2(1)            # print: 1 5
f2(x = 1)        #        1 5
f2(1, 2)         #        1 2

Another parameter attribute (OptionalAttribute) received similar treatment from IronPython: you do not need pass in values either; IronPython assign it a special value (the .NET run-time default value for most of primitive types, System.Type.Missing.Value for "object" type, null otherwise). Such supports are useful when using IronPython to call into the COM libraries (such as Office automation). The user can avoid typing in every arguments.

public void M6([Optional] int x, [Optional] object y) {
Console.WriteLine("{0} {1}", x, y);
}
public void M7([Optional] string x, [Optional] Type y) {
Console.WriteLine("{0} {1}", x == null, y == null);
}

o.M6()           # print: 0 System.Type.Missing
o.M6(10)         #        10 System.Type.Missing
o.M6(20, "hi")   #        20 hi
o.M7()           #        True True

IronPython: Grab the .NET Type

Haibo Luo - MSFT — Wed, 26 Sep 2007 00:58:00 GMT

After loading the assembly by clr.AddReference and then import namespace, we are ready to take hold of types and down-level namespaces using "attribute references".

>>> import System
>>> System.DateTime                    # type
<type 'DateTime'>
>>> System.Diagnostics                 # still namespace
<module 'Diagnostics' (CLS module, 2 assemblies loaded)>
>>> System.Diagnostics.Debug           # type again
<type 'Debug'>
>>> System.Diagnostics.CodeAnalysis    # namespace again
<module 'CodeAnalysis' (CLS module from mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)>
>>>

We can also access public nested type from the declaring type same way. (I was unable to find such example in mscorlib.dll/system.dll)

// lib.cs
public class C {
public class NestedC {}
}

>>> import clr
>>> clr.AddReference("lib")
>>> import C # C has no namespace here
>>> C.NestedC
<type 'NestedC'>

When the type is generic type, such attribute reference returns the open generic type. We need then call the index operation (subscriptions) to construct the closed generic type, and do something interesting with it. A type tuple is expected as the index, but the parentheses can be omitted most of time.

>>> System.Collections.Generic.Dictionary # open generic type
<type 'Dictionary[TKey, TValue]'>
>>> System.Collections.Generic.Dictionary[int, str] # closed generic type
<type 'Dictionary[int, str]'>
>>> System.Collections.Generic.Dictionary[(int, object)] # closed generic type
<type 'Dictionary[int, object]'>

Also the .NET type names could be "same" or similar; one example is System.Nullable and System.Nullable<T> in mscorlib.dll. Many experimental code of mine has types like C, C<T>, C<T1, T2> . As shown below, "System.Nullable" now returns a type group, including 2 types. In order to distinguish the non-generic type from the generic one, an empty tuple need be passed in as index.

>>> System.Nullable
<types 'Nullable', 'Nullable[T]'>
>>> System.Nullable[()] # get the non-generic type
<type 'Nullable'>
>>> System.Nullable[System.DateTime]
<type 'Nullable[DateTime]'>

IronPython: clr.AddReference

Haibo Luo - MSFT — Wed, 26 Sep 2007 00:57:00 GMT

In order to interop with .NET libraries, we need first load in the assemblies we want to play with. The family of AddReference methods in the clr module serves the purpose.

clr.AddReference
clr.AddReferenceByName
clr.AddReferenceByPartialName
clr.AddReferenceToFile
clr.AddReferenceToFileAndPath

clr.AddReference accepts System.Reflection.Assembly objects and/or assembly names. The typical usage looks like:

asm = ... # any approach of getting an Assembly object
clr.AddReference(asm)
clr.AddReference("System.Drawing, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")
clr.AddReference("System.Drawing")

If a string is passed in, Assembly.Load(string) is used to load the assembly. If failed, it re-tries with Assembly.LoadWithPartialName(string).

clr.AddReferenceByName basically is the wrapper of Assembly.Load, which means the assembly full name is expected. Similarly, clr.AddReferenceByPartialName is the wrapper around the obsolete Assembly.LoadWithPartialName: "System.Drawing" and "System.Drawing, Version=2.0.0.0" are examples of valid arguments.

A typical usage of clr.AddReferenceToFile looks like:

clr.AddReferenceToFile("my1.dll", "my2") # load my1.dll and my2.dll

It expects file name(s) without the directory path as the argument. To locate my1.dll, it searches each directory in sys.path until the specified file is found (it also tries to append ".dll" or ".exe"). Behind the scene, the loading is via Assembly.LoadFile. Normally sys.path includes the current working directory; so if my1.dll and my2.dll exist in the current directory, clr.AddReferenceToFile("my1.dll", "my2") should succeed. If you know the exact full path of a clr assembly, you may use clr.AddReferenceToFileAndPath to load. As a by-product, the path of the assembly file will be appended to sys.path. Note Assembly.LoadFrom is never used underneath by these 5 methods due to the loading context concern.

We can use clr.References to list which assemblies have been loaded. The output shows that mscorlib.dll and System.dll are implicitly loaded/added (without the need of calling clr.AddReference).

>>> import clr
>>> for r in clr.References: print r
...
mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
>>> clr.AddReference("System.Drawing")
>>> for r in clr.References: print r
...
mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
System.Drawing, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

By treating the namespace like the python module, IronPython extends the import statement semantics to bring in the clr namespace. In current implementation the "classical" python module is still preferred by import: given the statement "import MyLib", if the file "MyLib.py" is present under any directory of sys.path, that file will be imported; if not, for each loaded assembly, import tries to find the namespace "MyLib". Although not common in well-designed .NET frameworks, a type could have empty namespace; so "import MyLib" also tries to find type "MyLib" in the loaded assemblies.

The namespace of "MyLib" could exist more than 1 loaded assemblies. The following snippet shows that, "import System" adds a CLS module to sys.modules and this CLS module contains members from 2 assemblies: mscorlib.dll and System.dll (since both assemblies has types and down-level namespaces like System.*).

> ipy.exe
>>> import sys
>>> sys.modules.keys()
['sys', '__builtin__', '__main__', 'site']
>>> import System
>>> sys.modules.keys()
['sys', '__builtin__', '__main__', 'site', 'System']
>>> sys.modules['System']
<module 'System' (CLS module, 2 assemblies loaded)>
>>> System == sys.modules['System'] # better use "is" to check identity as Michael pointed out in the comment
True
>>> System.__file__
['mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089', 'System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089']
>>> System.Console.WriteLine('later')
later

IronPython: import clr

Haibo Luo - MSFT — Wed, 26 Sep 2007 00:56:00 GMT

clr is an IronPython built-in module, which provides some functionalities in order to interop with .NET. When writing "import clr" in a python module, it means you intend to leverage the .NET libraries. One classical example is python string's methods. In IronPython, python type str is implemented by the .NET type System.String. We can call System.String's methods on string objects after the line of "import clr" (see the example below); of course those python string methods are still available.

> ipy.exe
IronPython console: IronPython 2.0 (2.0.0.00) on .NET 2.0.50727.1378
Copyright (c) Microsoft Corporation. All rights reserved.
>>> s = "IronPython"
>>> s.upper()
'IRONPYTHON'
>>> s.ToUpper()
Traceback (most recent call last):
File , line 0, in ##14
File , line 0, in _stub_##15
AttributeError: 'str' object has no attribute 'ToUpper'

>>> import clr
>>> s.upper() # python method still works
'IRONPYTHON'
>>> s.ToUpper() # works
'IRONPYTHON'

Such context change is per-python-module. Although lib.py (below) imports the clr module and app.py then imports the lib module, app.py will not see the .NET methods. Running this app.py under C-Python will fail (ImportError) due to the usage of the clr module in lib.py. But if we make lib.py platform-neutral next time, we are ensured that no change in app.py is needed in order to run it under both C-Python and IronPython.

# lib.py
import clr
def my_upper(s): return s.ToUpper()

# app.py
import lib
s = "hello world"
print lib.my_upper(s) # print "HELLO WORLD"
#print s.ToUpper() # AttributeError would throw

DebuggerTypeProxy for IronPython Old-style Class and Instance

Haibo Luo - MSFT — Fri, 03 Aug 2007 22:37:00 GMT

First I want to make it clear that this post means nothing related to IronPython's plan about debugging python code in the future; it serves more as an interesting example to demonstrate DebuggerTypeProxyAttribute and how types can be created dynamically and be used in the VS debugger windows to display information the end users are interested in.

The python code below shows one old style class and some operations on the instance. After downloading the binary zip of IronPython 2.0 Alpha 3, we can create a solution, set it to launch ipy.exe for debugging, and step through the code inside Visual Studio; the locals window may show something like this after hitting the breakpoint.

The local variable alpha3's .NET type is IronPython.Runtime.Types.OldInstance, the way this object is displayed in the locals window is based on that type's fields and properties, which exposes many implementation details. While debugging, what the python users really want to know is the object attributes. My goal is to make python object attributes look like C# object's public fields; how can I achieve this?

DebuggerTypeProxyAttribute allows us to specify a proxy type for the target type. VS debugger uses the proxy type to display the target object. We already see such proxy types a lot: when viewing objects of most collection types, we are actually viewing their proxy type layouts.

In current IronPython implementation, every old style instance is of type I.R.T.OldInstance, so my first step is to define a proxy type "OldInstanceProxy" to expose the object attributes only. Clearly old style instances can have different attribute set of different names; but I can only specify one proxy type for the I.R.T.OldInstance type. [DebuggerBrowsable(DebuggerBrowsableState.RootHidden)] is part of the solution to achieve the look-and-feel. The proxy type (OldInstanceProxy) has an object field (m_oldInstance) decorated with that attribute so that m_oldInstance will be hiden in the debugger window, but the fields m_oldInstance has will be shown. What should m_oldInstance look like? A .NET type (let me call it the "real" proxy type) with as many public fields as python object attributes, each having the same name as the attribute name, and we are going to create such types on the fly. Also the "real" proxy type's constructor is emitted with the code to assign these public fields properly. We then instantiate the new type with the original OldInstance object, and assign it to m_oldInstance.

[assembly:DebuggerTypeProxy(
typeof(IronPython.DebuggingSupport.OldInstanceProxy),
  TargetTypeName = "IronPython.Runtime.Types.OldInstance, IronPython, Version=2.0.0.300, ...")
]

public class OldInstanceProxy{
[DebuggerBrowsable(DebuggerBrowsableState.RootHidden)]
public object m_oldInstance;

public OldInstanceProxy(object target) {
try {
   OldInstance obj = target as OldInstance;
TypeBuilder tb = CodeGen.CreateTemporaryType();
ConstructorBuilder cb = tb.DefineConstructor(MethodAttributes.Public, CallingConventions.Standard, new Type[] { typeof(OldInstance) });
ILGenerator il = cb.GetILGenerator();
il.Emit(OpCodes.Ldarg_0);
il.Emit(OpCodes.Call, typeof(Object).GetConstructor(Type.EmptyTypes));

foreach (KeyValuePair<object,object> pair in obj.Dictionary) {
    string key = pair.Key.ToString();
    Type fieldType = pair.Value == null ? typeof(object) : pair.Value.GetType();
    FieldBuilder fb = tb.DefineField(key, fieldType, FieldAttributes.Pub.Public);

        // emit code for the ctor ...
      }
      il.Emit(OpCodes.Ret);

      m_oldInstance = Activator.CreateInstance(tb.CreateType(), obj);

Few discussions:

The attached solution also includes another proxy type: OldClassProxy for IronPython.Runtime.Types.OldClass. With that, viewing class attributes feels like viewing C# class static fields. Such coding pattern can be applied to other similar scenarios.
It is a bit hard to apply this for IronPython new style instances. For each new style class, IronPython creates a new type on the fly. For example, "class C(object): pass" generates a IronPython.Runtime.NewTypes.Object_1. The proxy type (which will create the "real" proxy type) need be created at the same time.
Types created by Reflection.Emit are not GC-collectible. In my proxy type implementation, a simple caching mechanism is implemented to reuse those "real" proxy types; but considering the python dynamism, object attributes can be added, removed, their values' type can be changed; such "real" proxy type may have to be created frequently, each click to expand in the debugger window could cause one type creation, so use wisely if you use such proxy type in real world scenarios.
Unlike DebuggerVisualizer where VS provides a way to debug it, it is a bit inconvenient to debug DebuggerTypeProxy code. So I found having the constructor code wrapped inside try-catch and saving the exception if thrown is useful. The exception object can be viewed in the debugger window if the type proxy dll is compiled with the "Debug" configuration. The messages from Debug.WriteLine are helpful too.
The proxy type implementation depends on the rapidly changing code of the DLR/IronPython project. It is not surprising that the attached solution could be broken in the future. .

The following pictures show the proxy type in action. Compared to the previous picture, now you can easily check out the values of the attribute "url" and "version", as well as the class attribute "project_name" (shown in the first picture). The new attribute "release_date" appears in the second picture after stepping through the breakpoint line.

To use the attached solution, you need download IronPython 2.0 alpha 3 binary zip, and I assume you unzip it to C:\. Your comments are welcome.

Name of Array Type

Haibo Luo - MSFT — Wed, 29 Nov 2006 09:36:00 GMT

Two things related to array type name came to my mind while writing the previous post.

Given an one-dimensional non-zero based double array x, x.GetType().ToString() returns "System.Double[*]"; we can use Type.GetType("System.Double[*]") to get hold of such type easily. To get an instance of such type, instead of calling Array.CreateInstance, we can find ConstructorInfo first, and call ConstructorInfo.Invoke. Believe it or not, this type has 2 constructors, one is to create zero-based array, the other one for non-zero-based array.

Type t1 = Type.GetType("System.Double[*]");
ConstructorInfo[] ctors = t1.GetConstructors();
foreach (ConstructorInfo ci in ctors)
Console.WriteLine(ci); // print "Void .ctor(Int32)" and "Void .ctor(Int32, Int32)"

Array a1 = (Array)ctors[0].Invoke(new object[] { 10 });
Console.WriteLine(a1.GetType()); // print "System.Double[]"

Array a2 = (Array)ctors[1].Invoke(new object[] { -5, 10 }); // lowerbound: -5
Console.WriteLine(a2.GetType()); // print "System.Double[*]"

Type.GetType("System.Int32[,][]") returns a jagged array, more precisely, a one-dimensional array whose elements are two-dimensional arrays, This is different from the type "int[,][]" declared in C#, which is a two-dimensional array of one-dimensional arrays. The type name grammar used in Type.GetType is more close to how IL describes such array: you can see a3's type as "int32[0...,0...][]" in ildasm.

Type t2 = Type.GetType("System.Int32[,][]");
int[][,] a3 = new int[2][,] { new int[,] { { 1, 2 } }, new int[,] { { 3, 4, 5 }, { 6, 7, 8 } } };
int[,][] a4 = new int[1, 2][] { { new int[] { 1, 2 }, new int[] { 3, 4, 5 } } };
Console.WriteLine(a3.GetType() == t2); // True
Console.WriteLine(a4.GetType() == t2); // False

.locals init (
[6] int32[0...,0...][] a3,
[7] int32[][0...,0...] a4,

Late-bound Array SetValue

Haibo Luo - MSFT — Mon, 27 Nov 2006 08:04:00 GMT

The type "System.Array" provides us a set of API for the late-bound array operations, including array creation (Array.CreateInstance), read/write access to array element (Array.SetValue/Array.GetValue). They are convenient to use. Let me start with some code to create an one-dimensional integer array, and set the first 2 elements using early-bind assignment and Array.SetValue, respectively. To simplify my code, this post will only touch the late-bound set value on array (the similar discussion can be applied to get value).

// creation: new int[5]
Array x = Array.CreateInstance(typeof(int), 5);

// 0. early-bind set: explicit cast to int[]
int[] y = (int[])x;
y[0] = 100;

// 1. late-bound set via Array.SetValue
x.SetValue(200, 1);

From the signature of Array.SetValue(object, int), we know that the integer "200" has to be boxed first before passing into this API, which is a performance hit (among others). However, we can take a look at what kind of IL sequence C# compiler generates for the early-bound assignment, and build an equivalent dynamic method to avoid the boxing: C# compiler emits Stelem.I4 to store the 32-bit integer value.

// 2. late-bound set via DynamicMethod/Delegate/OpCodes.Stelem
DynamicMethod dm = new DynamicMethod("SetValueByStelem", typeof(void),
new Type[] { typeof(Array), typeof(int), typeof(int) }, typeof(Test));
ILGenerator ilgen = dm.GetILGenerator();
ilgen.Emit(OpCodes.Ldarg_0);
ilgen.Emit(OpCodes.Castclass, x.GetType());
ilgen.Emit(OpCodes.Ldarg_1); // index
ilgen.Emit(OpCodes.Ldarg_2); // value
ilgen.Emit(OpCodes.Stelem_I4);
ilgen.Emit(OpCodes.Ret);
MySetValue sv = (MySetValue)dm.CreateDelegate(typeof(MySetValue));

sv(x, 2, 300);

delegate void MySetValue(Array array, int index, int value);

The runtime (so is System.Array) supports non-zero lowerbound array (C# does not). The code snippet below creates a length-of-5 integer array with -3 as its' lower-bound, and then tries to set value for some elements.

We can no longer cast such array to int[].
Array.SetValue works. Note we need use the lowerbound value to access the first element.
The instructions "stelem/stelem.<type>" are for one dimensional array with zero-based index only. So the dynamic method approach with stelem instruction won't work.
For each concrete array type, runtime adds three special methods: Get/Set/Address. With reflection we can hold those MethodInfo, so you can call MethodInfo.Invoke or create another dynamic method wrapper around it. This works for zero-based array too.

// creation: new int[-3..1]
Array x = Array.CreateInstance(typeof(int), new int[] { 5 }, new int[] { -3 });

// 1. late-bound set via Array.SetValue
x.SetValue(200, 1 - 3);

// 3. late-bound set via MethodInfo.Invoke of "Set"
MethodInfo mi = x.GetType().GetMethod("Set");
mi.Invoke(x, new object[] { 3 - 3, 400 });

// 4. late-bound set via DynamicMethod/Delegate/"Set"
DynamicMethod dm = new DynamicMethod("SetValueBySet", typeof(void),
new Type[] { typeof(Array), typeof(int), typeof(int) }, typeof(Test));

ILGenerator ilgen = dm.GetILGenerator();
ilgen.Emit(OpCodes.Ldarg_0);
ilgen.Emit(OpCodes.Castclass, x.GetType());
ilgen.Emit(OpCodes.Ldarg_1);
ilgen.Emit(OpCodes.Ldarg_2);
ilgen.Emit(OpCodes.Call, mi);
ilgen.Emit(OpCodes.Ret);

MySetValue sv = (MySetValue)dm.CreateDelegate(typeof(MySetValue));
sv(x, 4 - 3, 500);

Using Vance's CodeTimers class, I measured these 5 approaches' performance on my Vista machine. My scenario is to set value on 1-dimensional zero-based integer array, the action loop contains 10 lines of set-value code (those dynamic method and delegate creation are prepared ahead). The fastest late-bound set approach is DynamicMethod with Stelem, about 90-times slower than the early-bound assignment; but it is 3-times faster than Array.SetValue. Here is the raw output:

Data units of msec resolution = 0.279365 usec
Early-bound assignment : count: 1000000 4.525 +- 1% msec
Late-bound set via Array.SetValue : count: 1000000 1138.182 +- 2% msec
Late-bound set via DynamicMethod/Stelem : count: 1000000 397.005 +- 0% msec
Late-bound set via Set MethodInfo.Invoke: count: 1000000 36352.360 +- 0% msec
Late-bound set via DynamicMethod/Set : count: 1000000 453.956 +- 7% msec

What about multi-dimensional array? Array.SetValue still works; for better performance, we might want use the special "Set" method. By the way, the following ildasm output shows what C# compiler generates for "array[0, 1] = 1000;", which uses "Set". Note "int32[0..., 0...]" is TypeSpec(0x1B).

IL_0009: ldloc.0
IL_000a: ldc.i4.0
IL_000b: ldc.i4.1
IL_000c: ldc.i4 0x3e8
IL_0011: call instance void int32[0...,0...]/* 1B000003 */::Set(int32, int32, int32) /* 0A000020 */

Take Two: IL Visualizer

Haibo Luo - MSFT — Fri, 17 Nov 2006 00:55:00 GMT

I was glad to hear many positive feedbacks about the DebuggerVisualizer for DynamicMethod; on the sad side, it shows our lack of good LCG debugging support (on which, Mike Stall is seeking your opinion).

Recently along with the ILReader update, I made a few changes on the visualizer:

Add support to show IL of RuntimeMethodInfo and RuntimeConstructorInfo (besides DynamicMethod). Actually Roy already had a version providing such functionality. Last year, I wanted to enable viewing IL on all MethodBase-derived types. Well, my daily duty and interests changed; I am afraid I won't ever meet that goal.
Add support to show IL of Delegate.MethodInfo, where the delegate might be created from DynamicMethod.
Simplify IL Visualizer UI. When the DynamicMethod IL code is incomplete, those branch instructions' label shown inside the visualizer could be incorrect. However it is easy to identify this fixup failure: they always branch to the next instruction. I removed the label-fix-succeed-failed information as well as the font ComboBox at the bottom of the dialog. Now changing the font is done by the RichTextBox's context menu.
Add another way to view IL: "Send to IL Monitor". DebuggerVisualizer dialog is a modal dialog, which means we have to dismiss it first to continue anything with Visual Studio. I found this is inconvenient when trying to track the whole code-gen process. You can now choose "Send to IL Monitor". Before clicking it, a separate application "ilmonitor.exe" need to be started first. Those IL information is sent to this application with TCP socket (and the listening port number is hard-coded as 22017). For one dynamic method, if we click the magnifying glass "Send to IL Monitor" button a couple of times at different code-gen stages, the same window will show the growing IL with different background colors. As you see from the pictures below, it can also track many dynamic methods' IL.

You may download the attachment and try to build the solutions in your VS. The images below were captured in my Vista box.

Disclaimer: THE CODE IS PROVIDED "AS IS", WITH NO WARRANTIES INTENDED OR IMPLIED. USE AT YOUR OWN RISK.

update (3/7/2008): VS2008 solution can be found at http://blogs.msdn.com/haibo_luo/archive/2008/03/07/8107924.aspx.

update (4/19/2010): VS2010 solution can be found at http://blogs.msdn.com/haibo_luo/archive/2010/04/19/9998595.aspx

Turn MethodInfo to DynamicMethod

Haibo Luo - MSFT — Wed, 08 Nov 2006 01:13:00 GMT

I do not know why anyone ever need this :) but few readers did ask me similar questions before. Solving this problem also demonstrates one more ILReader usage.

To build a DynamicMethod, we can choose either DynamicILGenerator or DynamicILInfo. My first impression of using DynamicILGenerator to emit code is slow: you need make calls to emit ILs instruction by instruction. Actually this approach is more difficult, a topic for future post. I already wrote a post about DynamicILInfo. I do not know the motivation behind the DynamicILInfo class (at that time I was not in the CLR team), but I heard this was a feature request from SQL.

To turn a MethodInfo to DynamicMethod, we can not simply call SetCode with the byte array returned by the static method body's GetILAsByteArray. Those tokens (if any) are metadata tokens, meaningful only in the scope of the module where that static method lives. The DynamicILInfo.GetTokenFor overloads are designed to build the relationship between an integer number (the new "token") and the runtime member (type, method, field, string, or signature...) in the scope of the dynamic method. We can first make a copy of the static method's IL byte array, and overwrite the old metadata token with the new token from GetTokenFor for each applicable IL instruction. So here comes ILInfoGetTokenVisitor. As you see, we override 6 visit methods in ILInfoGetTokenVisitor, doing the token replacements; the top-level code to prepare new code array looks simple.

class ILInfoGetTokenVisitor : ILInstructionVisitor {
private DynamicILInfo ilInfo;
private byte[] code;

public ILInfoGetTokenVisitor(DynamicILInfo ilinfo, byte[] code) {
this.ilInfo = ilinfo;
this.code = code;
}

public override void VisitInlineMethodInstruction(InlineMethodInstruction inlineMethodInstruction) {
OverwriteInt32(ilInfo.GetTokenFor(inlineMethodInstruction.Method.MethodHandle,
inlineMethodInstruction.Method.DeclaringType.TypeHandle),
inlineMethodInstruction.Offset + inlineMethodInstruction.OpCode.Size);
}
public override void VisitInlineFieldInstruction(InlineFieldInstruction inlineFieldInstruction) { ... }
public override void VisitInlineSigInstruction(InlineSigInstruction inlineSigInstruction) { ... }
public override void VisitInlineStringInstruction(InlineStringInstruction inlineStringInstruction) { ... }
public override void VisitInlineTokInstruction(InlineTokInstruction inlineTokInstruction) { ... }
public override void VisitInlineTypeInstruction(InlineTypeInstruction inlineTypeInstruction) { ... }
...
}

private static void SetCode(MethodInfo method, MethodBody body, DynamicILInfo ilInfo) {
byte[] code = body.GetILAsByteArray();
ILReader reader = new ILReader(method);
ILInfoGetTokenVisitor visitor = new ILInfoGetTokenVisitor(ilInfo, code);
reader.Accept(visitor);

ilInfo.SetCode(code, body.MaxStackSize);
}

JAY was asking how to set exceptions. Reflection does not provide API to return the exception blob directly, instead the exception information is exposed through the MethodBody.ExceptionHandlingClauses property. With this ExceptionHandlingClause list, we can follow the format described by Ecma 335 (Part 2, 25.4.5 and 25.4.6) and build the exception blob from scratch. To avoid worrying about the restriction of the small version of exception handling clauses, the exception array generated below is with the fat form layout. See some comments inside the code.

private static void SetExceptions(MethodBody body, DynamicILInfo ilInfo) {
IList<ExceptionHandlingClause> ehcs = body.ExceptionHandlingClauses;
int ehCount = ehcs.Count;
if (ehCount == 0) return;

// Let us do FAT exception header
int size = 4 + 24 * ehCount;
byte[] exceptions = new byte[size];

exceptions[0] = 0x01 | 0x40; //Offset: 0, Kind: CorILMethod_Sect_EHTable | CorILMethod_Sect_FatFormat
OverwriteInt32(size, 1, exceptions); // Offset: 1, DataSize: n * 24 + 4

int pos = 4;
foreach (ExceptionHandlingClause ehc in ehcs) {
// Flags, TryOffset, TryLength, HandlerOffset, HandlerLength: all size 4
OverwriteInt32((int)ehc.Flags, pos, exceptions); pos += 4;
OverwriteInt32(ehc.TryOffset, pos, exceptions); pos += 4;
OverwriteInt32(ehc.TryLength, pos, exceptions); pos += 4;
OverwriteInt32(ehc.HandlerOffset, pos, exceptions); pos += 4;
OverwriteInt32(ehc.HandlerLength, pos, exceptions); pos += 4;

// ClassToken or FilterOffset
switch (ehc.Flags) {
case ExceptionHandlingClauseOptions.Clause:
int token = ilInfo.GetTokenFor(ehc.CatchType.TypeHandle);
OverwriteInt32(token, pos, exceptions);
break;
case ExceptionHandlingClauseOptions.Filter:
OverwriteInt32(ehc.FilterOffset, pos, exceptions);
break;
case ExceptionHandlingClauseOptions.Fault:
throw new NotSupportedException("dynamic method does not support fault clause");
case ExceptionHandlingClauseOptions.Finally:
break;
}
pos += 4;
}

ilInfo.SetExceptions(exceptions);
}

If you are interested in the details, you may download the attachment and read the whole source code closely.

A related question is how to build a dynamic method from other sources. DynamicMethod is not Serializable, .NET framework built-in serializing/deserializing support will not work. One approach I can think of is to preserve the code array, the members/strings/... (in order to call GetTokenFor to get new tokens) used and their matching offsets (with one scan of the ILs). MethodInfo, FieldInfo, Type are marked "Serializable", but not sure whether this is the right way to save/restore those members. If the method handles exception, we can save the pre-calculated exception blob, and remember where we need replace token for what exception type.

Finally, let us play with the turn-static-method-to-dynamic-method call, and show off some IronPython code. The first 3 lines bring in the DynamicMethodHelper class. Then we use Reflection to get DateTime.get_Now's MethodInfo (sGetNow), and convert it to DynamicMethod (dGetNow) with "DynamicMethodHelper.ConvertFrom". The last part of the code is interesting: it converts the method "DynamicMethodHelper.ConvertFrom" to DynamicMethod (dConvert), and use that dynamic method (dConvert) to convert sGetNow to another dynamic method (dGetNow2). Just fyi: DateTime.get_Now has 5 instructions, DynamicMethodHelper.ConvertFrom has ~80 instructions and exception handlings clauses.

>>> import clr, System
>>> clr.AddReference('ClrTest.Reflection.DynamicMethodHelper.dll')
>>> from ClrTest.Reflection import *
>>>
>>> clr.GetClrType(System.DateTime).GetMethod("get_Now")
<System.Reflection.RuntimeMethodInfo object at 0x000000000000002B [System.DateTime get_Now()]>
>>> sGetNow = _
>>> DynamicMethodHelper.ConvertFrom(sGetNow)
<System.Reflection.Emit.DynamicMethod object at 0x000000000000002C [System.DateTime get_Now()]>
>>> dGetNow = _
>>> dGetNow.Invoke(None, None)
<System.DateTime object at 0x000000000000002D [11/7/2006 11:40:49 AM]>
>>>
>>> clr.GetClrType(DynamicMethodHelper).GetMethod("ConvertFrom")
<System.Reflection.RuntimeMethodInfo object at 0x000000000000002E [System.Reflection.Emit.DynamicMethod ConvertFrom(System.Reflection.MethodInfo)]>
>>> sConvert = _
>>> DynamicMethodHelper.ConvertFrom(sConvert)
<System.Reflection.Emit.DynamicMethod object at 0x000000000000002F [System.Reflection.Emit.DynamicMethod ConvertFrom(System.Reflection.MethodInfo)]>
>>> dConvert = _
>>>
>>> dConvert.Invoke(None, System.Array[object]([sGetNow]))
<System.Reflection.Emit.DynamicMethod object at 0x0000000000000030 [System.DateTime get_Now()]>
>>> dGetNow2 = _
>>> dGetNow2.Invoke(None, None)
<System.DateTime object at 0x0000000000000031 [11/7/2006 11:40:59 AM]>
>>>

System.Reflection-based ILReader

Haibo Luo - MSFT — Mon, 06 Nov 2006 23:02:00 GMT

Compared to what I posted previously here (or what was used in the DynamicMethod visualizer), this new version introduced the Visitor pattern.

A do-nothing visitor ILInstructionVisitor is included; the users can focus on their domain-specific logic by simply inheriting from it and overriding few methods.

public abstract class ILInstructionVisitor {
    public virtual void VisitInlineBrTargetInstruction(InlineBrTargetInstruction inlineBrTargetInstruction) { }
    public virtual void VisitInlineFieldInstruction(InlineFieldInstruction inlineFieldInstruction) { }
    ...
}

The abstract class ILInstruction now has an abstract method "Accept"; each derived ILInstruction class provides the override "Accept" method, which dispatches the call to ILInstructionVisitor's corresponding visit method. It saves a long switch statement when an ILInstruction is about to be processed.
The class ILReader has a shortcut to traverse all IL instructions: the instance method "Accept".

Class ILReader has 2 overload constructors. There are many mscorlib types derived from MethodBase (see the picture at the right side, from Reflector); only RuntimeMethodInfo or RuntimeConstructorInfo instance is supposed to pass into the first constructor. Internally it uses MethodBase.GetMethodBody and token resolution APIs from System.Reflection.Module. DynamicMethod does not provide public APIs to access the method body, nor the token resolution APIs. IILProvider and ITokenResolver are designed for easy plug-in of other IL Streams. If you are familar with the DynamicMethod visualizer source code, you already know, for tool authoring purpose, we could use Reflection to reflect some private members (which so depends on the implementation details, and could be likely broken in the future runtime version), and then come up dynamic method related classes (which implement both interfaces). We could figure out how Reflection.Emit manages those tokens at the bake time, but I have not gotten enough time on solving it.

public ILReader(MethodBase method);
public ILReader(IILProvider ilProvider, ITokenResolver tokenResolver);

Let us check out how we can build something "useful" with the ILReader. Instead of demo'ing with C# code, I'd like to write some Python code here (well, to promote IronPython ...).

The first IronPython code examines which System.String's public methods could throw ArgumentOutOfRangeException directly. For that, I can write a ILInstructionVisitor-derived class ThrowExceptionHunter, having one override method VisitInlineMethodInstruction, which checks whether the instructions's operand "Method" belongs to one of ArgumentOutOfRangeException's constructors. In order to stop hunting after hitting the ArgumentOutOfRangeException constructor, I need write a tailored loop, instead of using ILReader.Accept. From the output, we see 11 methods could throw ArgumentOutOfRangeException "directly", and they always call the constructor which requires 2 string arguments (i.e., paramName, message);

class ThrowExceptionHunter(ILInstructionVisitor):
    def __init__(self, method, exceptionType):
        self.method = method
        self.ctors = exceptionType.GetConstructors()
        self.found = False
    def VisitInlineMethodInstruction(self, instruction):
        if instruction.Method in self.ctors:
            print instruction.Method, '|', self.method
            self.found = True

clrStringType = clr.GetClrType(System.String)
exceptionType = clr.GetClrType(System.ArgumentOutOfRangeException)
for method in clrStringType.GetMethods():
    h = ThrowExceptionHunter(method, exceptionType)
    for il in ILReader(method):
        il.Accept(h)
        if h.found: break

Output:
Void .ctor(System.String, System.String) | System.String Join(System.String, System.String[], Int32, Int32)
...
Void .ctor(System.String, System.String) | System.String Remove(Int32)

Next visitor is to find callees (methods called by the current method). Early-bind method calls happen with either InlineMethodInstruction or InlineSigInstruction. Suppose you unlikely write code using "calli", only one ILInstructionVisitor method need to be overridden: to put the called method together into a python set (therefore there is no duplicate). The results shows String.StartsWith(String, StringComparison) internally used 10 different methods (including constructor call, property access).

class CalleeFinderNoDup(ILInstructionVisitor):
    def __init__(self):
        self.callees = set()
    def VisitInlineMethodInstruction(self, instruction):
        self.callees.add(instruction.Method)

method = clrStringType.GetMethod('StartsWith', System.Array[System.Type]([str, System.StringComparison]))
reader = ILReader(method)

v = CalleeFinderNoDup()
reader.Accept(v)
print 'How many:', len(v.callees)
for x in v.callees:
    print x, '/', x.DeclaringType

Output:
How many: 10
Void .ctor(System.String) / System.ArgumentNullException
...
Int32 CompareOrdinalIgnoreCaseEx(System.String, Int32, System.String, Int32, Int32) / System.Globalization.TextInfo

We can think of many scenarios related to IL code disassembly; the attached ILReader library provides 2 visitors for that purpose: ReadableILStringVisitor and RawILStringVisitor. Along the visit, we pass in IILStringCollector. ReadableILStringToTextWriter, which implements IILStringCollector, can be used to dump the readable IL string to the specified TextWriter (shown below). We can write customized ILStringCollectors to collect those IL strings in other manners.

v = ReadableILStringVisitor(ReadableILStringToTextWriter(System.Console.Out))
reader.Accept(v)

Output:
IL_0000: ldarg.1
IL_0001: brtrue.s IL_000e
IL_0003: ldstr "value"
IL_0008: newobj Void .ctor(System.String)/System.ArgumentNullException
IL_000d: throw
...
IL_00f7: call System.String GetResourceString(System.String)/System.Environment
IL_00fc: ldstr "comparisonType"
IL_0101: newobj Void .ctor(System.String, System.String)/System.ArgumentException
IL_0106: throw

By the way, you may think those method names inside the ILInstructionVisitor class are way too long - they all can simply be named "Visit". I agree, actually that was my first try; however to be dynamic language friendly, I renamed them by adding the matching instruction class name. When all methods are named "Visit", static language compiler (like C#) can still decide which method to bind based on the parameter type, while IronPython has to surrender itself to classes derived from such visitor: If we wrote one customized "Visit" method for ThrowExceptionHunter, which base "Visit" method is supposed to override? If you need to override several "Visit" methods from ILInstructionVisitor, there is no way to implement it in Python - defining any amount of "Visit" methods only keeps the last one alive.

TryILReader.py can be viewed here.

More tools (and discussion) related to this ILReader coming soon. Hope one day this could become System.Reflection.ILReader ...

Disclaimer: THE CODE IS PROVIDED "AS IS", WITH NO WARRANTIES INTENDED OR IMPLIED. USE AT YOUR OWN RISK

Always Peverify IL Code of your Dynamic Method

Haibo Luo - MSFT — Wed, 01 Nov 2006 07:39:00 GMT

Current dynamic method implementation does not have built-in support to pre-check whether the dynamic method code is verifiable. With bad IL sequence, very often you will get "System.InvalidProgramException: Common Language Runtime detected an invalid program" when it gets executed. See the code example below, where C1, C2 are 2 simple reference types. Btw I understand only me write such "fun" IL code.

LocalBuilder lb = ilgen.DeclareLocal(typeof(C2));
ilgen.Emit(OpCodes.Newobj, typeof(C1).GetConstructor(Type.EmptyTypes));
ilgen.Emit(OpCodes.Stloc_0, lb);
ilgen.Emit(OpCodes.Ldc_I4_0);
ilgen.EmitWriteLine("I got called");
ilgen.Emit(OpCodes.Ret);

Even if you did not get InvalidProgramException, it does not mean the code-gen is correct. When the following code get invoked, "I got called too" will be printed; but the code is not verifiable. When the SkipVerification permission is refused, executing this dynamic method causes JIT to verify first: "System.Security.VerficationException: Operation could destabilize the runtime" throws.

LocalBuilder lb = ilgen.DeclareLocal(typeof(C2));
ilgen.Emit(OpCodes.Newobj, typeof(C1).GetConstructor(Type.EmptyTypes));
ilgen.Emit(OpCodes.Stloc_0, lb);
ilgen.EmitWriteLine("I got called too");
ilgen.Emit(OpCodes.Ret);

However both exception messages contain no much useful information: they do not tell which instruction could be blamed. For any production code using DynamicMethod, to ensure the code-gen quality, it is a good practice to have another implementation which have Reflection.Emit to emit the same IL code to disk, and call Peverify.exe to verify it. IronPython's code-gen has both approaches: at the beginning when we enforced this check, we caught several code-gen issues. This is what peverify.exe shows for the 2 scenarios above:

[IL]: Error: [D:\snippets.dll : Sample::M][offset 0x00000005][found ref 'C1'][expected ref 'C2'] Unexpected type on the stack.
[IL]: Error: [D:\snippets.dll : Sample::M][offset 0x00000011] Stack must be empty on return from a void function.
----
[IL]: Error: [D:\snippets.dll : Sample::M][offset 0x00000005][found ref 'C1'][expected ref 'C2'] Unexpected type on the stack.

Shift Away from CLR Reflection Area

Haibo Luo - MSFT — Wed, 01 Nov 2006 07:35:00 GMT

After 2-year Reflection test ownership and 1-year babysitting for Reflection.Emit and DynamicMethod areas, I am gradually away from these CLR feature areas, and will be more focusing on IronPython (and maybe others later).

Before completely leaving them behind, first of all, I want to apologize to you if you sent me emails about this blog before and I did not reply. I plan to answer some of those emails in the coming posts.

Also I will try my best to share more of what I learned last 2 years, before I forget them. If you have anything particular you want to discuss, please leave a comment.

Thanks for listening.

ILGenerator.EmitCall Mainly For vararg Methods

Haibo Luo - MSFT — Mon, 14 Aug 2006 02:18:00 GMT

Vararg (variable arguments) methods accept argument lists of unknown length and type. CLR supports this by the IL instruction (arglist) and other BCL types, such as System.ArgIterator. C# compiler has undocumented keyword "__arglist" to support defining vararg methods, accessing variable arguments and calling them, as shown below.

static void VarargMethod(string headline, __arglist) {
ArgIterator ai = new ArgIterator(__arglist);

Console.Write(headline);
while (ai.GetRemainingCount() > 0)
Console.Write(TypedReference.ToObject(ai.GetNextArg()));
Console.WriteLine();
}

static void CallVarargMethod() {
VarargMethod("Hello world from ", __arglist(Assembly.GetExecutingAssembly()));
VarargMethod("Current time: ", __arglist(DateTime.Now, " (UTC ", DateTime.UtcNow, ")"));
}

Two MemberRef tokens are needed to specify each vararg method call site. The blob each MemberRef points to specifies the methodref signature: types of both fixed and variable arguments.

MethodName: VarargMethod (06000001)
MemberRef #1 (0a00000c)
2 Arguments
Argument #1: String
Argument #2: <ELEMENT_TYPE_SENTINEL> Class System.Reflection.Assembly
MemberRef #2 (0a00000f)
5 Arguments
Argument #1: String
Argument #2: <ELEMENT_TYPE_SENTINEL> ValueClass System.DateTime
Argument #3: String
Argument #4: ValueClass System.DateTime
Argument #5: String

ILGenerator.EmitCall(OpCode opcode, MethodInfo methodInfo, Type[] optionalParameterTypes) is designed for emitting call to such vararg method (for both Reflection.Emit and dynamic method scenarios). The optionalParameterTypes argument is passed in to create different MethodRef tokens.

Assume the MethodInfo of "VarargMethod" is miVarargMethod, the following code creates a DynamicMethod to perform the same functionality as previous "CallVarargMethod" does:

DynamicMethod dm = new DynamicMethod("CallVarargMethod", typeof(void), Type.EmptyTypes, typeof(object));
ILGenerator il = dm.GetILGenerator();

il.Emit(OpCodes.Ldstr, "Hello world from ");
il.Emit(OpCodes.Call, typeof(Assembly).GetMethod("GetExecutingAssembly"));
il.EmitCall(OpCodes.Call, miVarargMethod, new Type[] { typeof(Assembly) });

il.Emit(OpCodes.Ldstr, "Current time: ");
il.Emit(OpCodes.Call, typeof(DateTime).GetMethod("get_Now"));
il.Emit(OpCodes.Ldstr, " (UTC ");
il.Emit(OpCodes.Call, typeof(DateTime).GetMethod("get_UtcNow"));
il.Emit(OpCodes.Ldstr, ")");
il.EmitCall(OpCodes.Call, miVarargMethod, new Type[] { typeof(DateTime), typeof(string), typeof(DateTime), typeof(string) });

il.Emit(OpCodes.Ret);

As you see in the attached C# code (which uses ILGenerator.EmitCall for both Reflection.Emit and DynamicMethod), Type.GetMethod (and other APIs) can return MethodInfo of vararg methods; however MethodInfo.Invoke currently does not support to invoke on them. It is easy to get around with this: just create a dynamic method, and use EmitCall, and invoke on the dynamic method.

Although EmitCall can be used to emit a call to non-vararg method, you are encouraged to use ILGenerator.Emit(OpCode, MethodInfo) for that scenario, and use EmitCall mainly for vararg methods. With that said, the naming of this API does sound somewhat misleading.

Member Order Returned by GetFields, GetMethods ...

Haibo Luo - MSFT — Mon, 10 Jul 2006 09:04:00 GMT

As John mentioned in his post, every Reflection user should keep this in mind: our code should not count on the member order returned by GetFields, GetMethods and other similar GetXXXs calls.

Given two fields (F1, F2) in one type, I think the compiler is free to emit F1 before F2, or vice versa. Also reflection does not guarantee to return fields in the order of how they present in the metadata; GetXXXs could be returning the members in different order at different running moments, which is illustrated by the following code. (Of course, they should have the same set of members)

using System;
using System.Reflection;

public class C {
public int F1;
public int F2;

static void Main() {
typeof(C).GetField("F2"); // Line 9

PrintFields(); // Line 11
PrintFields(); // Line 12

GC.Collect();
GC.WaitForPendingFinalizers(); // Line 15

PrintFields(); // Line 17
PrintFields(); // Line 18
}

static void PrintFields() {
foreach (FieldInfo fi in typeof(C).GetFields())
Console.Write("{0} ", fi.Name);
Console.WriteLine();
}
}

Below is mostly like what you will get when running under .NET 2.0 (50727.42). You may download the attached C# file, and get it a try. (By the way, the PE file generated by C# compiler has F1 before F2 in the metadata).

> FieldOrder.exe
F1 F2
F2 F1
F1 F2
F1 F2

Behind the scenes is reflection 2.0's MemberInfo caching mechanism. Joel Pobar’s MSDN article has a nice overview about it. Here are some basic facts before diving into each important line.

Each type has its own MemberInfo caches. Reflection creates different caches for different member types, one for each (if necessary): FieldInfo, MethodInfo, ConstructorInfo...
The caches are created and populated lazily. If ConstructorInfos are never asked, the ConstructorInfo cache will not be created. If only one method is requested for MethodInfo, only that method’s MethodInfo will be populated into the MethodInfo cache, not with all other methods.
The type keeps a weak reference to its MemberInfo caches, so, for example, when there is no reference to this type’s FieldInfos, the runtime could reclaim the memory used by the FieldInfo cache.

Line 9: we are asking for FieldInfo of "F2". At that time, the type C’s FieldInfo cache is null, Reflection will create the cache, look for the field with name "F2", create an object RuntimeFieldInfo, and put it into the cache. Imagining the cache looks like an ArrayList, the first element of the cache now contains FieldInfo "F2".

Line 11: typeof(C).GetFields() asks for type C's all fields (with default binding flags). It goes through all fields, and returns an array of FieldInfo (in order of F1/F2). Before returning this array back to the user, it updates the cache: appending "F1" after "F2" ("F2" already exists as the result of line 9). It also marks this cache "complete".

Line 12: the 2nd GetFields notices that the cache was built completely, immediately scans the cache, and returns an array of FieldInfo F2/F1 in that order.

Line 17: the 3rd GetFields occurs after GC, where the FieldInfo cache was reclaimed; GetFields has to re-create the cache. It goes through all fields again, and the cache is filled with FieldInfos in the order of F1/F2.

Line 18: same reason as the 2nd GetFields call, it returns F1/F2 quickly.

The picture at right shows the appearance/changes of the FieldInfo cache. Note the cache after line 11 has F2 in front of F1, GetFields at that line still returns F1/F2.