Haibo Luo's weblog

ILVisualizer 2010 Solution

Haibo Luo - MSFT — Mon, 19 Apr 2010 17:38:00 GMT

The projects are set to be built against the .NET 4.0.

IronPython: System.Reflection.Assembly object

Haibo Luo - MSFT — Thu, 13 Mar 2008 06:12:00 GMT

IronPython offers a little bit more love to the Assembly object instance: we can directly access the assembly's top-level members (namespace, public type) via the dot operation. System.Reflection provides many ways to let you hold the assembly object, such as Assembly.Load method, Type.Assembly property, ...

For example, given assembly1.dll compiled from the C# file below, dir(a1) in the python below shows the class name "C2" and the top level namespace "NS1" in its' dictionary.

// assembly1.cs
namespace NS1 {
    namespace NS2 {
        public class C1 { }
    }
}
public class C2 {
    public class C3 { }
}
class C4 { }

# part 1 from file/test1.py
import System
a1 = System.Reflection.Assembly.Load("assembly1")

print dir(a1)  # ['C2', ..., 'NS1', ...]
print a1.NS1   # <module 'NS1' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print a1.C2    # <type 'C2'>

# access non-public type
print a1.C4    # AttributeError: 'Assembly' object has no attribute 'C4'

# access the down-level namespace, type with the dot operations
print a1.NS1.NS2.C1         # <type 'C1'>
# access the nested type
print a1.C2.C3              # <type 'C3'>

Normally I would use clr.AddReference to bring in the CLR assembly. The python code below gives the similar output.

# part 1 of file/test2.py
import clr
clr.AddReference("assembly1")
import NS1 
import C2

print NS1        # <module 'NS1' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print C2         # <type 'C2'>
print NS1.NS2    # <module 'NS2' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print NS1.NS2.C1 # <type 'C1'>
print C2.C3      # <type 'C3'>

However the first approach may help keep the global (and local) dictionary less "polluted" and sometimes can be used to avoid the namespace/type name collision. For example, suppose we need to load in another assembly built from the following C# code, and we use the clr.AddReference approach.

// assembly2.cs
namespace NS1 { 
    public class NS2 { } 
}

# part 2 of file/test2.py
clr.AddReference("assembly2")
print NS1.NS2    # <type 'NS2'>
print NS1.NS2.C1 # AttributeError: 'type' object has no attribute 'C1'

NS2 is a type now (no longer the namespace (or python module) NS1.NS2 from "assembly1"), and we lost the access to the type NS1.NS2.C1. If we use the assembly object approach, we can hold both the namespace NS1.NS2 from assembly1 and the type NS1.NS2 from assembly2; basically we can think the assembly variable name adds one more layer to prevent the name collision. Or in the other direction of thinking, what clr.AddReference does for you (related to implicit name merge/override) may not be what exactly you want.

# part 2 from file/test1.py
a2 = System.Reflection.Assembly.Load("assembly2")
print a2.NS1     # <module 'NS1' (CLS module from assembly2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print a2.NS1.NS2 # <type 'NS2'>
print a1.NS1     # <module 'NS1' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>
print a1.NS1.NS2 # <module 'NS2' (CLS module from assembly1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)>

Note that such name collision should rarely happen for well-designed framework assemblies. Also IronPython merge types/down-level namespaces under the same namespaces, even if they are from different assemblies. One simple example is System.Int32 from mscorlib.dll and System.Uri from System.dll, both peacefully under the "System" module.

import System
print System            
# <module 'System' (CLS module, 2 assemblies loaded)>
print System.__file__   
# ['mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089', 
#  'System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089']
import clr
print clr.GetClrType(System.Int32).Assembly  
# <Assembly mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089>
print clr.GetClrType(System.Uri).Assembly    
# <Assembly System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089>

IronPython: System.Array

Haibo Luo - MSFT — Thu, 13 Mar 2008 06:04:13 GMT

When you start interop'ing with .NET in IronPython, sooner or later, you will find that you are in need of creating an array as argument. There are mainly 2 ways to create array objects:

Array with type indexing (for one-dimensional array object only), and
classic reflection API: Array.CreateInstance

System.Array indexing with a type creates a concrete array type, which can then take a collection of the element objects (or an enumerable) and form the array object. There are half dozen Array.CreateInstance overloads, which allow us to create one-dimensional empty array, also multi-dimensional/non-zero lower-bound arrays.

import System
array_int = System.Array[int]
print array_int               # <type 'Array[int]'>

# list
print array_int([1, 2])       # System.Int32[](1, 2)
# tuple
print array_int((3, 4, 5))    # System.Int32[](3, 4, 5)
# xrange
print array_int(xrange(6,10)) # System.Int32[](6, 7, 8, 9)
# CLR List
print array_int(System.Collections.Generic.List[int]()) # System.Int32[]()

# one-dimensional array 
a1 = System.Array.CreateInstance(int, 5)
for i in range(5): a1[i] = i * 10
print a1        # System.Int32[](0, 10, 20, 30, 40)

# two-dimensional array 
a2 = System.Array.CreateInstance(float, 2, 2)
a2[1, 1] = 3.14
print a2        # System.Double[,](
                # 0.0, 0.0
                # 0.0, 3.14)

IronPython also supports some python list-like operation to the CLR array objects, such as indexing with slice, +, *, in...

a1 = array_int(range(5))
a2 = array_int([100])

# slice
print a1[1:-1]      # System.Int32[](1, 2, 3)
# +
print a1 + a2       # System.Int32[](0, 1, 2, 3, 4, 100)
# *
print a1 * 2        # System.Int32[](0, 1, 2, 3, 4, 0, 1, 2, 3, 4)
# +, upcast the result to object[]
print a2 + System.Array[str]('py')  # System.Object[](100, 'p', 'y')

# slice with step
print a1[1::2]      # System.Int32[](1, 3)
# assignment 
a1[1::2] = [11, 13]
print a1            # System.Int32[](0, 11, 2, 13, 4)

# in/__contains__
print 11 in a1      # True

IronPython: ipyw.exe

Haibo Luo - MSFT — Thu, 13 Mar 2008 06:03:48 GMT

Each IronPython binary release ships two executable files: ipy.exe and ipyw.exe. Their (only) difference is, ipy.exe is a console application and ipyw.exe is a windows application. So given the following winform.py,

## winform.py
import clr
clr.AddReference("System.Windows.Forms")
from System.Windows.Forms import *

class SimpleForm(Form):
    def __init__(self):
        self.Text = 'Simple Winform'

Application.Run(SimpleForm())

Running it with ipy.exe will keep the associated console; however, if I launch it with ipyw.exe in a console window, I can continue working on other stuffs within that console.

I found that Shawn wrote some background about WINDOW_CUI/WINDOW_GUI subsystem here.

SignatureResolver (unfinished)

Haibo Luo - MSFT — Sun, 09 Mar 2008 01:44:00 GMT

While writing the ILVisualizer for dynamic method late 2005, I'd like to show the local variable information as well; so I started working on the managed signature parser, at least to parse LocalVarSig (Ecma-335 23.2.6). It was 2+ years ago, and never got to a finished state. Now perhaps I will never spend more time on it, so I refreshed the code a bit and you may find it useful for some scenarios.

Here is an example of how to use SignatureResolver to get LocalVarSig in IronPython. The static method SignatureResolver.GetLocalVarSig takes the byte array (the signature blob) and the enclosing method, and returns an array of LocalVar (which includes mainly the runtime type) inside that method.

# show local variables for every Array.CreateInstance overload methods
import clr
import System
clr.AddReference("ClrTest.Reflection.SignatureResolver")
from ClrTest.Reflection import *

for method in clr.GetClrType(System.Array).GetMember("CreateInstance"):
    body = method.GetMethodBody()   # C# won't allow this
    tok = body.LocalSignatureMetadataToken
    sig = method.DeclaringType.Module.ResolveSignature(tok)
    print "> %s, 0x%x" % (method, tok)
    for lv in SignatureResolver.GetLocalVarSig(sig, method):
        print lv

Next is part of the output showing one Array.CreateInstance method's local variable types. The result is, of course, same as what Reflector tells us in the below picture. "(p)" means the local variable is pinned.

> System.Array CreateInstance(System.Type, Int32[], Int32[]), 0x11000009
System.RuntimeType 
System.Int32 
System.Int32& (p)
System.Int32& (p)
System.Array 
System.Int32[] 
System.Int32[] 
System.RuntimeTypeHandle

Yes, I understand the MethodBody.LocalVariables property gives us the similar information; but the dynamic method can not. That was why I wanted to decode the bytes, and to use DynamicScopeTokenResolver to get a meaningful type string. I admit this version of code diverts from its original goal: it only works for RuntimeMethodInfo/RuntimeConstructorInfo, and won't work for dynamic method; but if one day, you are in need of parsing blob, the functions like GetToken/GetInt32Uncompressed could be handy. GetToken basically is the reverse of TypeDefOrRefEncoded (ecma-335: 23.2.8)

int GetToken() {
    int token = GetInt32Uncompressed();
    return s_mapForTokenType[token & 0x03] | (token >> 2);
}

int GetInt32Uncompressed() {
    byte first = _sig[_pos];

    if (first <= 0x7F) {
        return _sig[_pos++];
    } else if (first <= 0xBF) {
        byte[] bytes = new byte[2];
        bytes[1] = (byte)(_sig[_pos++] - 0x80);
        bytes[0] = _sig[_pos++];
        return BitConverter.ToInt16(bytes, 0);
    } else {
        byte[] bytes = new byte[4];
        bytes[3] = (byte)(_sig[_pos++] - 0xC0);
        bytes[2] = _sig[_pos++];
        bytes[1] = _sig[_pos++]; ;
        bytes[0] = _sig[_pos++];
        return BitConverter.ToInt32(bytes, 0);
    }
}

My SignatureResolver also provide another static method GetMethodRefSig to get MethodRefSig (Ecma-335 22.25 and 23.2.2).

If you are interested in this topic, you may download the attached file, unzip it, and take a look at the source. Also try to use VS2008 to build it.

If you are an IronPython user, you may try the accompanying tests (in IronPython). I compare the SignatureResolver.GetLocalVarSig result with the official reflection API MethodBody.LocalVariables: they are same for all method/constructors from all simple-LoadFrom-successful framework assemblies. If you have ClrTest.Reflection.ILReader.dll (or know how to get one), you can try test_methodrefsig.py, which tests GetMethodRefSig.

As stated in the title, this is *unfinished* and could be buggy.

Disclaimer: THE CODE IS PROVIDED "AS IS", WITH NO WARRANTIES INTENDED OR IMPLIED. USE AT YOUR OWN RISK.

ILVisualizer VS2008 solution

Haibo Luo - MSFT — Sat, 08 Mar 2008 03:11:00 GMT

I attached the ILVisualizer VS2008 solution in this post. There is no source code update; the only change is to upgrade the Microsoft.VisualStudio.DebuggerVisualizers.dll reference from version 8.0.0.0 to version 9.0.0.0.

Just for your convenience, so you can see a successful build after downloading in VS2008.

Thanks Julien for leaving the instruction here.

IronPython: Accessing the .NET Field

Haibo Luo - MSFT — Mon, 29 Oct 2007 06:10:00 GMT

The .NET libraries normally do not expose the public field, but we can still find its' presence for certain scenarios. IronPython treats the .NET fields like python attribute. For python attribute, the typical operations are set, get and delete. In general, IronPython throws TypeError when deleting members of CLR types and their objects (so for fields). To do set and get operation, the IronPython code looks similar to C# code. Updating (setting) values to the read-only or literal fields throws as we expect.

public class C {
public static int StaticField;
public int InstanceField;
}

>>> ...
>>> C.StaticField = 1          # set
>>> C.StaticField              # get
1
>>> c = C()
>>> c.InstanceField = 2        # set
>>> c.InstanceField            # get
2

Guess what you will get for C.InstanceField? It is "field descriptor", same as C.__dict__['InstanceField'], upon which we can call __set__/__get__ or two equivalent helper methods SetValue/GetValue (feel like repeating CLR reflection FieldInfo API here). Although not shown below, we can also set/get the value of the static field too by calling these methods on C.__dict__['StaticField'].

>>> C.InstanceField
<field# InstanceField on C>
>>> C.InstanceField is C.__dict__['InstanceField']
True
>>> C.InstanceField.__set__(c, 3)
>>> C.InstanceField.__get__(c)
3
>>> C.InstanceField.SetValue(c, 4)
>>> C.InstanceField.GetValue(c)
4

IronPython allows accessing static field on instance too. On the other hand, C# compiler requires type name to access static member (otherwise CS0176).

>>> c.StaticField = 5
>>> c.StaticField, C.StaticField
(5, 5)

Updating fields of value types is not allowed. If interested, you may go ahead read the design decision here. Until recently I did not know that as a remedy, the feature of setting fields/properties in the constructor call should be able to set field for value types while still keeping value types effectively immutable after the creation. Unfortunately we had no tests to cover this part (which never worked). This code below illustrates the expected behavior. I expect it to work soon.

public struct MyPoint {
public int X, Y;
public override string ToString() { return string.Format("X: {0}, Y: {1}", X, Y); }
}

>>> p = MyPoint(X=1, Y=2)
>>> p
<MyPoint object at 0x000000000000002C [X: 1, Y: 2]>
>>> p.X = 3
# one kind of exception still throws

VisualStudio as my IronPython editor

Haibo Luo - MSFT — Wed, 17 Oct 2007 08:00:00 GMT

The following steps are what I did to get Visual Studio ready as my IronPython (and IronRuby) editor.

Install the latest internal dogfood build of Visual Studio 2008.
- you may use Visual Studio 2005 or download the VS 2008 public beta2;
Download and install ASP.NET futures release (July 2007). This will give me the nice syntax coloring and (well... limited) intellisense for python code.
- you may download VSSDK 2008 CTP, and build the IronPython integration sample, which will give you the similar editing experience;
Download the attached add-in binary zip (poorly named as DlrToolsAddin); for my Vista box, I extract them under "%USERPROFILE%\Documents\Visual Studio 2008\Addins". I wrote this add-in basically to allow me to run the script inside the Visual Studio, and show the output in the output window.
- if you consider using it for VS 2005, you will need put it in "Visual Studio 2005\Addins".

Say, I am trying the example in my previous post, I create a new C# file (sample.cs) without creating solution/project, I can press Ctrl+1 (or the first button in the DlrTools toobar) to get it compiled to sample.dll in the local directory. You see the compile result at the bottom output window. Then I create a python file (method.py) in the same directory, type in some code (shown in color). Again, I can press Ctrl+1, the result are shown in the output window. I do not need leave VS and run these files in the cmd.exe window.

Sometimes I want to run the same code with different tools (for example, to check IronPython compatibility, I often run the same .py file against C-Python 2.5 too). That is the first combo box comes to play.

The second combo box is to set the working directory: "." means the current directory where the active file lives, ".." for the parent directory of the current file, or you may use the absolute path. Such support is needed to run C-Python regression tests. (For the add-in implementation side, I feel what I really want is a combo box of type vsCommandControlTypeDynamicCombo, which seems not available for add-in development).

The delete button is to remove your tool choice for those files you pressed "Run Script". By clicking the last button, an XML file (specifying which tools for which file extension) is opened so you may change it. You must update it with your tool paths, since the default setting is suited for myself.

You may also run part of the file by selecting those lines first (A temporary file is created and will be deleted after VS shutdown).

One bad thing I noticed of this add-in is that sometimes I press the "run script" button to start debugging, both icons have similar shapes.

To uninstall it, delete DlrToolsAddin.* files under "Addins"; and then run once "devenv.exe /resetaddin DlrToolsAddin.Connect".

The source code (VS2008 project) is also attached. Disclaimer: THE CODE IS PROVIDED "AS IS", WITH NO WARRANTIES INTENDED OR IMPLIED. USE AT YOUR OWN RISK. Hope you like Visual Studio as your IronPython editor.

IronPython: explicitly choose one method

Haibo Luo - MSFT — Fri, 12 Oct 2007 06:32:47 GMT

C# allows method overloading. Given an argument list, the compiler performs the overload resolution to select the best method to invoke. The chosen method token is baked into the assembly. IronPython does the method resolution at the IronPython run time; for most scenarios, it has no problem in picking up the most reasonable one based on the argument list. The rules are a bit complicated, and I will leave them for a future post. If you believe IronPython chooses the "unexpected" method for your scenario, please do let us know.

IronPython also provides the "Overloads" (python) attribute for the bound method, on which we can do index operation with parameter types to get the exact method we want. In the example below, the .NET reflection API Type.MakeByRefType and MakeArrayType are used in order to get those constructed types. Since these calls are only available for System.Type object, clr.GetClrType is called first on the python type to get the underlying CLR type. However, I am able to use the python type for the first 2 index calls (thanks to the type conversion).

public class ChooseOverload {
   public void M(int arg) { Console.WriteLine(1); }
   public void M(byte arg1, byte arg2) { Console.WriteLine(2); }
   public void M(ref int arg) { Console.WriteLine(3); arg = 10; }
   public void M(int[] arg) { Console.WriteLine(4); }
}

>>> ... # add the assembly
>>> import ChooseOverload
>>> o = ChooseOverload()
>>> o.M
<built-in method M of ChooseOverload object at 0x000000000000002B>
>>> o.M.Overloads
{'M(self, int arg)': <built-in function M>, 'M(self, Byte arg1, Byte arg2)': <built-in function M>, 'int M(self, int arg)': <built-in function M>, 'M(self, Array[int] arg)': <built-in function M>}
>>> o.M.Overloads[int](1)
1
>>> o.M.Overloads[System.Byte, System.Byte](1, 2)
2
>>> o.M.Overloads[clr.GetClrType(int).MakeByRefType()](1)
3
>>> o.M.Overloads[clr.GetClrType(int).MakeArrayType()](None)
4

When generics come to the picture, o.M could mean the generic method. Similar to the generic type, IronPython chooses the index operation to hold the instantiated generic method. Currently IronPython has no support to specify parameter types which are (or made from) generic type parameters (of either the generic method or type). Expect this will be supported one day.

public void M2<T>(T arg) { Console.WriteLine(5); }
public void M2<T>(byte arg) { Console.WriteLine(6); }

>>> o.M2[int](1) # bind to the first M2 given the argument "1"
5
>>> o.M2[int].Overloads[System.Byte](1) # in order to get the second M2
6
>>> o.M2[int].Overloads[???](1) # unable to pick the first M2 currently

IronPython: Provide (or not) Argument for by-ref Parameter

Haibo Luo - MSFT — Fri, 05 Oct 2007 08:21:00 GMT

Python function may return multiple objects as a tuple. The .NET method can only return one object as the result of a call; in order to return more than one objects, by-ref parameters are needed. They are decorated with the parameter modifier (ref, out) in C#.

Dictionary.TryGetValue(TKey key, out TValue value) has an output parameter "value". The API returns true if the dictionary contains an element with the specified key (the element value itself is returned to the parameter "value"), otherwise it returns false. When making calls to such methods in IronPython, we may not pass in argument for the output parameter. Instead, the result of such .NET method call in IronPython will likely be a tuple (unless the .NET method's return type is void and the method has only one by-ref parameter), which contains the value of the output parameter (see the example below).

import System
d = System.Collections.Generic.Dictionary[int, str]()
d[1], d[2] = 'One', 'Two'

print d.TryGetValue(1)         # (True, 'One')
print d.TryGetValue(2)[1]      # Two
print d.TryGetValue(3)         # (False, None)

We may also pass a clr.Reference object for the output parameter. The clr.Reference object has only one member "Value", which will carry the output parameter value after the call. "clr.Reference" object is of the type identical to System.Runtime.CompilerServices.StrongBox<T> in the new .NET Framework 3.5. When encountering this type of argument, IronPython codegen/runtime does something special to update the "Value".

x = clr.Reference[str]()             # like "new StrongBox<string>()" in C#
print d.TryGetValue(1, x), x.Value   # True One
print d.TryGetValue(3, x), x.Value   # False None

For reference parameter, we are required to pass in something. If the argument is not clr.Reference object, such reference argument value will also be part of the returned tuple; otherwise, the by-ref change is tracked inside clr.Reference. The following C# code is my twisted way to find the maximum number; the IronPython snippet shows the behaviors of calling that method not using and using clr.Reference.

public class C {
public bool M(int current, ref int max) {
    if (current > max) {
      max = current; return true;
    } else return false;
}
}

import C
o = C()

print o.M(10, 20)             # (False, 20)
print o.M(30, 20)             # (True, 30)

x = clr.Reference[int](20)
print o.M(10, x), x            # False 20
print o.M(30, x), x            # True 30

o.M(1)                        # TypeError: M() takes exactly 2 arguments (1 given)

Note if the .NET method contains more than one by-ref parameters, IronPython expects the user to provide proper clr.Reference objects for either ALL or NONE of them. A mix of clr.Reference object and normal argument/omission (for out parameter) will cause error (as illustrated below).

public int M2(ref int x, ref int y) {...}

o.M2(clr.Reference[int](1), 2) # TypeError