Shri Borde's WebLog

GitDiff.bat - power-diffing with GIT

shrib — Mon, 13 Apr 2009 16:56:14 GMT

As mentioned in the GIT documentation, you can set the environment variable GIT_EXTERNAL_DIFF to use a custom diff utility. Try doing "set GIT_EXTERNAL_DIFF=echo" and running a "git diff" command to see how GIT_EXTERNAL_DIFF works.

One limitation of this technique is that GIT launches the diff utility program once for every file modified. What this means is that you do not get to see the list of all files changed upfront (if you want to see how many files have changed, etc), and are also forced to review the files in the order GIT chooses. If you want to see all the test files first for example, or if you need to go back to a file you have already diffed, you have to start the "git diff" command all over again. Also, if you want to end the review half way through, you cannot do that easily and are forced to manually close the diff program multiple times until the "git diff" command terminates.

Luckily, there is a simple solution to this. You can use a simple script. See GitDiff.bat (and GitDiff.rb). The arguments to the script are the same as "git diff". However, it will show you a diff of two folders, one with the old version of the files and one with the new version of the files. You can review the files in the order you chose...

Here is a screenshot using Beyond Compare as the diffing tool. The window at the top is the main window showing the list of files. The two windows at the bottom are diffs of two of the changed files.

Setup instructions:

Copy GitDiff.bat and GitDiff.rb to somewhere in your path
Make sure that ruby.exe is in the path
Go to a folder in a GIT repo
GitDiff.bat <old-sha> <new-sha>

Proposed spec for Ruby's Thread.critical=

shrib — Wed, 07 Jan 2009 17:24:31 GMT

Ruby has green threads, and so its implementation of Thread.critical= is speced as follows:

Sets the status of the global ``thread critical’’ condition and returns it. When set to true, prohibits scheduling of any existing thread. Does not block new threads from being created and run. Certain thread operations (such as stopping or killing a thread, sleeping in the current thread, and raising an exception) may cause a thread to be scheduled even when in a critical section. Thread::critical is not intended for daily use: it is primarily there to support folks writing threading libraries.

The requirement of descheduling other threads complicates things for Ruby implementations like IronRuby and JRuby which use native threads. Furthermore, most uses of Thread.critical= do not actually depend on the descheduling of other threads. This is canonical usage:

def get_unique_cookie()
    begin
        Thread.critical = true
        cookie = @@counter
        @@counter += 1
    ensure
        Thread.critical = false
    end
    cookie
end

So most usages expect Thread.critical to behave like a critical section. Given this, here is the proposed new behavior. This behavior can be efficiently implemented even on native threads, and also works for most usage scenarios providing users with a simple locking construct.

Sets the status of the global ``thread critical’’ condition and returns it. When set to true, blocks any other thread that tries to set Thread.critical=true. It is recommend to use an ensure clause to set Thread.critical to false to exit the "global critical section".

If a thread terminates while it is in the global critical section (either because of an uncaught exception, because of normal termination at the end of the block, or because of a call to Thread.kill), it leaves the process in an undefined state. Some implementations might leave the global critical section allowing other threads to enter it; some implementations might do nothing causing the app to hang if any other thread tries to set Thread.critical=true; some might exit the process.

Calling "sleep" in the critical section does nothing. Calling "Thread.stop" in the global critical section also does nothing (it sets Thread.critical to false in Ruby 1.8.6).

Thread.critical= is not reentrant. If a thread in the global critical section tries to set Thread.critical=true again, the behavior is undefined. Some implementations might require a single Thread.critical=false to leave the global critical section; some might require a matching number of Thread.critical=false; some might throw an exception.

The behavior when calling Thread.critical=false from a thread that is not in the global critical section causes undefined behavior. Some implementations might ignore the call; some might cause the thread in the global critical section (if any) to exit the critical section; some might throw an exception.

The code that executes in the global critical section should be kept to a minimum, and should ideally not call any user-defined code as it opens up the possibility of deadlocks. Since its hard to know when user-defined code can execute (because any library method may have been monkey-patched), Thread.critical= should be used very carefully. Thread::critical is not intended for daily use: it is primarily there to support folks writing threading libraries.

We intend to propose this to ruby-core for approval.

Update: Ruby 1.9 does not support Thread#critical! So this issue should go away in the future. For 1.8.*, here is the ruby-core discussion thread. There may be a few scenarios which rely on thread descheduling, but it does seem to be marginal. JRuby is going to try to implement Thread#critical= as a global mutex. IronRuby is likely to stay on that plan too.

OLE Automation support ON in IronPython 2.0 Beta 4

shrib — Wed, 30 Jul 2008 22:15:00 GMT

As I had mentioned in a previous post, we have added support in IronPython for accessing OLE Automation objects using the IDispatch interface, without having to rely on interop assemblies. This was enabled with the "ipy.exe -X:PreferComDispatch" command-line until IronPython 2.0 Beta 3. In Beta 4, this behaviour is on by default. This makes IronPython's support for OLE Automation more script-friendly akin to VBScript and JScript.

See the difference

In IronPython 1.X, trying to use Excel would cause IronPython to generate the inteorp assembly on the fly, which would take more than minute, before resulting in an exception. If you explictly added a reference to Microsoft.Office.Interop.Excel.dll, then things get back on track. However, note that not all machines with Office installed will necessarily have the primary interop assemblies (PIAs) installed in the GAC, as explained in this link.

c:\IronPython-1.1.2>ipy.exe -v
IronPython 1.1.2 (1.1.2) on .NET 2.0.50727.1433
Copyright (c) Microsoft Corporation. All rights reserved.
>>> import System
>>> t = System.Type.GetTypeFromProgID("Excel.Application")
>>> excel = System.Activator.CreateInstance(t)
>>> wb = excel.Workbooks.Add() # This will take a long time
Generating Interop assembly for 000208d5-0000-0000-c000-000000000046
Traceback (most recent call last):
  File , line 0, in <stdin>##16
SystemError: Could not load type 'Excel._Application' from assembly 'Excel, Version=1.6.0.0, Culture=neutral, PublicKeyToken=null'.
>>>
>>> import clr
>>> clr.AddReference("Microsoft.Office.Interop.Excel")
>>> wb = excel.Workbooks.Add()
>>>

In IronPython 2.0 Beta 3, things work much better, but as can be seen from the type of "wb", it still relies on the PIA. If the PIA is not installed or registered on the machine, IronPython would try to generate the interop assembly on the fly, again taking a very long time.

c:\IronPython-2.0B3>ipy.exe -v
IronPython 2.0 Beta (2.0.0.3000) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> import System
>>> t = System.Type.GetTypeFromProgID("Excel.Application")
>>> excel = System.Activator.CreateInstance(t)
>>> wb = excel.Workbooks.Add()
>>> wb
<Microsoft.Office.Interop.Excel.WorkbookClass object at 0x000000000000002B>
>>> wb.GetType().Assembly.CodeBase
'file:///C:/Windows/assembly/GAC/Microsoft.Office.Interop.Excel/12.0.0.0__71e9bce111e9429c/Microsoft.Office.Interop.Excel.dll'
>>> excel.Quit()
>>>

In IronPython 2.0 Beta 4, things just work, thanks to the shiny new OleAut support. "wb" is just a simple COM object that supports IDispatch, and there is no need for a PIA anymore!

c:\IronPython-2.0B4>ipy.exe -v
IronPython 2.0 Beta (2.0.0.4000) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> import System
>>> t = System.Type.GetTypeFromProgID("Excel.Application")
>>> excel = System.Activator.CreateInstance(t)
>>> wb = excel.Workbooks.Add()
>>> wb
<System.__ComObject (_Workbook) object at 0x000000000000002B>
>>> excel.Quit()
>>>

Different ways to get started

IDispatch is about invoking a method on an existing OleAut object. But how do you get to an OleAut object in the first place? VBScript and VBA have a method called CreateObject to create an OleAut object. There are multiple ways of doing the same in IronPython.

The first is to use the Type.GetTypeFromProgID method as shown here. This is the simplest solution.

>>> import System
>>> t = System.Type.GetTypeFromProgID("Excel.Application")
>>> excel = System.Activator.CreateInstance(t)
>>> wb = excel.Workbooks.Add()

The second way is to use the new "clr.AddReferenceToTypeLibrary" function. This currently requires hard-coding the GUID of the typelib (we could consider easier ways in the future). However, the advantage is that you can access the typelib just like a Python module. This is particularly handy if you need to access enum values in your program. Using the TLB, you can use the symbolc enum names instead of hard-coding int constants.

>>> import System
>>> excelTypeLibGuid = System.Guid("00020813-0000-0000-C000-000000000046")
>>> import clr
>>> clr.AddReferenceToTypeLibrary(excelTypeLibGuid)
>>> from Excel import Application
>>> excel = Application()

>>> wb = excel.Workbooks.Add()

Finally, you can use the PIA as before. There is no particular advantage to doing this. It just means that your old code will keep on working as before (though method invocation will use the new IDispatch mechanism).

>>> import clr
>>> clr.AddReference("Microsoft.Office.Interop.Excel")
>>> from Microsoft.Office.Interop.Excel import Application
>>> excel = Excel()
>>> wb = excel.Workbooks.Add()

All of the above techniques end up calling the Win32 function CoCreateInstance. So one more technique you could use is to make a pinvoke call to CoCreateInstance!

Once you have created a COM object, the rest of the code to access the object should work pretty much like before; it just wont require the presence of a PIA.

How to turn back the clock?

If you run into any bugs with the new OleAut support, there is a way to get to the old behavior. This is done by setting an environment variable called COREDLR_PreferComInteropAssembly to "TRUE" as show here. Do report the bug to users@lists.ironpython.com so that the issue can be fixed. COREDLR_PreferComInteropAssembly might go away in the future.

c:\IronPython-2.0B4>ipy.exe
IronPython 2.0 Beta (2.0.0.4000) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> import System
>>> t = System.Type.GetTypeFromProgID("Excel.Application")
>>> excel = System.Activator.CreateInstance(t)
>>> wb = excel.Workbooks.Add()
>>> wb
<System.__ComObject (_Workbook) object at 0x000000000000002B>
>>> ^Z 

 
c:\IronPython-2.0B4>set COREDLR_PreferComInteropAssembly=TRUE 

 
c:\IronPython-2.0B4>ipy.exe
IronPython 2.0 Beta (2.0.0.4000) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> import System
>>> t = System.Type.GetTypeFromProgID("Excel.Application")
>>> excel = System.Activator.CreateInstance(t)
>>> wb = excel.Workbooks.Add()
>>> wb
<Microsoft.Office.Interop.Excel.WorkbookClass (Workbook) object at 0x000000000000002B>
>>> wb.GetType()
<System.RuntimeType object at 0x000000000000002C [Microsoft.Office.Interop.Excel.WorkbookClass]>
>>>

Known issues

There are few expected differences to be aware of.

There is one important detail worth pointing out. IronPython tries to use the typelib of the OleAut object if it can be found, in order to do name resolution while accessing methods or properties. The reason for this is that the IDispatch interface does not make much of a distinction between properties and method calls. This is because of VB6 semantics where "excel.Quit" and "excel.Quit()" have the exact same semantics. However, IronPython has a strong distinction between properties and methods, and methods are first class objects. For IronPython to know whether "excel.Quit" should invoke the method Quit, or just return a callable object, it needs to inspect the typelib. If a typelib is not available, IronPython assumes that it is a method. So if a OleAut object has a property called "prop" but it has no typelib, you would need to write "p = obj.prop()" in IronPython to read the property value. This behavior is by design.
Calling a method with "out" (or in-out) parameters requires explicitly passing in an instance of "clr.Reference" as described in this blog, if you want to get the updated value from the method call. Note that COM methods with out parameters are not considered Automation-friendly. JScript does not support out parameters at all. If you do run into a COM component which has out parameters, having to use "clr.Reference" is a reasonable workaround. Another workaround is to leverage the inteorp assembly by using the full method name syntax of "outParamAsReturnValue = InteropAssemblyNamespace.IComInterface(comObject)".

Note that the Office APIs do have "VARIANT*" parameters, but there methods do not update the value of the VARIANT. The only reason they were defined with "VARIANT*" parameters was for performance since passing a pointer to a VARIANT is faster than pushing all the 4 DWORDs of the VARIANT onto the stack. So you can just treat such parameters as "in" parameters.
Real world examples of APIs with out parameters are ADODB.Command.Execute, most of the Microsoft Agent APIs. If you run into any others, let me know, so that I can compile a list here for reference. If this issue affects many scenarios, we could consider returning the out parameters as a return value tuple. This would require checking the ITypeInfo to check the type of parameters. We have avoided doing this so that we can have the same consistent code path irrespective of whether the typelib is available or not.

Currently, there are a few bugs in corner cases:

Passing a Python long (BigInteger) as an argument to a COM method expecting a LONGLONG causes a TypeError (InvalidCastException).
Passing some types of objects (eg "xrange(1)") to a COM method expecting an IUnknown results in a TypeError (InvalidCastException).
There are some corner cases where code that would have throw an exception before is now allowed. For example, it is possible to pass any object to a COM method expecting a BSTR. This is because the default IDispatch marshalling rules allow this. Noone should be affected by this because it is a negative scenario anyway.

Under the hoods

Running with "ipy.exe -X:ShowRules", you can see that the generated code makes pinvoke calls on the IDispatch interfaces. This is similar to what VBScript (or VBA late bound code) would have done. UnsafeMethods.IDisaptchInvoke is a helper function in Microsoft.Scripting.Core.dll that will dereference the vtable of the OleAut object, get the function pointer for IDispatch::Invoke, and make a call to the function pointer.

>>> wb = com.Workbooks
//
// AST: Rule
// .scope <rule> (
) {
    .if ((((.arg $arg0) != .null) && (Marshal.IsComObject)(
        (.arg $arg0),
    )) ) {.return .site (Object) GetMember Workbooks( // Python GetMember Workbooks IsNoThrow: False
        (ComObject.ObjectToComObject)(
            (.arg $arg0),
        )
        (.arg $arg1)
    );
    } .else {/*empty*/;
    }}

// AST: Rule
// 
.scope <rule> (
) {
    .if (((((.arg $arg0) != .null) && (((Object)(.arg $arg0)).(Object.GetType)() == ((Type)IDispatchComObject))
    ) && ((IDispatchComObject)(.arg $arg0).ComTypeDesc == (ComTypeDesc)System.Scripting.Com.ComTypeDesc)
    ) ) {.scope (
        System.Scripting.Com.IDispatchObject dispatchObject
        System.IntPtr dispatchPointer
        System.Int32 dispId
        System.Runtime.InteropServices.ComTypes.DISPPARAMS dispParams
        System.Scripting.Com.Variant invokeResult
        System.Object returnValue
    ) {
        .scope (
        ) {
            {
                (.var dispId) = (ComMethodDesc)Object&  Workbooks().DispId
                (.var dispParams).cArgs = 0
                (.var dispParams).cNamedArgs = 0
                (.var dispatchObject) = (IDispatchComObject)(.arg $arg0).DispatchObject
                (.var dispatchPointer) = ((.var dispatchObject)).(IDispatchObject.GetDispatchPointerInCurrentApartment)(
)
                .try {
                    .scope (
                        System.Scripting.Com.ExcepInfo excepInfo
                        System.UInt32 argErr
                        System.Int32 hresult
                    ) {
                        {
                            (.var hresult) = (UnsafeMethods.IDispatchInvoke)(
                                (.var dispatchPointer),
                                (.var dispId),
                                (INVOKEKIND)INVOKE_FUNC, INVOKE_PROPERTYGET,
                                (.var dispParams),
                                (.var invokeResult),
                                (.var excepInfo),
                                (.var argErr),
                            )
                            (ComRuntimeHelpers.CheckThrowException)(
                                (.var hresult),
                                (.var excepInfo),
                                (.var argErr),
                                "Workbooks",
                            )
                            (.var returnValue) = ((.var invokeResult)).(Variant.ToObject)()
                        }
                    }
                } .finally {
                    {
                        ((.var dispatchObject)).(IDispatchObject.ReleaseDispatchPointer)(
                            (.var dispatchPointer),
                        )
                        ((.var invokeResult)).(Variant.Clear)()
                    }
                }
                .return (.var returnValue);
            }
        }
    }
    } .else {/*empty*/;
    }}

CLS compilation of IronPython

shrib — Thu, 24 Jul 2008 05:01:39 GMT

One of the common feature requests for IronPython is to support static compilation. While the feature looks like a no-brainer initially, it does have a few wrinkles when you look at the details. Here are the different forms that a related question might look like?

Can I call IronPython code from C#?
Can I statically compile my IronPython source code to a binary?
Can I define custom attributes in IronPython?

Terminology

This has several aspects related to compilation, with different folks caring about different aspects of the issue. To clarify the issues involved, here is the terminology I will use.

Compiling to an assembly on disk - This is useful if you do not want to ship your source code. This by itself does not imply if the IL it contains is CLS compliant or not. IronPython 1.X does support this feature, and IronPython 2 will be restoring the feature in Beta 4 (due shortly).
Generating CLS compliant IL - Common Language Specification (CLS) specifies the requirements on IL (and metadata) so that it can be seamlessly consumed from other statically compiled languages like C# and VB.Net. IronPython normally generates mangled IL to implement all the dynamic features of Python which cannot be represented with CLS compliant IL in general. However, users are willing to put up with some restrictions in order to get CLS compliant code (which can be either on disk or in-memory).
Calling IronPython code from C# - Many folks assumes this implies compilation of IronPython code, but this is not true. See the Alternatives section below.

CLS compilation usage scenarios

This blog is about the CLS compliant compilation aspect. While CLS compliance is a nice goal by itself, it is interesting to think of the different ways that the generated IL could be consumed.

Authoring .NET assemblies - The compiled assembly would be referenced from statically typed language like C# or VB.Net. This would be consumed using something like "csc /r:PythonWidgetAssembly.dll app.cs". The author of PythonWidgetAssembly.dll would probably want IronPython to support the full range of CLS features as she wants to use IronPython as a first-class languages completely on par with C# and VB.Net.
Reflection on the generated assembly - This is used by tools like NUnit which look for all types in a test assembly with a given custom attribute. Similarly, a host might look in a plugin assembly for a type inherting from some known interface.
Reflection on an object at runtime - Some .NET API might inspect its incoming argument object for a method with given name, or a property with a custom attribute. LINQ to SQL looks for custom attributes like TableAttribute on the class of the object being used in a LINQ query to determine how to map the LINQ query to a database operation. Normally, IL generanted by Python is mangled so that a Python class method does not really exist on instances of the Python class. Also, Python does not have a concept of custom attributes.

Issues

First two usage patterns, you need to explicitly compile your Python code to an assembly. This takes away one of the important qualities of Python which is the ability to just run your source code (as "ipy.exe foo.py") without an extra compile step. In general, you want the code to have the same semantics irrespective of whether you precompile it to as assembly on disk, or not.
Extensions to Python syntax - Expressing the full range of CLS features will need adding new syntax to IronPython. For example, Python has no syntax for accessibility of members (public, protected, private, internal, etc), method type like virtual or abstract, the type of arguments or the return type, custom attributes on assembly, types, methods, and arguments, specifying events, etc. See the description of CLS for the complete list. It could stay valid Python code by expressing the features as comments or interpreting existing Python elements in creative ways. But at some point, it will start becoming P# in appearance (ie. not quite Python). At that point, would you be better off writing those parts of your application in C# or VB.Net?
Extensions to Python semantics - All Python name resolution is dynamic. For eg, if you had "System.Console.WriteLine("foo")", then the name "System" is usually resolved at runtime. For the first two usage patterns, IronPython would need to attempt to bind this at compile-time which does bend the rules. In general, IronPython's support of .NET is done in a very Pythonic way. IronPython code using .NET code can in theory also run under CPython if the .NET libraries were implemented as Python libraries. There is an experimental project called Python-System which attempts to do this. This would be a tall task, but the point is that IronPython has not bent the core semantics of the language.

Alternatives

I do want to provide people with some possible alternative solutions that might work for their scenarios in the meantime

Writing part of the application in C# or VB.Net, and then extending it from IronPython - For example, you could define the SQL-related types in C# or VB.Net so that you could set the custom attributes from the System.Data.Linq namespace. (This would not work as IronPython does not support authoring LINQ queries yet, but it illustrates the point)
Using the DLR hosting APIs to call into IronPython code from C# or VB.Net
In the future, languages could possibly provide better support to call into IronPython as discussed in this blog.

Conclusion

Usage 3 above (Reflection on an object at runtime) seems like the main issue to me as the user is not trying to persist an assembly to disk (which is not Pythonic).

Also, some features might provide big bang for the buck (like custom attributes on types and members as required by LINQ to SQL), whereas supporting the full range of CLS features may not be warranted. I would be curious to hear of real world use-case of how people want to use "static compilation" to understand what the right design point is.

If you have run into a wall with trying to use IronPython because of one of the issues above, do leave a comment about your exact scenario, which libraries were involved, what your workaround was, etc.

Signals on Windows

shrib — Fri, 23 May 2008 23:01:00 GMT

While looking at signal support in IronRuby, I played with the signal and raise functions that are available in the C runtime. It turns out that these functions are much more limited than their Unix counterparts. On Unix, kill can be used to send a signal to another process, enabling cross-process communication.

However, the CRT raise function does not take a process id. It can only send a synchronous signal on the same thread to the process it is being called from. This does not seem to be of much use. SIGINT (Ctrl-C) can also be dealt with using SetConsoleCtrlHandler in unmanaged code, or Console.CancelKeyPress in managed code, and the Windows runs the handler on a separate thread. SIGILL, SIGSEGV, and SIGTERM are not generated on Windows NT. So that only leaves SIGABRT and SIGFPE. Not sure how important these are.

If asynchronous signals were truly supported on existing threads, this would present challenges to using managed code to handle the signals. The thread may not be in an interruptible state when the handler is run. If the handler triggers a GC or stack-walk, that would cause trouble. Also, running managed code can allocate memory since the code may not have been JITed. Doing such heavy-weight operations in the handler would also lead to trouble. Python ("they can only occur between the ``atomic'' instructions of the Python interpreter") and Perl ("to avoid these problems signals are deferred") seem to defer running of the signal handler from a safe point, or run it on a new thread like SIGINT on Windows.

I am attaching a test case to use signals from managed code if anyone wants to play with it. All handlers are run synchronously. Ctrl-C is run asynchronously but on a new thread.

Accessing IronPython objects from native JScript (using IReflect)

shrib — Tue, 13 May 2008 21:29:00 GMT

The DLR aims to enable dynamic languages like IronPython and IronRuby to access and minipulate objects created by each other. But what about dynamic languages like JScript implemented using unmanaged code? It is sometimes useful for IronPython and native JScript to interact with each other. Consider the following scenarios:

IronPython could be used to host a System.Windows.Forms.WebBrowser control which loads a Web page with native JScript. See this MSDN article for a description of the WebBrowser control. Srivatsn discusses some of the issues related to using it from IronPython here.
Native Web browser JScript could talk with IronPython running in a Silverlight 2 application embedded in the Web page. (Note that unfortunately, this scenario will not currently work. See "Issues and Limitations" #1 below.)
VBA macros running inside Office could load and run IronPython scripts.

IronPython is now able to access native JScript objects because of the IDispatch support we have added to the DLR.

For the reverse direction of native JScript accessing IronPython objects, the IronPython objects need to be marshalled to unmanaged code as IDispatch (and ideally also IDispatchEx). I had previously discussed how managed types can implement System.Reflection.IReflect to be marshalled as IDispatch. Reading Srivatsn's blog, I decided to see how I could build on this IReflect <-> IDispatch mapping. This turns out to be fairly straightforward. I have implemented a Python modules called IReflectUtils. This allows an IronPython class to trivally indicate that it wants to make itself accessible to native JScript. The IronPython class declaration looks like the following:

from IReflectUtils import IReflectBase
class TestClass(IReflectBase):
def foo(self):
print "In TestClass.foo"

All it takes is inheriting from a single class IReflectBase. No other change is needed. At that point, the Python class becomes accessible from native JScript. JScript can invoke any of the methods of the Python class, including dynamically added methods. JScript can also read and write any of the attributes defined on the object. Tada!

Note that because of a bug in IronPython, IReflectBase needs to be the first base type specified. So the class declaration in Srivatsn's blog would become:

class MyForm(IReflectBase, Form):

How it works

Under the hoods, IReflectBase implements IReflect. It needs to implement the following two main functions:

IReflect.GetMethods - this is implemented by calling "dir" on the object and returning all the attributes as method names. The function needs to return the method names as an array of MethodInfo, and so we have to create a subtype of MethodInfo which basically just knows the name it corresponds to.
IReflect.InvokeMember - this turns into a "getattr(target, name)", possibly followed by a call to the returned object. The invokeAttr parameter that is passed into the method indicates whether the Python attribute should just be returned (if BindingFlags.GetProperty is set), or if it should be invoked (if BindingFlags.InvokeMethod is set)

Issues and limitations

This solution will not work on Silverlight 2 which currently does not include any COM Support. Hence, it does not support the IReflect <-> IDispatch mapping. It would be possible to port this functionality from .NET Frameworks to a future version of Silverlight.
This solution will only work with IE, since only Microsoft's JScript implementation is based on IDispatch. Firefox's implementation uses a similar but different interface that is part of XPCOM. I do not what the XPCOM interface looks like, but if it is similar to IDispatch, it should be possible to map IReflect to it as well.
Python classes with __getattribute__ will not work. The CLR calls IReflect.GetMethodNames to get a full list of methods supported by the object. This is different than IDispatch.GetIDsOfNames which only supports querying whether a single given method is supported or not (which is similar to how __getattribute__ works). Because of this, such Python classes will not be able to expose all the method names that __getattribute__ recognizes. IReflect does have a GetMethod method that could have been used to query whether a specific method existed, but the IReflect <-> IDispatch mapping does not use this. A theoretical solution is for IReflect.GetMethodNames to return the complete infinite set of all character combinations, but this does not really work. Ruby's method_missing will run into the same issue. This will need a creative fix to the IReflect <-> IDispatch mapping, or for a new interface to be added to the CLR which more closely follows the IDispatch model.
This feature could be shipped in different ways. The attached sample show a simple Python module. This could even be used with IronPython 1.X. However, it requires that all Python classes explicitly opt in, which may not work well for large existing code bases. Also, builtin types with dynamic behavior implemented by IronPython.dll will not work as the user cannot make them implement IReflect. The feature could be built into the IronPython runtime so that all IronPython user-defined classes would implicitly inherit from IReflect, and any builtin types could be made to implement IReflect.
The current solution relies on inheriting from IReflect (via IReflectBase). This can lead to name collisions if the user defines method with names which are the same as some significant IReflect method.
VBScript does not require the use of paranthesis while calling a method with no arguments. This makes ambiguous the syntax for an instance method call and a property access. This can cause some issues since the IReflect.InvokeMember needs to know whether a property should be accessed or if the attribute should be invoked. Similarly, VBScript and IDispatch are case-insensitive whereas Python is case-sensitive. The IReflect <-> IDispatch mapping seems to do a case-insensitive search and randomly picking an attribute if there are multiple matches.
We could support richer interaction between managed and unmanaged dynamic languages by taking advantage of the System.Runtime.InteropServices.Expando.IExpando <-> IDispatchEx mapping that the CLR supports. This would allow addition and deletion of attributes. This is currently left as an exercise for the user. Also, note that this feature has bugs in it, and the support may not be sufficient for all the scenarios you would care about.

About the attached file

IReflectUtils.zip contains the following files:

IReflectUtils.py - The main Python module file. This file can be used in your own IronPython projects.
TestClass.py - A sample Python class that inherits from IReflectUtils.py
ManagedFactory.cs - A bridge between test.js/test.vbs and TestClass.py. It includes a class that is ComVisible and hence can be instantiated by test.js and test.vbs. The class loads TestClass.py and passes the IronPython object back to test.js and test.vbs. See the top of the file for instructions on compiling and registering the file.
test.js and test.vbs - The driver files which instantiate ManagedFactory, use it to get an IronPython object, and then access the object. See the top of the file for instructions on running the files.

OLE automation (IDispatch) support in IronPython

shrib — Thu, 10 Apr 2008 21:57:52 GMT

IronPython 1.X

IronPython 1.X supported COM interop by building on top of the COM interop support built into the CLR. This relies on the use of interop assemblies for managed code to access COM objects. The interop assembly can be accessed in different ways:

clr.AddReference("interopAssembly") - This is the preferred mechanism so that the interop assembly does not have to be created multiple times. The down-side is that it complicates deployment. You have to ensure that the interop assembly exists on the machine.
Having IronPython automatically generate it on the fly - if you have a OLE automation object, IronPython tries to find its type library (TLB) and then to convert the TLB to an interop assembly on the fly using the TypeLibConverter class. The advantage is that this works seamlessly as long as there is a type library on the machine. The down-side is that this conversion can take a long time for large TLB files.

This is how Word could be accessed in IronPython 1.X. It requires the interop assembly Microsoft.Office.Interop.Word.dll to be available on the machine.

c:\>ipy.exe
IronPython 2.0 Beta (2.0.0.1000) on .NET 2.0.50727.1433
Copyright (c) Microsoft Corporation. All rights reserved.
>>> import clr
>>> clr.AddReference("Microsoft.Office.Interop.Word") # load interop assembly
>>> from Microsoft.Office.Interop.Word import ApplicationClass
>>> word = ApplicationClass() # load class defined in interop assembly
>>> word.Visible = True # this works via CLR's COM interop support
>>> doc = word.Documents.Add()
>>> doc.Range().Text = "Hello from IronPython"
>>> word.Quit()
>>>

IronPython 2.0

IronPython 2.0 has added support for interacting with OLE automation objects using the IDispatch interface. This completely eliminates the need for interop assemblies. This is the same mechanism that VBA late-bound code and also WSH use to interact with OLE automation object. This feature can currently be accessed using the -X:PreferComDispatch command line option.

Here is an interactive session showing access to Word. There are no interop assemblies in the picture.

c:\>ipy.exe -X:PreferComDispatch
IronPython 2.0 Beta (2.0.0.1000) on .NET 2.0.50727.1433
Copyright (c) Microsoft Corporation. All rights reserved.
>>> import clr
>>> import System
>>> wordTypeLibGuid = System.Guid("00020905-0000-0000-C000-000000000046")
>>> clr.AddReferenceToTypeLibrary(wordTypeLibGuid)
>>> from Word import Application
>>> word = Application() # this uses Type.GetTypeFromCLSID and Activator.CreateInstance under the hood
>>> word.Visible = True # this uses IDispatch.Invoke
>>> doc = word.Documents.Add()
>>> doc.Range().Text = "Hello from IronPython"
>>> word.Quit()
>>>

The feature is not complete yet, and so is not currently (April 08) enabled by default. The default behavior is still the same as IronPython 1.X. I can't say for sure when we will enable it by default, but its already quite stable for many real-world uses. We run all of our COM tests in both modes - using interop assemblies and using IDispatch. There are a few differences between the two cases in the minute details, but the mainline scenarios work well in both. This feature is implemented in the DLR, and so should be accessible from all DLR-based languages with little effort on the part of the language-implementors.

What if a C# snippet does not work when transliterated to IronPython?

shrib — Mon, 24 Mar 2008 21:54:49 GMT

Say you have a working piece of C# code. You want to include it in your IronPython app. So you transliterate it to IronPython code. And it stops working. What do you do?

You might feel lost at this point since the IronPython code is "supposed" to work. You have the following options:

The first step ofcourse is to double-check your transliteration.

The next step is simple but sometimes not obvious when you are in the middle of many other ongoing issues. This solution is to convert the C# code a function at a time to IronPython, or even a line at a time if needed. C# and IronPython can call each other very easily. So its easy to move small pieces of C# code over to IronPython. At every step, you check that the combined code works. You can use a binary search by speculatively moving bug chunks of code from C# to IronPython, and backing off when such a move exposes a problem. You will hopefully be able to narrow the problem down to a single line of IronPython code, which you can leave in C# until you figure out why IronPython cannot deal with it.

This isnt a perfect solution. Some problems can be non-deterministic, and the problem might repro at random times. However, in many cases, you will be able to narrow down the problem to a single line of IronPython code.

You could manually write C# code that uses Reflection.Emit to do the same work as the original C# code (once you have narrowed down the issue to a few line of code). IronPython code also gets compiled to Reflection.Emit code on the fly. Any issues that affect Reflection.Emit will be surfaced by your C# + Ref.Emit code.

IronPython cannot call AutomationElement.FromHandle

shrib — Mon, 24 Mar 2008 06:10:20 GMT

If you use IronPython to call AutomationElement.FromHandle, it will return null. This happens because FromHandle uses Assembly.GetCallingAssembly and expects to get a statically compiled assembly with references to the correct version of the UIAutomation libraries. However, since IronPython code gets compiled on the fly using Reflection.Emit and DynamicMethods, it confuses AutomationElement.

This UIAutomation team knows about the issue and will be fixing it. I am not sure which particular release will have the fix. The workaround is to narrow down the specific call which is failing, and then perform that call from C#/VB.Net assembly.

I had added this comment to the Assembly.GetCallingAssembly docs. I am including it here to explain the details of the problem

The common case is for the caller to have directly called currently executing method. However, the caller may have invoked the currently executing method via a delegate, or in a late-bound way using MethodBase.Invoke. The delegate or the MethodInfo might have been created by one assembly, and then another assembly could have invoked it. GetCallingAssembly will not indicate who created the delegate or the MethodInfo. So if you use GetCallingAssembly to make security decisions, you will need to be extremely careful to ensure that you handle the cases of late-bound invocation.
Another unusual case is if the caller is a DynamicMethod. Since DynamicMethods can potentially be associated with arbitrary assemblies during creating, GetCallingAssembly will return the associated assembly.

Saving a file to the local disk in Silverlight

shrib — Tue, 25 Sep 2007 04:10:00 GMT

All user code in Silverlight runs in a sandbox. Hence, for security reasons, there are no APIs to directly open local files from disk. However, there is a OpenFileDialog class which allows a Silverlight app to open files on disk. Human intervention is required to interact with the dialog box. This ensures that the user trusts the app to open the files selected in the dialog box.

However, currently there is no equivalent to writing to a file to disk, even if the user wants to allow such an operation. This is unfortunate. If you implemented a text or image editor in Silverlight, the user would not be able to save the edited file to disk. However, there is a workaround - but at the cost of extra network traffic. The solution involves the following steps:

The Silverlight client app running in the browser sends the data that needs to be saved to disk to the web server using a POST operation using System.Windows.Browser.Net.BrowerHttpWebRequest::GetResponse
The web server saves the data to a temporary file and returns a URL for the temporary file in the response
The Silverlight client app reads the URL that is returned in the System.Net.HttpWebResponse object.
It navigates to the URL using System.Windows.Browser.HtmlPage::Navigate.
This will result in the browser invoking the default action for the given file type. This is often to prompt the user to save the file to disk. If the browser opens the file directly, the user could then have the browser save the file to disk. If multiple files need to be saved, the web server could save them all in a zip file using System.IO.Compression.GZipStream.

This solution is obviously not ideal as it result in unnecessarily sending the data back and forth over the network. For large files, this could be prohibitive. For small files, its a reasonable solution until there is a SaveFileDialog API in Silverlight.

I do not know if there will be a SaveFileDialog in Silverlight 1.1, but I hear that the Silverlight team is looking into it...

Late-binding and ByRef parameters

shrib — Mon, 10 Sep 2007 06:53:00 GMT

VB, VBA, and VBScript pass arguments by reference by default. A subroutine definition can explicitly override the default by specifying that a parameter should be passed by value. This does raise two issues in the face of late-binding:

How are arbitrary expressions passed by reference? The answer is that expressions which cannot be the target of assignments, for eg, x + 1, get evaluated into a temporary, and the temp variable then gets passed byref to the subroutine. So any modifications to the temp variable are not noticeable to the user. See the sample below.
How does a call-site determine if the target subroutine expects the argument as byref or byval. The answer is that the callsite cannot know. It always passes the arguments as byref. The target function then takes care of reading the value out of the byref parameter and storing it in a temporary variable. Any updates made by the body of the target function only affect the temp variable.

Sub Foo(param)
    param = "Foo"
End Sub

Sub AssertValue(actualValue, expectedValue)
    If actualValue <> expectedValue Then
        MsgBox "Error! Expected " & expectedValue & " but got " & actualValue
    End If
End Sub

Dim x

x = 1
Foo x
AssertValue x, "Foo"

x = 1
Foo (x)
AssertValue x, 1

x = 1
Foo x+1
AssertValue x, 1

Oleview hangs on Vista when run from elevated cmd prompt

shrib — Mon, 10 Sep 2007 06:33:00 GMT

Oleview has been hanging and taking up all the CPU when I tried to open a typelib on a Vista machine. I saw http://support.microsoft.com/kb/176938 and tried to follow the recommended solution (regsvr32 iviewers.dll), but that did not fix it either. It turns out the solution is to run oleview from a non-elevated cmd prompt. For example, you can just run it from the Run dialog box. I was always running it from an elevated cmd prompt since that's what I am have open for during development.

You wouldn't think that something that works in non-elevated mode would break in elevated mode, but apparantly it happens because of some registry shimming...

Using IReflect to expose a type as IDispatch to COM

shrib — Tue, 04 Sep 2007 23:40:00 GMT

The CLR supports marshalling of objects that support the IReflect interface as IDispatch COM objects. Similarly, IExpando gets marshalled as IDispatchEx. Here is a sample of a managed type called ManagedIDispatch which is used from VBScript and used in a late-bound way. VBScript just deals with IDispatch, and under the hoods, the CLR routes the calls to the IReflect methods implemented by ManagedIDispatch, and also takes care of providing a stable mapping between method names and dispids.

IReflect.GetMethods gets called every time COM instantiates the managed type. The CLR caches the method names for the given object, and the names are then available to COM from the IDispatch interface. If COM accesses a method name that is not in the cached list, IReflect.GetMethods get called again.

The C# code should be compiled and registered as:

csc /t:library ManagedIDispatch.cs

regasm /codebase ManagedIDispatch.dll

Here is the significant contents of ManagedIDispatch.cs

public class ManagedIDispatch : IReflect {

...

public MethodInfo[] GetMethods(BindingFlags bindingAttr) {

Console.WriteLine("In ManagedIDispatch.GetMethods (_getMethodsDone={0})", _getMethodsDone);

// dynFoo is a method that is not always present

if (_getMethodsDone) {

MethodInfo[] methods = new MethodInfo[_methodInfos.Length + 1];

_methodInfos.CopyTo(methods, 0);

methods[_methodInfos.Length] = new MyMethodInfo("dynFoo");

return methods;

} else {

_getMethodsDone = true;

return _methodInfos;

}

public object InvokeMember(string name, BindingFlags invokeAttr, Binder binder, object target, object[] args, ParameterModifier[] modifiers, CultureInfo culture, string[] namedParameters) {

Console.WriteLine("In ManagedIDispatch.InvokeMember({0} : {1})", name, GetArgs(target, args));

switch (name) {

case "[DISPID=0]": return this;

case "raiseException": throw new Exception("raiseException has been invoked");

case "foo": return "foo-return";

case "dynFoo": Debug.Assert(_getMethodsDone); return "dynFoo-return";

default:

if (name == ("bar" + _index)) return "bar-return";

throw Utils.NotImplemented("This is unreachable");

};

}

Here is the VBScript used to script the type:

Sub AssertError(errNum, errDescr)
    If Err.Number <> errNum Then
        MsgBox "Error number is incorrect:" & Err.Number & ", expected " & errNum
    End If

    If Err.Description <> errDescr Then
        MsgBox "Error description is incorrect:" & Err.Description & ", expected " & errDescr
    End If

Err.Clear
End Sub

c1 = CreateObject("TestFactory")
m = c1.GetManagedObject()
m.foo()
m.dynFoo()
m.bar0()

On Error Resume Next

m.nonexistent()
AssertError 438, "Object doesn't support this property or method"

m.raiseException()
AssertError -2146233088, "" ' 0x80131500 : CLR exception

The output of the script is:

c:\bugs>cscript vbs1.vbs
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.

In ManagedIDispatch.GetMethods (_getMethodsDone=False)
In MyMethodInfo.Name.get(foo)
In MyMethodInfo.Name.get(bar0)
In MyMethodInfo.Name.get(raiseException)
In ManagedIDispatch.InvokeMember([DISPID=0] : ManagedIDispatch)
In ManagedIDispatch.InvokeMember(foo : ManagedIDispatch)
In ManagedIDispatch.GetMethods (_getMethodsDone=True)
In MyMethodInfo.Name.get(dynFoo)
In ManagedIDispatch.InvokeMember(dynFoo : ManagedIDispatch)
In ManagedIDispatch.InvokeMember(bar0 : ManagedIDispatch)
In ManagedIDispatch.GetMethods (_getMethodsDone=True)
In MyMethodInfo.Name.get(dynFoo)
In ManagedIDispatch.InvokeMember(raiseException : ManagedIDispatch)

Tail call performance on x86

shrib — Tue, 25 Jan 2005 20:58:00 GMT

MSIL includes the tail. prefix to be used with call instructions (call, callvirt, calli), and also the specialized form jmp. The tail. prefix is a hint that the stack frame of the current method should be popped from the stack before transferring control to the target of the call. This is important for functional languages like Scheme which rely on recursion. Without tail calls, they would run into StackOverflows on a routine basis.

The CLR implementation of tail calls on x86 is not the most optimal it could be. We have improved the performance of tail calls in Whidbey (V2). Most tail calls are about 50% faster. However, there is still some work to be done. This blog explains why implementing tail calls is not as straight-forward as it might seem.

The CLR has an accurate tracing Garbage collector - as opposed to a conservative one like the Boehm collector. This puts some requirements on the native code compiled from the IL, and also on the tracking information corresponding to it (similar to pdata on RISC platforms). One of the requirements is that the CLR be able to determine exactly where the return address of the method is on the stack. This is required so that the CLR can “hijack” the return address by temporarily overwriting it with the address of one of the hijacking function in the CLR. This is done so the CLR will get control when the method returns, and the CLR can perform a garbage-collection.

This requirement makes it difficult to move the return address around. Moving the return address around is needed while doing a tail call to a method with a different number of stack-based arguments as the return address gets pushed after the stack arguments. The following diagram shows how the stack looks at various points when a method foo is called normally, and it does a tail call to bar which expects a different number of arguments on the stack. Note that the return address needs to be moved by foo before it does the tail call.

As soon as foo While foo When foo is about to
is called is executing tail call to bar

Furthermore, we tend to be really careful when changing the hijacking code as any bugs introduced in this code are not easily caught, and instead can become GC holes (corrupt GC references). Most programs will seem to work fine, but if a GC is triggered at just the right spot, some poor object reference will get corrupted.

Another (secondary) concern is GC starvation. If two methods recursively tail call to each other, it causes an infinite loop and the hijacking mechanism does not get a chance to kick in. This reduces the opportunity for GC to suspend the thread and do a collection. This could cause other threads to block longer while they wait for GC to free up some heap memory.

We do know how to make this even faster, and would have loved to have fixed this. However, because of schedule pressure and the risk associated, further improvements will have to wait until after Whidbey (V2.0).

FWIW, the jmp IL instruction is faster since the return address does not have to be shuffled in this case. Also, tail calls will be much faster on 64-bit platforms as the arguments for most methods fit in registers (and so the return address does not need to be moved around). In fact, on 64-bit platforms, the CLR tries to do tail calls even if the tail. prefix is not specified. This is because platform-independent IL programs will get the same amount of stack space reserved for them by the OS, even though the 64-bit process will consume stack at a faster rate. Hence, we try to reduce stack usage by doing tail calls.