Srivatsn's Blog

Visual Basic comes to Windows Phone!

Srivatsn N — Thu, 23 Sep 2010 17:16:52 GMT

Ah.. It’s such a relief to be able to finally talk about it I’ve been working on getting Visual Basic supported in the Windows Phone development tools. It’s been hard to not talk about it, especially when I’d seen comments from users like “The phone dev tools are great but it’s sad that there is no VB support”.. You can read about the announcement here on Soma’s blog. Jason Zander has a post showing a walkthrough of a basic VB windows phone application. The VB Team blog has a more detailed walkthrough. I’ll post about the details of the release later. Have fun developing VB apps for the phone!

Solution Navigator: Get rid of your toolwindows

Srivatsn N — Wed, 21 Jul 2010 00:37:39 GMT

Adrian Collier has just blogged about the release of the Solution Navigator extension for VS 2010. One of the things I hate about Visual Studio is the number of tool windows that I have to juggle around. I have two big monitors at work and I still run out of screen real estate every so often. I hate working from home because on my laptop screen, after all the toolwindows, there’s like seven pixels left for actual code. So, anything that promises to reduce the need for multiple toolwindows excites me. Solution navigator, in one fell swoop gets rid of a bunch of toolwindows that I use regularly. There is a solution navigator pane that takes the place of solution explorer and makes it much richer. The tooltips (pinnable!) are the killer feature for me though, hover over any artifact in your code and everything you’d want to know about that is present in a tooltip.

This works for both C# and VB projects. I also happened to write the initial support for VB projects before the Solution Navigator team took over ( It was a very fun week writing the basic plumbing and seeing everything light up!). Check it out and give the team your feedback.

Relax! so what if the return type is missing?

Srivatsn N — Wed, 14 Jul 2010 04:18:27 GMT

Visual Basic generally goes out of its way to help out the programmer. Delegate relaxation is a good example of this need to please. Simply put, this feature is about being able to assign to delegates when the signatures don’t exactly match. The canonical example goes like this:

Sub ClickHandler() Handles Button1.Click

Button1.Click takes two parameters whereas ClickHandler takes no parameters. This is quite a convenience if you don’t plan to use the eventargs.

Another example that’s easier to examine is something like:

Dim a As Action = AddressOf String.Empty.ToString

Action is a delegate that takes no parameters and has no return type. We are assigning a function to it which has a return type of String. Here is the code that’s generated for this case.

Dim $VB$Closure_ClosureVariable_14_D As New _Closure$__1
$VB$Closure_ClosureVariable_14_D.$VB$Local_VB$t_string$S1 = String.Empty
Dim a As Action = New Action(AddressOf $VB$Closure_ClosureVariable_14_D._Lambda$__1)

Public Sub _Lambda$__1()
    Me.$VB$Local_VB$t_string$S1.ToString
End Sub

What is all this messy code doing? What happens is that when the compiler sees a delegate of type <expr>.method it breaks it down into temp = <expr> and temp.Method. The temp var is then hoisted into a closure if needed. The closure class also has a method which has the right signature needed for the target delegate. This function simply calls temp.Method. If there was no relaxation needed (meaning no lambda is generated) then the temp is simply a temp and doesn’t have to be hoisted.

How I landed up in the Visual Basic team.

Srivatsn N — Sun, 11 Jul 2010 01:27:31 GMT

My last post was in September 2008. It’s been close to two years since I posted anything. My rationalization for such unforgivable behaviour is that I started working on a project that wasn’t public at that time. This was an incubation effort and I was part of a very small team of fantastic people that adopted agile practices and is easily one of the best teams for which I’ve worked. After that effort reached it’s conclusion, I, changed roles and moved to the Visual Basic .NET team as a developer and during the Dev10 cycle I owned intellisense, the expression editor and even code model for a while. Recently I’ve started working more with the compiler codebase and there is a quite a bit of blog fodder when you work in the compiler space. So I’m reviving this blog to write about all the corner cases of the language that I’m learning – or atleast that is the reason I give myself for doing something that I should just have been doing all along for the past two years.

Hosting IronPython made easier

Srivatsn N — Wed, 17 Sep 2008 01:30:34 GMT

With 2.0 Beta 5 coming out very soon, there is a new hosting helper class in there called IronPython.Hosting.Python . It has a few helper methods to create a ScriptRuntime or ScriptEngine and adds some Python-specific extension methods to ScriptRuntime and ScriptEngine.

Creating a ScriptRuntime the normal way now requires a LanguageSetup as Sesh explains here. The Python class has helpers called CreateRuntimeSetup and CreateLanguageSetup which return a pythonic ScriptRuntimeSetup and LanguageSetup respectively. But if you are not interested in the configuration options of the DLR then you can simply use Python.CreateEngine/ Python.CreateRuntime.

ScriptRuntime runtime = Python.CreateRuntime();
runtime.ExecuteFile("foo.py");

There are also three extension methods tacked on to ScriptRuntime and ScriptEngine called GetSysModule, GetBuiltinModule and GetClrModule which directly return a ScriptScope for these modules. You can do something like this for example:

using IronPython.Hosting;

ScriptEngine engine = Python.CreateEngine();

ScriptScope sys = engine.GetSysModule();
var platform = sys.GetVariable("platform");
Console.WriteLine(platform);

ScriptScope builtins = engine.GetBuiltinModule();
var pow = builtins.GetVariable<Func<double, double,double>>("pow");
Console.WriteLine(pow(2,3));

ScriptScope clr = engine.GetClrModule();
var getPythonType = clr.GetVariable<Func<Type, PythonType>>("GetPythonType");            
Console.WriteLine(PythonType.Get__name__(getPythonType(typeof(string))));

This will then print out cli, 8 and str respectively.

Passing keyword args to C# methods from IronPython

Srivatsn N — Wed, 10 Sep 2008 00:13:14 GMT

In Python one can call functions with either the argument itself or a named argument i.e f(3) or f(x=3). Python also provides syntax to absorb excess arguments for either style of argument passing with either * or **. So for example for this function f:

def f(x, *args, **kwargs):
    pass
    
f(3,4,5,y=6,z=7)

The value passed to x is 3 and args then absorbs the extra positional arguments and is passed as a tuple (4,5). kwargs then takes up the keyword arguments and is passed as a dictionary {'y':6, 'z':7}.

Variable number of arguments in the CLS

Now if 'f' were a C# method would this still work? Passing the positional and named parameters from IronPython definitely works. The question is can C# accept variable number of parameters? The CLS clearly states that the only calling convention supported is the standard managed calling convention and variable length argument lists are not allowed. So how does C# support the params keyword then? The C# compiler applies an attribute (System.ParamArray) and passes the extra positional parameters as an array. Great, so can we then do the same thing for dictionary params as well?

ParamDictionaryAttribute

There is a ParamDictionaryAttribute in Microsoft.Scripting.dll that serves the purpose of the System.ParamArray for keyword arguments. But since the C# compiler doesn't know about it there is no syntactic sugar here and it needs to be manually applied. We can, therefore, define a function like this:

using System;
using Microsoft.Scripting;

public class foo {
    public void bar(int x, [ParamDictionary] IAttributesCollection kwargs) {
        foreach (SymbolId key in kwargs.SymbolAttributes.Keys) {
            Console.WriteLine("key: {0}, value: {1} ", key, kwargs[key]);
        }
    }
}

The function bar now accepts a dictionary of excess keyword parameters when called from IronPython. IAttributesCollection is another interface defined in Microsoft.Scripting which represents a dictionary that can be accessed using SymbolIds and also arbitrary objects. The SymbolAttributes property returns just the SymbolIds. This can now be called from IronPython like so:

import clr
clr.AddReference("test.dll")
import foo
foo().bar(2,y=4, z=6)

And you'll see y:4 and z:6 were indeed passed into the method as items in the dictionary.

Static Compilation of IronPython scripts

Srivatsn N — Wed, 06 Aug 2008 09:28:01 GMT

The ability to compile IronPython scripts into .NET IL and to save them to disk existed in IronPython 1.0 but has been missing in 2.0 so far. With IronPython 2.0 Beta4 this has been added back.

Why would I compile dynamic language scripts?

There are a lot of reasons to compile scripts into a binary form. Shri talks about some of them here. For folks who don't want to distribute source code in plain text this provides one level of obfuscation. To that end, a function called CompileModules has been added to the clr module to compile scripts into executable IL. The signature of the function is:

CompileModules(str assemblyName, dict kwArgs, Array[str] filenames)

So to compile a file foo.py into foo.dll you would do this:

import clr
clr.CompileModules("foo.dll", "foo.py")

This can now be brought in using the regular clr.AddReference. When clr.AddReference sees a compiled assembly, it publishes the module as well. So one can simply import the module into the code.

clr.AddReference("foo.dll")
import foo

Multiple files and main

The function can take multiple python files and compile them into one dll. What if you want it to be a standalone executable? There are two things to be done. First, a stub exe is needed that can load the dll. Second, a way to distinguish the main module is needed. The keyword args that CompileModules can take comes in handy here

import clr
clr.CompileModules("foo.dll", "foo.py", "bar.py", mainModule="main.py")

Now a stub exe can be written that loads up this compiled dll. The IronPython sample pyc.py has code that does shows how to generate a stub exe.

Wait, what is -X:SaveAssemblies mode then?

When IronPython is started with -X:SaveAssemblies, it generates a dll containing IL corresponding to the code it executed. Sounds an awful lot like compilation doesn't it? The difference is one is executable IL and the other isn't.

To understand the difference, one needs to understand that IronPython under normal course of its execution generates IL anyway. Every statement is converted to the DLR AST and IL gets spit out for the ASTs which is then executed. The SaveAssemblies mode simply dumps the generated IL into a dll. It is meant as a debugging device. So what is missing from this IL that prevents it from being re-executable code? The short answer is Dynamic Sites. The sites that are generated during the execution are not persisted. The compilation feature does exactly this - it persists the dynamic sites as well. Lets look at an example here and compare the generated IL in reflector. (Only the relevant code is copied over from reflector). This python code:

print 2 + 5
print 2 * 5
print 3 / 5

when run with -X:SaveAssemblies mode produces this code:

public static CallSite<DynamicSiteTarget<int, int, object>> #Constant207;
public static CallSite<DynamicSiteTarget<int, int, object>> #Constant208;
public static CallSite<DynamicSiteTarget<int, int, object>> #Constant209;

$lineNo = 1;
PythonOps.Print(__global_context, #Constant207.Target(#Constant207, 2, 5));
$lineNo = 2;
PythonOps.Print(__global_context, #Constant208.Target(#Constant208, 2, 5));
$lineNo = 3;
PythonOps.Print(__global_context, #Constant209.Target(#Constant209, 3, 5));

Notice that the Constants defined here are actually defined as fields on the generated type and this type doesn't get instantiated anywhere and therefore the sites don't get assigned anywhere. The same python code when compiled with clr.CompiledModules produces this code:

object[] objArray = new object[] { 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Add")), 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Multiply")), 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Divide")) 
};

line = 1;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[0]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[0], 2, 5));
line = 2;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[1]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[1], 2, 5));
line = 3;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[2]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[2], 3, 5));

You can see that all the dynamic call sites are being created here and their targets are being invoked. This then is perfectly executable code - maybe not as succinct as the python code but it does the same thing :)

IronPython 2.0 Beta 3 is out!

Srivatsn N — Sat, 14 Jun 2008 01:58:07 GMT

The next stop on the path to IronPython 2.0 is Beta 3 and we pushed it out today. You can get the sources and binaries of IronPython here. As Jimmy mentioned in his release notes for Dynamic Silverlight last week, this release also contains the sources and binaries of IronPython and Dynamic Silverlight needed to build Silverlight applications with IronPython. This release builds against Silverlight 2.0 Beta 2 – so make sure that is installed.

Head over to the release notes to find out what has changed since the last release and the list of bugs that have been fixed. Congratulations to the team and the community on the release.

Accessing IronPython objects from native Javascript

Srivatsn N — Fri, 09 May 2008 10:16:52 GMT

Today I got a question by e-mail from someone who wanted to implement something like this MSDN article in IronPython. The gist of the article is that there is a property called ObjectForScripting on a WinForms WebBrowser control. If one sets the property to an object, then that object will be available to javascript that executes in the page through window.external. The Form needs to add the ComVisible attribute.

.NET attributes cannot be added to a class from IronPython but fortunately objects created are ComVisible by default. So we can try a direct translation of the code in the article :

import clr
clr.AddReference("System.Windows.Forms")
from System.Windows.Forms import Application, Form, WebBrowser, MessageBox

class MyForm(Form):
    def __init__(self):
        self.wb = WebBrowser()
        self.Controls.Add(self.wb)
        self.wb.ObjectForScripting = self
        self.wb.DocumentText = """
                               <html>
                                <body>
                                 <button onclick="window.external.Test('hello')">
                                   call client code from script code
                                 </button>
                                </body>
                               </html>
                               """
    def Test(self, msg):
        MessageBox.Show(msg)
    
        
Application.Run(MyForm())

Unfortunately this doesn't work because ObjectForScripting is being set to an instance of the python type MyForm. In the CLR-world this type will be generated dynamically and will be something like IronPython.NewTypes.System.Windows.Form_1$0 and will not contain any of the python methods. You can verify this by doing clr.GetClrType(MyForm).GetMembers(). The easiest solution then is to define an interface in C# with the methods that you want on your type and implement that interface on your type. So define a simple interface like this:

public interface IWebBrowserInterop
{
    void Test(string message);
}

and in the python code make your class implement that interface:

from ClassLibrary1 import IWebBrowserInterop

class MyForm(Form, IWebBrowserInterop):
...

Now if you fire up the app and click on the button the MessageBox will popup. You just reached across from unmanaged javascript through COM/.NET to python

Caveat(s): This is not type-safe. The IronPython runtime caches the types that are generated. So if there is another class in the python code that is "similar" (same base classes/interfaces and not different __slots__-wise) then this type can be reused. So if you are not bound by other factors, you should try to use Managed JScript instead and the hosting APIs to interact with the python objects.

Turning your .NET object models dynamic for IronPython

Srivatsn N — Sat, 12 Apr 2008 05:16:00 GMT

Say you want to interop with a .NET library but you also want it to behave like objects in dynamic languages do. You want to be able to add/delete methods/properties to the object dynamically. In python you can do something like this:

class x(object):
    pass

y = x()
y.z = 42
dir(y)

And the dir(y) will contain z in it. Now if x were a .NET class instead of a python class would you still be able to do the same? Let’s try a simple .NET class

public class TestExt
{
}

From IronPython you would do something like this:

import clr
clr.AddReference("TestExtensions.dll")
from TestExtensions import TestExt
y = TestExt()
y.z = 42

But this code throws a AttributeError saying ‘TestExt’ object has no attribute ‘z’. What now? Enter Stage Right DLR’s extension mechanism. There are five methods that a .NET class can implement that have a special meaning to the binder if you tell it so. The methods are:

GetCustomMember – runs before normal .NET lookups
GetBoundMember – runs after normal .NET lookups
SetMember – runs before normal .NET member assignment
SetMemberAfter – runs after normal .NET member assignment
DeleteMember – runs before normal .NET operator access (there’s no .NET version – so its the only one)

A .NET class can implement these functions and mark them with a SpecialName attribute. The rule generated for binding now makes a call to Getter/Setter before and after the normal .NET binding is done. GetCustomMember/SetMember is called first and if it returns a value that is treated as the result of the member-lookup. This overrides any .NET members that may exist. But if you return OperationFailed.Value from the function, then it will proceed with the normal lookup as well. GetBoundMember/SetMemberAfter is called if the normal binding fails – that is there is no member of the name that it is trying to bind to. So with that in mind let’s modify the .NET class and add this:

Dictionary<string, object> dict = new Dictionary<string, object>();
[SpecialName]
public object GetBoundMember(string name)
{
    if (dict.ContainsKey(name))
        return dict[name];
    else
        return OperationFailed.Value;
}

[SpecialName]
public void SetMemberAfter(string methodName, object o)
{
    dict.Add(methodName, o);
}

Now if I try to do a y.z = 42, it works. I can assign y.z to a function as well and will be able to call y.z(). I could have overridden GetCustomMember and SetMember instead and the behaviour would be exactly the same since OperationFailed.Value is returned for member that are not in the dict but then there is this overhead involved for any .NET member lookup.

The SetMember function can choose to return a bool instead of void and in that case, the returnvalue would control whether further binding lookups happen or not.

So, what exactly is the use of all this, why would I want to do this? Consider writing an object model for something like the following xml file:

<foo>
<bar>baz</bar>
</foo>

Now I want to access this xml as foo.bar and the value of that should be baz. To implement this all that I would need to do is add the GetBoundMember method to the .NET XmlElement class to do a search and return another XmlElement or add a extension method to XmlElement. Now here is the part where things come a little unstuck in IronPython. Extension methods dont show up in reflection, so the IronPython support for that currently isn’t where we want it to be. There is a way around though. You can decorate an assembly with an ExtensionType attribute describing what type you are extending and with what class. After that you would need to register the assembly once so that those methods get injected at the right places. This might change in the future with a better way but this works for now. This is the code you’d need to implement:

[assembly: ExtensionType(typeof(System.Xml.XmlElement), typeof(TestExtensions.ExtClass.XmlElementExtension))]
namespace TestExtensions
{    
    public class ExtClass
    {
        static ExtClass()
        {
            Microsoft.Scripting.Runtime.RuntimeHelpers.RegisterAssembly(typeof(ExtClass).Assembly);
        }

        public static XmlElement Load(string fileName)
        {
            XmlDocument doc = new XmlDocument();
            doc.Load(fileName);
            return doc.DocumentElement;
        }
        public static class XmlElementExtension
        {
            [SpecialName]
            public static object GetCustomMember(object myObj, string name)
            {
                XmlElement xml = myObj as XmlElement;

                if (xml != null)
                {
                    for (XmlNode n = xml.FirstChild; n != null; n = n.NextSibling)
                    {
                        if (n is XmlElement && string.CompareOrdinal(n.Name, name) == 0)
                        {
                            if (n.HasChildNodes && n.FirstChild == n.LastChild && n.FirstChild is XmlText)
                            {
                                return n.InnerText;
                            }
                            else
                            {
                                return n;
                            }

                        }
                    }
                }
                return OperationFailed.Value;
            }
        }
    }
}

Now from IronPython do this:

import clr
clr.AddReference("TestExtensions.dll")
from TestExtensions import ExtClass
foo = ExtClass.Load("test.xml")
print foo.bar

This prints out “baz”. Sweet!