DLR Hosting and related stuff...

Codeplex.com\dlr is now setup to receive regular source updates

2008-12-05T02:42:00Z

In my previous blog post I had mentioned that we were in the process of setting up regular automatic source pushes to the codeplex DLR project. I am happy to announce that work is now complete and our DLR sources are now available here. Going forward, the sources from our internal main source control branch will be available on a regular basis at codeplex.

We haven't yet decided on the update frequency and should finalize this in a day or two. Currently, we are considering either weekly or daily source pushes.

DLR v0.9 beta released in CodePlex

2008-11-27T05:16:00Z

We are happy to announce the release of the DLR Project in Codeplex. This beta of the v0.9 of the DLR consists of a source and binary release. This excellent release note by our PM Bill Chiles gives a nice short overview of what's in this release. We plan to release the RTW version of the 0.9 shortly in sync with IronPython v2.0.

Going forward, we will update this project regularly with binaries, weekly source drops and related documents.

Please try this release and let us know your feedback. You can contact me directly through comments or email or send an email to dlr@microsoft.com with any questions related to the DLR.

P.S : The codeplex.com site also lets you subscribe and follow the discussions on this project.

Using LINQ and the DLR in the same application

2008-09-20T05:36:00Z

The recent beta release of the DLR allows the usage of LINQ and other CSharp 3.0 features in an application that hosts the DLR. Harry, our IronPython PM has a detailed post explaining why this was not possible before and whats changed now.

DLR Hosting API : How to create a ScriptRuntime object without using App.Config files using language specific hosting API

2008-09-15T02:39:00Z

This previous post demonstrates the use of App.Config files to create script runtimes to host dynamic languages in an application. It is also possible to create runtimes without using a config file.

The API to create a runtime and a python engine without using a config file is

ScriptRuntime runtime = IronPython.Hosting.Python.CreateRuntime();

A reference to the IronPython assemblies – IronPython.dll and IronPython.Modules.dll are needed for this call to work. The previous sample already has the proper references in place and this call would automatically work in that project.

DLR Hosting Sample : Simple dlr host using the new App.Config based ScriptRuntime creation

2008-09-11T20:58:00Z

Due to the recent changes to the DLR Hosting API, dlr hosts now have to use the App.Config file to configure the languages they intend to use in the application. It’s probably a little bit more complex than before to get started on Hosting the DLR after this change.

The attached self contained sample(for VS 2008) addresses this and tries to make it easier to ‘get started’. The sample is a very simple console application that -

1) Hosts the dlr

2) Uses the ScriptRuntimeSetup.ReadConfiguration() and the ScriptRuntime constructor to create the runtime based on the entries in the App.Config file

3) Creates 2 language engines – one for IronPython and 1 for IronRuby

4) Creates a variable in IronPython and prints the value in C#.

The sample contains a pre configured App.Config file. You should be able to just extract the files locally and build the sample and execute it. The dlr and language binaries are also included (in the ‘lib’ folder). Note that these post IronPython V2 beta4 binaries are for demo purposes only and come with no warranty or support.

DLR Hosting API : latest version of the spec is available online (includes changes to runtime initialization)

2008-09-08T00:32:00Z

The latest version of the Hosting API spec is available here. The most recent source code drop to the IronPython codeplex site conforms to this spec (except for some open bugs).

The biggest change this spec introduces to the beta4 binaries is the way the hosting runtime is initialized.

Prior to this update, the hosting runtime is initialized by creating a new ScriptRuntime object. Now, a typical runtime initialization involves 2 things.

A quick 5 min how-to on the new initialization is as follows

1) Setting up entries in the application’s config file. Here is a sample config file

<?xml version="1.0" encoding="utf-8" ?>

<section name="microsoft.scripting"

type="Microsoft.Scripting.Hosting.Configuration.Section, Microsoft.Scripting, Version=1.0.0.5000, Culture=neutral, publicKeyToken=31bf3856ad364e35"

requirePermission="false" />

</configSections>

<microsoft.scripting>

<language names="IronPython,Python,py"

extensions=".py"

displayName="IronPython 2.0 Beta"

type="IronPython.Runtime.PythonContext,IronPython, Version=2.0.0.5000, Culture=neutral, publicKeyToken=31bf3856ad364e35" />

<language names="IronRuby;Ruby,rb"

extensions=".rb"

displayName="IronRuby 1.0 Alpha"

type="IronRuby.Runtime.RubyContext, IronRuby, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>

</languages>

</microsoft.scripting>

</configuration>

2) Calling the following methods

ScriptRuntimeSetup srs = ScriptRuntimeSetup.ReadConfiguration();

ScriptRuntime runtime = new ScriptRuntime(srs);

DLR Hosting API : How to create ScriptScopes using backing symbol dictionaries to implement dynamic variable lookup

2008-08-16T06:04:00Z

The factory methods used to create a ScriptScope object have overloads that take a 'IAttributeCollection' object to create the scope using the collection as a backing store. This feature is very useful and any app that needs non trivial Hosting would mostly use this feature.

The easiest way to implement the IAttributeCollection interface is to implement the CustomSymbolDictionary defined in the the Hosting API. Once this is done, all variable access by a script executing in the context of this scope will result in a call to the set or get methods that were overriden.

You can do some interesting things like chaining of scopes, inspecting variable values upon script access, value validation etc using this symbol dictionary implementation.

The following example uses a custom symbol dictionary to chain scopes to create a pseudo hierarchy.

using System;

using Microsoft.Scripting;

using Microsoft.Scripting.Hosting;

using Microsoft.Scripting.Runtime;

namespace ScopeOps {

public class ScopeHiererchy {

public static void Main() {

ScriptEngine eng = new ScriptRuntime().GetEngine("py");

ScriptScope baseScope = eng.CreateScope();

var dict = new DynamicLookupDictionary(baseScope);

//set variables in the base scope.

baseScope.SetVariable("base_int", 100);

baseScope.SetVariable("base_string", "from base");

ScriptScope useScope = eng.CreateScope(dict);

var src = eng.CreateScriptSourceFromString(

"print base_int\nprint base_string",

SourceCodeKind.Statements);

//'useScope' doesn't have the variables accessed by the script yet.

//The overriden 'get' method dynamically looks up the variable in the base scope.

src.Execute(useScope);

}

internal class DynamicLookupDictionary : CustomSymbolDictionary {

private ScriptScope _baseScope;

public DynamicLookupDictionary(ScriptScope baseScope):base() {

_baseScope = baseScope;

}

//This method would be invoked whenever a script executing in the context of this scope

//accesses a variable

protected override bool TryGetExtraValue(SymbolId key, out object value) {

//If the variable is part of the base scope, return that value - this effectively

//creates a 'scope chain'

if (_baseScope.TryGetVariable(SymbolTable.IdToString(key), out value)) {

return true;

}

value = null;

return false;

}

public override SymbolId[] GetExtraKeys() {

return null;

}

protected override bool TrySetExtraValue(SymbolId key, object value) {

return false;

}

Quick notes on ScriptScope

2008-08-07T05:30:00Z

ScriptScope is one of the key types available in the DLR Hosting API. They act as a context in which hosts execute code and record the effects of execution like changes in object values, creation of new objects etc.

Prior to any code execution, ScriptScopes can be populated with variables by hosts using methods defined in this type and these variables are then available to any code that’s executed in its context. ScriptScope also acts as a unit of isolation - it maintains state, isolates side effects from code and can be reused as a context to execute another code snippet.

The Hosting API spec describes this type in detail. The following list is a short summary of some of the key points described by the spec

This class represents a namespace essentially. Hosts can bind variable names in ScriptScopes, fetch variable values, etc. Hosts can execute code within scopes for distinct name bindings.
You create instances of ScriptScopes using the CreateScope and ExecuteFile methods on ScriptRuntime or CreateScope on ScriptEngine.

o ScriptScope scope = pythonEngine.CreateScope();

o ScriptScope scope = runtime.CreateScope();

o ScriptScope scope = runtime.ExecuteFile(@"C:\Foo.py");

If a ScriptScope were created with a default language, then members such as Execute and IncludeFile use that language's engine to perform these operations. If the ScriptScope has no default language, then these convenience methods throw an exception, and the Engine property returns null.
At runtime, the DLR manifests a ScriptScope object as a dynamic object. This means that normal object member access sees the variables stored in the scope, not the members of the static .NET ScriptScope type.
Hosts can use ScriptScopes to execute any kind of code within their namespace context. This method offers an easy and convenient way to execute a piece of script code and get a result from the execution. The variables from the execution are also bound to this ScriptScope and this lets hosts examine the state after code execution

o public object Execute(string code)

ScriptScopes has members to support operations on variables like set, edit, remove, clear etc.

o public bool ContainsVariable(string name)

o public object GetVariable(string name)

o public void SetVariable(string name, object value)

o public bool RemoveVariable(string name)

o public void ClearVariables()

In addition to the above items, the spec also contains details about each of the method defined in this type and their arguments.

IronPython 2.0 beta 4 released

2008-08-07T02:45:00Z

IronPython 2.0 Beta 4 has been released here. Since the DLR and Hosting API components ship with IronPython, this beta also contains newer versions of Hosting API. So you can use this version to try out any of the Hosting API samples from this blog’s earlier entries.

4 different ways to execute IronPython code using the DLR Hosting API

2008-07-29T06:55:00Z

The Hosting API offers 4 different sets of methods to execute a script. These methods provide hosts - managed application in C#, VB and other .NET languages - with powerful, convenient and versatile ways to execute scripts from the DLR languages like IronPython, IronRuby etc. ( Though the title mentions just IronPython, this post applies to others like IronRuby, ToyScript etc as well)

I have put together a short summary of these methods and the types that define them. The descriptions found here are shortened versions of the descriptions found in the Hosting API spec. Please refer to the same for more specific details about the type and the method.

The types that define these method families are ScriptRuntime, ScriptScope, ScriptSource and CompiledCode.

ScriptRuntime

This class is the starting point for hosting. ScriptRuntime represents global script state. This includes referenced assemblies, a "global object" (ScriptRuntime.Globals), "published" scopes (scopes bound to a name on Globals), available language engines, etc.

public ScriptScope ExecuteFile(string path);

This method executes the source identified in the path argument and returns the new ScriptScope in which the source executed. ExecuteFile determines the language engine to use from the path's extension and the ScriptRuntime's configuration, comparing extensions case-insensitively.

The path is an absolute pathname or a filename that naturally resolves with standard .NET BCL file open calls.

ScriptScope

This class represents a namespace essentially. Hosts can bind variable names in ScriptScopes, fetch variable values, etc. Hosts can execute code within scopes for distinct name bindings.

public object Execute(string code);

public T Execute<T>(string code);

These methods return objects that result from executing code in the string argument.

ScriptSource

ScriptSource represents source code and offers a variety of ways to execute or compile the source. You can get ScriptSources from factory methods on ScriptEngine, and ScriptSources are tied to the engine that created them.

public object Execute(ScriptScope scope);

public ObjectHandle ExecuteAndWrap(ScriptScope scope);

public T Execute<T>(ScriptScope scope);

public int ExecuteProgram();

These methods execute the source code and return the result. All but one method executes the code in the specified ScriptScope.

CompiledCode

CompiledCode represents code that has been compiled to execute repeatedly without having to compile it each time, and it represents the default ScriptScope the code runs in. You can also execute the code in any ScriptScope if you need it to execute in a clean scope each time, or you want to accumulate side effects from the code in another scope.

public object Execute() { }

public object Execute(ScriptScope scope) { }

public ObjectHandle ExecuteAndWrap() { }

public ObjectHandle ExecuteAndWrap(ScriptScope scope) { }

public T Execute<T>() { }

public T Execute<T>(ScriptScope scope) { }

Unlike the ScriptSource’s execute methods, these do not compile the code again.

A few of these methods can be found in the samples from the earlier blog posts. For more details and documentation about these methods and types please refer to the Hosting API spec.

How to create IronPython objects of types defined in C#

2008-07-24T05:45:00Z

The DLR Hosting API lets the hosted script instantiate objects of types defined in the hosting C# / VB module. This post describes how to do that using the appropriate methods from the API. Here are the simple steps -

1) Load the assembly containing the type in to the ScriptRuntime object

ScriptRuntime runtime = ScriptRuntime.Create();

runtime.LoadAssembly( Assembly.GetAssembly( typeof( n1.MyType1)));

2) Import the namespace/type inside the script

from n1 import *

3) Create an object of the type and invoke members

n=MyType1()

print n.Name

Here’s the full sample

using System.Scripting;

using Microsoft.Scripting.Hosting;

using System.Reflection;

namespace n1 {

public class MyType1 {

public string Name { get; set; }

public MyType1() {

Name = "MyType1";

}

public class Program {

static void Main(string[] args) {

string code = @"from n1 import *

n=MyType1()

print n.Name";

ScriptRuntime runtime = ScriptRuntime.Create();

runtime.LoadAssembly( Assembly.GetAssembly( typeof( n1.MyType1)));

ScriptEngine eng = runtime.GetEngine("py");

ScriptScope scope = eng.GetEngine("py").CreateScope();

ScriptSource src = eng.CreateScriptSourceFromString( code, SourceCodeKind.Statements);

src.Execute(scope);

}

IronEditor - Editor for IronPython, IronRuby and other DLR languages

2008-07-23T04:06:00Z

Ben Hall has released IronEditor - an IDE for the DLR languages like IronPython, IronRuby etc as a CodePlex project. Here are the links to the mail threads in the lists where he announced the release - IronPython & IronRuby. You will find some interesting followup comments and conversation about the editor in these threads.

Apart from being a useful editor for the DLR languages, this application is also a practical demonstration of using the DLR Hosting API in a managed application.

DLR Hosting Sample : QikScript - Handle Visual Studio 2008 extensibility events in IronPython

2008-07-18T00:01:00Z

The DLR Hosting API offers a well defined interface for managed applications to host and utilize the power of the DLR (Dynamic Language Runtime) to enable end user scripting. The objective of this sample (and the forthcoming series of posts based on this sample) is to demonstrate and explain the usage of the API in doing those things.

QikScript is a Visual Studio addin that hosts the DLR enabling end users to write python scripts to handle events fired by Visual Studio.

As part of its extensibility model, Visual Studio defines events that can be subscribed to through an add-in. Add-ins can also access ‘live’ objects like the project system, editor, menus, output window etc in the hosting instance of visual studio. By Hosting the DLR and routing the events appropriately, QikScript lets IronPython scripts to handle these events and access VS objects at runtime.

How to Install

1) Extract the contents of the attached zip file to a folder on disk( say C:\QikScript)

2) Copy the ‘QikScript.AddIn’ file to the VS 2008 AddIns folder. is the Addin folder is present in C:\Documents and Settings\<user name>\My Documents\Visual Studio 2008\Addins or C:\users\<user name>\My Documents\Visual Studio 2008\Addins

3) Open the AddIn file in a text or XML editor. Edit the value of the <Assembly> tag to indicate the full path to the ‘QikScript.dll’ file you copied in step 1.

4) If you extracted the files to a disk folder other than ‘C:\QikScript’ do the following. Go to the folder where you extracted the binary files. Open up the ‘Settings.XML’ file. Edit the ‘FilePath’ node in each of the ‘handler’ nodes to refer to the full path of the ‘handler.py’ file. (You can do a search and replace over the entire file).

5) The ‘handler.py’ contains the event handlers. You can write your own code in these methods and it will be invoked the next time VS fires that event.

Once you install the add-in, the next instance of VS that you start will invoke functions defined in the handler file. The simplest way to check if the addin is loaded succesfully is to take a look at the 'outpu' window of the VS. The dropdown box would contain an entry with the name 'QikScript' if loading is succesful.

Depending on your settings, it is possible for the addin not to load on startup. In such cases, you can go to the 'Tools->Addin Manager' menu item and manually select the addin, make it autoload and restart VS. Please do not hesitate to let me know via email or comments if you are stuck with the installation steps.

How to disable or remove the addin

The easiest way to disable this addin is to move or delete the QikScript.addin file from the AddIns folder. You could also change the extension of this file if you want to temporarily stop the VS from loading this. If you want to load VS without *any* addin you can use the ‘devenv /safemode’ from the Run prompt to start VS.

Package details

The deployment zip file contains the binaries that implement the functionality, an event handler definition XML file (settings.xml) and a python file (handler.py) that implements the event handlers. The DLR and ironPython binaries are from the IronPython 2.0 Beta 3 release

Settings.xml

· Settings.xml contains the details about the event handlers. You can edit the individual events to run a specific function in any given python file.

· The settings.xml must reside in the same folder as the QikScript.dll file.

Handler.py

· The Handler.py file contains the event handlers.

· Each of the handler defined in this file uses the correct signature needed to work with VS. Some of the arguments are primitive data types – string, bool , int and some are VS specific types.

· Note the commented out line just above the event handler. That line indicates the types of the arguments passed to the event handler.

The event handlers can also access the host VS object through the ‘hostApp’ variable in the script. The type of the object is DTE2. All output (like print calls) from the script is redirected to a new output window (with the name ‘QikScript’) that the add-in creates.

Notes

· This is strictly intended as a demo tool and comes with zero warranty or support.

· This addin doesn’t make VS an IronPython editor. You would use an external editor to edit the IronPython code. See this post if you are interested in enabling VS to edit IronPython sources

· I have used this addin only as part of Visual Studio 2008. It may or may not work in earlier versions.

· Theoretically, all operations supported by the VS through the extensibility model( the DTE2 interface) can be performed from any of the handlers. However, I did not attempt any testing to see which operations work and which don’t. So you are on your own when trying out adventurous thingsJ.

· Currently only ‘IronPython’ is available as the scripting language

This post is very light on implementation details deliberately. I am planning to put up a separate post with more details about the parts of the Hosting API this component uses.

Please let me know of any comments, questions, feedback or suggestions you have regarding this sample, DLR and/or the Hosting API.

Using the 'Find' box in VS to do some cool stuff

2008-07-12T07:27:00Z

Baoping from my ex team* has blogged about a couple of cool IDE tricks you can do using the 'Find' box in visual studio. Here is the post.

While you are there check this post about a tool called 'Code Injector'. This tool helped us find lots and lots of bugs in the C# IDE and the Compiler during the VS 2008 product cycle.

* I moved to the DLR QA team from C# about 5 months back

How to redirect output from python using the DLR Hosting API

2008-07-08T22:58:00Z

This is the 3rd in the series of 'How To' posts on using the DLR Hosting API to host languages like IronPython, IronRuby, Toyscript etc in a managed application. The first 2 posts are here and here.

The DLR Hosting API provides the ScriptIO class to give you complete control over the input, output and error streams used by the scripting languages. The Hosting API describes ScriptIO like this - “This class let's you control input and output by default for dynamic code running via DLR hosting. You can access the instance of this class from the IO property on ScriptRuntime.”

Here's the step by step guide to redirect the output to a Stream object using the ScriptIO class.

1) Create the runtime, the python engine and a MemoryStream object

ScriptRuntime runTime = ScriptRuntime.Create();

ScriptEngine pyEngine = runTime.GetEngine("py");

MemoryStream ms = new MemoryStream();

2) Set the memory stream created above as the output stream for this script runtime.( The API can redirect output to any Stream object. So you can also use a FileStream object to capture output).

runTime.IO.SetOutput(ms, new StreamWriter(ms));

3) Execute a python code snippet that would output something.

ScriptSource ss = pyEngine.CreateScriptSourceFromString(

"print 'hello world'",

SourceCodeKind.Statements);

ss.Execute();

4) Read from the memory stream object created in step (2).

string str = ReadFromStream(ms);

5) ‘str’ will now contain the output from the script.

The input and error streams can also be set using the above technique and calling the appropriate method in ScriptIO. Refer to the DLR Hosting API spec to learn more about the ScriptIO and other related classes.

The full sample follows. Normally you wouldn’t care to redirect the output, input or error streams in a console application. In spite of this, I am sticking to a console application as it lets me demonstrate the API usage succinctly and provide a full copy & paste enabled sample

using System;

using System.IO;

using System.Scripting;

using System.Text;

using Microsoft.Scripting.Hosting;

namespace ConsoleApplication1 {

class Program {

public static void Main(string[] args) {

ScriptRuntime runTime = ScriptRuntime.Create();

ScriptEngine pyEngine = runTime.GetEngine("py");

MemoryStream ms = new MemoryStream();

//runTime.IO refers to the current runtime's IO property

runTime.IO.SetOutput(ms, new StreamWriter(ms));

ScriptSource ss =

pyEngine.CreateScriptSourceFromString(

"print 'hello world'",

SourceCodeKind.Statements);

ss.Execute();

string str = ReadFromStream(ms);

ms.Close();

Console.WriteLine(str);

}

private static string ReadFromStream(MemoryStream ms) {

int length = (int)ms.Length;

Byte[] bytes = new Byte[length];

ms.Seek(0, SeekOrigin.Begin);

ms.Read(bytes, 0, (int)ms.Length);

return Encoding.GetEncoding("utf-8").GetString(bytes, 0, (int)ms.Length);

}