Jason Zander's WebLog : CLR

IronRuby Source Available for Download

Jasonz — Mon, 23 Jul 2007 18:55:00 GMT

John Lam and team have been working hard on getting IronRuby ready for public release, and today is the day! Today we are announcing two key things:

Availability of the IronRuby source code
Our plans to accept contributions to IronRuby libraries (to start) through Rubyforge by August

You can read all of the details on John's blog here.

There are two things that really excite me about this release. The first is getting to this important milestone with IronRuby, which we first demonstrated at Mix '07. Ruby is a powerful language which is not just for the web server ;-) When you combine Ruby with Silverlight, you can create really cool RIA applications as well. And as always, IronRuby, IronPython, and all of our dynamic languages will run in all .NET environments by the time we are finished.

The second thing that excites me personally is building another offering with the open source community. I can tell you from a lot of personal experience that getting the IronPython license approved took a lot of effort. That release and others like it paved the way for the Microsoft Permissive License. In DevDiv, we've then used that license to release the ASP.NET AJAX Control Toolkit and IronRuby. I'm looking forward to continuing these kinds of efforts in the future.

Congratulations to the team on the release!

Code Sample: Is Your Process Using the Silverlight CLR?

Anonymous — Sat, 12 May 2007 01:23:00 GMT

In my previous post I described how multiple CLRs can now be run in the same process starting with Silverlight. In the sample video I had a small console application that could be used to track managed processes on the machine and which version(s) of the CLR they had in them. I’ve included the code for this tool below.

The CLR debugger APIs implement ICorPublishProcess to find managed processes, but these APIs have not been updated for Silverlight and are likely to be deprecated in the future. Instead, in my first version I used System.Diagnostics.Process. I found these wrappers to be extremely slow due to grabbing all file version information from all modules loaded. I’m sure this is pretty useful for a one time snapshot, but my tool needed to refresh itself very frequently. Because of this I wound up using p/invoke on top of kernel32!CreateToolhelp32Snapshot (first part of the code).

The rest of the code is pretty simple: every 3 seconds take a process list snapshot and search for CLR instances by name, add them into the process list, then display them to the console. The code recognizes the CLR as follows:

· mscorwks.dll is the name of the CLR engine for version 1.0, 1.1 and 2.0 (3.0 and 3.5 build on top of 2.0)

· coreclr.dll is the name of the Silverlight CLR engine

I didn't spend a huge amount of time on this code and would put more structure in it if I ever put it in production (I'd also replace the dipslay code with a decent GUI). It’s nothing fancy, but it got the job done. Enjoy!

using System;

using System.Collections.Generic;

using System.Text;

using System.Runtime.InteropServices;

using Microsoft.Win32.SafeHandles;

using System.Runtime.ConstrainedExecution;

namespace EnumClrConsole

{

// The safe handle class is used to safely encapsulate win32 handles below

public class ToolHelpHandle : SafeHandleZeroOrMinusOneIsInvalid

{

private ToolHelpHandle()

: base(true)

{

}

[ReliabilityContract(Consistency.WillNotCorruptState, Cer.MayFail)]

override protected bool ReleaseHandle()

{

return NativeMethods.CloseHandle(handle);

}

// The following p/invoke wrappers are used to get the list of process and modules

// running inside each process.

public class NativeMethods

{

[DllImport("kernel32.dll", SetLastError = true)]

static public extern bool CloseHandle(IntPtr hHandle);

[DllImport("kernel32.dll")]

static public extern bool Module32First(ToolHelpHandle hSnapshot, ref MODULEENTRY32 lpme);

[DllImport("kernel32.dll")]

static public extern bool Module32Next(ToolHelpHandle hSnapshot, ref MODULEENTRY32 lpme);

[DllImport("kernel32.dll")]

static public extern bool Process32First(ToolHelpHandle hSnapshot, ref PROCESSENTRY32 lppe);

[DllImport("kernel32.dll")]

static public extern bool Process32Next(ToolHelpHandle hSnapshot, ref PROCESSENTRY32 lppe);

[DllImport("kernel32.dll", SetLastError = true)]

static public extern ToolHelpHandle CreateToolhelp32Snapshot(SnapshotFlags dwFlags, uint th32ProcessID);

public const short INVALID_HANDLE_VALUE = -1;

[Flags]

public enum SnapshotFlags : uint

{

HeapList = 0x00000001,

Process = 0x00000002,

Thread = 0x00000004,

Module = 0x00000008,

Module32 = 0x00000010,

Inherit = 0x80000000,

All = 0x0000001F

}

[StructLayoutAttribute(LayoutKind.Sequential)]

public struct PROCESSENTRY32

{

public uint dwSize;

public uint cntUsage;

public uint th32ProcessID;

public IntPtr th32DefaultHeapID;

public uint th32ModuleID;

public uint cntThreads;

public uint th32ParentProcessID;

public int pcPriClassBase;

public uint dwFlags;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]

public string szExeFile;

};

[StructLayoutAttribute(LayoutKind.Sequential)]

public struct MODULEENTRY32

{

public uint dwSize;

public uint th32ModuleID;

public uint th32ProcessID;

public uint GlblcntUsage;

public uint ProccntUsage;

IntPtr modBaseAddr;

public uint modBaseSize;

IntPtr hModule;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 256)]

public string szModule;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]

public string szExePath;

};

}

// The ProcessList is used to track all running processes that have managed code

// running in them. Each RootProcess item in turn tracks all CLR instances running

// inside the process, which as of Silverlight could be more than one.

class Program

{

public Dictionary<string, RootProcess> ProcessList;

public Program()

{

ProcessList = new Dictionary<string, RootProcess>();

}

// Build will find all running processes and search them for an instance of

// the CLR engine. mscorwks.dll is the enging for CLR V1.0 -> 2.0 (which includes V3.0 and 3.5).

// coreclr.dll is the new Silverlight CLR engine.

public void Build()

{

// Reset the old list of running processes and then get a list of everything running.

ProcessList.Clear();

ToolHelpHandle hProcessSnap;

NativeMethods.PROCESSENTRY32 pe32 = new NativeMethods.PROCESSENTRY32();

hProcessSnap = NativeMethods.CreateToolhelp32Snapshot(NativeMethods.SnapshotFlags.Process, 0);

if (hProcessSnap.IsInvalid)

{

Console.WriteLine("Failed on call to CreateToolhelp32Snapshot");

return;

}

pe32.dwSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(pe32);

if (NativeMethods.Process32First(hProcessSnap, ref pe32))

{

// For each running process, get the list of modules. If it has CLR running

// inside, add it to the running list.

List<ClrInstance> list = new List<ClrInstance>();

if (GetClrList(ref list, pe32.th32ProcessID))

{

string key = String.Format("{0}:{1}", pe32.szExeFile, pe32.th32ProcessID.ToString());

// If this process is already in the list, use that instance.

RootProcess p;

if (ProcessList.ContainsKey(key))

p = ProcessList[key];

// Otherwise create a new entry.

else

{

p = new RootProcess(pe32.th32ProcessID.ToString(), pe32.szExeFile);

this.ProcessList.Add(key, p);

}

foreach (ClrInstance c in list)

{

p.AddClr(c);

}

} while (NativeMethods.Process32Next(hProcessSnap, ref pe32));

}

hProcessSnap.Close();

}

// Counter data for display.

int iCounter = 0;

int iLastLineCount = 0;

int iLineCount;

private void WriteLine(string str, params Object[] o)

{

++iLineCount;

Console.WriteLine(str, o);

}

// The display method dumps the process tree to the screen in a pretty

// dumb way. This routine could easily be replaced with a GUI.

private void DisplayProcessList()

{

iLineCount = 0;

Console.CursorLeft = 0;

Console.CursorTop = 0;

WriteLine("{0}", ++iCounter);

foreach (KeyValuePair<string, RootProcess> rp in ProcessList)

{

String ProcPid = String.Format("{0} - {1}", rp.Value.ProcName, rp.Value.pid);

WriteLine("{0,-79}", ProcPid);

foreach (ClrInstance c in rp.Value.rgClr)

{

WriteLine(" {0,-75}", c.ClrName);

}

for (int i = iLastLineCount - iLineCount; i > 0; --i)

{

Console.WriteLine("{0,-79}", " ");

}

iLastLineCount = iLineCount;

}

// Returns true if this is a managed process.

bool GetClrList(ref List<ClrInstance> list, uint pid)

{

bool IsManaged = false;

ToolHelpHandle hModuleSnap;

NativeMethods.MODULEENTRY32 me32 = new NativeMethods.MODULEENTRY32();

hModuleSnap = NativeMethods.CreateToolhelp32Snapshot(NativeMethods.SnapshotFlags.Module, pid);

if (hModuleSnap.IsInvalid)

return (IsManaged);

me32.dwSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(me32);

if (NativeMethods.Module32First(hModuleSnap, ref me32))

{

// mscorwks.dll == a CLR of version V1.0, V1.1, V2.0

// coreclr.dll == Silverlight

if (String.Compare(me32.szModule, "mscorwks.dll", true) == 0 ||

String.Compare(me32.szModule, "coreclr.dll", true) == 0)

{

ClrInstance c = new ClrInstance(me32.szExePath);

list.Add(c);

IsManaged = true;

}

} while (NativeMethods.Module32Next(hModuleSnap, ref me32));

}

hModuleSnap.Close();

return (IsManaged);

}

static void Main(string[] args)

{

Program p = new Program();

while (true)

{

p.Build();

p.DisplayProcessList();

System.Threading.Thread.Sleep(3000);

}

public class RootProcess

{

public string pid;

public string ProcName;

public List<ClrInstance> rgClr;

public RootProcess(string inpid, string inproc)

{

pid = inpid;

ProcName = inproc;

rgClr = new List<ClrInstance>();

}

public void AddClr(ClrInstance clr)

{

rgClr.Add(clr);

}

public class ClrInstance

{

public string ClrName;

public ClrInstance(string strClr)

{

ClrName = strClr;

}

Side by Side In Process CLRs Start with Silverlight

Anonymous — Thu, 10 May 2007 22:31:00 GMT

When we started working on managed code for Silverlight one scenario I wanted to see working was the ability to create a new browser or shell in managed code that could still browse to and execute rich web content. We spent (endless) time debating if this scenario should simply target the desktop CLR if it were installed and just use one CLR. In the end my main concern about that solution was ensuring 100% behavior compatibility. We strive really hard to reach this on the desktop already, and at some point just being able to execute what you wrote against is the best solution. So instead we went down the road of creating a separately factored CLR which could run side by side in the same process. To test this, I wrote a small managed application which hosted shdocvw.dll (IE’s core browser engine) and browsed to a Silverlight enabled web site. The first few times this was not a pretty experience <g>, but that was quickly fixed.

Enabling side by side CLRs in the same process not only enables this scenario, but it removes a long standing complaint from developers trying to write applications that implement add-in models for extensibility. In these cases, you often want to load and bind to the version of the CLR your add-in was built for. Previously one had to roll forward to the latest and greatest version to pull this off. It does work, but can be a little painful.

This video gives a demonstration of running two CLRs in the same process using a WPF demo I blogged about here:

There are a few things to notice in this diagram:

· There is no retroactive change to Versions 1.0, 1.1, or 2.0 of the CLR. Only one of these may be in the process at the same time. The diagram shows V2.0 (3.0 and “3.5” are built on top if used).

· Silverlight can run side by side with any one of the V1.0, V1.1, or V2.0 CLRs.

· Future versions of the CLR (which are under development already) will run side by side with the original desktop CLR instances and Silverlight.

Now that we have this feature, you may ask why it wasn’t there from the beginning. There were always two main considerations we had historically for not going down this road. The first is thread suspension. You need to ensure that if two CLRs are suspending threads to do a GC they don’t suspend each other in the process (there are some XML dlls out there today which do a poor job of this). The second consideration is the performance of the resulting process. Each CLR has its own GC heap and loaded code; they do not share data with each other (any interop between them must be done through unmanaged code / COM).

Given all of this, it is still our recommendation that if you are designing a new extensibility API for your application, try to make it work well with the add-in running out-of-process. This really does provide the cleanest way to isolate 3^rd party code and keep the best possible compatibility. However if you are using an existing API extensibility model (sidebar, browser, etc) that was not designed for this, this new feature does simplify your life a lot.

Red Bits and Compatibility

Anonymous — Mon, 22 May 2006 07:30:00 GMT

Doesn't Side by Side Cure All?

Soma posted in his blog how we plan to produce an updated version of the .NET Framework that maintains compatibility. This particular subject always draws a lot of passionate opinions. I've heard the opinions from the polar extremes. This is a common example:

"There is absolutely NO reason to maintain backwards compatibility with previous versions of .NET with new versions. First you're artificially tying your hands, second, since the frameworks can live side-by-side (except the web.config bug between .net 2.0 and .net 1.1 virtual directories) there is absolutely no reason for for it. (ship the version that your application requires with your application!)"

Automatic roll forward is the biggest problem with this. When you do a LoadLibrary (or cause one such as in COM activation) and the CLR has never been loaded into the process, we have to pick one to execute the code. We could pick the version that matches the file you are loaded. But if that version were older and you subsequently loaded an assembly that required a new version, it would fail. To fix this, we automatically roll forward to the latest and greatest CLR on the machine and maintain backwards compatibility to the best of our ability.

I have also talked with Fortune 100 companies who like to reduce their TCO by standardizing on one version of the .NET Framework. To these groups, it is cheaper to have their IT staff manage one flavor in their data center. The cost to target a new version can be expensive, especially if each new version were to take carte blanche with compatibility. By keeping our compatibility high between versions, we make the transition easier.

I posted a great deal of information on how we do our compatibility work (see this link). If you do find compatibility issues between the versions, please do report them!

Trade Offs

All that said, I do agree with some underlying sentiment from the comments. I would like to see the .NET Framework be able to evolve over time and get rid of legacy. We created System.ObsoleteAttribute to help.

There are a few attributes that any solution should have:

Simplicity - Any solution needs to be easy to describe and figure out. Ideally using a design pattern that is easy to understand and doesn't require a lot of moving parts.

Predictability - Developers like to know what to expect, especially in large enterprises. Having a system that allows you to know how many more compatible versions to expect helps you plan your releases and roll outs accordingly.

Pragmatic - There needs to be a sunset policy for old versions to avoid have too many permutations to track and service.

We have been working on a solution that factors in these attributes called Compatibility Blocks. Think of a block as a set of compatible versions where an application written for a block can safely roll forward to the newest version in the block. Such a concept allows developers (and in particular the enterprise) to plan around a set of compatible versions. At the same time, it allows us to pick a transition point to obsolete old functionality (e.g. at the creation of a new block). You can think of V1.0, V1.1, and V2.0 as a compatibility block. New versions we create (including Red Bits) that continue this pattern extend the block. At some point we will pick an inflection point to create a new block (we'll give you plenty of warning when that happens; there is nothing formally planned at this point).

We'll have more details to share as designs firm up. In the meanwhile, keep your opinions and requirements coming. They are truly valuable to help define the technology as it evolves!

In Closing

Finally on Don's comment about our favorite Canadian sitcom ever to hit PBS:

"This sounds like a really dumb idea, and the whole "Red Green" thing just makes it sound worse. Ever seen the "Red Green" show? Not the guy I want planning my version compatibility scenarios."

As it would have it, I am the Approver for Red Bits changes in Orcas and am reviewing each with a small group of experts. The DDCPX team owns driving a high level of quality for the release thereafter. I can assure you there is no duct tape being used in this production ;-)

Catchy headlines? Sure, but get the facts

Anonymous — Fri, 27 May 2005 06:54:00 GMT

Well it's been a fun and creative couple of days for headlines:

Microsoft Recruits for .Net Framework Compatbility Testing (started off being called "Microsoft .NET Framework Breaks Apps")
Ouch! .Net Framework 2.0 Breaks Apps
.Net Developers: 'What, Me Worry'?
Microsoft Wrestles With .Net Framework Compatibility

While cute headlines may grab advertising clicks, you need to read past the titles to get the real story here:

Side by Side: Joel Spolsky has it right in the Marj Jo Foley article. The default binding behavior for an application is to bind to the version of the runtime it was built for. This includes, for example, your web sites in Asp.Net, where it is an explicit decision by the administrator to change the .NET Framework version for a vroot.

Minor Code Updates Required: I blogged already on all the work we have done to keep the API set compatible to minmize your developer related upgrade costs. As Soma points out in his blog, the first eWeek article explains that Infragistics ported 1.8 million lines of code without major issues. Mary Jo Foley's article also quotes Ryan Hoffman as having taken only one hour to update 20-30 lines of code out of thousands. I'm very happy to see these kinds of results. It validates all the effort we put into maintaining a high compatibility bar.

Only One Official .NET FX Redist: I understand the general care that Greg DiMichillie mentions one should take around shipping software, including his reference to GDI+ (remember my number one reason why I said I didn't want to build a linker?). However you should understand there is only one official redist, only one copy of the code on your machine, we don't statically link the .NET FX into your code, and we service all of our binaries in place. There is no case where we do not maintain strict control over the location of our binaries or our ability to service them.

I strongly encourage you to read through the entire content of these articles. Then go read this MSDN white paper which gives more technical details. And finally, give it a try on your application. It's really easy to see if your 1.x application will just work by adding the following to the <appname>.exe.config file:

<startup>
<requiredRuntime version="v2.0.50215"/>
</startup>

where "v2.0.50215" is the version string for Beta 2 (as found in your %windir%\Microsoft.Net\Frameworks directory). I've tried this on countless applications. I have found issues, but the vast majority of applications simply boot up happily and run. Don't stop at the headlines. Give it a try yourself and let us know what you find.

[Note: I made a quick edit to fix the linker url]

Why should your language target the CLR?

Anonymous — Thu, 24 Mar 2005 15:19:00 GMT

In my post on a meeting with some JVM experts, I mentioned a few high level notes on our dynamic language support. James Barnes asked for a response to Patrick Logan's article on the subject. Patrick points out my post wasn't deep on details, and could really be looked at as trying to force types deeper into typeless languages.

First, I'm very glad to see the interest in Dynamic Languages in the context of the .NET Framework. I'm very excited about the growing set of developers using dynamic languages being able to further leverage Visual Studio and the rich libraries of the .NET FX as well as new technologies like Avalon and Indigo. Second, I should clarify that my remarks were geared more towards a compiler language author rather than a language user (an important distinction which determines what a user sees in the language itself).

The Managed Environment

Managed languages are typically made up of three parts:

Compiler

The language implementation targeting the Engine.

Platform Libraries

The Platform Libraries are for accessing databases, networking, and the usual kinds of features you need to be productive as a programmer. Ideally these are written using the language itself and target its instruction set, but often they are written in unmanaged code with an interop layer.

Engine

The engine functionality probably includes some kind of collector, support for exceptions and language intrinsics, potentially support to interop with unmanaged code (be it native C or COM or whatever), etc.

A programmer typically lives in the world of their compiler (often through an IDE) and the platform libraries. Ideally the machine level features aren't in your face. For example, you expect to allocate objects and use them without having to know every nuance of how the collector itself works. Ideally the language gives you a nice way to deal with its constructs (like the using clause in C#). We're not always as successful as we'd like to be, but it is a general goal.

Why Target the CLR When You Already Have a Language VM?

I'd like to see dynamic languages used in more places. Stand alone utilities like BitTorrent are cool. But can I utilize the technology in other places, like plug-ins for major applications? Yes, this is possible to do today. But here's the world I would hate to see: a commonly used host application (like a browser) winds up with multiple garbage collectors competing for memory and machine cycles to collect it, tons of duplicated platform libraries clogging up working set, basically making the app a pig.

I believe that while most compiler writers are interested in garbage collection algorithms (hey, who wouldn't be?), I do think they would like to spend more time innovating in the language. If you author a compiler that targets the CLR, you've extended your development team to include ours working on your behalf. I believe that user's of your language get the win as well, through access to a rich set of new platform libraries and better performance.

The next conversation you can have is a blow by blow comparison of features the CLR provides vs what the language environment itself had already. I would rather work with the language authors on that and get their feedback first hand. That is why I was so excited about getting Jim on the CLR team in the first place: it brings a world class expert to help us accomplish the goal.

Access to Platform Libraries

It is obviously important to a language user to get access to the libraries they already know, and to which they've already written a lot of code. For those libraries which target the language itself, this should just work. For example, IronPython runs those libraries which are written in Python. Libraries which are written in native C/C++ are trickier, because their integration with the system may be tightly coupled to the machine level and its programming interfaces.

The next thing that is important is having a rich set of platform libraries that you didn't have easy access to before. We bring you that with the .NET FX. The string object you use in your language is a System.String underneath, and is the string you'll use in C#, VB.Net, or COBOL. Finally, you get access to new managed libraries that vendors like Microsoft are writing (beyond the already rich .NET Framework). Avalon and Indigo are two examples of that reach. Jim did a demo at PyCon showing an IronPython Avalon application.

This notion of composing an application from components written in multiple languages has been a core tenant of the CLR since day one (duh, spell out the acronym).

Tools Support

Once you have a language which targets the CLR, you can take advantage of a ton of great integrated tools support. As an example, when you write Python code which utilizes libraries written in C# (VB.Net...), Visual Studio will give you integrated call stacks between the languages automatically. It's all just managed code to VS. Check out the Nemerle project for another non-Microsoft example. The same thing applies for profilers, static analysis tools (like FxCop), etc. I will also point out that there is a rich set of tools from non Microsoft partners as well.

Have rich tools existed in the past for other environments? Of course! But those tools typically only support a specific language. For example, if you wanted to use code analysis to enforce a set of coding guidelines for your organization, you might have had to use a lint variant for C/C++, and may or may not have had a tool for your VB6 UI code. It's the integration that works across all the code your organization owns that is exciting, not the fact that we have tools in the first place. Having the entire Visual Studio team and a huge user community working on tools that will work with your new language is just a bonus.

Are We Forcing Types on You?

They call them typeless languages for a reason. The compiler is the first key towards making this possible. The Python code you wrote yesterday will work on IronPython today unaltered. The first demo in the IronPython readme is "2 + 2", not "int x=2; int y=2; print x+y;". You aren't required to add strong typing into the language.

However, objects on the CLR do carry some type information (called the “runtime type”). The types act as a barrier to what can be done to an object at runtime. But that is because a secure platform such as the .NET Framework really must hide some details of object representations from client programs. Dynamic languages do have to take this into account, as they already have to when interacting with an operating system or with code written in other languages (e.g. C). This means some things may need to happen when a dynamic language calls a .NET component: nearly all the time a dynamic language will help the programmer by automatically converting data to the necessary format, but sometimes the programmer may need to ensure that the data is in the right format in the first place.

That said, I do agree that there are improvements we can make here so that typeless languages feel very natural in the environment and tools. There is work to be done here and I'm looking forward to working with the community to get there. Your feedback is very important to us!

As an aside, projects like F# and Nermele are beginning to show that even statically typed languages can be essentially typeless, in the sense that you don’t have to write lots of types in your source code: these languages can reconstruct the types for most of your code without having type annotations.

Details on Whidbey (V2.0) Features

You've asked for details on what we've added which assist compiler authors. Here are some examples:

Extensions to Delegates

- We have a new feature called delegate relaxation. You can now call on a delegate which points to a method which is covariant on the return type and contravariant on its parameters. In more general terms, the parameter types of the target method can be more generalized than the parameter types of the delegate and the return type of the target method can be more specialized than the return type of the delegate.

- We’ve also allowed for more delegate binding scenarios, including: open-instance (a delegate to an instance method that is bound to a particular instance only at invocation time) and closed-static delegates (a delegate to a static method where the first parameter to be passed to the target method is specified once at delegate creation time).

Reflection token/handle Resolution API’s

- Reflection has exposed runtime identity abstractions and new API’s that interact with these abstractions specifically for compiler writers and those that want more control over Reflection. We’ve exposed managed constructs for metadata tokens and runtime handles. Exposing metadata tokens gives a compiler writer unique metadata level identity, allowing them to do things like build real symbol table abstractions. Runtime handles give languages unique runtime level identity (among other things), perfect for late binding scenarios commonly found in dynamic languages

Lightweight Code Generation

- New support for runtime method level code generation. These LCG methods are first class citizens of the runtime and for construction of these methods we utilize the familiar Reflection.Emit MethodBuilder API’s. LCG methods are also GC reclaimable. This allows a language author to cheaply create functions that get used and then thrown away. In V1.x, the closest thing we had was Method Rental which came with the overhead of metadata declaration. You can find some examples on Jole's blog.

Generics

- Generics increase the richness of runtime type information that is attached to objects: you can query an object to detect if it is a List<string> or List<Button>. Some dynamic languages will ignore this information, but some may wish to use it in order to do different things when a method is invoked on different kinds of objects. Generics also mean that dynamic languages will occasionally have to specify some extra information or insert some extra “impedance matching” conversions to ensure objects get the right runtime type information attached.

But the cool thing about generics is they will help other languages use code written in dynamic languages. This is really important, and may take a while to appreciate, since people who advocate using dynamic languages aren’t really used to people wanting to call into their code. But in the long run I expect lots of great components to be written in IronPython and other languages, and once that happens it’s interesting to think how the C# and VB programmers are going to use those components.

For example, let’s consider a hypothetical dynamic language PyLisp, and say the C# programmer comes across a really cool module written in this language. Let’s assume PyLisp represents all values using a runtime type “DynamicObj”. What will a C# programmer do to convert a List<string> to a “DynamicObj”? How about a List<int>? Of course, he or she will write a generic function

DynamicObj ConvertToPyLisp<T>(List<T> myData)

Highly efficient versions of this function may be possible according to what “T” is, and the implementation of .NET generics will generate especially efficient versions of functions when instantiated at value types. Most importantly they let you efficiently detect what “T” is at runtime, and this increases your range of options for writing this kind of “static-to-dynamic transformation” code.

So, .NET generics will make it easier and more efficient for other .NET programmers to use functionality defined in dynamic languages. In the long run this can only be a win for the dynamic languages themselves.

Summary

In closing, thank you for reading this far <g>. I'm hoping you'll play with IronPython and get some ideas on how you too can make your language work on the CLR. I'd love to see the new applications you can write using libraries like Avalon and Indigo. I'm looking forward to hearing your feedback to make the system better!

Announcing IronPython 0.7 Ready for Download

Anonymous — Wed, 23 Mar 2005 19:05:00 GMT

Today at PyCon, Jim Hugunin announced the availability of IronPython 0.7 which you can download from this site. We are releasing the code under a BSD style Shared Source license. This version runs on the Whidbey CLR (V2.0) which has features like LCG designed for dynamic languages as well as numerous performance improvements. Because IronPython targets the CLR, 0.7 also works directly with other new managed technologies like Avalon and Indigo (also recently release for usage, see the the March CTP).

I want to congratulate Jim for getting this release up on the site and ready for use! We're looking forward to hearing your feedback. I'd love to hear about the cool apps you write. Let us hear from you!

Microsoft Technology Summit Notes on the CLR

Anonymous — Thu, 17 Mar 2005 17:49:00 GMT

This morning I met with a group of industry influencers who are more exports in the JVM community. I brought with me a few folks from the CLR team, including Jim Miller, Brad Abrams, and Jim Hugunin.

Rather than do a lot of presenting (I had only 9 slides including the title page), we spent most of the time fielding some really good questions. I wanted to jot down a few additional pointes here:

Garbage Collection

There were lot's of questions on how our collector works. This article by Rico is a great resource for full details. A few notes:

We do have both a client (built for interactive responsiveness) and server (built for CPU scaling) version of the collector.
Our collector doesn't use handle indirection for references, so if you write a for() loop we will enregister the reference and do all the heavy lifting to track/update the roots ourselves.
Our philosphy is not to have a ton of knobs on the GC. We think that for many typical programmers it is too easy to get into trouble. The response from the audience was two fold on this point: (1) they generally fealt that the JVM had overdone this with far too many switches and developer configured settings, but also (2) some developers do want more power. We do support advanced settings with our hosting API's if you need them.
There was a question about whether we were adding the ability for developers to plug in their own GC routines. We are not planning to do this now or very likely ever. These kinds of customizations can make it harder to write a successful tuning GC (changes heuristics too much).

Memory Overhead and Sharing

I walked through how an Assembly is found on the machine, how the CLR is bootstrapped, and how we use the GAC for versioning work. A few intrestesting notes on this:

At runtime, any IL, Metadata, etc in your Assembly will be shard by Windows using the normal PE loading techniques.
We have the NGen compiler which allows you to pre-compile your code and share pages at runtime. I have a long article on this here on this subject. This means that NGen'd code is also shared between processes.
In practical terms, it means if one CLR app is already running on the machine, the next one will start up a lot faster because the system is "warm". You are running the same native code for System.String in every process and don't have to wait for it to be recompiled.
Parts of the runtime metadata (v-table and field layout information used by the engine) can also be shared with NGen'd processes.
If you JIT compile, then the code is per process, as is the runtime metadata.
We've done a ton of work in V2.0 to further reduce managed code overhead. This is really a big win for client applications.

Memory Model and Volitility

A question came up about how we handle aggressive memory scheduling on various CPU's. The CLR does support the concept of volitility and read order semantics. This is another example of a complicated trade off based on developer segment. Lots of people would be very happy to not ever have to understand the concepts involved. But other segments are going to be looking for every ounce of performance and want access to those features. Our high level principle is to make the system easy to use and program against by default, but not to totally hide things either.

Versioning of Assemblies

Versioning is a very tricky subject. A few notes we talked about:

We support side by side CLR's on the machine, but only one per process.
We capture strong versioning information at compile time and store it in the metadata.
At deployment time, you can place your strongly versioned assemblies into the Global Assembly Cache (GAC) where we can find them later.
At bind time, we can look at your reference and the version information to figure out which one you required and load that version.
We do support "unification" for the core .NET FX libraries themselves. This allows you to get one consistent stack at the base.
There is a lot of good lower level material: on Suzanne's blog and Junfeng's blog.

Dynamic Languages

We had several questions about our philosphy on dynamic languages. I think I'll actually write a separate blog post on this one because it is such an interesting subject. A few quick notes for now:

I believe we already have a great platform for dynamic languages. You get the GC, exceptions, interop, and access to a huge set of libraries for free when you taget the CLR.
In Whibey we add Lightweight Code Gen (LCG) which further eases the ability to author the languages.
Because we also built generics deeply into the runtime, you can leverage that support dynamically at runtime without having to "fake it out". This is a distinct advantage over techniques that use "erasure" to replace syntax with expanded code.
Because you target our MSIL and type system, we also enable easy access to tons of tools that work across langauges seemlessly.
I expect in the future to do a Project 7 style approach to adding more improvements to the engine, just like we did in the late 90's to form the current engine.
And finally a quick plug for Jim Hugunin who is doing a keynote at Pycon next week.

In closing, I want to thank the attendees for their great questions and feedback!

March CTP of WinFX (Avalon, Indigo, .NET FX) now posted on MSDN

Anonymous — Wed, 16 Mar 2005 22:31:00 GMT

I'm back! I was OOF for a bit after getting pretty sick coming back from Bucharest and Tel Aviv. But I had to report that the March CTP was just posted today on MSDN. This is the first combined drop of Avalon and Indigo targeting the same CLR, as well as the first version to be dropped with SDK support. You can find the docs for the release up on MSDN as well.

This drop represents a ton of work that we started last September. Obviously the Avalon and Indigo deserve a lot of credit for this drop. But I want to make sure and congratulate the other teams that worked very hard to pull this together: the CLR team, the SDK team, the UE folks, and build and release teams (sorry if I missed anyone!). In particular, Jason Sutherland, has been kicking butt running the Tactics team. It takes a lot of folks to deliver a release like this.

Great job folks!

I got pretty sick after coming back from Romania/Israel. Being a stubborn workaholic, I actually did go to work with pneunomia. But finally had to post my OOF mail:

It seems my bronchitis has turned into pneumonia.

So why am I still doing email? Well my wife insists it’s because I’m crazy.

I prefer to think instead of my ancestors who settled Minnesota and North Dakota, building sod homes by night and planting the north 40 by day. Would pneumonia slow them down? No way! Besides, sudden rapid weight loss is quite slimming.

So I’ll be working from home doing my email, watching the next 10 lbs melt away. Worst case scenario, I’m sure Overlake Medical has added wireless to the rooms by now.

Gotta run, my wife wants to spray more Lysol on my keyboard.

[Edit: fixed the indent font thing that was messing up the rest of my page]

Bucharest .NET User's Group

Anonymous — Sun, 30 Jan 2005 09:02:00 GMT

We just wrapped up our recruiting trip to Bucharest. On Monday 24-January, Brand, Ronald, and myself attended the .NET User's Group hosted by Dorin Badea of the Microsoft office and Microsoft MVP Aurelian Popa who did a talk on OSS for the group.

The session was well attended with about 50 to 60 people. Here is a smaller group that was able to fit on the stage for a group photo:

Brandon Bynum of the Winforms team did a presentation on new Whidbey (v2.0) features. He included a few cool demos of using Winforms to create the same look and feel as Outlook and MSN Messenger.

Finally I gave a presentation on CLR internals (borrowed from Joel Polbar's TechEd deck). We wrapped up the session with about an hour of Q&A with a lot of details on the product. We got a lot of great questions. Dorin taped the event and is going to send me the DVD which Scoble has offered to turn into a Channel 9 session. Stay tuned!

Thanks to Aurelian and Dorin for hosting us while in Romania and giving us a chance to meet the MVP's and user group. We had a blast!

New tiny CLI implementations hit the market

Anonymous — Mon, 20 Dec 2004 20:21:00 GMT

Here are a couple of interesting new items based on the CLI:

Axalto has produced a smartcard based on the ECMA standard.
.netcpu Corp has produced a development kit for the their embedded CLR CPU Module. Haven't tried it, but defintately looks cool. My wife has been looking for gift ideas, hmmm...

Happy Holidays!

November Community Tech Preview (CTP) with Standard Edition available

Anonymous — Tue, 14 Dec 2004 23:02:00 GMT

The November Community Tech Preview (CTP) has now been released. This drop contains a version of the Standard Edition. I'm in the process of uninstalling the October release and putting this one on. (Unfortunately that always seems to involve rmdir and regedit to find all traces of the old stuff, so it could take a while.)

Make sure to report any issues, suggestions, etc to the Ladybug site. Enjoy!

Oracle 10g Release 2 to support the CLR

Anonymous — Wed, 08 Dec 2004 02:19:00 GMT

Oracle is having their Openworld conference right now and has announced support for CLR stored procedures in an upcomming release. Read more in the press release. This means that Oracle, DB/2, and Microsoft SQL Server will support CLR managed stored procedures. Cool!

New 64-bit material available on MSDN

Anonymous — Wed, 01 Dec 2004 22:10:00 GMT

This new site covers common 64-bit questions about the .NET FX and gives examples. I'm often asked a few FAQ's, so let me answer them right here:

Version 2.0 (Whidbey) is the first version to support 64-bit.
Versions 1.1 (SP1) will run only as 32-bit in the WOW
V2.0 supports x64 processors (AMD Opertron, Intel EM64T), as well as the Intel Itanium (IA64). In fact, you can see a tour of our lab running all this hardware on Channel 9.
Whidbey Beta 1 and subsequent CTP's already support 64-bit, so you can down load and try it out today
Pure and type safe MSIL code (VB.Net, C# without the unsafe keyword, etc) can be JIT compiled automatically to 64-bit instructions. We only ship one copy of the BCL and System.* with the FX using this technique.
The most important piece of advice to write 64-bit capable code is to use IntPtr in your interop definitions and run FxCop. More hints on the site.
If you use pure code and run the FxCop rules cleanly, your code should generally just work on 64-bit. We had a lot of good experience while bootstrapping the system. The more unmanaged code and interop you do, the more of a traditional bootstrap it will feel like for you. But if you ever lived through something like Win32s or the 16-bit macro crackers, this will feel like heaven <g>

If you have feedback on the 64-bit support or questions, please do let us know.

So tell me, do you have an application that can use the extra processing power and address space of 64-bit? What chips do you think you'll be running on over the next five years?

How much power is too much?

Anonymous — Fri, 19 Nov 2004 00:00:00 GMT

In the second section of the Channel 9 tour videos, Rico Mariani talks about performance and I once again butcher his last name (I once made the same mistake at a BillG review, sorry man <sigh>; it's pronounced "Mary-annie", got it now <g>).

One thing we've seen a lot is that it is really easy to take one or two lines of managed code and really blow your working set and startup time. It's a double edged sword. On the one hand, you get a really easy to use class library, on the other you don't feel the pain that would otherwise make you think twice about using an expensive feature.

Two concrete examples include the XML serializer and compiled Regex expressions. I've seen requests for Assembly.Unload() that really are due to having these extra dynamically generated assemblies around. This performance report on the internal Headtrax application explains how that application invoked the C# compiler 16 times at startup! (this is discussed in the interview section with Scoble coming up in a new segment). If your application is slow to start, see if you are making this very easy mistake.

We've kicked around a bunch of ideas on this in the past. Obviously our first goal is that you always get the fastest solution by default (you should precompile your XML serializer code for example). We've toyed with giving you a "red/yellow/green" gradient in the docs or Intellisense that would hint at where you are using an expensive feature. Profilers are also useful for you to figure this kind of stuff out; but do require work and planning.

So my question for you: how much power is too much? Should we be making the library much more "in your face" or harder to use for things that are really going to cost you? What are your favorite "feature rich" API's that might be causing this unexpected/unwanted overhead?