Lucian's VBlog : tools

The limitations of Reflection.Emit

ljw1004 — Sun, 29 Nov 2009 05:51:00 GMT

Question: Is Reflection.Emit a comprehensive API that lets you build any assembly you want?

Answer: No. There are some things that are valid IL, and yet can't be built with Reflection.Emit. Thanks to Eric Lippert for showing me the following example.

Why do we care about this? Partly it's just curiosity. Partly it's because we're porting the VB compiler from C++ into VB, and we wanted to know if we could use Reflection.Emit for the back-end (answer "no"). Partly because we're exploring the idea of a "REPL loop" and want to know how best to implement it (no solid answers yet).

Attempt 1: Wrong Order

Imports System.Reflection
Imports System.Reflection.Emit

' Goal: to emit this class structure through Reflection.Emit:

Public Class X

Inherits Y

Public Interface I

End Interface

End Class

Public Class Y

Implements X.I

End Class

' Implementation: this is our attempt. We can't actually do it,

' because of limitations in Reflection.Emit

Module Module1

Sub Main()

Dim name = New AssemblyName("test")

Dim ba = Threading.Thread.GetDomain.DefineDynamicAssembly(name, AssemblyBuilderAccess.RunAndSave)

Dim bm = ba.DefineDynamicModule(name.Name, name.Name & ".dll")

Dim by = bm.DefineType("Y", TypeAttributes.Public Or TypeAttributes.Class)

Dim bx = bm.DefineType("X", TypeAttributes.Public Or TypeAttributes.Class, by)

Dim bi = bx.DefineNestedType("I", TypeAttributes.NestedPublic Or TypeAttributes.Interface Or TypeAttributes.Abstract)

by.AddInterfaceImplementation(bi)

' The question now is: in what order do we create the types X/Y/I ?

' http://msdn.microsoft.com/en-us/library/system.reflection.emit.typebuilder.createtype.aspx

' * X before I - "create enclosing type before creating the nested type"

' * Y before X - "create parent type before creating the current type"

' * I before Y - "create interface type before creating the current type"

' It's impossible to satisfy all three constraints

Dim ti = bi.CreateType()

Dim ty = by.CreateType()

Dim tx = bx.CreateType()

ba.Save(name.Name & ".dll")

End Sub

End Module

Attempt 2: Callbacks

Update: Thank you for the excellent discussion in the comments. Kvb86 pointed out the use of "ResolveEventHandler" callbacks. With them we can emit the code at the top of this article:

Imports System.Reflection

Imports System.Reflection.Emit

' Goal: to emit this class structure through Reflection.Emit:

Public Class X

Inherits N.Y

Public Interface I

End Interface

End Class

Namespace N

Public Class Y

Implements Global.X.I

End Class

End Namespace

' Implementation: this is our attempt. We can't actually do it,

' because of limitations in Reflection.Emit

Module Module1

Sub Main()

Dim ad = Threading.Thread.GetDomain()

Dim name = New AssemblyName("test")

Dim ba = Threading.Thread.GetDomain.DefineDynamicAssembly(name, AssemblyBuilderAccess.RunAndSave)

Dim bm = ba.DefineDynamicModule(name.Name, name.Name & ".dll")

Dim by = bm.DefineType("N.Y", TypeAttributes.Public Or TypeAttributes.Class)

Dim bx = bm.DefineType("X", TypeAttributes.Public Or TypeAttributes.Class, by)

Dim bi = bx.DefineNestedType("I", TypeAttributes.NestedPublic Or TypeAttributes.Interface Or TypeAttributes.Abstract)

by.AddInterfaceImplementation(bi)

' The question now is: in what order do we create the types X/Y/I ?

' http://msdn.microsoft.com/en-us/library/system.reflection.emit.typebuilder.createtype.aspx

' * X before I - "create enclosing type before creating the nested type"

' * Y before X - "create parent type before creating the current type"

' * I before Y - "create interface type before creating the current type"

' It's impossible to satisfy all three constraints

Dim yc = by.DefineDefaultConstructor(MethodAttributes.Public Or MethodAttributes.PrivateScope Or MethodAttributes.HideBySig Or MethodAttributes.SpecialName Or MethodAttributes.RTSpecialName)

Dim xc = bx.DefineConstructor(MethodAttributes.Public Or MethodAttributes.PrivateScope Or MethodAttributes.HideBySig Or MethodAttributes.SpecialName Or MethodAttributes.RTSpecialName, CallingConventions.Standard, Type.EmptyTypes)

Dim ilg = xc.GetILGenerator()

ilg.Emit(OpCodes.Ldarg_0)

ilg.Emit(OpCodes.Call, yc)

ilg.Emit(OpCodes.Ret)

Dim eh = Function(o As Object, e As ResolveEventArgs) As Assembly

Select Case e.Name

Case "X" : bx.CreateType()

Case "Y" : by.CreateType()

Case "I" : bi.CreateType()

End Select

Return ba

End Function

AddHandler ad.TypeResolve, eh

Dim ti = bi.CreateType()

Dim ty = by.CreateType()

Dim tx = bx.CreateType()

RemoveHandler ad.TypeResolve, eh

ba.Save(name.Name & ".dll")

End Sub

End Module

Attempt 3: Lack of precision

Update: but as Rolf points out, the existing ResolveEventHandler in Reflection.Emit is inadequate. That's because it doesn't qualify the names. So if you were trying to emit this code, where everything is called "X", then the ResolveEventHandler wouldn't tell you which "X" it has in mind!

Public Class X

Inherits N.X

Public Interface X

End Interface

End Class

Namespace N

Public Class X

Implements Global.X.X

End Class

End Namespace

Conclusion: if Reflection.Emit were to be used seriously, it'd have to be fixed first.

Where are the SDK tools? Where is ildasm?

ljw1004 — Sat, 15 Nov 2008 04:48:00 GMT

C:\Program Files\Microsoft SDKs\Windows\v6.0\bin\ildasm.exe
C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\ildasm.exe
C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\ildasm.exe
C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\x64\ildasm.exe
C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\ildasm.exe
C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\x64\ildasm.exe
C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\ildasm.exe
C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\x64\ildasm.exe
C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\ildasm.exe
C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\x64\ildasm.exe

What are all these different versions for? Which one should I use? Sumit Kumar is the Project Manager for “Windows SDK”, so I sat down with him to find out. The Windows SDK blog is at http://blogs.msdn.com/windowssdk/.

(Note: the last four directories exist but don't actually contain a copy of ildasm.exe. I just put them there for dramatic effect.)

What are the SDK tools?

The SDK tools are a sort of grab-bag of utilities that advanced programmers will find useful. Some are general tools put forward by the Windows and C++ teams. Others are for .Net development from the CLR or Visual Studio teams. There are over 100 tools but so far I’ve only used these:

                ildasm.exe – dumps out the IL bytecode from a .net assembly.
                peverify.exe – verifies that Visual Basic produces valid .net assemblies.
                gacutil.exe – checks which assemblies are in the Global Assembly Cache
                guidgen.exe – generates a new GUID
                consume.exe – consumes memory or CPU or page-file or disk, as a stress test

The Windows SDK blog is a great place to find out about them. Bonus: a .net game called Terrarium!

The “Windows release track of the SDK tools” means that whenever a new version of Windows is released, they release a new version of the SDK tools as a free download. The intention is that this release of the tools is good for targeting this release of Windows. So when Version 6.0 of Windows was released, i.e. Vista, they released the tools in “C:\Program Files\Microsoft SDKs\Windows\v6.0”.

The “Visual Studio release track” means that whenever a new version of Visual Studio is released, it incorporates a version of the SDK tools as well. The intention is that this release of the tools is good for targeting anything that this release of Visual Studio can target. (Because VS can target most prior versions of the .Net framework, it means that the VS version of the SDK tools will also include prior versions of the .Net-specific tools.) For the VS release track, the tools directory name has the suffix “A” after it. So Visual Studio 9 (which came out after v6.0 of Windows) went in “C:\Program Files\Microsoft SDKs\Windows\v6.0A”.

Sharp readers will look at the list above and think “Ahah! In the list above he wrote v7.0A! The “A” means it’s a release of Visual Studio. And the 7.0 means that the next Windows will be released before Visual Studio 2010!” Hold your horses! I’m sorry to disappoint you, but we picked these numbers and directory names for the SDK a long time in advance, there's no guarantee that they're final, and the numbers end up bearing no relation to release schedules.

.Net 4.0 ? 64 bit ?

Visual Studio 2010 will ship with a new version of .Net, version 4.0. But it supports "multi-targeting", where you can target older versions of .Net as well. And so it has to ship .Net3.5 versions of the tools as well as .Net4.0 versions of the tools.

Also, some of the tools have 64bit versions; others are 32-bit only. The 64bit versions are only installed if you have a 64bit operating system. The 64bit versions are found in subdirectories called "...\x64". If the x64 directories does not contain a particular tool, then it means that we haven't shipped a 64bit version of that tool, and you're expected fall back to the 32bit version.

So here's the complete table. Only the shaded rows actually exist. The rest are all potential future versions that Microsoft won't commit to until it ships them.

	Release "A"	Version	32/64bit	CLR
C:\Program Files\Microsoft SDKs\Windows\...
...\v6.0\bin\ildasm.exe	Windows	Vista	x86	.Net3.5
...\v6.0A\bin\ildasm.exe	VS	2008	x86	.Net3.5
...\v6.1\bin\ildasm.exe	Windows	Server2008	x86	.Net3.5
...\v7.0\bin\ildasm.exe	Windows	?? next version	x86	??
...\v7.0\bin\x64\ildasm.exe	Windows	?? next version	x64	??
...\v7.0A\bin\ildasm.exe	VS	2010	x86	.Net3.5
...\v7.0A\bin\x64\ildasm.exe	VS	2010	x64	.Net3.5
...\v7.0A\bin\NETFX 4.0 Tools\ildasm.exe	VS	2010	x86	.Net4.0
...\v7.0A\bin\NETFX 4.0 Tools\x64\ildasm.exe	VS	2010	x64	.Net4.0

C:\Program Files (x86)\Microsoft SDKs\Windows\...
This directory only exists on x64 systems, and it doesn't contain any tools executables. Ignore it.

Registry and folder locations

When you launch the "Visual Studio Command Line Tools" shortcut from the start menu, it sets the PATH environment variable to point to appropriate versions of the tools.

But if you have your own tooling and want to point it to the correct locations, you should use the registry. For release 7.0A of the tools we expect to use these registry keys. Once again, these are not final and might change when we ship Visual Studio 2010.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx35Tools-x64]
"InstallationFolder"="C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\x64\"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx35Tools-x86]
"InstallationFolder"="C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx40Tools-x64]
"InstallationFolder"="C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\x64\"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx40Tools-x86]
"InstallationFolder"="C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-VSHeadersLibs]
"InstallationFolder"="C:\Program Files\Microsoft SDKs\Windows\v7.0A\"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-VSTools]
"InstallationFolder"="C:\Program Files\Microsoft SDKs\Windows\v7.0A\"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-VSWin32Tools]
"InstallationFolder"="C:\Program Files\Microsoft SDKs\Windows\v7.0A\"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDKIntellisenseRefAssys]
"InstallationFolder"="C:\Program Files\Microsoft SDKs\Windows\v7.0A\"

The set of registry keys isn't complete for all past versions of the tools. To compute the path to earlier versions of the tools is awkward. That's because different languages use different paths (e.g. Italian versions of Windows use C:\Programmi) and because users can themselves choose different paths (e.g. E:\Apps).

Notionally in VB you should use My.Computer.FileSystem.SpecialDirectories.ProgramFiles. This returns "C:\Program Files" or its equivalent, and works great most of the time. The only exception is if you're running on a 64bit operating system and your assembly is set to target "CPU:x86" rather than AnyCPU or x64. In this case it returns "C:\Program Files (x86)" or its equivalent, which isn't any use.

Anyway, the Windows SDK tools use non-localized directory and filenames. So once you get to the path to the program files directory, you can append "\Microsoft SDKs\Windows\v6.0\bin\ildasm.exe" or similar to point to a particular tool.