Ziad's Rants and Ramblings

MSIL Injection: Rewrite a non dynamic method at runtime

2009-03-30T03:38:00Z

The .Net Framework contains classes to create dynamic methods at runtime but what about rewriting existing methods at runtime? Is it even possible? It turns out yes, but with some stringent constraints. We are going to create a sample program where we do just this. In our example we are going to create two static methods. The signatures for these methods are below. We will talk more about these later.

ReplaceMethod ( MethodBase source, MethodBase dest );

ReplaceMethod ( byte[] source, MethodBase dest );

Before any code is executed in a program it must be loaded into memory by the loader. Our first step will be to find where in a memory a particular method exists. Lucky for us CLR assemblies are filled with gobs of metadata that has from referenced types and attributes to the names of fields in a type. The CLR metadata can also tell us where we can find the body of a particular method either in memory or on disk.

CLR assemblies are in the Portable Executable or PE format. Besides program code PE files contain other data used by the OS to execute a program. We are just interested in the CLR metadata but an understanding of the PE format is needed for us to get there. All of the structures we need to parse a PE header are defined in windows.h. Unfortunately we do not have time to go in to PE in depth in this article. We will just be going over the parts that are relevant to us getting to the CLR metadata. If you want to learn more about the PE format you should check out the links at the bottom of this article.

Many of the address in a PE file are relative but not to the start of the file, but relative to the layout of the file when it is loaded in memory. When the file is loaded in memory the OS will align the various sections of the file to pages in memory. The addresses are called relative virtual addresses or RVA. These sections are not aligned on the disk though. The PE file Image section can be used to convert an RVA to a disk offset. In order to find our CLR method body we will need to convert a RVA in to an actual address.

How can we identify a CLR assembly? The first thing we should check if the PE header’s DataDirectories field. The field contains a list of IMAGE_DATA_DIRECTORY’s that contain the size and start address for various sections of the assembly. The 15^th entry should tell us the address of the CLR header. If this address is zero then we know right away this is not a CLR assembly. Another thing we look at is the Import Address Tables. If we look at a CLR assembly in dependency walker or a similar tool we will always see one entry in the import table to mscoree.dll for either “_CorDllMain” or “_CorExeMain”. The OS does not know how to execute MSIL, so one of these functions must be imported so the program can yield execution to the CLR vm.

Below is the CLR header structure. As you can see there is lots of interesting data here. We have the CLR version, EntryPointToken, Managed Resources location, etc. We are interested in just the MetaData location. The metadata IMAGE_DATA_DIRECTORY contains the starting location of metadata in the assembly. We are going to jump to that address.

typedef struct IMAGE_COR20_HEADER

{

// Header versioning

DWORD cb;

WORD MajorRuntimeVersion;

WORD MinorRuntimeVersion;

// Symbol table and startup information

IMAGE_DATA_DIRECTORY MetaData;

DWORD Flags;

DWORD EntryPointToken;

// Binding information

IMAGE_DATA_DIRECTORY Resources;

IMAGE_DATA_DIRECTORY StrongNameSignature;

// Regular fixup and binding information

IMAGE_DATA_DIRECTORY CodeManagerTable;

IMAGE_DATA_DIRECTORY VTableFixups;

IMAGE_DATA_DIRECTORY ExportAddressTableJumps;

// Precompiled image info (internal use only - set to zero)

IMAGE_DATA_DIRECTORY ManagedNativeHeader;

} IMAGE_COR20_HEADER, *PIMAGE_COR20_HEADER;

The CLR metadata starts with the header below. This header contains a signature and some version information that we can use to verify we are in the correct spot. The metadata header is followed by a storage header. The storage header contains the number of metadata streams in a file. The storage header is immediately followed by a list of stream headers that contain the stream name, offset from start of metadata section, and size.

typedef struct COR_METADATA_HEADER {

char Signature[4]; // BSJB

WORD MajorVersion;

WORD MinorVersion;

DWORD ExtraData;

DWORD VersionLength;

byte Version[1];

} *PCOR_METADATA_HEADER;

CLR metadata is defined in six streams. These streams are listed below.

1. #~ Optimized Metadata Stream

2. #- Unoptimized - Metadata Stream

3. #US User defined strings

4. #Strings Internal strings (type names, namespaces, etc).

5. #GUID Internal Guids

6. #Blob Internal Blob

The #GUID, #String, and a metadata stream (#~ or #-) will always be present the rest of the streams are optional. The most important of these streams are the optimized #~ and unoptimized #- metadata streams. These are mutually exclusive, only one can exist at a time. These streams contain all the CLR type information like class names, class namespace, method names, class members, etc. CLR metadata is composed in tables very similar to a relational database. The unoptimized version contains several tables that are not available in the optimized version (FieldPtr, ParameterPtr, MethodPtr, etc).

There are 45 different metadata tables in total. These tables are listed below. A CLR assembly will also contain schema information for each table. It will tell us the name of a table, the size of a row, the index column in the table, the number of columns, the names of the columns, etc. Unfortunately I could not find too much information online about these tables. I recommend you check out Serge Lidin’s .NET IL Assembler book which goes over these tables in detail.

Module	TypeRef	TypeDef	FieldPtr	Field
MethodPtr	Method	ParamPtr	Param	InterfaceImpl
MemberRef	Constant	CustomAttribute	FieldMarshal	DeclSecurity
ClassLayout	FieldLayout	StandAloneSig	EventMap	EventPtr
Event	PropertyMap	PropertyPtr	Property	MethodSemantics
MethodImpl	ModuleRef	TypeSpec	ImplMap	FieldRVA
ENCLog	ENCMap	Assembly	AssemblyProcessor	AssemblyOS
AssemblyRef	AssemblyRefProcessor	AssemblyRefOS	File	ExportedType
ManifestResource	NestedClass	GenericParam	MethodSpec	GenericParamConstraint

At first I was going to manually parse the assembly metadata but that was too much work. I stumbled across the unmanaged metadata api. You can find more information about the metadata api here Metadata (Unmanaged API Reference). Here are going to be using the IMetadataTables interface to find out where a method exists in an assembly. The method below takes a MethodBase object and returns the starting address in memory of the method body. We use the MetadataToken property of the MethodBase to get the row number of the method in the method table. Then we read the RVA column and add it to the base address to get starting location.

static byte* GetMethodStart(MethodBase^ method)

{

PIMAGE_NT_HEADERS header;

LPVOID imageSectionStart;

CComPtr<IMetaDataDispenserEx> metaDataDispenser;

CComPtr<IMetaDataTables> metaDataTables;

CComPtr<IUnknown> unknown;

COR_METADATA_TABLE_INFO tableInfo;

HRESULT hr;

// Get the module base address in memory.

byte* moduleStart = GetModuleAddress(method->DeclaringType->Assembly);

if ( moduleStart == 0 )

{

throw gcnew Exception("Module not found!");

}

// Read the nt hewaders.

ReadNtHeader(IntPtr((void*)moduleStart),&header,&imageSectionStart);

// Create the metadata dispenser

hr = CoCreateInstance(

CLSID_CorMetaDataDispenser,

NULL, CLSCTX_INPROC_SERVER,

IID_IMetaDataDispenserEx,

(void**)&metaDataDispenser

);

if ( FAILED(hr) )

{

throw gcnew Exception("Failed to create metadata dispenser");

}

// Read the metadata start location.

byte* metaDataStart;

int metadataSize;

ReadMetaDataStart(header,moduleStart,&metaDataStart,&metadataSize);

// Validate the start location.

PCOR_METADATA_HEADER metaDataHeader = (PCOR_METADATA_HEADER)metaDataStart;

if ( memcmp(metaDataHeader->Signature,"BSJB",4) != 0 )

{

throw gcnew Exception("Invalid metadata header");

}

// Open memeory scope and get the IMetadataTables interface

hr = metaDataDispenser->OpenScopeOnMemory(

metaDataStart,

metadataSize,

CorOpenFlags::ofReadWriteMask,

IID_IMetaDataTables,

&unknown

);

if ( FAILED(hr) )

{

throw gcnew Exception("Open scope memory failed");

}

if ( FAILED(unknown->QueryInterface(IID_IMetaDataTables, (void**)&metaDataTables)) )

{

throw gcnew Exception("failed to get the metadata table");

}

// Get the row in the metadata table

PCOR_METADATA_METHOD_ROW row;

int rowIndex = 0xFFFFFF & method->MetadataToken;

tableInfo = ReadTableInfo(metaDataTables,CorTable::CorTable_Method);

hr = metaDataTables->GetRow(tableInfo.ixTbl,rowIndex,(void**)&row);

if ( FAILED(hr) )

{

throw gcnew Exception("failed to read metadata row");

}

if ( row->Rva == 0 )

{

throw gcnew Exception("method has no body.");

}

return moduleStart + row->Rva;

};

Before the start of the method body there is a method header. The method header tells us how big the method is in bytes, max stack size, a signature for local vars, and some flags to tell us if we are going to have a SEH table following the method body, etc. There are two types of method headers, a tiny and fat header. A tiny header is only 1 byte and can be used when a method does not have any local variables and the methody body byte size is less than 64 bytes. The first two bits in the first byte of the header will tell us of it is fat or tiny. Below are the tiny and fat structures from Cor.h.

/* Used when the method is tiny (< 64 bytes), and there are no local vars */

typedef struct IMAGE_COR_ILMETHOD_TINY

{

BYTE Flags_CodeSize;

} IMAGE_COR_ILMETHOD_TINY;

/************************************/

// This strucuture is the 'fat' layout, where no compression is attempted.

// Note that this structure can be added on at the end, thus making it extensible

typedef struct IMAGE_COR_ILMETHOD_FAT

{

unsigned Flags : 12; // Flags

unsigned Size : 4; // size in DWords of this structure (currently 3)

unsigned MaxStack : 16; // maximum number of items (I4, I, I8, obj ...), on the operand stack

DWORD CodeSize; // size of the code

mdSignature LocalVarSigTok; // token that indicates the signature of the local vars (0 means none)

} IMAGE_COR_ILMETHOD_FAT;

typedef union IMAGE_COR_ILMETHOD

{

IMAGE_COR_ILMETHOD_TINY Tiny;

IMAGE_COR_ILMETHOD_FAT Fat;

} IMAGE_COR_ILMETHOD;

Our replace method is listed below. The first we will need to do is get a process handle with the PROCESS_VM_WRITE and PROCESS_VM_OPERATION access so we can write to program memory. You can learn more about these permissions here Process Security and Access Rights. Next we run into our first constraint. The source method must be smaller or equal in bytes to the destination method. We will talk more about this later and maybe see if we can fix in a future article. At first I just tried to write the source method over the destination method without worrying about size difference but this did not seem to work. Addig some nop instructions to the start of the method seemed to resolve the issue.

static void ReplaceMethod(byte* source, byte* dest)

{

HANDLE processHandle;

try

{

// Get token with write process memory access

processHandle = GetElevatedProcessHandle();

// Get the code size.

int sourceCodeSize = GetMethodCodeSize(source), destCodeSize = GetMethodCodeSize(dest);

bool isSourceFat = HasFatHeader(source), isDestFat = HasFatHeader(dest);

// Get the headers.

IMAGE_COR_ILMETHOD* sourceMethodHeader = (IMAGE_COR_ILMETHOD*)source;

IMAGE_COR_ILMETHOD* destMethodHeader = (IMAGE_COR_ILMETHOD*)dest;

// See if we have enough space

if ( sourceCodeSize + (isDestFat || isSourceFat? sizeof(IMAGE_COR_ILMETHOD_FAT):sizeof(IMAGE_COR_ILMETHOD_TINY)) >

destCodeSize + (destCodeSize? sizeof(IMAGE_COR_ILMETHOD_FAT):sizeof(IMAGE_COR_ILMETHOD_TINY)))

{

throw gcnew Exception("Cannot replace a method if the destination is less than the source !");

}

SIZE_T bytesWrote = 0;

int size = 0, sizeDiff = 0;;

bool useFat, result;

byte* buffer;

// Get size diff

if ( destCodeSize > sourceCodeSize )

{

destCodeSize - sourceCodeSize;

}

useFat = isDestFat || isSourceFat || ((sizeDiff + sourceCodeSize) > MAX_TINY_METHOD_SIZE);

// Write the header

try

{

// If source or dest are fat then use fat.

if ( useFat)

{

IMAGE_COR_ILMETHOD_FAT* fat;

size = sizeof(IMAGE_COR_ILMETHOD_FAT);

buffer = new byte[size];

fat = (IMAGE_COR_ILMETHOD_FAT*) buffer;

FillHeader(fat, sourceMethodHeader);

fat->CodeSize += sizeDiff;

}

else

{

size = 1;

buffer = new byte[size];

*buffer = (byte)(sourceMethodHeader->Tiny.Flags_CodeSize);

IMAGE_COR_ILMETHOD_TINY* tiny = (IMAGE_COR_ILMETHOD_TINY*)buffer;

tiny->Flags_CodeSize = (tiny->Flags_CodeSize & 0x3) |

(((tiny->Flags_CodeSize >> 2) + sizeDiff) << 2);

}

result = WriteProcessMemory(

processHandle,

dest,

(LPCVOID)buffer,

size,

&bytesWrote

);

// Move to start of il.

dest += bytesWrote;

source += isSourceFat ? sizeof(IMAGE_COR_ILMETHOD_FAT) : 1;

}

finally

{

if ( buffer != 0 )

{

delete buffer;

}

// Add padding

if ( sizeDiff > 0 )

{

try

{

// Create buffer filled with nop instruction and write.

buffer = new byte[sizeDiff];

ZeroMemory(buffer,sizeDiff);

result = WriteProcessMemory(

processHandle,

dest,

(LPCVOID)buffer,

sizeDiff,

&bytesWrote

);

dest+=sizeDiff;

}

finally

{

if ( buffer != NULL )

{

delete buffer;

}

//replace the method il.

result = WriteProcessMemory(

processHandle,

dest,

(LPCVOID)source,

sourceCodeSize,

&bytesWrote

);

}

finally

{

// Close the process handle.

if ( processHandle != 0 )

{

CloseHandle(processHandle);

}

};

Lets test our replace method now. Each of these test methods will just print write a string to the console. The first method shows the JIT in action running the method and keeping a cached compiled copy. Even after we replace the IL the method gives us the same results as the first time we ran it. Our call to TestFour works correctly and we see “TestThree” printed to the console.

// Get method handles for the test methods.

MethodBase[] methods = new MethodBase[]

{

typeof(TestClass).GetMethod("TestOne",BindingFlags.Static|BindingFlags.Public),

typeof(TestClass).GetMethod("TestTwo",BindingFlags.Static|BindingFlags.Public),

typeof(TestClass).GetMethod("TestThree",BindingFlags.Static|BindingFlags.Public),

typeof(TestClass).GetMethod("TestFour",BindingFlags.Static|BindingFlags.Public),

typeof(TestClass).GetMethod("TestFive",BindingFlags.Static|BindingFlags.Public),

typeof(TestClass).GetMethod("TestSix",BindingFlags.Static|BindingFlags.Public),

typeof(TestClass).GetMethod("TestSeven",BindingFlags.Static|BindingFlags.Public)

};

// Call TestOne

TestClass.TestOne();

// Replace test one with test two.

MethodReplacer.ReplaceMethod(methods[1], methods[0]);

// Call test one again. Same result. JIT already compiled.

TestClass.TestOne();

// Replace TestFour with TestThree.

MethodReplacer.ReplaceMethod(methods[2], methods[3]);

// Call TestFour. JIT has not yet cached this method, we get expected results.

TestClass.TestFour();

Why not just allocate some new memory, write some IL, and the update the RVA? This way we don’t have to worry about any size constraints. This was my first approach but it did not work. I suspect the issue is with the IMetadataTables->GetRow() function. The address returned for the row does not appear to be in the same address space as the assembly in memory. Maybe in the future I will try to manually locate the row, update the RVA, and see if I get different results. If we can write to the metadata tables and the VM recognizes our changes then we can do all kinds of interested things, like maybe trick the vm into running some x86 assembly code instead of IL.

Rewriting methods at runtime does not seem to work to well. We have to deal with the size issues and JIT keeping a compiled copy of the method after it invoked for the first time. Let's not forget about NGEN too.

Cecil is a library developed by Mono that allows one to inspect and rewrite assemblies before they are loaded them into memory. I have not had a chance to place with Cecil yet but I hear nothing but good things about it. If you want to play around with metadata and rewrite an assembly before it is loaded I suggest you give Cecil a try.

Peace

An In-Depth Look into the Win32 Portable Executable File Format

Microsoft Portable Executable and Common Object File Format Specification

Metadata (Unmanaged API Reference)

Cecil

New Open source Console Emulator in C#

2009-02-23T02:24:00Z

Been a while seen I had a change to get on here. I have been really busy at work. I will try to start posting more.

I started a new open source project today on codeplex. A couple years ago I wrote a semi working nintendo emulator in C#. I got it to the point where it boot and ran several games. I decided I want to finish it up especially if I can find some people to help me ;) Today I made some optimizations to the memory read code and wrote a directdraw graphics render. Increased ~ fps from 40 to 120-175+ on my laptop :) Check it out http://www.codeplex.com/Nintender

I will have some new posts this week. Peace

Ziad

Cuda and Stream Programming - The next big thing?

2009-01-04T02:13:00Z

Today while researching which type of video card to buy I kept seeing the word CUDA in the specs but I did not know what it meant. Nvidia in the last year or so bought a company called PhysX. PhysX created a physics runtime and a special add in card that the runtime uses instead of the CPU. Now that Nvidia owns PhysX and CUDA enabled video cards are also supported by the PhysX runtime. Games that use the PhysX runtime off load some of their physics math to the PhysX runtime in order to do neat effects in games like realistic water, realistic cloth effects, smoke/gas effects, etc. Read Nvida's page for more info http://www.nvidia.com/object/nvidia_physx.html.

CUDA stands for Compute Unified Device Architecture. CUDA is a parallel programming focused runtime/api that allows developers to write programs that can partly be ran on a their GPU. Modern Nvidia gpus have hundreds of stream processors. The new gtx 295 has a whooping 480 stream processors ! These programs need not be graphics specific. CUDA unleashes a ton of power (4 terraflops on high end Nvidia solutions) to us developers. CUDA/stream processing puts super computer power in to the our hands at a fraction of the cost.

Lets talk a little about stream processing. In stream programming the stream is the data set you are working on. The stream processors perform the same operation on each element or thread in the stream in parallel. You can kinda of think of stream programming as the relative of SIMD (Single Instruction Multiple Dataset - mmx, 3dnow, sse, etc). Nvidia calls their technology Single Instruction Multiple Thread or SIMT. With SIMD the size of the dataset/vector is fixed. Grouping data so it can be executed in parallel can also be difficult as well. With SIMT just need to write code that can be executed in parallel and CUDA takes care of the rest. Efficient CUDA code will easily scale if more processors are available.

So what is the catch? Programming in CUDA/Stream Programming is completely different experience than most developers are used to. Developers need to break the code in to the simplest operations that can be run in parallel without any dependencies on other threads. Not every program can take advantage of CUDA, on consumer computers multimedia programs (image processing, 3d, sound, etc) and maybe encryption? Only operations that deal with large amounts of data and are normally CPU intensive will see improvements. Data compression could be faster but it is very I/O intense and the data has to be read in and of the video card (I would love to see 7zip running on CUDA since it is very very CPU intensive).

If you look at Nvidia CUDA user submitted demo site you see some really, really interesting applications claiming huge speed increases. Math libraries claiming 40x speed increases over CPU (not sure which gpu and CPU they are comparing), Password crackers claiming 50x speed increase, quick sort 10x, googles MapReduce algorithm 100x speed increase, etc. http://www.nvidia.com/object/cuda_home.html

CUDA program are written in a special version of C. Developers create special functions called kernels. When invoked a kernel function is ran X amount of time on N number of threads. In side functions you use a special variable called threadIdx to identify which thread you are in. threadIdx is vector with an x,y,z component I guess this for doing 3d math which is what these cards are made for ;) The threadIdx variable is very useful when writing your kernels. Threads are divided in to groups called blocks. Threads have access to three different kinds of memory. There own private thread local storage, block storage that is in scope for the thread block, and global memory which is accessible by all threads. Read the sdk documentation for more info.

Unfortunately I really don't have to much experience with C or C++. A want to make a simple hello world example but I don't really think their is an equivalent in CUDA. We need to create something that will make use of the massive amount of parallel processors a gpu has. I am not a graphics developer, math expert, or anything like that so I decided to make something really simple.

It is possible to work with the CUDA driver api using C#. You can work with the nvcuda.dll using P/invoke but the driver api does not support emulation so you will need an actual card to begin developing (I just ordered mine). You write your CUDA code and compile it with nvcc to generate a cubin file. You can then use the driver api to load the cubin file, execute kernels, read/write to processor memory, etc. There is a third party wrapper for nvcuda developed by the Company for Advanced Supercomputing Solutions called Cuda.net. Cuda.net is free so we will be using it in our example code.

When I get my new video card next week we will write some really really simple cuda examples and compare performance results. Peace.

Creating a WCF IDispatchMessageInspector

2008-12-29T12:30:00Z

Sorry I have not had a chance to post in a while. I had a really bad flu and then there was a snow storm where I live. Everything thing is cleared up now and I will be going to work tomorrow.

I am still a little new to wcf. Today I was presented with an interesting task. I need the ability to read a custom http header on each request to a wcf service and deny the user access depending on the value in the header. No problem. Add a reference to System.ServiceModel.Web, turn on asp.net compatibility mode, and use the WebOperationContext to read the header on request and set the status code on the response.

            string myHeader = WebOperationContext.Current.IncomingRequest.Headers["MyHeader"];
            if (!Authenicate(myHeader))
            {
                WebOperationContext.Current.OutgoingResponse.StatusCode = HttpStatusCode.Forbidden;
                return;
            }

I have to add this code to every service method though. I want something modular that I can just to any service with no code. I want to add this ability to any service just by changing some configuration. I probably should research the Wcf authentication apis to see how I can hook in. Unfortunately I did not see how yet but I will research more later on this week.

I did notice a interface called IDispatchMessageInspector. I thought maybe I could use this to hook in to each request and validate the user. To make this completely configuration based I need to create a service behavior that I can add in the config file. The library has some good document on how to create a IDispatchMessageInspector and IServiceBehavior class so we can expose it in the config.

My first step was creating a class that implements both IDispatchMessageInspector and IServiceBehavior. I stubbed out all the interface methods, implemented IServiceBehavior.ApplyDispatchBehavior like in the library example, and added the necessary configuration. It looked like I would have a small issue. I wanted to pass in some configuration values to the service behavior. I added the configuration below and was treated with an error :\

<system.serviceModel>
<behaviors>
   <serviceBehaviors>
    <behavior name="SampleProviderService.SampleServiceBehavior">
      <serviceMetadata httpGetEnabled="true" />
     <serviceDebug includeExceptionDetailInFaults="false" />
      <TestInterceptor />
    </behavior>
   </serviceBehaviors>
</behaviors>
   <extensions>
     <behaviorExtensions>
       <add
         name="TestInterceptor"
         type="SampleProviderService.TestRequestInterceptor, SampleProviderService"
        />
     </behaviorExtensions>
   </extensions>
<services>
   <service behaviorConfiguration="SampleProviderService.SampleServiceBehavior"
    name="SampleProviderService.SampleService">
    <endpoint address="" binding="wsHttpBinding" contract="SampleProviderService.ISampleService">
    </endpoint>
   </service>
</services>
</system.serviceModel>

The error message said the type needed to inherit from BehaviorExtensionElement. I must have missed this part in the library documentation. The BehaviorExtensionElement class is very straight forward. It is an abstract class that has a Type property and a create method. The type property returns the type of the class that implements IServiceBehavior and the Create method creates the service behavior. The BehaviorExtensionElement class also inherits from ConfigurationElement so I can specify my own configuration. I created a new class that inherits from BehaviorExtensionElement but I still got an error. It looks like you must specify the complete fqn with publictokenkey, version, and culture of the assembly when adding behaviorExtensions. Also make sure you don't have any line returns and a single space after every comma. After making this change everything was working.

<system.serviceModel>
<behaviors>
   <serviceBehaviors>
    <behavior name="SampleProviderService.SampleServiceBehavior">
      <serviceMetadata httpGetEnabled="true" />
     <serviceDebug includeExceptionDetailInFaults="false" />
      <TestInterceptor providerName="testProvider" />
    </behavior>
   </serviceBehaviors>
</behaviors>
   <extensions>
     <behaviorExtensions>
       <add
         name="TestInterceptor"
         type="SampleProviderService.RequestInterceptorBehaviorExtension, SampleProviderService, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
        />
     </behaviorExtensions>
   </extensions>
<services>
   <service behaviorConfiguration="SampleProviderService.SampleServiceBehavior"
    name="SampleProviderService.SampleService">
    <endpoint address="" binding="wsHttpBinding" contract="SampleProviderService.ISampleService">
    </endpoint>
   </service>
</services>
</system.serviceModel>

Now that we our wcf interceptor we need to find out how to read the http headers.   The IDispatchMessageInterceptor.AfterReceiveRequest has a Message object parameter that has a Headers property. It looks like these are soap headers though. I put a break point in the AfterReceiveRequest method and found what I was looking for. The Message class has a dictionary property called Properties. In the dictionary I saw a key called 'httpRequest' and an object called HttpRequestMessageProperty. The HttpRequestMessageProperty gave me access to all the http headers amoung other things.

We now need the ability to cancel the request and send the client a response with a error http status code. Since the AfterReceiveRequest takes the Message property by ref I thought maybe we can replace the message or set it to null. Setting the message to null worked great and it caused us to jump right to the BeforeSendReply without invoking the service method. How will I know there was an error though when the BeforeSendReply method is invoked? Luckily we have the ability to pass some state from AfterReceiveRequest in to this method. AfterReceiveRequest returns an object and that object is passed into the BeforeSendReply method.

        public object AfterReceiveRequest(ref Message request,
            IClientChannel channel, InstanceContext instanceContext)
        {
            object correlationState = null;
            HttpRequestMessageProperty requestMessage = request.Properties["httpRequest"] as HttpRequestMessageProperty;
            if (request == null)
            {
                throw new InvalidOperationException("HttpRequestMessageProperty 'httpRequest' property not found !");
            }
            string authHeader = requestMessage.Headers["MyHeader"];
            if (string.IsNullOrEmpty(authHeader)|| !Authenicate(authHeader))
            {
                WcfErrorResponseData error = new WcfErrorResponseData(HttpStatusCode.Forbidden);
                correlationState = error;
                request = null;
            }

            return correlationState;
        }

        public void BeforeSendReply(ref Message reply, object correlationState)
        {
            WcfErrorResponseData error = correlationState as WcfErrorResponseData;
            if (error != null)
            {
                HttpResponseMessageProperty responseProperty = new HttpResponseMessageProperty();
                reply.Properties["httpResponse"] = responseProperty;
                responseProperty.StatusCode = error.StatusCode;

                IList<KeyValuePair<string, string>> headers = error.Headers;
                if (headers != null)
                {
                    for (int i = 0; i < headers.Count; i++)
                    {
                        responseProperty.Headers.Add(headers[i].Key, headers[i].Value);
                    }
                }
            }
        }

That handles everything. Attached is the sample code. Later on this week I look at wcf security apis to see if there is a better to hook in through there. I will also look at what client needs to do to do to consume these services. I am new to wcf so I am sure if this is the best way to do this. If anyone has any ideas or examples please send me an email. Until next time peace.

Update : Example config had the IServerBehavior type and not the BehaviorExtension type. Fixed.

Create dynamic WCF Clients without any configuration/service references

2008-12-16T00:51:00Z

The other day I wanted to do some simple perf tests against a bunch of different wcf services that I am working on. I wrote a little test tool to do this but I had some problems with my test tool. My test application was very tightly coupled to the services. For each service I had to add a service reference. The ide auto generates some code for each service reference so each time something changed I had to update the service reference in the ide. A recompile for a service update :( On top of that the services are dependent on a ton of configuration setting in the app config.

I wanted the ability to test any service without any configuration or dependencies on ide generated code. The test application had to be generic enough to work with any wcf service. I wanted the application to work the following way : user picks an assembly, we scan the assembly for interfaces that have a service contract attribute, look for types that implement the contract interface, give the user a selection of the services found, user scripts up service calls to single or multiple services, clicks run, we spawn the services on a random endpoint on the localhost, create the client, and start hammering the service. No configuration or ide generated code required is the goal.

I was able to get everything work in a couple hours but the code is little bit sloppy :( I am not going to share the entire application but only the code that creates the wcf client. Maybe later if I have a chance to clean up the rest of the code I will post it here too ;)

In my application since the service and client existed in the same process we already have the service contract type for the client. This is not exactly dynamic. Next time we will look at generating a wcf client based off a wsdl. In this example though the contract must already be know.

Where should I start? I started by adding a service reference the normal way and observing what the ide does. Bunch of stuff in the config and some auto generated code. The ide creates a class called ClientBase<> which takes a generic type parameter called TChannel. The parameter needs to be an interface that has some service method attributes. ClientBase<T> implements the TChannel interface and wraps calls to a TChannel Channel property. We can easily generate this class using reflection.emit.

To do this I made a little utility class with a static method that takes the contract type, binding, and endpoint as parameters and returns an object that can be used to communicate with the service. The signature for our helper method is below.

        public static ICommunicationObject GetClient(Type contractType, Binding binding,
                EndpointAddress address)

Lets look at some of the code we will use to generate the client. We are going to create a new type that inherits from ClientBase<TChannel> with TChannel being the contractType that was passed in. We will loop through the methods in the interface and add them to our new type. The new methods will simply wrap calls made to the Channel property. The ChannelBase has several constructors but we will only be supporting the constructor that takes a binding and endpoint address. This is all we really need. The code for generating ClientBase<TChannel> in below.

ILGenerator ilGen = null;

            // Create the type and add the interface.
            TypeBuilder typeBuilder = builder.DefineType("TestClient_" + contractType.Name, TypeAttributes.Class | TypeAttributes.Public, clientBaseType);
            typeBuilder.AddInterfaceImplementation(contractType);

            ConstructorInfo constructorInfo = clientBaseType.GetConstructor(BindingFlags.Instance | BindingFlags.NonPublic, null, new Type[] { typeof(Binding), typeof(EndpointAddress) }, null);
            ConstructorBuilder constructorBuilder = typeBuilder.DefineConstructor(MethodAttributes.Public, CallingConventions.HasThis, new Type[] { typeof(Binding), typeof(EndpointAddress) });
            ilGen = constructorBuilder.GetILGenerator();
            ilGen.Emit(OpCodes.Ldarg_0);
            ilGen.Emit(OpCodes.Ldarg_1);
            ilGen.Emit(OpCodes.Ldarg_2);
            ilGen.Emit(OpCodes.Callvirt, constructorInfo);
            ilGen.Emit(OpCodes.Ret);

            // Get the interface methods
            MethodInfo[] interfaceMethods = contractType.GetMethods();
            MethodInfo channelGetMethod = clientBaseType.GetProperty("Channel", BindingFlags.Instance | BindingFlags.NonPublic).GetGetMethod(true);
            MethodBuilder methodBuilder = null;
            Type[] methodParmeterTypes = null;

            // Loop through all the methods.
            for (int i = 0; i < interfaceMethods.Length; i++)
            {
                // Get the types
                methodParmeterTypes = Array.ConvertAll<ParameterInfo, Type>(interfaceMethods[i].GetParameters(), p => p.ParameterType);

                // Create method in our wrapper type.
                methodBuilder = typeBuilder.DefineMethod(
                    interfaceMethods[i].Name,
                    MethodAttributes.Public | MethodAttributes.Virtual,
                    CallingConventions.HasThis,
                    interfaceMethods[i].ReturnType,
                    methodParmeterTypes
                        );
                ilGen = methodBuilder.GetILGenerator();

                // Push the channel property value onto the stack
                ilGen.Emit(OpCodes.Ldarg_0);
                ilGen.Emit(OpCodes.Callvirt, channelGetMethod);

                // Load the method parameters on the stack
                for (int x = 0; x < methodParmeterTypes.Length; x++)
                {
                    switch (x + 1)
                    {
                        case 1:
                            ilGen.Emit(OpCodes.Ldarg_1);
                            break;
                        case 2:
                            ilGen.Emit(OpCodes.Ldarg_2);
                            break;
                        case 3:
                            ilGen.Emit(OpCodes.Ldarg_3);
                            break;
                        default:
                            ilGen.Emit(OpCodes.Ldarg_S, x);
                            break;
                    }
                }

                // Call the method.
                ilGen.EmitCall(OpCodes.Callvirt, interfaceMethods[i], null);
                // Return the value
                ilGen.Emit(OpCodes.Ret);

// Mark the method.
typeBuilder.DefineMethodOverride(methodBuilder, interfaceMethods[i]);

}

To speed things up we will use some of the techniques we talked about in previous posts and create a factory object to create these clients. Our factory interface is below. We will gen some il for the factories and store them in a hash table using the contract type as the hash key.

    public interface IClientFactory
    {
        object Create(Binding binding, EndpointAddress address);
    }

This il for our factory is very simple. We create a new type that implements IClientFactory and have it return a new instance of the new ClientBase<> type we just defined.

            // Create the type and add the interface
            TypeBuilder typeBuilder = modBuilder.DefineType(wrapperType.Name + "Factory", TypeAttributes.Public | TypeAttributes.Class);
            typeBuilder.AddInterfaceImplementation(typeof(IClientFactory));

            // Create the method.
            MethodBuilder methodBuilder =
            typeBuilder.DefineMethod("Create", MethodAttributes.Public | MethodAttributes.Virtual,
                CallingConventions.HasThis, typeof(object), new Type[] { typeof(Binding), typeof(EndpointAddress) });

            // Get the constructor for our new wrapper type
            ConstructorInfo info = wrapperType.GetConstructor(new Type[] { typeof(Binding), typeof(EndpointAddress) });

            // Gen the il
            ILGenerator gen = methodBuilder.GetILGenerator();
            gen.Emit(OpCodes.Ldarg_1);
            gen.Emit(OpCodes.Ldarg_2);
            gen.Emit(OpCodes.Newobj, info);
            gen.Emit(OpCodes.Ret);

            // Mark the method.
            MethodInfo createMethod = typeof(IClientFactory).GetMethod("Create");
            typeBuilder.DefineMethodOverride(methodBuilder, createMethod);

// Return the type.
return typeBuilder.CreateType();

The code for WCF client factory is attached. I am sorry I did not have a chance to make a sample application this time around. The code should be self explanatory though. Next time we will look at generating on WCF client off a wsdl. Peace :)

Performance when creating objects using generic new constraint

2008-12-14T05:29:00Z

I got an email asking about the performance when creating objects using the generic new constraint. You can learn more about the generic new constraint here http://msdn.microsoft.com/en-us/library/sd2w2ew5.aspx and here http://msdn.microsoft.com/en-us/library/d5x73970.aspx. The generic new constraint is used to specify a generic type parameter must implement a default public constructor. You can then create new instances of the generic type in your generic class. Here is the sample from the MSDN documentation.

class ItemFactory<T> where T : new()
{
    public T GetNewItem()
    {
        return new T();
    }
}

What happens under the covers though? First lets try a quick performance test. We are going to use the code from the last post. We are going to add an additional test though using the generic new constraint to create the objects. We will create an extremely simple GenericNewFactory class to create the objects. The code for that is below.

    public static class GenericNewFactory
    {
        private static Dictionary<Type, IFactory> _cache = new Dictionary<Type, IFactory>();

        public static object CreateObject(Type type)
        {
            return CreateFactory(type).Create();
        }

        public static IFactory CreateFactory(Type type)
        {
            IFactory factory = null;
            if (_cache.TryGetValue(type, out factory))
            {
                return factory;
            }
            if (type.GetConstructor(Type.EmptyTypes) == null)
            {
                throw new ArgumentException("type", "The type must have a public default constructor.");
            }
            Type genericFactoryType = typeof(TestFactory<>).MakeGenericType(type);
            factory = Activator.CreateInstance(genericFactoryType) as IFactory;
            _cache[type] = factory;
            return factory;
        }
        private class TestFactory<T> : IFactory
            where T : class, new()
        {
            public TestFactory()
            {
            }

            public T Create()
            {
                return new T();
            }

            object IFactory.Create()
            {
                return Create();
            }
        }

        public interface IFactory
        {
            object Create();
        }
    }

Lets run the tests and see what happens. I upped the iterations to 10 million. The results are below.

Running 10000000 interations of creation test.
Direct Call                     00:00:00.5320932
Delegate Wrapper                00:00:00.8127212
Generic New                     00:00:20.2164442
Activator.CreateInstance        00:00:43.3707797

Generic new is faster then Activator.CreateInstance but still way slower then direct call. Let open up our test code in reflector. First thing we notice is new T() is replaced with a call to Activator.CreateInstance<T>(). Why don't we see a newobj call here instead? The newobj instruction has a operand type of InlineMethod and requires the handle for the constructor. I guess the compiler cannot resolve the constructor since the generic type is unknown till jit time maybe?? Why is Activator.CreateInstance<T> faster than Activator.CreateInstance(). I am not really sure. I will try to find out and we will talk about it in another post. Peace :)

Reflection : Fast Object Creation

2008-12-13T10:47:00Z

Reflection is a powerful api in the .net framework. Reflection allow us to inspect types at runtime, dynamically invoke methods, create objects, etc. Using reflection developers can make highly modular applications where functionality can be replaced and defined at runtime.

One of the drawbacks of reflection is performance. Using reflection to set a property, invoke a method, create an object, etc is several times slower than normal code. There are ways to get around these performance issues though. If you have not already done so I recommend that you read the MSDN Magazine article Reflection : Dodge Common Performance Pitfalls to Craft Speedy Applications. The article goes over some really good techniques on how to get your late-bound code running as fast as possible.

In most applications developers will use some type of strongly typed "contract" to talk to the late-bound object. This contract could be an interface, abstract class, or delegate. The MSDN article goes over this and talks about how to use this technique to mitigate most of the performance issues. One thing that is not covered is object creation. There is no way in .Net to get a strongly typed contract for an object constructor. To create a late bound object at runtime you must call Activator.CreateInstance, ConstructorInfo.Invoke, etc. Most application are not trying to create millions of late-bound objects every second. It's nice to know we have the ability though if we need it :)

What if the .net framework had a way to expose a strongly typed contract to a constructor? Here is how I imagine it would work. There would be a new method on the ConstructorInfo called CreateDelegate. This method will take the delegate type and return a delegate that can be used to create the object. Below is a sample.

            ConstructorInfo constructorInfo = typeof(MyClass).GetConstructor(new Type[]{typeof(int)});
            Func<int,MyClass> del = constructorInfo.CreateDelegate(typeof(Func<int,MyClass>)) as Func<int,MyClass>;
            object j = del(1);

This feature does not exist in the .net framework but we can add it ourselves. A ConstructorInfo inherits from MethodBase. Can we just bind the constructor directly to a delegate? I don't think so. When a new object is created the newobj opcode is used opposed call, calli, and callvirt opcodes. A constructor does not return the object. The object or "this" is a hidden argument that is passed in to the constructor and every non static method. The newobj opcode/new operator creates the object, then calls the constructor to initialize the object, and returns the result to the caller. One opcode for two separate things per say. Let try to bind to a delegate to a constructor and see what happens. There is a internal CreateDelegate method on the delegate class that takes a runtime method handle that we will be using. The sample code is below.

class Program
{

static void Main(string[] args)
{

            // Create the object
            MyClass testObj = new MyClass(1);
            // Write the value.
            Console.WriteLine("Class Value {0}", testObj.Value);
            // Get a delegate to the constructor.
            ConstructorInfo constructorInfo = typeof(MyClass).GetConstructor(new Type[]{typeof(int)});
            Action<int> del = constructorInfo.BindDelegate(typeof(Action<int>), testObj) as Action<int>;
            // Call the constructor with a new value
            del(2);
            Console.WriteLine("Class Value {0}", testObj.Value);
            // Call the constructor with a new value
            del(3);
            Console.WriteLine("Class Value {0}", testObj.Value);
            Console.ReadLine();
        }
        public class MyClass
        {
            private readonly int _value;
            public MyClass (int value)
            {
                Console.WriteLine("Constructor Invoke Current Value {0}", _value);
                _value=value;
                Console.WriteLine("Constructor Invoke New Value {0}", _value);
            }
            public int Value
            {
                get
                {
                    return _value;
                }
            }
        }
    }

    public static class ConstructorHelper
    {
        private delegate Delegate CreateDelegateHandler(Type type, object target, RuntimeMethodHandle handle);
        private static CreateDelegateHandler _createDelegate;

        static ConstructorHelper()
        {
            MethodInfo methodInfo =
                typeof(Delegate).GetMethod("CreateDelegate", BindingFlags.Static|BindingFlags.NonPublic, null,
                new Type[] { typeof(Type), typeof(object), typeof(RuntimeMethodHandle) }, null);
            _createDelegate = Delegate.CreateDelegate(typeof(CreateDelegateHandler), methodInfo) as CreateDelegateHandler;
        }
        public static Delegate BindDelegate(this ConstructorInfo constructor,
            Type delegateType, object obj)
        {
            return _createDelegate(delegateType, obj, constructor.MethodHandle);
        }
    }

A couple interesting things happening here. First off the code works. That really surprised me. The constructor is invoked 3 times on the same object. The next interesting thing is we are able to write to a readonly member variable several times. This is getting us off track though. Lets go back to creating a strongly typed contract for a constructor.

Our next method is using the reflection emit to create a new method that will create and return the object. We can bind a delegate to this new method and we are done :) We have a strongly typed contract that we can use to invoke our constructor not directly (the new method calls the constructor) but still several times faster than the other techniques.

        public static Delegate CreateDelegate(this ConstructorInfo constructor,
            Type delegateType)
        {
            if (constructor == null)
            {
                throw new ArgumentNullException("constructor");
            }
            if (delegateType == null)
            {
                throw new ArgumentNullException("delegateType");
            }

            // Validate the delegate return type
            MethodInfo delMethod = delegateType.GetMethod("Invoke");
            if (delMethod.ReturnType != constructor.DeclaringType)
            {
                throw new InvalidOperationException("The return type of the delegate must match the constructors delclaring type");
            }

            // Validate the signatures
            ParameterInfo[] delParams = delMethod.GetParameters();
            ParameterInfo[] constructorParam = constructor.GetParameters();
            if (delParams.Length != constructorParam.Length)
            {
                throw new InvalidOperationException("The delegate signature does not match that of the constructor");
            }
            for (int i = 0; i < delParams.Length; i++)
            {
                if (delParams[i].ParameterType != constructorParam[i].ParameterType || // Probably other things we should check ??
                    delParams[i].IsOut)
                {
                    throw new InvalidOperationException("The delegate signature does not match that of the constructor");
                }
            }
            // Create the dynamic method
            DynamicMethod method =
                new DynamicMethod(
                    string.Format("{0}__{1}", constructor.DeclaringType.Name, Guid.NewGuid().ToString().Replace("-","")),
                    constructor.DeclaringType,
                    Array.ConvertAll<ParameterInfo, Type>(constructorParam, p => p.ParameterType),
                    true
                    );

            // Create the il
            ILGenerator gen = method.GetILGenerator();
            for (int i = 0; i < constructorParam.Length; i++)
            {
                if (i < 4)
                {
                    switch (i)
                    {
                        case 0:
                            gen.Emit(OpCodes.Ldarg_0);
                            break;
                        case 1:
                            gen.Emit(OpCodes.Ldarg_1);
                            break;
                        case 2:
                            gen.Emit(OpCodes.Ldarg_2);
                            break;
                        case 3:
                            gen.Emit(OpCodes.Ldarg_3);
                            break;
                    }
                }
                else
                {
                    gen.Emit(OpCodes.Ldarg_S, i);
                }
            }
            gen.Emit(OpCodes.Newobj, constructor);
            gen.Emit(OpCodes.Ret);

// Return the delegate :)
return method.CreateDelegate(delegateType);

}

Below are some results from some performance tests. I ran it twice to give the runtime a chance to "warm up". As we can see creating the object using the delegate is much, much faster than Activator.CreateInstance and close to the performance of direct call.

Running 1000000 interations of creation test.
Direct Call                     00:00:00.0643122
Delegate Wrapper                00:00:00.0691446
Activator.CreateInstance        00:00:06.8385526

Attached is some of the sample code. We might look at building on this in a later post. Peace until next time :)

Reflection : Dodge Common Performance Pitfalls to Craft Speedy Applications
http://msdn.microsoft.com/en-us/magazine/cc163759.aspx

Why are these nop instructions in my debug build?

2008-12-11T23:12:00Z

If you run the sample code from the last post in release mode and debug mode you will notice that the msil for the test method are different.

Method Program.TestMethod -- DEBUG
-------------------------------
0x0000 nop
0x0001 ldc.i4.0
0x0002 stloc.0
0x0003 ldloc.0
0x0004 ldc.i4.1
0x0005 add
0x0006 stloc.0
0x0007 ldloc.0
0x0008 ldc.i4 0x000008ff 2303
0x000d add
0x000e stloc.0
0x000f ldloc.0
0x0010 ldc.i4.2
0x0011 div
0x0012 stloc.1
0x0013 ldloc.1
0x0014 call 0x0a000019 Void WriteLine(Int32)
0x0019 nop
0x001a ldloc.0
0x001b call 0x0a000019 Void WriteLine(Int32)
0x0020 nop
0x0021 ldstr 0x700000df test
0x0026 call 0x0a000013 Void WriteLine(System.String)
0x002b nop
0x002c ret

Method Program.TestMethod -- RELEASE
-------------------------------
0x0000 ldc.i4.0
0x0001 stloc.0
0x0002 ldloc.0
0x0003 ldc.i4.1
0x0004 add
0x0005 stloc.0
0x0006 ldloc.0
0x0007 ldc.i4 0x000008ff 2303
0x000c add
0x000d stloc.0
0x000e ldloc.0
0x000f ldc.i4.2
0x0010 div
0x0011 stloc.1
0x0012 ldloc.1
0x0013 call 0x0a000019 Void WriteLine(Int32)
0x0018 ldloc.0
0x0019 call 0x0a000019 Void WriteLine(Int32)
0x001e ldstr 0x700000df test
0x0023 call 0x0a000013 Void WriteLine(System.String)
0x0028 ret

The debug code has all these nop instructions and the release mode does not. What is the nop instruction? Nop mean no operation. Why would these nop instructions exist in debug mode and not in release? After some searching on the internet it looks like the nop command is used in debug mode for break points. Try this in visual studio. Change your solution to debug mode and put a break point on a bracket. In code below I put the breakpoint on the top bracket after the word get. Run the solution in debug mode then change the solution to release mode and run. Notice how in debug mode you able to break on the actual bracket but in release mode you can only break on the line with actual code.

            public int Value
            { // Breakpoint here
                get
                {
                    return _value;
                }
            }

        private static void TestMethod()
        { // Break Point here

        }

Check out the links below for more info.

Debugging and nop Instructions

Visual Basic .NET allows you to set breakpoints on non-executing lines of code such as End If, End Sub, and Dim statements. To facilitate this popular debugging technique, the compiler inserts nop instructions as placeholders for the non-executing lines of code (since non-executing lines are not translated into IL instructions). The nop instruction is a "no operation" instruction—it does not perform any meaningful work yet can consume a processing cycle.

http://msdn.microsoft.com/en-us/library/aa289509.aspx -- Scroll down to Debugging and nop Instruction section

http://msdn.microsoft.com/en-us/library/system.reflection.emit.opcodes.nop.aspx

Msil Parser Quick Update

2008-12-11T20:55:00Z

Here is a quick update. The code will now resolve any tokens in the msil so now you can see the methods, strings, types, etc. I added a new method called GetData that takes a module, opcode, and the raw data byte array. As you can see they are very few data types that are inline in the msil. We use the OperandType enumeration to determine what the operand is for the given opcode. To resolve the inline tokens in the msil we use some of the methods on the Module class. These methods are listed below.

http://msdn.microsoft.com/en-us/library/system.reflection.module.resolvefield.aspx
http://msdn.microsoft.com/en-us/library/system.reflection.module.resolvemember.aspx
http://msdn.microsoft.com/en-us/library/system.reflection.module.resolvemethod.aspx
http://msdn.microsoft.com/en-us/library/system.reflection.module.resolvesignature.aspx
http://msdn.microsoft.com/en-us/library/system.reflection.module.resolvestring.aspx
http://msdn.microsoft.com/en-us/library/system.reflection.module.resolvetype.aspx

When I have some free time this week I am going to look at writing the msil back out using reflection.emit and possibly some other cool things.

Update : Fixed small bug with MsilInstruction.ToString() and RawData byte array.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Reflection;
using System.IO;
using System.Reflection.Emit;

namespace TestMsilReader
{


    class Program
    {

        static void Main(string[] args)
        {

            PrintMethod(typeof(Program).GetMethod("TestMethod", Type.EmptyTypes));
            PrintMethod(typeof(String).GetMethod("IsInterned", new Type[] { typeof(string) }));
            PrintMethod(typeof(String).GetMethod("IsNullOrEmpty", new Type[] { typeof(string) }));
            PrintMethod(typeof(String).GetMethod("Copy", new Type[] { typeof(string) }));
            
            Console.WriteLine("Press any key to exit.");
            Console.ReadLine();
        }
        private static void PrintMethod(MethodInfo methodInfo)
        {
            Console.WriteLine("Method {0}.{1}", methodInfo.DeclaringType.Name, methodInfo.Name);
            Console.WriteLine("-------------------------------");
            MsilReader reader = new MsilReader(methodInfo);
            while (reader.Read())
            {
                Console.WriteLine(reader.Current);
            }
            Console.WriteLine("-------------------------------");

        }


        public static void TestMethod()
        {
            int i = 0;
            i += 1;
            i += 2303;
            int x = i / 2;
            Console.WriteLine(x);
            Console.WriteLine(i);
            Console.WriteLine("test");
        }

        public class MsilReader
        {
            private static Dictionary<short, OpCode> _instructionLookup;
            private static object _syncObject = new object();
            private BinaryReader _methodReader;
            private MsilInstruction _current;
            private Module _module;
            static MsilReader()
            {
                if (_instructionLookup == null)
                {
                    lock (_syncObject)
                    {
                        if (_instructionLookup == null)
                        {
                            _instructionLookup = GetLookupTable();
                        }
                    }
                }
            }
            public MsilReader(MethodInfo method)
            {
                if (method == null)
                {
                    throw new ArgumentException("method");
                }
                _module = method.Module;
                _methodReader = new BinaryReader(new MemoryStream(method.GetMethodBody().GetILAsByteArray()));
            }


            public MsilInstruction Current
            {
                get
                {
                    return _current;
                }
            }

            public bool Read()
            {
                if (_methodReader.BaseStream.Length == _methodReader.BaseStream.Position)
                {
                    return false;
                }
                int index = (int)_methodReader.BaseStream.Position;
                int instructionValue;

                if (_methodReader.BaseStream.Length - 1 == _methodReader.BaseStream.Position)
                {
                    instructionValue = _methodReader.ReadByte();
                }
                else
                {
                    instructionValue = _methodReader.ReadUInt16();
                    if ((instructionValue & OpCodes.Prefix1.Value) != OpCodes.Prefix1.Value)
                    {
                        instructionValue &= 0xff;
                        _methodReader.BaseStream.Position--;
                    }
                    else
                    {
                        instructionValue = ((0xFF00 & instructionValue) >> 8) |
                            ((0xFF & instructionValue) << 8);
                    }
                }
                OpCode code;
                if (!_instructionLookup.TryGetValue((short)instructionValue, out code))
                {
                    throw new InvalidProgramException();
                }
                int dataSize = GetSize(code.OperandType);
                byte[] data = new byte[dataSize];
                _methodReader.Read(data, 0, dataSize);

                object objData = GetData(_module, code, data);


                _current = new MsilInstruction(code, data, index, objData);
                return true;
            }

            private static object GetData(Module module, OpCode code, byte[] rawData)
            {
                object data = null;
                switch (code.OperandType)
                {
                    case OperandType.InlineField:
                        data = module.ResolveField(BitConverter.ToInt32(rawData,0));
                        break;
                    case OperandType.InlineBrTarget:                    
                    case OperandType.InlineSwitch:
                    case OperandType.InlineI:
                        data = BitConverter.ToInt32(rawData, 0);
                        break;
                    case OperandType.InlineI8:
                        data = BitConverter.ToInt64(rawData, 0);
                        break;
                    case OperandType.InlineMethod:
                        data = module.ResolveMethod(BitConverter.ToInt32(rawData, 0));
                        break;
                    case OperandType.InlineR:
                        data = BitConverter.ToDouble(rawData, 0);
                        break;
                    case OperandType.InlineSig:
                        data = module.ResolveSignature(BitConverter.ToInt32(rawData, 0));
                        break;
                    case OperandType.InlineString:
                        data = module.ResolveString(BitConverter.ToInt32(rawData, 0));
                        break;
                    case OperandType.InlineTok:
                    case OperandType.InlineType:
                        data = module.ResolveType(BitConverter.ToInt32(rawData, 0));
                        break;
                    case OperandType.InlineVar:
                        data = BitConverter.ToInt16(rawData, 0);
                        break;
                    case OperandType.ShortInlineVar:
                    case OperandType.ShortInlineI:
                    case OperandType.ShortInlineBrTarget:
                        data = rawData[0];
                        break;
                    case OperandType.ShortInlineR:
                        data = BitConverter.ToSingle(rawData, 0);
                        break;
                    
                        
                        
                    
                }
                return data;
            }




            private static int GetSize(OperandType opType)
            {
                int size = 0;
                switch (opType)
                {
                    case OperandType.InlineNone:
                        return 0;
                    case OperandType.ShortInlineBrTarget:
                    case OperandType.ShortInlineI:
                    case OperandType.ShortInlineVar:
                        return 1;

                    case OperandType.InlineVar:
                        return 2;
                    case OperandType.InlineBrTarget:
                    case OperandType.InlineField:
                    case OperandType.InlineI:
                    case OperandType.InlineMethod:
                    case OperandType.InlineSig:
                    case OperandType.InlineString:
                    case OperandType.InlineSwitch:
                    case OperandType.InlineTok:
                    case OperandType.InlineType:
                    case OperandType.ShortInlineR:


                        return 4;
                    case OperandType.InlineI8:
                    case OperandType.InlineR:

                        return 8;
                    default:
                        return 0;

                }
            }


            private static Dictionary<short, OpCode> GetLookupTable()
            {
                Dictionary<short, OpCode> lookupTable = new Dictionary<short, OpCode>();
                FieldInfo[] fields = typeof(OpCodes).GetFields(BindingFlags.Static | BindingFlags.Public);
                foreach (FieldInfo field in fields)
                {
                    OpCode code = (OpCode)field.GetValue(null);
                    lookupTable.Add(code.Value, code);
                }
                return lookupTable;
            }
        }



        public struct MsilInstruction
        {
            internal MsilInstruction(OpCode code, byte[] rawData, int index, object data)
            {
                Instruction = code;
                RawData = rawData;
                Index = index;
                Data = data;
            }
            public readonly OpCode Instruction;
            public readonly byte[] RawData;
            public readonly int Index;
            private readonly object Data;


            public override string ToString()
            {
                StringBuilder builder = new StringBuilder();
                builder.AppendFormat("0x{0:x4} ", Index);
                builder.Append(Instruction.Name);
                if (RawData != null && RawData.Length > 0)
                {
                    builder.Append(" 0x");
                    for ( int i = RawData.Length - 1; i >= 0; i-- )
                    {
                        builder.Append(RawData[i].ToString("x2"));
                    }
                }
                if (Data != null)
                {
                    builder.Append(" " + Data.ToString());
                }
                return builder.ToString();
            }
        }
    }
}

Msil Parser

2008-12-11T15:33:00Z

I just moved into a new apartment last Saturday and have had no internet access till today. Life without internet is like being cut off from the world. I can't check my email, I can't work from home, I got to call and talk to people to order food ;{ It has been horrible :( Despite not having any internet yesterday night I decided to get on my computer and look for something fun to do. I am going to share that search with you ;)

I am quite a big user of reflector. Whenever I want to know how something works or why something doesn't work I just load it up in reflector and look at the code. How does reflector work though? There is no api included in the .net framework that will allow you to read msil (as far as I know). Unfortunately the reflector code is super obfuscated and very hard to read (reflector can't reflect on reflector lol) so I decided to start from scratch. I want to write something that can parse msil

I noticed in .net 2.0 a new class called MethodBody and a new method on MethodBase called GetMethodBody() were introduced. Instead of trying to parse PE headers and the .net meta data we will just use this class and only try to parse the msil. The MethodBody class has a method called GetILAsByteArray. We will parse these bytes and turn them into some objects that represent the msil.

In the parser we will use a dictionary to look up the instructions. The .net framework has a class OpCode used to represent a msil instruction when using the reflection emit apis. The OpCodes class has each OpCode as a public field. We will use these fields to fill our dictionary.

                Dictionary<short, OpCode> lookupTable = new Dictionary<short, OpCode>();
                FieldInfo[] fields = typeof(OpCodes).GetFields(BindingFlags.Static | BindingFlags.Public);
                foreach (FieldInfo field in fields)
                {
                    OpCode code = (OpCode) field.GetValue(null);
                    lookupTable.Add(code.Value, code);
                }
                return lookupTable;

Cool. After this is just looping though the bytes. Lookup up the instruction in our lookup table, figuring out the size of the data for each instruction, and move to the next instruction. One thing that got me was that msil instructions can be 1 or 2 bytes in length. Since I was out of internet access I had no way of looking up the exact format. I noticed that all the instructions that are 2 bytes start with 0xfe. In the OpCodes class Prefix1 has this value. After figuring this out everything worked great. The is also a prefix2-7 but I don't see any instructions that use these so I did not implement.

                int instructionValue;
                if (_methodReader.BaseStream.Length - 1 == _methodReader.BaseStream.Position)
                {
                    instructionValue = _methodReader.ReadByte();
                }
                else
                {
                    instructionValue = _methodReader.ReadUInt16();
                    if ((instructionValue & OpCodes.Prefix1.Value) != OpCodes.Prefix1.Value)
                    {
                        instructionValue &= 0xff;
                        _methodReader.BaseStream.Position--;
                    }

                    if ((instructionValue & OpCodes.Prefix1.Value) != OpCodes.Prefix1.Value)
                    {
                        instructionValue &= 0xff;
                        _methodReader.BaseStream.Position--;
                    }
                    else
                    {
                        instructionValue = ((0xFF00 & instructionValue) >> 8) |
                            ((0xFF & instructionValue) << 8);
                    }
                }

                OpCode code;
                if (!_instructionLookup.TryGetValue((short)instructionValue, out code))
                {
                    throw new InvalidProgramException();
                }

                int dataSize = GetSize(code.OperandType);
                byte[] data = new byte[dataSize];
                _methodReader.Read(data, 0, dataSize);
                _current = new MsilInstruction(code, data);

Below is a complete sample console application. Pass in any MethodInfo into PrintMsil to see the msil. We will continue the rest later. Next time we will look at parsing the data byte array and resolving tokens ( method, type, field, etc ) that are in the msil. The Module class has some methods that allow us to do this. Then we will look at some cooler things like rewriting the msil that we read in. Peace until next time :)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Reflection;
using System.IO;
using System.Reflection.Emit;

namespace TestMsilReader
{

    class Program
    {

        static void Main(string[] args)
        {
            PrintMethod(typeof(Program).GetMethod("TestMethod", Type.EmptyTypes));
            PrintMethod(typeof(String).GetMethod("IsInterned", new Type[]{typeof(string)}));
            Console.WriteLine("Press any key to exit.");
            Console.ReadLine();
        }


        private static void PrintMethod(MethodInfo methodInfo)
        {
            Console.WriteLine("Method {0}.{1}", methodInfo.DeclaringType.Name,methodInfo.Name);
            Console.WriteLine("-------------------------------");
            MsilReader reader = new MsilReader(methodInfo);
            while (reader.Read())
            {
                Console.WriteLine(reader.Current);
            }
            Console.WriteLine("-------------------------------");

        }


        public static void TestMethod()
        {

            int i = 0;

            i += 1;

            i += 2303;

            int x = i / 2;

            Console.WriteLine(x);

            Console.WriteLine(i);

        }



        public class MsilReader
        {

            private static Dictionary<short, OpCode> _instructionLookup;

            private static object _syncObject = new object();

            private BinaryReader _methodReader;

            private MsilInstruction _current;

            private Module _module;// Need to resolve method, type tokens etc



            static MsilReader()
            {

                if (_instructionLookup == null)
                {

                    lock (_syncObject)
                    {

                        if (_instructionLookup == null)
                        {

                            _instructionLookup = GetLookupTable();

                        }

                    }

                }

            }

            public MsilReader(MethodInfo method)
            {

                if (method == null)
                {

                    throw new ArgumentException("method");

                }

                _module = method.Module;

                _methodReader = new BinaryReader(new MemoryStream(method.GetMethodBody().GetILAsByteArray()));

            }





            public MsilInstruction Current
            {

                get
                {

                    return _current;

                }

            }



            public bool Read()
            {

                if (_methodReader.BaseStream.Length == _methodReader.BaseStream.Position)
                {

                    return false;

                }

                int instructionValue;



                if (_methodReader.BaseStream.Length - 1 == _methodReader.BaseStream.Position)
                {

                    instructionValue = _methodReader.ReadByte();

                }

                else
                {

                    instructionValue = _methodReader.ReadUInt16();

                    if ((instructionValue & OpCodes.Prefix1.Value) != OpCodes.Prefix1.Value)
                    {

                        instructionValue &= 0xff;

                        _methodReader.BaseStream.Position--;

                    }
                    else
                    {
                        instructionValue = ((0xFF00 & instructionValue) >> 8) |
                            ((0xFF & instructionValue) << 8);
                    }


                }

                OpCode code;

                if (!_instructionLookup.TryGetValue((short)instructionValue, out code))
                {

                    throw new InvalidProgramException();

                }

                int dataSize = GetSize(code.OperandType);

                byte[] data = new byte[dataSize];

                _methodReader.Read(data, 0, dataSize);



                _current = new MsilInstruction(code, data);

                return true;

            }









            private static int GetSize(OperandType opType)
            {

                int size = 0;

                switch (opType)
                {

                    case OperandType.InlineNone:

                        return 0;

                    case OperandType.ShortInlineBrTarget:

                    case OperandType.ShortInlineI:

                    case OperandType.ShortInlineVar:

                        return 1;



                    case OperandType.InlineVar:

                        return 2;

                    case OperandType.InlineBrTarget:

                    case OperandType.InlineField:

                    case OperandType.InlineI:

                    case OperandType.InlineMethod:

                    case OperandType.InlineSig:

                    case OperandType.InlineString:

                    case OperandType.InlineSwitch:

                    case OperandType.InlineTok:

                    case OperandType.InlineType:

                    case OperandType.ShortInlineR:





                        return 4;

                    case OperandType.InlineI8:

                    case OperandType.InlineR:



                        return 8;

                    default:

                        return 0;



                }

            }





            private static Dictionary<short, OpCode> GetLookupTable()
            {

                Dictionary<short, OpCode> lookupTable = new Dictionary<short, OpCode>();

                FieldInfo[] fields = typeof(OpCodes).GetFields(BindingFlags.Static | BindingFlags.Public);

                foreach (FieldInfo field in fields)
                {

                    OpCode code = (OpCode)field.GetValue(null);

                    lookupTable.Add(code.Value, code);

                }

                return lookupTable;

            }

        }







        public struct MsilInstruction
        {

            public MsilInstruction(OpCode code, byte[] data)
            {

                Instruction = code;

                Data = data;

            }

            public readonly OpCode Instruction;

            public readonly byte[] Data;





            public override string ToString()
            {

                StringBuilder builder = new StringBuilder();

                builder.Append(Instruction.Name + " ");

                if (Data != null && Data.Length > 0)
                {

                    builder.Append("0x");

                    foreach (byte b in Data)
                    {

                        builder.Append(b.ToString("x2"));

                    }

                }

                return builder.ToString();

            }

        }

    }
}

Intro

2008-12-11T15:32:00Z

My name is Ziad Elmalki. I have been a software developer for about 5 years. I started programming vb in 1999 when I was in high school and then learned C# when .Net 1.1 came out. I am a SDE in Microsoft's STO group. I have been working here at Microsoft for about 9.5 Months. I am always trying to learn new things. I decided to start a blog to talk about any cool code I come across at work or on the side. Stay tuned for cool stuff :)