With the Mix2009 release of Windows Azure tools, we now support native code execution. This also includes the ability to debug native code. As you would imagine, you can do PInvoke’s to system DLL’s or to your packaged native DLL with your role.
The image in Windows Azure runs on a 64bit OS. Role hosts (web and worker hosts) are also 64 bit executables which result in your web and worker roles runing as 64bit binaries. That means if you want to PInvoke directly into your native DLL, your native DLL must be compiled for 64bit platform, otherwise you would get the error:
An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B)
The simplest solution is to rebuild the assembly we are trying to load to 64bit, but there are some cases that we cannot do that like if we don’t own the source code. One way is to host the 32bit dll in a 32bit program that runs on WoW. Then we can use WCF to marshal all calls between the role process (which is 64bit) and our out-of-proc dll (which is 32bit).
I will show you an example on how to do that with a WebRole. My demo involves a 32bit NativeLibrary.dll that contains an Add function; Also, a managed DllHost86.exe that acts as a WCF server and as a host for the Native Library. The managed DllHost86 is configured to be built for x86 specific:
Both the NativeLibrary and the Dll-Host are configured to output their binaries to the webrole Redist folder. Also, we added these files as contents to the web role project to make sure they are packaged when launching under the devfabric or deployed to the Cloud:
The interactions between the Dll-Host and the web role goes like the following:
- On any request to the web role (using Application_BeginRequest event), we check if the DllHost86 process is running.
- If not running, launch the process.
- Also, on any request to the web role, try to connect the WCF client to the WCF server hosted inside the DllHost86 process. Once connected, cache the client object in the server current session (Application session). Next time we actually get the client from the session and just check if the connection is valid.
- The binding used on WCF is NetNamedPipeBinding for optimal performance on the same machine communication.
- The time out on the WCF link is set to infinity.
The code (from Global.asax) is shown next:
private const int ClientInitTimeOut = 20; // in seconds
protected void Application_BeginRequest(object sender, EventArgs e)
{
// Make sure that our dll host is running
EnsureDllHostRunning();
// Make sure the client is connected
EnsureCalcServiceClientConnected();
}
private void EnsureDllHostRunning()
{
Process[] p = Process.GetProcessesByName(Path.GetFileNameWithoutExtension(dllHostPath));
if (p.Length == 0)
{
Application["CalcServiceClient"] = null;
ProcessStartInfo psi = new ProcessStartInfo(dllHostPath);
Process dllHost = Process.Start(psi);
}
}
private void EnsureCalcServiceClientConnected()
{
CalcServiceClient client;
client = (CalcServiceClient)Application["CalcServiceClient"];
if (client == null || client.State != System.ServiceModel.CommunicationState.Opened)
{
client = GetCalcServiceClient();
Application["CalcServiceClient"] = client;
}
}
private CalcServiceClient GetCalcServiceClient()
{
CalcServiceClient serv = null;
int retryCount = 0;
bool connected = false;
while (retryCount < ClientInitTimeOut * 10)
{
try
{
EndpointAddress address = new EndpointAddress("net.pipe://localhost/CalculatorService");
NetNamedPipeBinding binding = new NetNamedPipeBinding();
binding.ReceiveTimeout = TimeSpan.MaxValue;
serv = new CalcServiceClient(binding, address);
serv.Open();
Debug.WriteLine("state = " + serv.State);
if (serv.State == System.ServiceModel.CommunicationState.Opened)
{
connected = true;
break;
}
}
catch (Exception e)
{
Debug.WriteLine(e.Message);
}
retryCount++;
System.Threading.Thread.Sleep(100);
}
if (!connected)
{
throw new TimeoutException("Couldn't connect to the calculator service.");
}
return serv;
}
So far, we setup the web role to startup the dll host and setup the client connection to the WCF server inside the DllHostx86. The code there is just a standard code that uses ServiceHost and just waits forever. Here is the code from the Program class:
static void Main(string[] args)
{
Uri address = new Uri("net.pipe://localhost/CalculatorService");
NetNamedPipeBinding binding = new NetNamedPipeBinding();
binding.ReceiveTimeout = TimeSpan.MaxValue;
using (ServiceHost host = new ServiceHost(typeof(Calculator)))
{
host.AddServiceEndpoint(typeof(ICalcService), binding, address);
ServiceMetadataBehavior metadata = new ServiceMetadataBehavior();
host.Description.Behaviors.Add(metadata);
Binding mexBinding = MetadataExchangeBindings.CreateMexNamedPipeBinding();
Uri mexAddress = new Uri("net.pipe://localhost/CalculatorService/Mex");
host.AddServiceEndpoint(typeof(IMetadataExchange), mexBinding, mexAddress);
host.Open();
Console.WriteLine("The receiver is ready");
Console.ReadLine();
}
}
So, the service we expose contains an Add methods that is implemented internally to call the native method using PInvoke. Here is a sample of the implementation:
class Calculator : ICalcService
{
[DllImport("NativeLibrary.dll", EntryPoint="Add")]
static extern UInt32 NativeAdd(UInt32 a, UInt32 b);
#region ICalcService Members
public int Add(int num1, int num2)
{
Console.WriteLine("Received numbers: {0}, {1}", num1, num2);
return (int)NativeAdd((uint)num1, (uint)num2);
}
#endregion
}
I tested the sample above in the cloud and it works fine. I also attached it for your reference. Here is how it looks like:
So, in this blog, I demonstrated how to use a 32bit dll in the Cloud by means of hosting it under a 32bit process and then use WCF communication to do the marshaling.
Hope it helps.
I will demonstrate here how we can use the functional programming feature in C# to program something like a statement/expression translation. Basically, we will translate a statement tree into a function that we can call to run the program. I am not using Linq Expression as demonstrated in my previous blog. Instead, I will show how to use functional programming.
The expression list we support is a very small subset of a real programming language, the definition in detail is:
| int |
an integer constant |
| variable |
variable reference |
| plus |
expression + expression |
| multiply |
expression * expression |
The classes are:
// base class for the expression node
abstract class Expression
{ }
// defines a constant integer
class Int : Expression
{ public Int(int i)
{ Value = i;
}
public int Value { get; set; } }
// defines a variable reference
class Variable : Expression
{ public Variable(string name)
{ Name = name;
}
public string Name { get; set; } }
// defines a plus expression
class Plus : Expression
{ public Plus(Expression left, Expression right)
{ Left = left;
Right = right;
}
public Expression Left { get; set; } public Expression Right { get; set; } }
// defines a multiplication expression
class Multiply : Expression
{ public Multiply(Expression left, Expression right)
{ Left = left;
Right = right;
}
public Expression Left { get; set; } public Expression Right { get; set; } }
Our supported statements are just a very small subset of the whole programming language. It contains, an assignment, if, while, skip, and a sequence of statements. The definition in detail is:
| Assignment |
variable := expression |
| if |
if (expression) then statement else statement |
| while |
while (expression) then statement |
| seq |
statement, statement |
| skip |
does nothing, e.g. could be used with the else statement. |
The classes that represent our statements:
abstract class Statement
{ }
class Skip : Statement
{ }
class Seq : Statement
{ public Seq(Statement s1, Statement s2)
{ S1 = s1;
S2 = s2;
}
public Statement S1 { get; set; } public Statement S2 { get; set; } }
class Assign : Statement
{ public Assign(string variable, Expression expression)
{ Variable = variable;
Expression = expression;
}
public string Variable { get; set; } public Expression Expression { get; set; } }
class If : Statement
{ public If(Expression condition, Statement @then)
: this(condition, @then, new Skip())
{ }
public If(Expression condition, Statement @then, Statement @else)
{ Condition = condition;
Then = @then;
Else = @else;
}
public Expression Condition { get; set; } public Statement Then { get; set; } public Statement Else { get; set; } }
class While : Statement
{ public While(Expression condition, Statement body)
{ Condition = condition;
Body = body;
}
public Expression Condition { get; set; } public Statement Body { get; set; } }
After we defined our language, we can just use it to construct our program, or use a help of a scanner/parser. What we care here is just to translate both expressions and statements nodes.
The way we are going to access our variables is through a heap. Our heap is a function that takes a variable name as a string and returns an integer (I assume we deal only with integer types)
heap := var => int
We will translate an expression into a function that takes a heap (heap function) and returns an integer value. For example, the addition expression will use the heap to return the result of the sum.
expression := heap => int
Statement will get translated into a function that takes a heap and returns another heap. For example, the assignment statements, will take the current heap and return a new heap that contains the new variable.
statement := heap => heap
The HeapFunc is declared as a delegate:
delegate int HeapFunc(string variable);
The code for expression translation is simple and straight forward:
// returns a function that takes a heap and returns an integer
Func<HeapFunc, int> TranslateExpression(Expression e)
{ if (e == null)
throw new ArgumentNullException("Expression cannot be null");
if (e is Int)
{ // whatever the heap is, return the value
return (h => ((Int)e).Value);
}
else if (e is Variable)
{ // use the heap to lookup the variable name
return (h => h(((Variable)e).Name));
}
else if (e is Plus)
{ // translate the left expression into a function
var f1 = TranslateExpression(((Plus)e).Left);
// translate the right exression into a function
var f2 = TranslateExpression(((Plus)e).Right);
// run the functions on the given heap to get integers and sum them
return (h => f1(h) + f2(h));
}
else if (e is Multiply)
{ // similar to the plus expression
var f1 = TranslateExpression(((Multiply)e).Left);
var f2 = TranslateExpression(((Multiply)e).Right);
return (h => f1(h) * f2(h));
}
else
{ throw new NotSupportedException("Not supported expression."); }
}
The code for statement translation is shown next:
// returns a function that takes a heap and returns a heap.
Func<HeapFunc, HeapFunc> TranslateStatement(Statement s)
{ if (s == null)
throw new ArgumentNullException("Expression cannot be null");
if (s is Skip)
{ // just return the same heap
return (h => h);
}
else if (s is Assign)
{ Assign assign = (Assign)s;
// translate the expression
var f = TranslateExpression(assign.Expression);
// the resulted heap will check if the variable is the one responsible for
// if so return the result of the expression, else use the previous heap to look up.
return (h) => (var => var == assign.Variable ? f(h) : h(var));
}
else if (s is Seq)
{ Seq seq = (Seq)s;
// translate the first statement
var f1 = TranslateStatement(seq.S1);
// translate the second statement
var f2 = TranslateStatement(seq.S2);
// apply the first translation on the heap, which will return a new heap
// that will be applied to the second translation.
return (h) => f2(f1(h));
}
else if (s is If)
{ If @if = (If)s;
// translate the condition expression
var fcond = TranslateExpression(@if.Condition);
// translate the then statement
var fthen = TranslateStatement(@if.Then);
// translate the else statement
var felse = TranslateStatement(@if.Else);
// apply the heap to the condition function. If the value is not zero, use
// the heap returned by applying fthen, otherwise use the heap returned by
// applying felse.
return (h) => (fcond(h) != 0 ? fthen(h) : felse(h));
}
else if (s is While)
{ While @while = (While)s;
// translate the condition expression
var fcond = TranslateExpression(@while.Condition);
// translate the body statement
var fbody = TranslateStatement(@while.Body);
// apply the body multiple times then return the final heap.
return (h) => { while (fcond(h) != 0)
{ h = fbody(h);
}
return h;
};
}
else
{ throw new NotSupportedException("Statement not supported."); }
}
Sample:
I wrote a simple example to show how we can translate a simple program into a functional code. My example returns the sum of numbers between 0 and 10.
// this is our initial heap. our heap is empty, so calling this function
// sould just fail because we have no variable
int InitialHeapFun(string variable)
{ throw new Exception("cannot find variable '" + variable + "'"); }
void Run()
{ // sum := 0
// i := 10
// while (i)
// sum := sum + i
// i := i - 1
// end
Statement myProgram = new Seq(
new Assign("sum", new Int(0)), new Seq(
new Assign("i", new Int(10)), new While(new Variable("i"), new Seq(
new Assign("sum", new Plus(new Variable("sum"), new Variable("i"))), new Assign("i", new Plus(new Variable("i"), new Int(-1))) )
)
)
);
// translate the program into (heap=>heap)
var f = TranslateStatement(myProgram);
// run the program using our initial heap
var heap = f(InitialHeapFun);
// print the sum value. this outputs 55
Console.WriteLine(heap("sum")); }
There are a lot of other cool examples that would benefit from this method of translation. For example, we could translate an XML file that contains different actions. The result function will run much faster using just function calls. You can think of it as an XML compiler.
The code is attached for your convenience.
The last post, I showed a simple way to build a lexicon scanner in .NET. We can now extend on the scanner and build a parser that generates a Lambda expression that can be compiled and called for further use. The parser I used is just a top-down recursive parser that builds up the Expression tree. The main method is BodyExpression that returns the Expression object.
The class LambdaCalc, holds all the logic. You can pass a string that will get parsed into a lambda expression. For example:
LambdaCalc lc = new LambdaCalc("(a, b) => -a + 2 * b");
Console.WriteLine(lc.Function(1, 2));
Function is a property of a delegate type declared as:
public delegate double Func(params double[] args);
So, the way we pass arguments to our lambda calculator is through an array of doubles. Internally, in the generated lambda expression, all parameter references are converted into array references. The previous example would generate the following lambda expression:
args => (-args[0] + (2 * args[1]))
To generate a Lambda Expression, you need to use one of the Lambda static methods inside the Expression class. Using the generic version, allows us to strongly define the delegate that the lambda expression will map to. For our case, this is how I generated the Lambda expression:
Expression<Func> lambdaExpr = Expression.Lambda<Func>(body, this.funcParamsArg);
The BodyExpression method eventually calls also other methods inside the Linq Expression class, for example, we use Expression.Add(leftExpr, rightExpr), to generate the additive expression, and the same goes for multiplication and so on.
The thing that worth mentioning is that in order to use the argument from inside the body, you need to use the same ParameterExpression object, otherwise you would get the following exception when compiling your Lambda Expression:
System.InvalidOperationException: Lambda Parameter not in scope
I attached the code bellow for the parser. You need the code from LambdaCalcScanner for this to work.
I came across Linq and the new Lambda Expression classes in .NET 3.0. I thought, it would be a good idea to dynamically build a lambda expression from a math string, something like: (a + b) * 1.5. The string is parsed into a lambda expression and then is compiled into a delegate. Then, if you well, you can call the delegate to get the result of the formula.
So, in other words, we are building a scanner and a parser. The first part that I will talk about here is building a scanner using the Regex class in .NET.
The tokens that we need to scan for are defined next:
// Tokens that represent the input
internal enum Token
{ OpenParan, CloseParan,
Arrow,
Comma,
Plus, Minus, Multiply, Divide,
Constant,
Variable,
Other // Represents unrecognized charachters
}
The regular expression pattern then is defined next:
// The pattern used with the regular expression class to scan the input
const string Pattern = @"
(?'OpenParan' \( ) | (?'CloseParan' \) ) |
(?'Arrow' => ) |
(?'Comma' , ) |
(?'Plus' \+ ) | (?'Minus' - ) | (?'Multiply' \* ) | (?'Divide' / ) |
(?'Constant' (\.\d+|\d+(\.\d+)?) ) |
(?'Variable' [a-zA-Z]\w* ) |
(?'Other' [^ \r\n\t])";
// Regular expression used to scan the input
private static Regex MathRegex = new Regex(Pattern, RegexOptions.IgnoreCase | RegexOptions.IgnorePatternWhitespace |
RegexOptions.Singleline | RegexOptions.Compiled);
The pattern basically tries to find any of the groups and name each group by the given name. We also assign the group ‘Other’ to anything else that we couldn’t match.
The scanner then can be implemented as an enumerable method that returns the found tokens.
// Enumurable to get tokens from the given expression (scanner)
public static IEnumerable<TokenEntity> GetLambdaCalcTokens(this string exp)
{ Token[] tokens = Enum.GetValues(typeof(Token)).OfType<Token>().ToArray();
foreach (Match m in MathRegex.Matches(exp))
{ // Check which token is matched by this match object
foreach (Token token in tokens)
{ if (m.Groups[token.ToString()].Success)
{ yield return new TokenEntity(
token,
m.Index,
m.Value);
}
}
}
// return the end string token, to indecate we are done
yield return new TokenEntity(Token.Other, exp.Length, "\0");
}
The only problem with using enumerable is that it doesn’t allow us to peek forward without moving next. This could be a problem with complex parsers, but the parser that I am writing here is simple enough that can be written without this feature.
The previous method is written as an extension on the string, so it can be easily called on any string like “(a) => a + 1.5”.GetLambdaCalcTokens()
The class TokenEntity is defined as:
// Holds token info
internal class TokenEntity
{ public TokenEntity(Token token, int startPos, string value)
{ this.Token = token;
this.StartPos = startPos;
this.Value = value;
}
// Token type
public Token Token { get; private set; }
// Start position in the original string
public int StartPos { get; private set; }
// Value
public string Value { get; private set; }
public override string ToString()
{ return string.Format("{0} at {1}: {2}", Token, StartPos, Value); }
}
Running the previous code on: "(a, b) => ; a + b * 0.1" , results with the following tokens:
OpenParan at 0: (
Variable at 1: a
Comma at 2: ,
Variable at 4: b
CloseParan at 5: )
Arrow at 7: =>
Other at 10: ;
Variable at 12: a
Plus at 14: +
Variable at 16: b
Multiply at 18: *
Constant at 20: 0.1
Other at 21:
I am attaching the source code for the simple scanner. I will follow next with the parser and how we can build lambda expression dynamically.
