Patterns for Great Architecture

Why Task Based Programming (Parallel Extensions)

2009-10-14T21:06:37Z

Hi,

Today is October 15th 2009 and I am writing this post to express my understanding about parallel programming using Parallel Extensions (or Task based parallel programming); specifically to express why should we go for this approach.

Since the Parallel Extension announcement, I was excited to get my hands dirty with it but after working with Parallel Extensions, I had few initial questions that I thought everyone would have and there is less content available online that targets against benefits of Parallel extensions. Below are a couple of questions that I am targeting in this post.

Is Threading going to die? Are Parallel Extensions replacements to Threading?
Sometimes Threading gives me better performance than Task based programming. Why?

Before we come to these questions, it would be worth understanding options other than parallel extensions that we used to follow for parallel programming. Before Parallel Extensions, we would use primarily any of the below two modes for parallel programming:

Directly Creating Threads:
Here we create Threads directly in our code something similar to the following code:

Thread t = new Thread(DoSomeWorkMethod);
t.Start(someInputValue);

Benefits:
The benefit of this approach is that each of the thread will run at all times. .NET Framework’s scheduling algorithm will make sure that all the threads that we created will execute in similar to RoundRobin algorithm (though actual algorithm is different). This approach is especially useful in cases where I want to provide justice to all threads.
Another benefit associated with this approach is the fact that we have control over these created threads like aborting a thread in the middle.

Drawbacks:
The drawback of this approach comes into picture when we have more threads than the available CPUs (cores). As an example, if we have two cores and I am running 10 threads, context switching, invalidation of each thread’s cache etc. So we need to keep a balance between how many CPUs we have and how many Threads should we create for optimum performance.

Using Thread Pool:
.NET Framework has a thread pool (configurable) which limits the number of Threads that can be run at a time. Now when we want to execute something in a separate thread, we simply queue our work on a thread of thread pool instead of creating our own separate thread. Code looks something similar to below:

ThreadPool.QueueUserWorkItem(
DoSomeWorkMethod, someInputValue);

Benefits:
Thread Pool size for .NET Framework can be set by doing a simple calculation using number of CPUs the machine has. After this, Thread Pool makes sure to run only those number of threads that are optimal for a particular machine. This prevents any context switching and other overheads. So we get best performance.

Drawbacks:
The drawback of this approach is that developer doesn’t have much control over the Queued tasks over Thread Pool. In other words, once we have queued a work item, we do not get a reference to it and there is no explicit support for knowing when it completed, or for blocking until it completes or for cancelling it, or for identifying it in the debugger via some sort of ID etc. Plus, you are not guaranteed when your Task will start to run, so progress is unknown.

Using Parallel Extensions:
This approach is the one that we are here to discuss. Task based programming combines the benefit of both the approaches with some new additions. Using task Based programming, you create Tasks instead of Threads and start executing them. Programmatically they are almost identical to Thread Based programming with additions features. Here is a code snippet that uses the new Task class:

Task t = Task.Create(DoSomeWorkMethod,
someInputValue);

Benefits:

With Task Based programming, by default, you can be sure that we are not actually decreasing the performance by doing excessive multitasking since Task Manager (central class in Parallel Extensions) takes care of this issue by queuing any additional tasks if all the existing CPUs are busy.
With Task based programming you have complete control over the Task. You can pause, abort, wait, join etc.
The scheduling algorithm is improved over Thread Pool approach with new features like “Task Stealing”.
You get additional constructs to easily work with, like Parallel LINQ, Parallel For Loop etc.

Drawbacks:

Personally I see the only drawback is that you are not guaranteed when the Task will execute. So if you want to guarantee of starting your Task execution as soon as it is created, use plain old Threading Way. UI multitasking as an example should be done with Threading and not Thread Pool or Task based programming.

I will also share one interesting incident that happened while I was developing a Proof Of Concept. I was running the samples on a 4 Core Machines and created 2 Threads to do a Task and then 2 Tasks to do the same task and I could not see any improvement. Then I realized that if I am creating Threads which are less in number than available free CPUs I might actually get some greater performance sometimes, but that is not a guarantee. With task based programming, I can guarantee that either all my Tasks are running in parallel or all CPUs are busy executing my tasks.

I will try to cover benefits approach in future posts but for now, it is done.

I hope this post was useful. If you are a Microsoft Partner, please feel free to involve our team for all your development consultation needs by contacting us here. Comments, corrections; I Love them.

Rahul Gangwar

Microsoft Codename "Gemini"

2009-07-28T19:47:25Z

Hi,

Today is July 28th 2009 and I am writing this small post to share my understanding on Gemini and Business Intelligence arena and specifically around Analysis Service.

Talk about data in Microsoft and the very first thing that comes in mind is SQL Server. Microsoft hardly released SQL 2008 and we are looking forward for next version of SQL Server. That is very ambitious.

When it comes to Business Intelligence problem, things like Reporting, Analysis, data-mining come automatically forms part of the solution. But there are few patterns that should be considered for developing an efficient BI solution. Before talking about Analysis Service, let's understand how operations are different in OLTP vs OLAP databases.

In OLTP databases, data is written in form of rows (relational databases). So for example, if I want to store two records of two column structure having (name and email as columns) the storage will go in a fashion:
"Rahul" "rahul@contoso.com", "Sabiha" "sabiha@contoso.com".
Eeffectively, data is written over disk one row at a time. This makes typical OLTP transactions efficient since I can quickly read/write a all columns of ONE record somewhere in middle of 50 million rows. But when it comes to reading ALL records (let's say email of all 50 Million records), relational databases fail miserably in terms of performance.

In OLAP databases, data is stored in a different fashion. It is stored in the form of columns rather that rows. So if I want to store the same set of records in columnar fashion, it would be stored this ways:
"Rahul" "Sabiha", "rahul@contoso.com" "sabiha@contoso.com".
The benefit and drawbacks are here.
If I want to read columns of ONE record somewhere in middle of 50 million rows, it will fail miserably in comparison to relational databases. But when it comes to reading a column of ALL records, this will be surprisingly fast (and this requirement is very common in Analysis: to read large number of records rather than one record).

So one thing is clear that columnar databases are better than relational databases for data analysis.

Another thing that need to be considered while designing analysis solution is the storage medium. Databases that use disk as the storage medium will perform very slow for performing analysis algorithms than the databases that use in memory medium as storage. Storing data in memory is fast because of the medium speed as well as saving CPU cycles. Although the durability property of ACID is lost in this process (since memory is volatile) but this still holds a good candidate for analysis solutions. Even hybrid storage mechanisms can be considered depending on the requirement scenario.

Now comes Microsoft's project codename "Gemini". Gemini is next version of Analysis Service based on the concepts of using columnar database and in memory storage, making analysis of data hell fast. Gemini will be shipped in the next version of SQL Server (2008 R2/Kilimanjaro).
The good thing is that Excel users can use its capability by installing a plug-in. Hence the term "self service analysis". Every user in an organization will be able to analyse data and take decisions using simple Excel. More here:
http://www.microsoft.com/presspass/press/2008/oct08/10-06BI08PR.mspx

I hope this gives a peek about what Gemini is and how it will work.

Comments/feedbacks welcome.

Rahul Gangwar

Physics Based Games in Silverlight

2008-09-30T04:00:00Z

Hi,

Today is September 29 - 2008 and I am writing this short post (though big, but short for writing anything over Farseer/SL) to give an idea about one of the immense possibilities with Silverlight - Game Development. I will focus on simplicity to target the beginners and specifically cover possibilities with Farseer Engine. The Farseer Physics Engine is an easy to use 2D physics engine designed for Microsoft’s XNA and Silverlight platforms.

I do not know why it is named Farseer but possibly because it refers to followers of Eldar; one of the most ancient and advanced races in the universe's history. Their armies usually have the advantages of speed and technology. Possibly Jeff Weber can tell.

Physics Engine:

There are several games that simulates their motion like a real world entity. For example, if you have used Windows Live SkyDrive, while uploading a file, you are presented with a screen having a Baal. You can push this ball and it collides with walls, bounces back pretty much the same way as if it was a real ball. The ball has all the properties of a real world - it always feels a gravitational pull downwards, on experiencing a force i would follow a parabolic path similar to a real world ball. Now, no one has written the logic on how a ball should move when hit. such simulation is done by associating certain features to the ball (like mass, friction etc.), rest all is handled by the underlying Physics Engine. Physics based games are more quality games and closer to real world experience and form a different group of games altogether.

In short statement, any piece of logic that could provide simulation of Newtonian Physics is a physics engine. It doesn't necessarily has to be a game engine, though the most common application is game industry, it is also used in various other fields such as research, advertisements etc. Primarily there are two kinds of Physics Engines - "Real Time" and "High Precision". Both the engines aim to simulate the laws of physics (like friction, moment of inertia, force, gravity (which is again a force) etc.) but since there is less availability of computation over most of the end users machines, "Real Time" engines eat up some of the details and simulate the bare minimum required to provide a real world feeling. On the other hand, "high precision" engines take into account of almost all the factors that exist in our world like air speed etc. and are commonly used in scientific projects like defence, space etc. and are run over high end machines like super computers.

most of the machines out there depend entirely on CPU alone for most of the computational needs. In areas where high graphics intensive tasks are required, we can install specialized GPU (Graphics Processing Units)- we also know them by common name - graphics card. Similarly, in high end applications which use Physics Engines, we can also install PPU (Physics Processing Unit) for shedding off the load from CPU giving a better performance.

In case of Farseer, it is not an easy task to write a Physics Engine for browser based scenarios since you have to think 100 times before planning for using resources. Physics based calculations cannot be done easily when the machines are poor. That is the reason why you might find some glitches in terms of performance on the physics games over Silverlight and Farseer will improve continually over time - sometimes to cope up with new releases of Silverlight, at other time to add new demanded features and performance restrictions. Hats off to the people out there who are contributing to the developer community by creating such product. Now I can start my post after paying respect to those great people.

Silverlight as gaming platform:

Games have few components. Let us examine these and see what Silverlight has to offer w.r.t. these requirements.

1. A nice light and programmable UI - Silverlight supports these features through various concepts like Path, Vector shapes etc. and most importantly, they are based on XAML so you can talk to your graphics during runtime which you might not be able to do with a normal graphics (like jpeg etc.).

2. A way to provide simulation to the UI elements - Silverlight supports these through various subtle concepts like animations. More importantly, since we can work in C# we can use all the good features and usability of the same. You don't have to learn any new tool/language.

3. A way to interact with end user in sufficient real time fashion - Silverlight makes this possible through its optimised CLR (called AgCLR; and do you know that CLR of SL is almost same as that of a the .NET Framework). Also adding to the fact that entire processing will be done at client side, we can expect SL based games to be sufficiently real time. And if you don't see it, probably its time to upgrade the machine.

4. Game Logic - Now don't expect SL to write game logic, otherwise it would have been a Product rather than a Platform.

5. Game Engine - Game engine is kind of another framework sitting on top of the underlying framework and it exposes functionality to your game to ease the development effort. It is similar to the fact that .NET APIs, provides a framework where we build other applications on top of them. However it has to be noticed that a game engine is not something that is generic. One game engine works only with a particular genere of games. It is similar to the concept of Software Factories which could be applied to a specific kind of applications only. Silverlight doesn't provide any. You have to write your own (if required) or depend on others available engines in market.

6. Generalized Game Engines - Physics engine is a good candidate to fit in this scenario. These engines help us to add certain features rather acting as a specific game engine. For example, Silverlight doesn't provide any API to simulate Physics, so we have to Physics Engines to add those features to our games/apps. Another example could be 3D rendering engine. Again, Silverlight doesn't support natively any 3D based APIs, so we have to take help of 3D rendering engines like Kit3D. By using Kit3D, we can develop 3D games over Silverlight. Combining Kit3D with Farseer could give us 3D, physics based games on Silverlight. Kit3D is my next topic of interest that I will take up.

Exploring Farseer:

I will not go into details of Silverlight/any other details and assume that people reading from here onwards have sufficient knowledge about concepts like Silverlight, Storyboards in Silverlight, C# etc. instead, we will straight away look at some common capabilities of Farseer engine to get started since there are lots of material over SL but very few on Farseer.

Out of the many capabilities of Farseer, I would explain only few capabilities like, PhysicsSimulator class, movement using "Body" class, Collision using "Geom" class. Apart from this I will also cover on how to achieve a Game Loop using Storyboards (although a better method exist in RC0 onwards of SL which is using CompositionTarget) and an organized way to Handling Keyboard Inputs. First we will discuss three classes and then we will move towards Logic of how to apply them.

PhysicsSimulator Class:

This is the main class that developer calls to apply the Physics he wants to apply. Physics Simulator class does many things but for now I could say that you can use it to specify Gravity to the world. It takes values through which you could specify how much gravity any object should experience in X/Y direction. Let's say we have Gravity of 5000 on Y direction and the scene is to represent a constant air flow in X direction, we can apply 100 units of force in X direction. So that ways whenever you will simulate any object, apart from the forces that you have applied, this gravitation and Air will also be applied. This can be specified in its constructor similar to this: PhysicsSimulator physicsSimulator = new PhysicsSimulator(new Vector2(0, 5000));

Movement (Body Class):

Body class is the central class that applies different Physics concepts like Force, torque, etc. It doesn't have any UI. We tie the Body object to a specific XAML object (or sprite) by writing a code which makes changes to different properties of the XAML code based on changes in the Body object. So, for example, if we want to give a physics motion to a rectangle XAML element, we will create a rectangle Body object and write code (lets say a method called Update) whose main job will be to keep the properties of Body object in sync with properties of Rectangle (like Canvas.Left etc.). So the concept is: Farseer world never does any changes directly to XAML (neither body object nor PhysicsSimulator). Physics Simulator only simulates the body object, it is we who make changes to the actual sprite based on the changes that Physics Simulator has done to body. So, Physics simulator will move the Body object according to the laws of physics and our code will merely keep our sprite in sync with Body object. This is important because now we can decide which properties to change based on the changes in Body object and not necessarily tied upon the Farseer Engines default functionality.

This feature essentially provides a loose coupling between the Engine and application making it reusable in different ways. Jeff could have written logic to change specific properties depending on the type of object in question. But then, we would have problems like:

It would have been impossible to use Farseer with the newly added classes/objects/third party XAMLs etc.
It would have become more time consuming for Jeff to build Farseer and also the engine could have become more heavy (which is a No Go for SL based applications). there are endless list of problems that could have got associated but thanks to Farseer designers for making it very generic and usable in different scenarios.

Collision (Geom Class):

We enrich a body further by providing it a geometry.

Collision is achieved through Geom (or Geometry) class. I don't know why Farseer implementers changed its name from Geometry to Geom so I will use this name interchangeably. Geom class is responsible to decide how a body should react when someone comes in its path. There are several factors in Physics which decide the result of collision. Like Friction (if its not a head on collision), Coefficient of Restitution etc. which we can set and then associate it with a Body.

In its creation process(basically using factory pattern) itself, we can specify the Physics Simulator to be used and the Body on which it has to act. You don't have to write any logic to handle collision. Based on the properties set to the Geometry of a Body, Physics simulator decides what should happen to other bodies 9which should also be controlled by the Physics Simulator) when they collide. The collision result will be calculated automatically. As said earlier, we have to keep our sprite in sync with Body object in the game loop since Body is the central place where Physics Simulator keeps its end result of any computation. We will see this shortly in a code segment. I will try to upload the code as soon as I finish this post but I might get busy so, if someone needs code sample for the explained concepts, please feel free to contact me by clicking on the email link on my home page of blog/share email via a comment (though that will be visible to everyone).

Game loop:

Game Loop is something that keep things going. It is kind of a timer mechanism. faster the game loop, the more real time the end user experience will be but at same time, a fast game loop will consume more memory (and sometimes its unnecessary). You have to decide the game loop frequency on your personal need basis.

In Silverlight, you could achieve the effect of time through DispatcherTimer but that is not a recommended approach, so people usually go for Empty Storyboard approach. But with release of Silverlight 2 RC, we have a better approach. Though I will show here the empty storyboard mechanism.

In empty storyboard approach, we create a storyboard in the code, set its duration, associate a event handler for completion and begin the storyboard. The code looks similar to below.

void GameLoop()
        {
            timer = new Storyboard();
            timer.Duration = TimeSpan.FromMilliseconds(15);
            timer.Completed += new EventHandler(timer_Completed);
            timer.Begin();
        }

When storyboard completes, it reached its completed event handler as show below and there we do all that we want to do with the game and at the end re-begin the storyboard.

void timer_Completed(object sender, EventArgs e)
{
//Write all logic that you need to execute every 15 milliseconds (in this case).

timer.Begin();
}

Simple!

Handling Keyboard:

Keyboard inputs can be handled directly by handling key down, key up events of the Canvas of our XAML. But it is not advised to handle it that way (you will do it if you want a real real realtime response). I am telling this because SL based applications has to be least resource intensive so it is good to share good practice. otherwise you could ignore this section and write canvas event handlers manually.

In a more optimised approach, we handle key inputs of canvas, and instead of responding to it directly, store the key stroke details in a shared array. When game loops itself (maybe 15 milliseconds in this case) we will simply fetch the key press activities from the array and react to it. this will allow us to move at our own pace rather than getting things hanged/behave abnormally because of an impatient end user hitting arrow keys more like a madman. Attached is a class file that is used to wrap this logic in the form of a class. the class file is present in the getting started samples of Farseer available from Codeplex web site. So, instead of reinventing the wheel, I am representing it as is. see the attached KeyHandler.zip file which has classes associated with it.

Farseer Sample Logic:

Okay, so enough of theories, here is one logic that you can apply to create a simple rectangle and circle and then apply motion to it along with handling collisions.

You need the sprites. In our case these would be XAML files for rectangle and circle having associated code behind files. The code behind files would contain nothing else than an object of type Body and a public function called Update (remember? This is the function which will keep different properties of rectangle/circle in sync with the contained Body object). here is a sample XAML and code behind class structure:

Rectangle XAML and Code Behind:

</UserControl>

public partial class MyRectangle : UserControl
    {
        //We could have implemented like a property but for simplicity I am not making it very complex.
        //This is the RectangleBody object that is changed by Physics simulator.
        //Our update method syncs the coordinates of rectangle with properties of this RectangleBody
        public Body RectangleBody;
        float x, y, rotation;

        public MyRectangle()
        {
            InitializeComponent();

}

        //Exposing size as a property so that size of rectangle can be set from external code
        public Vector2 Size
        {
            set
            {
                //Vector class is just a structure of packaging things. Its upto us to use it the way we like

                //We are seeting the width and height of rectangle and the user control
                //equal to the value supplied by external code
                this.Width = value.X;
                rectangle.Width = this.Width;
                this.Height = value.Y;
                rectangle.Height = this.Height;

                //Required to match the coordinate systems
                translateTransform.X = -this.Width / 2;
                translateTransform.Y = -this.Height / 2;
                rotateTransform.CenterX = this.Width / 2;
                rotateTransform.CenterY = this.Height / 2;
            }
            get
            {
                return new Vector2((float)this.Width, (float)this.Height);
            }
        }

        //Method that keeps properties of the rectangle in sync with the RectangleBody properties
        //Look how we are simply mapping the properties of Canvas with body properties
        public void Update()
        {
            if (RectangleBody == null) return;

            if (x != RectangleBody.Position.X)
            {
                x = RectangleBody.Position.X;
                this.SetValue(Canvas.LeftProperty, Convert.ToDouble(x));
            }

            if (y != RectangleBody.Position.Y)
            {
                y = RectangleBody.Position.Y;
                this.SetValue(Canvas.TopProperty, Convert.ToDouble(y));
            }

            if (RectangleBody.Rotation != rotation)
            {
                rotation = RectangleBody.Rotation;
                rotateTransform.Angle = (rotation * 360) / (2 * Math.PI);
            }
        }
    }

Circle XAML and Code Behind:

XAML and code behind of circle is almost similar to rectangle above. Please see the attached code for details.

Main Page Code Behind (Page.xaml.cs)

Important details added:

public partial class Page : UserControl
    {
        PhysicsSimulator physicsSimulator;
        MyRectangle rect;
        Body rectBody;
        Geom rectGeom;
        Body circleBody;
        MyCircle circle;
        Geom circleGeom;

Storyboard timer;
Vector2 force;

        public Page()
        {
            InitializeComponent();
            //Create Physics Simulator
            physicsSimulator = new PhysicsSimulator(new Vector2(0, 5000));
            //Create Body. Note the constructor uses physics simulator
            rectBody = BodyFactory.Instance.CreateRectangleBody(physicsSimulator, 128, 128, 1);

            //Rectangle will not move
            rectBody.IsStatic = true;
            circleBody = BodyFactory.Instance.CreateCircleBody(physicsSimulator, 64, 1);//fix

            //Create Geometry and set different physics factors
            rectGeom = GeomFactory.Instance.CreateRectangleGeom(physicsSimulator, rectBody, (float)this.Width, (float)this.Height / 5);
            rectGeom.RestitutionCoefficient = .5f;
            rectGeom.FrictionCoefficient = .2f;
            circleGeom = GeomFactory.Instance.CreateCircleGeom(physicsSimulator, circleBody, 64, 20);
            circleGeom.RestitutionCoefficient = .5f;
            circleGeom.FrictionCoefficient = .2f;

//Add these bodies to canvas
AddtoCanvas();

            //Attach the bodies to physics simulator so that they can be managed by it.
            physicsSimulator.Add(rect.RectangleBody);
            physicsSimulator.Add(circle.CircleBody);

            //Call their update method once during load tiem so that their initial position and size are set
            circle.Update();
            rect.Update();
        }

         //This method adds the XAML (sprites) to the main page's XAML
        void AddtoCanvas()
        {
            //Create a rectangle sprite and give it a body, initial size etc.
            rect = new MyRectangle();
            rect.RectangleBody = rectBody;
            rect.RectangleBody.Position = new Vector2((float)this.Width / 2 + 400, (float)this.Height / 2 + 200);
            rect.Size = new Vector2((float)this.Width, (float)this.Height / 5);
            //Create a circle sprite and give it a body, size, position etc.
            circle = new MyCircle();
            circle.Radius = 60;
            circle.CircleBody = circleBody;
            circle.CircleBody.Position = new Vector2((float)this.Width / 2 +500, (float)this.Height / 2 - 100);
            //Add these sprites to main canvas
            this.LayoutRoot.Children.Add(circle);
            this.LayoutRoot.Children.Add(rect);

}

        void GameLoop()
        {
            timer = new Storyboard();
            timer.Duration = TimeSpan.FromMilliseconds(15);
            timer.Completed += new EventHandler(timer_Completed);
            timer.Begin();
        }
        void timer_Completed(object sender, EventArgs e)
        {
            Button_Click(null, null);
            timer.Begin();
        }

        private void Button_Click(object sender, RoutedEventArgs e)
        {
            //Apply force
            force = Vector2.Zero;
            force.Y = -force.Y;
            force += new Vector2(0, 5000);
            //apply force in y direction
            circle.CircleBody.ApplyForce(force);
            //Rotate the ball so that it falls down
            circle.CircleBody.ApplyTorque(-5000);
           //update simulation
            physicsSimulator.Update(0.005f);
            //update the effect of physics simulator by calling update methods of sprites
            rect.Update();
            circle.Update();
            //Start the game loop
            GameLoop();
        }
    }

Please see the attached code for more details.

I hope this would give you a quick start for moving ahead with Physics based animations and Games.

If you are an ISV and want our team to be engaged for training/development projects, please feel free to connect with me here.

Happy Gaming!

Rahul Gangwar

Silverlight and The Application Design

2008-08-06T00:24:52Z

Today is August 5th 2008 and I am writing this short post just to convey how Silverlight changes the overall design of a conventional browser based application. Current public version of Silverlight is 2 Beta2.

There are many reasons to use Silverlight;

To enhance the UI of existing browser based applications; When we install Silverlight plug-in, browser can understand more set of tags (XAML) apart from HTML tags. Moreover we can use the capabilities of the presentation engine of Silverlight to draw better UI.
To scale the application; Since now most of the end user interaction and processing will happen at client side itself, there will be less number of hits on the server. So now the application could support more number of concurrent users.
To do things beyond JavaScript; Not many things can be done with JavaScript which can be achieved with Silverlight. one example could be rendering HD video, which requires us to use graphics card capabilities to render it.
To use existing skill set of developers; Silverlight supports multiple languages (like C#, VB.NET, IronRuby, IronPython etc.) we could reuse the existing capabilities of developers to create better applications over web rather than learning new technologies to enhance the UI of browser based applications.

Apart from these and many other features, one thing that attract me most is the way application deployment design changes with Silverlight development.

Almost every business application has three layers: The presentation/UI layer, Business Logic layer and data Layer. If we talk in context to browser based applications, the presentation layer sits over Browser, while the rest of layers (Business + data) sits at server side. Whenever there is a requirement to do some business processing or access data we go back to the server and this gives a bad end user experience.

With Silverlight, this design changes a little bit, but that little makes huge difference. With Silverlight, Presentation and Business Logic Layer can sit at Browser side while Data Layer can sit at server side. This means that now we have to go back to server ONLY when we need to get some data, otherwise we can sit happily at client side and attend them in real time fashion; just like windows applications do. On top of that, whenever we want to talk to server for data, we do so asynchronously which means end user doesn't come to know about whether server interaction is happening or not.

The above idea is conveyed in figure given below.

Any comments/suggestions/questions more than welcome.

Rahul's

ADO.NET Sync Services: Services Based Architecture's Step-by-Step Demo

2008-06-02T22:25:00Z

Today is June 3rd 2008 and I am writing this blog post to demonstrate a step by step guide to developing an ADO.NET Sync Services Sample Application that is using Service Based Architecture. Though I planned to first write a post on ADO.NET Sync Services as a whole from architecture perspective and then provide a step by step instructions for the sample code, but now, when I have completed the step by step instructions before the architecture document, I am posting the same.

In almost every practical case of disconnected applications architecture, the server side central database is exposed via http end points like web services. Moreover with the release of technologies like ADO.NET Data Services, it is a matter of few clicks to expose the entire data model to the external world using web services.

So I thought of writing down the steps that I took in Visual Studio 2008 to create a client windows application which will be talking to the central database server via a http WCF Service. If you are interested in the sample code as well, please feel free to get in touch with me.

Open Visual Studio and create a new windows application which will store all the client specific files.
Add a new class library project to the solution which will hold all the server side code.
Add a third web application project to the solution which will expose a web service. The web service will be used as a broker between the client windows application and the server side class library. So also add a new WCF service to the web application project. You can delete the Default.aspx page which is created by default by Visual Studio 2008. Also since this will be Cassini based, right click the project and select properties. On the properties page, select “Web*” tab and select “Specific part” radio button. This will fix the port on which the web server will run and we can have a fixed URL to the web service. Click on save.
In the windows application, add the SQL Server Table that you want to cache. You can do this by adding a LocalDatabaseCache file from the project templates. This will present you with a wizard where you will specify the SQL Server Tables. In the first form of wizard, specify the Server Connection to tell the Visual studio about the SQL Server with which you want to do the synchronization. Clicking on the drop down will present you with already known connections, if you want to specify a new connection, click on “New Connection” button adjacent to the Server Connection drop down and fill the required details.
Once you specify the Server Connection details, it will automatically suggest you with a Client Connection name. This basically specifies the location and name of SQL CE database file to be created at client side.
Once you specify the Client Connection details, you can see “Add” button enabled. This button allows you to select the tables on the SQL Server to be cached on the SQL CE database that you just specified in above steps. Going a little deep, this process creates a SQLCE database at client side and simply copies the schema from SQL Server to the CE database. So click on Add button, select tables that you want to cache/synchronize and click OK. Please explore other options shown if needed.
The moment you finish this wizard, you will be prompted with another wizard of creating a DataSet for the SQL CE database just created. If you wish to write your own ADO.NET code skip it, otherwise choose the Tables for which you want Visual studio designer to write the ADO.NET code. For first demo, I would recommend to go through the dataset wizard rather than writing your own ADO.NET code.
Now, we write the server side synchronization code into the class library created. We will do this by using the wizard again. Though we could have done it in previous steps but just to make it simple, I am doing it in this step. Double-click .sync file in the windows application. This will open the wizard that we have seen earlier. Here click on the “Advanced” button to see more options on the same page. Then on the “Server Project location” drop down select the class library project and click OK. This will transfer all the server side code from the client application to the Class Library which will be used by the web service.
Add a reference of the ClassLibrary project into the web application project because web service will use it. Also add references to Microsoft.Synchronization.Data and Microsoft.Synchronization.Data.Server assemblies into the web application project.
On the windows application add “service reference” and add the web service reference to the windows application.
On the class library, you will see a .SyncContract.cs file which has code for an Interface and class. We will have to duplicate the same structure on the WCF files. So copy the method declarations of the interface of .SyncContract.cs file (along with [OperationContract()] declarations) and paste them to the Interface file of WCF Service (in the web application project you will see a file starting with “I” if you have not added any additional files starting with “I” letter). You can delete the default method called “DoWork()” provided by Visual Studio. Copy also the “using” declarations so that there are no build errors.
Similarly copy all the methods and the private variable of the class from .SyncContract.cs file into the .svc.cs file of web service. Copy also the “using” declarations. The code also contains a constructor method. Just change the constructor name in .svc.cs file to the class name of the .svc.cs file. You can remove [System.Diagnostics.DebuggerNonUserCodeAttribute()] declarations (if present) from the methods.
In the same .svc.cs file add another using statement for the class library project (e.g., using ClassLibrary1;). Try to compile and see if there are any errors. Remove any errors if I have missed any logical step.
On the windows application, right click the .sync file and click “view Code”. Here, you will see one partial class already created with an empty OnInitialized() method. Create another class in same file (let us call it RemoteProviderClass) implementing the ServerSyncProvider class. Add following “using” declarations on the top of file:

using Microsoft.Synchronization;
using Microsoft.Synchronization.Data;
using System.Data;
using System.Collections.ObjectModel;

And the class declaration looks similar to this:
public partial class RemoteProviderClass:ServerSyncProvider

After copying the using statements, copy only the methods (not private variable) of the .svc.cs file of web application to this file inside the new class created (RemoteProviderClass). Again, one of the methods will be a constructor, so change its name appropriately. Notice that each of the method is calling methods from a private variable whose name starts with an underscore. Declare a private variable inside the RemoteProviderClass corresponding to the proxy of the web service. The variable name should be same as the one that each method is using and starting with an underscore. Also, inside the constructor method, delete existing code and write a line to instantiate this proxy variable.
Try to build the solution and you will see errors inside GetSchema() method. This is because Collection<string> is transferred as string[] array. So write code to convert the Collection into string array. Also notice the override keyword; this should be used with all methods instead of virtual keyword. The code should be similar to this:

public override SyncSchema GetSchema(Collection<string> tableNames, SyncSession syncSession)
{
string[] tables = new string[tableNames.Count];
tableNames.CopyTo(tables, 0);
return this._serverSyncProvider.GetSchema(tables, syncSession);
}

Everything should compile well now.
On the OnInitialized method write code to create an instance of this new class and assign to the RemoteProvider property of the sync agent. Also specify the sync direction of the tables. Here I have one table in my SQLCE database called Employee. The code looks similar to this

partial void OnInitialized()
{
this.RemoteProvider = new ProxyForWebServiceCall();
this.Employee.SyncDirection = SyncDirection.Snapshot;
}

Now we will create a UI on the windows form and invoke the synchronization process on timer tick event. Calling synchronization is straightforward but calling inside a timer that is invoked every 1 second is a little bit tricky. Practically, web service might return response in 1 second, and also till web service returns back, the UI thread will be unresponsive, so we will invoke the synchronization code asynchronously and only when one web service call has returned. I will illustrate this in following steps and also show the sample code to make things more comprehensive.

Open Form1 design and from data menu click on “Show Data Sources”. Drag and drop the table on the Form designer surface (in my case Employee table). Visual Studio will automatically create UI for end user editing and also write ADO.NET code for performing those operations on the local SQLCE database. Also drag and drop a timer control and set its enabled property to true and set its interval to 1000 (1 second). On the code behind of Form1 add using Microsoft.Synchronization.Data; declaration.
On the code behind of Form1, declare three private variables belonging to the class of the Form. A Boolean variable to see whether we have to call the synchronization code or not. A delegate to the synchronize method of the Sync Agent class. A variable of the sync agent class. To see the class name of your sync agent navigate to the .cs file of the .sync file and see the class name of the OnInitialized() method. The declarations look similar to this:

public partial class Form1 : Form
{
bool IsContinue = true;
private delegate SyncStatistics SynchronizeMethod();
LocalDataCache1SyncAgent syncAgent = new LocalDataCache1SyncAgent();

Now we will write code in the timer tick event handler to invoke synchronization. Here on the very first line we will check whether IsContinue is true or not, if it is not true, this means another web service call is pending and we have to wait, so we will return. But if IsTrue allowed us to enter further into synchronization code, let us block all other ticks of timer to enter. So the following code:

if (!IsContinue)
return;
else
IsContinue = false;

Then we will specify the synchronization direction of our global sync agent variable. I have set it to Bidirectional here. So this code:

syncAgent.Employee.SyncDirection = Microsoft.Synchronization.Data.SyncDirection.Bidirectional;

Next we will instantiate the delegate with the sync agent’s synchronize() method so that we can invoke it asynchronously. Then invoke it asynchronously. Notice that in the second line, we have passes a method name, MyCallBack, (which we will create in coming steps) which will be invoked once web service returns response. MyCallBack will simply set the IsContinue to true so that now timer can start for a fresh synchronization execution. So the code below:

SynchronizeMethod syncMethod = new SynchronizeMethod(syncAgent.Synchronize);
IAsyncResult result = call.BeginInvoke(new AsyncCallback(MyCallBack), null);

After invoking the webs ervice, we can collect the resulty of web service by using EndInvoke. The result will be used to see if the DataGrid should be refreshed or not. If there were changes between two databases, then only form should be reloaded. So the code below:

Microsoft.Synchronization.Data.SyncStatistics syncStats = syncMethod.EndInvoke(result);
if (syncStats.TotalChangesDownloaded > 0)
Form1_Load(null, null);

And finally the callback method:

private void MyCallBack(IAsyncResult result)
{
IsContinue = true;
}

Compile and run the application by setting windows application as startup project. Make changes from windows application and you can see them at SQL server and similarly make changes in SQL Server and they will be visible in Windows Application datagrid. Make sure that you do not change primary key field. This is just a start, go ahead and implement conflict resolution and try to make changes from multiple machines at same time.

I would love to see your comments/suggestions/corrections if any.

Rahul Gangwar

High Scalability Architecture For Data Types Like Image

2008-04-03T21:12:58Z

Today is April 3rd 2008 and this small article focus on the approach of implementing a scalable solution which involve serving large amount of data such as Images. This is an infrastructure pattern.

Ever wondered how a large web site serve content such as Images to end user? Take for example you have a photo sharing web site. Generally we use SQL Server as the back end storage solution for any application architecture. However, not all data types can be stored in the SQL Server. When it comes to data types such as images, SQL Server stores them in the form of BLOB. But BLOB in SQL Server has its own inherent problems. If you do read or write to BLOB, the whole of the BLOB being transferred over the wire and stored in memory. This can cause network congestion and memory pressure, particularly when there is a considerable load of concurrent users.

If you need to read or write BLOB data such as images, you should first consider the options of storing them directly on a hard disk and storing the physical path or the URL in the database. This reduces load on the database. This basically requires two components. One, a File Server (for storing actual data) and then the Database management System to store references of the same. Microsoft provides solution for both the components in the form of Windows Storage Server and SQL Server. The following diagram explains this solution in a better way:

Now, when we understand that dedicated File Server is the solution for storing datat types such as Images, how do we typically implement it? Such a solution is implemented in form of Storage Area Network (SAN) where Network Attached Storage (NAS) Devices are used to store the large data and act as a File Server. In such a solution even SQL Server resides on SAN.

However, if you still want to store BLOB data in the database (maybe because you don’t want to have complex infrastructure of Storage Server), consider the following options to reduce the performance cost:

--> Use chunking to reduce the amount of data transferred over the wire. Chunking involves more round trips, but it places comparatively less load on the server and consumes less network bandwidth. You can use the DataReader.GetBytes (from Application code) to read the data in chunks or use SQL Server-specific commands, such as READTEXT and UPDATEDTEXT, to perform such chunking operations.

--> Avoid moving the BLOB repeatedly because the cost of moving them around can be significant in terms of server and network resources. Consider caching the BLOB on the client side after a read operation.

Comments and suggestions, as always, more than welcome.

Rahul Gangwar

Patterns used for navigating between pages in ASP.NET:

2007-12-21T02:15:00Z

ASP.NET offers different APIs and ways (like Server.Transfer, Response.Redirect etc.) through which you can navigate between pages in your web applications. But before choosing any of these ways, we must have an understanding about the pattern that we want to follow. Depending on the pattern used, our choice about selecting an API changes.

Broadly, there are two patterns used for navigating between pages depending on the requirement:

1. Loosely Coupled Architecture:
In this pattern, each page has its own processing unit. So if you have a page that has some input controls, its code behind will have all the logic to form SQL Queries based on input and get the result also from database. Then when it comes to displaying the result, another page (the results page) is invoked and the data of previous page are passed to the new page. This is called loosely coupled architecture since each page is given a responsibility and it performs its work without being dependent on any other parts of the system. So one page is given the responsibility to gather user input and prepare the result and another page is given the responsibility to display data. The picture given below illustrates this architecture in a better way:

When to use this pattern:

Now when we know that what this architecture means, we can appreciate the scenarios where we can use it. The scenarios are too wide, and one possibility that I can think of is the scenario where you want a decentralized approach. Whenever we have scenarios where processing logic is different for each page, we go for this approach since there is no point in delegating the processing logic to results page. If we delegate the processing logic to results page, we will have many results page so to avoid this, we choose to keep the processing logic in the input gathering page itself.

How to implement:

The default model of ASP.NET provides this facility where each page posts to itself until we specifically configure the controls to post to another page. The steps involved are:

a) Implement the input page with all the data preparation logic in its code behind. The controls on this page will post to itself.
b) Invoke another page when the processing is done and pass the processed data to the target page. This second step can be achieved in many ways depending on the requirements. Below I show few techniques that can be used.

a) On the first page’s code behind (having input controls), when all the processing is over, add the data to be transferred to next page into the “Context” collection (Context.Items.Add() method can be used for adding into Context collection) and then invoke the target page using Server.Transfer() . On the target page (the results page) retrieve the data from the Context Collection simply by using Context.Items[] collection and typecast it if needed. Context collection remains into scope this ways on the target page as well (remember we cannot use response.redirect() method if we want to use Context collection to store our data since context dies the moment the request thread leaves the server).

b) Another API to achieve this is to use Request property. This basically gives you reference to the current context and then using Request.Form we can access the source page.

c) If you want to use the data between multiple requests, you can store data into Session collection instead of context collection. Everything else remains same (and yes we can use response.redirect() method if we are using session to store our data).

d) A third way could be to get the reference of the source page on the target page. Whenever we use Server.Transfer() method to navigate to another page, we can access the source page from Context.Handler property. So all we need to do is to create storage on the first page (for example by creating a property called data on source page) and store the result in that property. Then, on the target page access the properties (in fact everything that is public on the first page including controls) by using Context.Handler. This approach is used in cases where we have many discrete data elements to be transferred to the target page. So instead of adding them to Context Collection on first page, we add into the page itself by creating properties. Then we can access all the properties of source page on the target page. Another good thing in this approach is that we do not need to typecast in contrast to Context/Session approaches. More can be found here:
http://msdn2.microsoft.com/en-us/library/6c3yckfw(vs.71).aspx

2. Coupled Architecture (Centralized approach):
In this pattern, we designate a central page for processing the input and displaying the results. We form separate pages for gathering input. The input pages gather data from end user and post it to central processing unit (the results page). This is the approach where you post the source page directly to another page (the cross post scenarios). The following figure illustrates this:

When to use this pattern:

This approach is used whenever we want to reuse the code. If the logic can be reused across multiple pages, then there is no point in writing the same logic on multiple pages and making code unmanageable. In that case we create a central processing unit (the results page) and all other pages just collect data from end users and post themselves to the results page.

How to implement:

We can use PostbackURL property of controls to specify the target page. In this approach, on the source page we specify the target page URL in the PostbackURL property of controls (for example a submit button). This ways, when end user clicks on the submit button all the data of the input page is sent to the target page directly. So no need to write any code behind on source page. A very clean way to implement this pattern.

Yet another model can be followed which involves, first posting to the source page itself and then source page transferring to the target page but no processing is done at source page. We can do this by using Response.Redirect() and then passing the input data using query strings.

Rahul's

ADO.NET Entity Framework

2007-12-08T03:48:14Z

ADO.NET Entity Framework is Microsoft's step towards its Data Platform Vision. In this blog, I am going to talk about the architecture of the ADO.NET Entity Framework and the way it will change application development style. Though it is in Beta Phase right now but is expected to be part of Visual Studio 2008 during mid of year 2008.

The Vision:

If we closely look at the new development initiatives from Microsoft, we can clearly see where Microsoft Development Platform is heading towards. Be it the new "language features" in C# 3.0 like LINQ or new "components" in framework like Communication Foundation, Workflow Foundation or the new features in "products" like Biztalk, SQL Server etc., one thing is common in all of them: "Abstraction". By the term abstraction I mean hiding all the complex details and exposing simple interfaces to achieve one goal of allowing the developers to concentrate on "what" do they need rather than "how" do they need.

Take example of LINQ. Simply put, the primary aim of LINQ is to develop a common querying language within the programming language so that developer does not have to learn new querying languages (like SQL, XPath etc.) varying on the kind of data source they are dealing with. Today with LINQ, you don't need to know SQL or XPath to search data from a database or XML data source. You just need to know LINQ and search every data source with it.

Now look at the structure of a typical query expression in LINQ. You will find the structure similar to SQL having keywords like "Select", "From", "Where" etc. along with a fleet of operators to work with. It is typically the smell of any 4GL language. SQL is a 4th Generation Language, similarly LINQ is a 4th Generation Language. And what is a 4th Generation language in anyway? Any construct that offers me the ability to specify "what" without asking me "how" is a 4th Generation language.

Take another example of WCF. You can now create generic services in WCF within no time that can work on REST principle (REST is like backbone of today's web) and that can also work on a completely proprietary protocols with very little efforts. In WCF we call them "Service" which is as generic as we have "object" type in any OO Programming Language. The same service can behave as a web service, can use message queues, can use TCP with just simple changes in configuration. Developer does not have to worry about intrinsic details. In fact whenever I personally design an architecture for Microsoft Partners, I either think of WCF or Biztalk whenever communication comes into picture. These two technologies are key to enable communication framework within software components as well as enabling integration between different disparate systems.

The examples can go on and on since the technologies offered by Microsoft are many. But the point that I am trying to make is that Microsoft has this vision to introduce abstraction to such an extent that things become extremely simple and we can produce great software within no time, no cost.

With ADO.NET Entity Framework, the Microsoft vision remains the same: To abstract data access with programmers so that they spend their time in writing "what" rather than "how". Imagine if the 4GL concept spans across all layers, application development will be over in the requirement gathering phase itself since the moment we specified 'what" do we need from application, application is ready. ADO.NET Entity Framework is a little step towards that vision.

Introduction:

A typical application has those 3 layers: Presentation, Business and Data Access. ADO.NET Entity framework will change the way Data Access layer is written. One way of looking at it is to consider as an ORM (Object Relational Modelling) framework. Any ORM framework aims to bridge the gap between the way data is represented in database (or better data source) and the way it is represented inside the application. An ORM framework does that by providing a framework that does the mapping between your classes and properties within it with database tables and their columns. This framework is an extra layer between the data access layer and data source. So developer do not have to write code against database, fetch data in some format, convert it to fit with objects in application and then use the data. Instead, developers do all their modifications on objects only and the changes are translated by the ORM framework into appropriate database calls. ADO.NET Entity Framework is also an ORM framework that provides this functionality. ORM frameworks are not new concepts, we have plenty of them (like nHibernate) in market. But what is different with ADO.NET Entity Framework is that it is much more than simply an ORM framework. Not only does it bridges gap between representation of data within application and database, but also it aims to improve upon performance (unlike other ORM frameworks that hits to performance) and more importantly it integrates with the programming language itself (we have LINQ to entities for the same reason) making this integration more flawless and easy to use.

Architecture:

I would take the picture (shown below) from one of the "data points" column of MSDN Magazine that clearly shows the high level developer architecture of ADO.NET Entity Framework. Let us dissect each element to appreciate the architecture better.

Entity Framework Layers (Logical + Mapping + Conceptual):

The three entity framework layers shown in above diagram are the core of whole functionality. These layers are also called as EDM (Entity Data Model). Entity framework engine basically takes decisions on the basis of the EDM. The first step that we have to do in order to use the entity framework is to generate (optionally edit based on requirements) these application specific layers. But do not worry, generating these layers (EDM) is not more than a matter of clicks. You can generate an EDM using a database as a starting point. You can then modify the XML manually (or possibly using a modelling tool that may be available in a future release of Visual Studio). When you add an ADO.NET EDM to your project, the wizard walks you through the process of creating the EDM.

The Logical layer contains the entire database schema in XML format. This represents the story at database side.

The Conceptual Layer is again an XML file that defines the entities (your custom business objects) and the relationships (the custom ones) that your application knows. So if you say that each Customer can have many associated orders, that relationship is defined in the conceptual layer even if this relationship does not exist at database side.

The Mapping layer is also an XML file that maps the entities and relationships defined at conceptual layer with the actual relationships and tables defined at logical layer. This is the core of entire Entity Framework that bridges the gap.

Within application we always interact with Conceptual layer. To talk to Conceptual Layer we can use

A new language called Entity SQL directly (actually Entity SQL queries are fired using Entity Client provider which is very similar to ADO.NET model).
LINQ to Entities, which is again another language integrated querying facility.
Using Object Services which in turn uses an ObjectQuery Object.

Whenever we commit changes (like invoking SaveChanges() method) to databases from application, appropriate queries are fired (it also takes care of the relationships). We don’t have to think about relationships at application level since all the mapping is done at EDM layers. So, Entity Framework handles Relationships quiet neatly.

Wrapping UP:

The best wrap up that I can write is what John Papa gave in his article:

"The Entity Framework allows developers to focus on the data through an object model instead of the logical/relational data model. Once the EDM is designed and mapped to a relational store, the objects can be interacted with using a variety of techniques including EntityClient, ObjectServices, and LINQ.

While the traditional objects such as the DataSet, DataAdapter, DbConnection, and DbCommand are still supported in the upcoming release of ADO.NET available in Visual Studio "Orcas," the Entity Framework brings major additions that open ADO.NET to new and exciting possibilities."

There are other Microsoft initiatives in the data platform vision like having a framework to expose data as service and tools for developing for an occasionally connected architecture. More on those initiative in my future blog entries once I get my hands dirty with them as I have done with Entity Framework :).

Rahul's

Technorati Tags: ADO.NET Entity Framework Architecture,LINQ to Entities

Microsoft Vision for development platform - Codename Oslo

2007-11-20T03:23:04Z

If someone asks me Microsoft's strategy on application development, I would simply say SOA spanning across the servers, clients and the cloud. Microsoft is hitting on its strategy of SOA by releasing new products and platforms at a rapid rate. The next big thing from Microsoft towards this goal will be "Oslo".

Today in any typical application development scenario, there are various experts involved at different levels. Each if these experts try to develop a model of the area in which they have expertise. For example, business analyst might model the software requirements, architects model the high-level designs, administrative staff might model the deployment and maintenance requirements etc. Then developers develop the code corresponding to the model.

The approach is perfect where each person is doing his job in the field the he is expert but at the same time there are some impedance mismatch between these different sets of people. The models developed by an analyst is developed using entirely different set of tools than the tools used by an architect or developer. Not only this, if someone wants to be an end-to-end expert of the software lifecycle he will have to learn a vast range of tools, modeling languages etc. Even bigger than this is the problem when it comes to development. Developers have very less usage of the models that other experts have created as far as the development efforts are concerned. Models just remain a source of reference and this is the reason why modeling is still only at the surface of the software lifecycle.

To bridge all sorts of mismatch problems, Microsoft plans to bring an integrated environment for everyone. Analysts, Architects, IT professionals, Developers can all work together, and not only work together but also benefit from the model created by others practically. Microsoft is making it possible by building a general-purpose modeling language that everyone can use, common tools to automate the tasks and a repository where all the models will reside.

"Oslo" is the codename for all the technologies that will make this possible. With Oslo we can imagine a development environment where users will design a model for an application that then becomes the application itself. This will shorten the time from design to development and bridge the different phases of software lifecycle to make it a continuous process. The "Connected Tools Team" in Microsoft is responsible for the new modeling tool that will make this process a reality.

Oslo includes:

--> Future release of .NET Framework 4. So we can expect that .NET framework 4 will have further enhancements to the WCF Workflow Foundation Technologies.

--> Biztalk Server family will evolve for enabling a highly scalable SOA and BPM solutions. The Biztalk Server is expected to be built on top of WCF and WF and will also have the capabilities to develop, deploy and manage composite applications.

--> We will also see the release of Biztalk Services that will allow the development of cross organizational composite applications.

--> We will see a wide variety of tools that allows the dream of model driven development to turn into reality. So this is basically the extension of Visual Studio Team System capabilities which will be the unified environment for all class of experts bridging each phase's output with input to next phase.

--> Better repository system for metadata.

http://www.microsoft.com/soa/products/oslo.aspx

Rahul's

Technorati Tags: Oslo , M.NET Framework 4 , Composite Application , Model Driven Development

Silverlight

2007-08-29T22:49:00Z

Conceptual Silver Light:

I am writing this entry because I feel Silver Light to be the sword of developers in Web 2.0/S+S world.

I remember my first introduction to HTML programming in my school days. One of my seniors was doing all the HTML stuff in an attempt to design an attractive web page. When I saw the way he wrote html tags that was a surprise to me (more because it seemed quite terse to me). But then he told me that its actually very easy and intuitive if you know the elements that you have to use.

That said, I started learning HTML and came to know about various buzzwords like DHTML, JavaScript, DOM Object Model etc.

I request your patience for reading the HTML stuff because this will suddenly and finally open up the doors off Silver light.

One beautiful concept that I like about HTML programming is the Object Model. In HTML programming there are several HTML elements like <html>, <head>, <Table> and list goes on.

The basics of HTML programming lies in the fact that there are fixed (though large) number of elements that a standard browser understands. And these elements are nothing more than a way to tell the browser that “Hey! Organize the UI in this way”. So in effect, HTML can also be considered as a formatting mechanism used to format the UI on browser. I always wondered what if I want to have a UI that is not represented by any of the available elements… then the basics of composition (nesting in HTML) solved the purpose. So, if we want to create a UI that is not represented by any standard element in HTML, we can create the required UI by combining the basic HTML elements in appropriate form.

The browsers evolved in terms of the capabilities, they started understanding DHTML elements also for some moving UI, then browsers also supported scripting languages like JavaScript for both making UI flexible as well as interacting with end user and manipulating HTML elements. Then came more advanced capabilities in browser like asynchronous client script callbacks, XMLHTTP etc which led to cropping up of many new technologies similar to AJAX.

Still, I feel a bit limited in the way I program windows application and a web application. The complexity becomes even more complex when we have to do the multimedia, animation etc. IMHO, the primary reasons to why web programming is still complex are following:

1. There are two sides of the web programming framework, client and server. Both talk to each other using http. HTTP has inherent problems like being stateless etc. and we came up with different solutions (rather patches like session state etc.).

2. Handling end user interactions involve either roundtrips (which has another set of drawbacks) or through client side languages like JavaScript etc. This personally makes me hesitant in doing more things at client side and instead delegating it to server side. We can do stuff like asynchronous callbacks etc. but they again require lots of plumbing.

3. The UI controls of web programming are less rich than the windows programming because we have to get limited by the capabilities that browser provides. We really cannot do anything beyond what browser permits even if .NET or any server side platform has immense capabilities.

These and many other factors make web programming a little poor.

For me, most of the problems will be solved if we have the same capabilities on client side that we have at server side. So that said, if we have .NET on client side as well, we just need to program the application the way we used to and the application will run at server side similar to a windows application.

In this attempt Microsoft started off with smart client applications (click once) with added benefits like automatic updates etc. Great! But they did not pick success in developer world since not all clients are expected to have a .NET framework installed.

To overcome this functionality, Microsoft designed a new .NET framework which they called as Silver Light and this is a small size, mini version of .NET framework that can be installed on client side as a plug-in to most of the modern browsers. This mini .NET framework was geared towards making UI at client side more attractive and is currently oriented to provide the multimedia and animation capabilities on browsers similar to technologies like flash.

However I see this from a different angle because this makes me program it in a better way and appreciate the vision.

When I install Silver Light on a browser apart from considering as if I have the power of .NET at client side, I also look it as increasing the capability of browser all of a sudden. I install Silver Light and Boom! – My browser can now support more tags than ever before. Apart from understanding <HTML> <HEAD> <TABLE> etc. it will now also understand <CANVAS> <BRUSH><TEXTBLOCK> and many more. Now I don’t have to peek around for getting things done for my complex requirements but instead, now I have increased the element set of browser. Not only the element set; now I can also use either Java Script or my favorite C#. Isn’t this great? Well for me this is like a phenomenon.

The following diagram illustrates this more clearly:

We can see that Inside the DOM we have another DOM like object model which is the Silver Light Object Model. The tags that we write in DOM are called HTML while the tags that we write inside the Silver Light DOM are called XAML. Simple! Now just learn XAML and the way to access XAML and HTML elements using C# and we are set to program Silver Light.

Though Silver Light holds lots of promises for developers in its current stage it’s fighting with some initial hiccups like, right now we cannot use C# but instead we will have to still use JavaScript in the 1.0 version. However in Alpha version we can write in .NET compliant languages like C#, etc.

Moreover, at present, Silver Light is geared for Multimedia but we can see more controls in Silver Light class library to have most of the capabilities of any windows application. To me Silver Light holds promise for next generation Mash ups in web 2.0 or S+S world since it holds most capabilities and flexibility out of all the available development technologies from Microsoft.

Rahul's

SaaS-Value Proposition

2007-07-25T01:13:00Z

Is the Total Cost of Ownership of a SaaS application really low than a traditional application for an end user?

SaaS is buzzword in today’s delivery model of software. Many ISVs are hastily looking for converting their existing traditional applications on to the SaaS strategy. Once the product finally moves onto SaaS delivery model, then starts the process of attracting customers/end-users. The primary reason that customers (especially SMBs) are told to move to the SaaS product is that the total cost of ownership is low for them. The tenants don’t have to invest on the hardware, maintenance, employing IT staff etc. Moreover, customers are charged on per month/pay as use etc. model. SaaS offers so much of flexibility in terms of acquiring the software that customers fall prey of SaaS product immediately. The value proposition of TCO is so much tempting. That’s a good thing but at the same time this value proposition is basically confusing the customers about the other value propositions about SaaS. Not only this, at few instance the value proposition of TCO is actually wrong. There has to be a good understanding among both ISVs as well as customers about the concept and philosophy of SaaS in order to fully appreciate the SaaS model.

Let’s talk about the TCO of a SaaS application for a tenant. When a tenant buys a SaaS application, definitely, the TCO is low as compared to its traditional approach version. But how far? If we take an example where the customer is going to use the software for typically longer period it is quite possible (not necessary) that SaaS application will cost the customer much more than the traditional version. It will actually be cheaper for the end user to buy the software, build infrastructure and maintain it rather than paying on a monthly basis. Yes that is true.

I can give an analogy to the real world to understand the concept. When I go out to another city (maybe for vacation) I will need a vehicle to move around the city. I have two options there, buy a new car for me or hire a cab. Of course since I am there for a less period, the cab will intuitively be a better option. But what if I use cab even in the city where I live? Definitely as time moves from one year to another and so on, I will realize that buying a Car was a better option. Similar is the concept of SaaS application. In software field, some things change and are not same as the real world of Cabs and Cars. Those differences are the real value proposition for SaaS according to me.

For example, in software world, our needs/business requirements changes and we will need a framework to accommodate these changes into our existing infrastructure without investing heavily. Of course, it will not be a good idea to get a new version of application according to our business changes. SaaS application is built from ground up keeping these business changes in mind. A SaaS application provides the flexibility to customers to change the business rules.

Among other features that make SaaS approach more valuable include ease of sharing Data to third parties, availability of backup data centers for disaster recovery and, especially for smaller businesses, having access to the enterprise level application services that would be near impossible to achieve in-house.

Among other benefits of moving to SaaS, if I start counting I see the list endless. Things like achieving on-demand scalability just by a little extra, turning variable cost into fixed costs by delegating the variable costs on to the hosting provider, Integration with other LOB applications… list is endless.

The idea that I want to convey here is that for end users cost is not the sole value proposition, but when we look at the total value that SaaS can deliver, SaaS becomes even more compelling.

Rahul

How to SaaS

2007-03-27T22:20:00Z

This is my first blog entry and I am happy to get this blog meant only for Microsoft Full Time Employees.

Okay coming on to the title for this blog post. Today I am going to write down on Sofware as a Service (SaaS) architecture strategy.

I will un-cover the technical aspects involved in building a SaaS application and very high level overview of the technical aspects of a SaaS solution.

These concepts can be applied to any Application which is a good candidate for a SaaS application.

Contents:

1. SaaS- Introduction

2. SaaS -Challenges

3. SaaS -Solution to challenges

SaaS-Introduction

A SaaS application can be defines as any “Software deployed as a service and accessed using internet technologies”. In order to realize a SaaS solution (in fact any solution) we need to do two things:

1. Build our application which can be used as a service over internet by different clients. This could range from a programmatic service (accessed programmatically by other softwares) to a stand-alone application used by different clients. This step is involves us to follow certain steps that are different than developing a normal “On-Premise” or simply called a non-SaaS application.

Typically when we develop an application (a non-SaaS application) then we do assume two things:

· Application will be installed at client side (e.g., a desktop application) be it through a CD or downloading via internet. This also implies that the client will have to do the maintenance (like upgrading to new version, apply patches, maintain databases etc.) of application even after he has bought the application (not true for Application Service Provider model). This fact is not true in terms of a web application. So we can say that any web application forms a different category of application since it is not installed at client side. But still we cannot call ANY web application a SaaS application.

· Application will be served to requirement of a particular client. If another client needs some changes in the application, we will make changes in the application source code and then run another instance of that application for the new client. So in nutshell, we assume that one application is meant for one client only. This assumption is true even for a web application.

Here comes the difference! The core of a SaaS application is based on the opposites of these two (above mentioned) assumptions. The following are assumptions that are true for a SaaS application.

· Application will be typically deployed and maintained by a hosting provider. So, clients don’t have to invest in terms of buying hardware resources and employing the IT staff to manage the application. This will be done by the hosting provider.

· Application will be accessed by clients using internet. In special cases where a SaaS application is hosted inside the enterprise itself this is not true but still the clients accessing the hosted application will still use internet technologies to access the application.

· A single SaaS application (in fact a single database too) serves more than one client having different needs. So the application will be designed in such a way that only a single application instance will be able to provide different functionalities to different clients. This model is more popularly known as Multi-tenant model.

Among these assumptions, the last assumption related to multi-tenancy requires a SaaS application to be architected in a special way keeping in mind about certain aspects of a multi-tenant application. These aspects form the challenges of building a SaaS application and also our next topic of discussion.

2. SaaS world does not end just by building the application that satisfies SaaS characteristics. Deploying a SaaS application forms another set of challenges. Typically a SaaS application is deployed by a SaaS hosting provider and the hosting provider is responsible of maintaining the application. So, not only clients get rid of maintaining the application but also ISVs get rid of that aspect.

Microsoft provides end to end resources in developing and deploying a SaaS application. It provides resources ranging from development frameworks and technical resources used to build and design the application to hosting options that assist in deploying the application.

SaaS-Challenges

As we have seen that one of the important characteristics of a SaaS application is multi-tenancy, this aspect implies further challenges in building a SaaS application. Let’s see those challenges after trying to define the multi tenancy.

Multi Tenant Architecture:

A single SaaS application serves more than one client having different needs. This means that you run only one instance of the application and that single instance will be accessed by all clients. For example, you built a CRM SaaS application and hosted it over a SaaS hosting provider. Now many organizations have different CRM needs depending on the nature of business they do. We will not build any separate application for all the companies but our single application will have the option using which each client will customize the SaaS application according to his need. Not only that a single database can be used (not always) for storing data of different clients. So security is also a major concern. Let’s see all the concerns and challenges that we need to consider while building a SaaS application.

Challenges

Here are some important challenges that we need to consider while building a SaaS application. It is important to note that not all challenges are required in a single SaaS application. These challenges depend on your application’s requirement. Here only the challenges are presented. Next section will deal about ways to address those challenges.

Business Agility:

Different organizations can have (in fact they do have) different business needs. For example process of approving a loan in two banks may be different in a way that Bank-X requires a Loan Manager to approve the loan request document before sanctioning the loan while Bank-Y requires both the Loan Manager and Bank Manager to approve the loan request. Now if we are building a SaaS application to target these different business requirements, then, we must provide a way through which different clients can configure the SaaS application according to their business requirement. This is what we mean by Business Agility (we can call it as ability of an application to configure its business rules).

User Interface Configurability:

Each client of our SaaS application might have the requirement to change the UI of the application according to their need. This could range from configuring color, font, Company Logo etc. to positioning , ordering of UI elements (for example web parts) etc. incorporating this capability of configuring the UI of a SaaS application according to client’s requirements form another requirement and thus challenge to building a SaaS application.

Database configurability:

Since typically a SaaS application uses a single database (not always) for every client, all the data of different clients can be stored in a single database. This approach makes a SaaS application more efficient since only a single database is used but this also poses challenges on how to design the database so that different data model needs can be accommodated in a single database approach. This approach is commonly known as Multi-tenant database.

Security:

Since a single application instance caters to requests of all clients, it is important (in fact a must) that we have some mechanism to make sure that data of each client remains secured and users of a particular client can access data meant for them alone.

Scalability:

A single application can be used by different clients and each client can have a number of users accessing the SaaS application. This puts a constraint that the application must be able to handle increased number of users. Scalability is a challenge that has to be handles at 3 different levels: The SaaS application (your code) must take care of scalability issues, the SaaS application server (generally done by hosting providers) must be scalable and then finally the database server must be scalable. Scalability at all these three different levels is achieved using different techniques that we will see later on.

SaaS-Solution to Challenges

We will take all the challenges, discussed in previous section, one at a time and propose techniques used to address these challenges. To some techniques we will provide the direct links to already existing resources instead of dwelling those techniques in detail out here.

Business Agility:

This aspect involves designing your SaaS application architecture in such a way that clients of the application can modify the business logic written in your application. Now, how do we provide the modification of business logic of application without writing new code or even modifying existing code?

The most commonly used approach is Metadata driven architecture. Let’s see how it work and what ways (maybe tools etc.) do Microsoft helps in designing such applications.

Concept of metadata driven architecture is quite simple and complex aspects are handled by Microsoft technologies and tools. In a metadata driven system there are three major steps involved:

Metadata: Whatever features of an application are to be configured during runtime, we define those features of the application in the form of Metadata (it could be an XML file stored on hard drive or a backend database). This metadata can be extended by writing more data to it.
Metadata Services: A set of programmatically accessible services are designed that provides an interface to access/modify the metadata. This can be thought of as an interface layer or façade layer. We provide these services so that metadata can be exposed (maybe over internet) to any application/tool while keeping the integrity, validity etc. of the metadata intact. Imagine a situation where an application (by mistake) modified the metadata that violates some core semantics of metadata. Providing an interface layer to any data store is a common principle in building a distributed application and any SaaS application is on first place a distributed application.
An Engine: The business layer of our SaaS application will be nothing more than an engine that runs the business processes defined in the metadata store and accessed using metadata services. These steps could be a bit complex depending on the configurability requirement of your SaaS application therefore Microsoft provides a number of options to build such systems.

Most commonly used technology is using BizTalk Server. BizTalk server has two main components:

A process execution engine that manages the steps, applies the business logic, and invokes the supporting applications of a complex process and/or transaction set.
A multi-transport messaging hub for transforming and routing information to and from applications and process steps.

Another technology seen by developers in their most loved development environment (Visual Studio) is Windows Workflow Foundation (Part of .NET Framework 3.0) used to model business processes as workflows.

Yet another technology from Microsoft is Microsoft Web Services used to build Service oriented applications which in turn is essential to deliver the business agility and IT .

For more details on these technologies you can refer to following resources:

http://www.dmreview.com/article_sub.cfm?articleId=1075079

http://www.microsoft.com/technet/prodtechnol/biztalk/biztalk2004/planning/business-rules-framework-overview.mspx

http://www.microsoft.com/technet/prodtechnol/biztalk/biztalk2004/planning/bpm-solution-overview.mspx

http://msdn2.microsoft.com/en-us/architecture/aa699456.aspx

http://msdn2.microsoft.com/en-us/architecture/aa948857.aspx

User Interface Configurability:

The process involved in allowing end users to modify the user interface according to their need is similar to achieving Business Agility- Using Metadata Driven Architecture. End-user preferences are stored in form of Metadata and UI is composed using this metadata. User interface metadata defines connections into user interface elements, such as menus, Windows Forms controls, and HTML.

http://msdn2.microsoft.com/en-us/library/ms954610.aspx

http://www.microsoft.com/technet/solutionaccelerators/cits/iwp/ibf/soldev/moibf1_sdg_07.mspx

http://msdn2.microsoft.com/en-us/library/aa480019.aspx#aj1metadat_topic13

http://msdn2.microsoft.com/en-us/library/ms136029.aspx

Database Configurability:

For a SaaS application we can use different databases for each client and also use single database for every client depending on the need of the application. This asks us to design a Multi-Tenant Data Architecture (where tenant is equivalent to client).

This involves choosing an approach to in a given scenario and also how to realize that approach. There is a very good article dedicated to developing a Multi-Tenant database system. I would recommend going through that article while going to develop the data model for your application.

http://msdn2.microsoft.com/en-us/library/aa479086.aspx#mlttntda_topic4

Arcast:

http://www.skyscrapr.net/blogs/arcasts/archive/2006/07/18/232.aspx

Security: (Most of the content of this heading is from the famous Catching the Long Tail article)

As we discussed earlier, security is an important aspect of a SaaS application (in fact any application). Security requires special attention for a SaaS application because a single application instance is serving different clients. This poses two major challenges:

How will the administrative tasks related to security handles? For example, creating new users, security policies, and permissions etc. for every client is obviously different.
How to keep client specific request secured? This also involves keeping data for clients secured even if we have data for different clients in same table.

There are different techniques and patterns such as filtering, encryption etc. Let’s see an overview of how are security requirements handles by a SaaS application.

Common approach is to create security services for the application. So security is also applied in the form of services (sole reason is to incorporate the distinct requirement of SaaS application). SaaS application is built on the foundations of SOA (service oriented architecture) and services are building blocks of any SOA application. Typically we divide security services into two components:

Authentication Services
Authorization Services

The SaaS provider typically delegates to each tenant the responsibility for creating and maintaining its own user accounts, a process known as delegated administration. Delegated administration creates a situation in which the customer is responsible for creating individual user accounts, but the vendor has to authenticate them. To accommodate this delegated-administration model, SaaS designers use two general approaches for handling authentication: a centralized authentication system, or a decentralized authentication system. The approach that you choose will have ramifications for the complexity of your architecture and the way end users experience the application, and you should consider what your business model says about the needs of the application, customers, and end users when making a decision.

In a centralized authentication system, the provider manages a central user account database that serves all of the application's tenants. Each tenant's administrator is granted permission to create, manage, and delete user accounts for that tenant in the user account directory. A user signing on to the application provides his or her credentials to the application, which authenticates the credentials against the central directory and grants the user access if the credentials are valid.

This approach requires a relatively simple authentication infrastructure that is comparatively easy to design and implement, and that does not require any changes to the tenant's own user infrastructure. An important disadvantage to this approach is that a centralized authentication system makes it much more difficult to implement single sign-on, in which the application accepts the credentials that the user has already entered to gain access to his or her corporate network. Without single sign-on, users are frequently presented with an inconvenient login prompt when logging in to the application, and they must enter their credentials manually.

In a decentralized authentication system, the tenant deploys a federation service that interfaces with the tenant's own user directory service. When an end user attempts to access the application, the federation service authenticates the user locally and issues a security token, which the SaaS provider's authentication system accepts and allows the user to access the application

This is an ideal approach when single sign-on is important, because authentication is handled behind the scenes, and it doesn't require the user to remember and enter a special set of credentials. The decentralized approach is more complex than the centralized approach, however, and a SaaS application with thousands of customers will require individual trust relationships with each of the thousands of tenant federation services.

In many cases, the SaaS provider may want to consider a hybrid approach—using the centralized approach to authenticate and manage users of smaller tenants, and the federated approach for larger enterprises that demand, and will pay for, the single sign-on experience.

To understand more about security concepts and how to apply them you can refer following resources:

http://download.microsoft.com/download/5/c/4/5c49d9c8-a3e9-4806-8c6d-a621b8d2402b/Windows-basedHosting_CentralizedManagement.doc

http://www.microsoft.com/technet/security/guidance/identitymanagement/idmanage/P1Fund_4.mspx?mfr=true

To configure security at database level use guidelines given in following resource:

http://msdn2.microsoft.com/en-us/library/aa479086.aspx#mlttntda_topic4

Scalability:

Scalability refers to the capability of an application to handle large concurrent requests. There are two approaches generally used to scale an application:

Scale Up: Scaling-up is process of upgrading the existing computer. For example, you can upgrade your Server computer from a 4 processor machine to an 8 processor.

Scale Out: Scaling-out is the process of adding additional computers. This ways if a server reaches its maximum number of requests, another server takes up further requests. SaaS solutions generally use Scale out. A Common techniques used for scaling out is clustering.

Scaling an application can typically be done at 3 levels: