• Patterns for Great Architecture

    Internet of Things (IoT) on Microsoft


    What is IoT: Internet of Things (IoT) has become the buzzword today (even though it was coined long time back… in 1990s by MIT professors). IoT simply means: “all entities connected”. Everything connected ranging from your home appliances (like TV, refrigerator, Microwave, Car etc.) to office appliances (tablets, laptops, phones etc.) to industry specific devices (manufacturing units, sensors, CCTVs etc.).

    When we go to buy grocery, the point of sales personnel scans barcodes from each item. This updates backend inventory and all kinds of reports are generated. This labelling of even technically incapable items like fruits/soaps is nothing but a simple example of IoT in retail. Microsoft has worked with London Underground transport system on this concept where not only rails but also things like lifts, escalators, cctvs, temperature/humidity monitoring units etc. are all connected and providing intelligence in real-time.

    IoT and StreamInsight: With large number of entities connected, there are IoT scenarios where we need to analyse data in near real-time. We cannot wait to store data coming from small devices to be first stored in database and then analysed later. Microsoft’s StreamInsight (that we released back in SQL 2008 R2) which is also available as an Azure Service holds important place in such IoT scenarios. Using StreamInsight we can perform BI on data on real-time before it even gets stored in database which is a must for many scenarios.

    IoT and BigData: With IoT, it is natural to have a very large data storage with wide data formats (not just relational) that traditional SQL Server were never designed for. With such large storage of data it is also expected to analyse this data at a speed that SQL Server cannot deliver (SQL Server is typically good for Terabytes only). Many organizations cannot afford solutions like SQL Parallel Data warehouse (PDW). So, the only choice remaining with large data associated with IoT is using Big Data techniques like Microsoft’s HDInsight (Hadoop implementation on Azure) or HDP for windows (Hadoop implementation on premise).

    IoT and Embedded Systems: With the need for every device to be connected (by IoT philosophy), it is natural to build intelligent small devices. Technologies like .NET Micro framework/ .NET Compact frameworks or Windows embedded 8.1. Microsoft’s Intelligent Systems Service is the central place to stop by for building such devices.

    IoT and Service Bus: Microsoft is also creating the base for developing custom solutions that are very much relevant in IoT. Networking Technologies (like VPN) may not be sufficient enough to realize IoT in scenarios where small devices cannot be part of VPN (lets say) but with technologies like Service Bus, we can expect connected device vision.

    IoT and Industry Verticals: Microsoft’s point of view about IoT is not just connecting everything (Internet of everything), rather connecting all those things that matter to us most. This creates many scenarios for IoT like IoT in retail, IoT in healthcare, IoT in manufacturing etc.

    IoT and Microsoft: With Microsoft’s April 2014 announcement of Azure Intelligent Systems Service, we’ve provided a practical solution towards IoT demonstrating how IoT solutions could be built using HDInsight, PowerBI, embedded devices, Analytics Platform System (combining SQL (PDW) and Big Data (HDInsight)).


    -Rahul Gangwar


  • Patterns for Great Architecture

    Live ID Authentication


    June 27 – 2011


    I assume the reader understands the concept of Claims Authentication and related terms like STS, IP...


    This post explains my understanding about Live ID authentication that you might want to integrate in your applications like Win Forms, SharePoint, CRM or a custom ASP.NET application.Live ID exposes different ways of authenticating against it. The primary ones being:

    ·         Using WS-Federation:

    If you want to use WS-Federation, then you need to (1) Register your application with Live ID STS as trusted RP (use MSM https://msm.live.com/wizard/default.aspx ). This mechanism is also known by name Relying Party Suite. (2) Configure your application to start using Live ID: Live ID STS exposes its Federation Metadata which makes it easy for you to use FedUtil.exe in your application to establish this trust from RP to Live ID STS. the Federation metadata is found here: Metadata XML URL Seems easy. BUT the very first step of configuring your application as RP using MSM is not exposed to external customers. It is available only for Microsoft Partners. So as per my conclusion, even though Live ID supports WS-Federation, you won’t be able to use it “directly”. I will explain in next option why I have used the term “directly”. I am not sure why do we have this article published when we cannot actually use it publicly: http://technet.microsoft.com/en-us/library/ff973117.aspx

    ·         Using Azure ACS:

    Microsoft Azure Access Control Service (ACS) is an cloud STS. It offers Live ID authentication as out of box identity provider. In order to use this path for Live ID authentication, you need to (1) Configure ACS to start trusting your application as trusted RP. (2) Add Live ID as Identity Provider in ACS (3) Configure your application to start trusting ACS as trusted STS. Both these steps are pretty easy to be followed with minimal coding. Here is one post that explains in step by step manner: http://msdn.microsoft.com/en-us/wazplatformtrainingcourse_introtoacslabsv2_topic2.aspx
    If this is so easy, then why doesn’t one always use it?

    §  Cost: Azure ACS doesn’t come for free. You need to pay for it.

    §  It adds an extra layer between your application and Live ID (a less agreed reason).

    ·         Using APIs offered by Microsoft:

    So, isn’t there any free, publicly available way of directly working with Live ID authentication? Well, yes there is and you can do it via APIs that Microsoft offered. These set of APIs (previously known with different names) are clubbed under the common name called as “Messenger Connect”. Messenger connect can not only be used to authenticate users with Live ID but also access their information stored on Live ID Servers like pictures, contacts etc. Here are details on how to achieve it with an ASP.NET application: http://msdn.microsoft.com/en-us/library/ff748287.aspx

    You can also develop a custom claims provider that uses Messenger Connect and configure it with AD FS 2.0 so that you don’t have to code it every time you plan to use Live ID authentication.

    Rahul Gangwar


  • Patterns for Great Architecture

    ApplicationPoolIdentity and IIS


    Ever faced the requirement of giving permissions to any resource (like folder) while your ASP.NET application’s pool is running under ApplicationPoolIdentity? Few points to understand:

    • There is no fixed account for this so you would not find any account under in built accounts while trying to set ACL permissions.
    • Corresponding to each AppPool that you have a new account is created when the AppPool starts. The naming convention is: “IIS APPPOOL\your_app_pool_name”. So if you have an application is running under “Classic .NET AppPool” then the local user account created is IIS APPPOOL\Classic .NET AppPool.
    • Just grant permissions to this account and you are set to go.
    • The benefit of this approach is that without you taking the trouble of creating any local user account under a system, you can configure to have your application run under different identity. Different applications can run simultaneously without any possibility to access each other’s data. Automatic provisioning also becomes easy.
    • Since this account actually doesn’t exist on the system there are less chances that you would have given any other rights to this account and as a result any hacking threat doesn’t compromises your system.
    • But in order to best use this feature you need to make sure that for such compartmented security requirement, you need to run your application under a dedicated custom Application Pool.

    Rahul Gangwar

  • Patterns for Great Architecture

    ADFS 2.0; The Service is Unavailable Error



    I just came to understand that while using the Step by step guides for ADFS 2.0 alongwith combination of the Virtual machines that are present at connect Microsoft website, you might receive "Service Unavailable" errors while trying to navigate to ADFS asmx services or FederationMetadata.xml file.

    The virtual Machines present at Microsoft connect location (below) have the Token-signing certificate and token-decrypting certificates on ContosoDC server expired on April 22 2011. Because of this ADFS is not able to build the certificate chain for these operations and as a result metadata endpoints are not exposed. An end user trying to work with these labs simply doesn’t see federationmetadata.xml file.

    Temporary Resolution:

    While it would take time to get the VMs updated, you can work ahead by renewing the certificates manually. In order to do so folow below steps:

    1. Log in as Administrator on ContosoDC

    2. Open Powershell in administrative mode and enter the command: "Add-PSSnapin Microsoft.Adfs.Powershell"

    3. Enter following commands one by one. After hitting enter, you need to wait till the command prompt returns back.

    Update-ADFSCertificate -CertificateType: Token-Signing -Urgent:$true

    Update-ADFSCertificate -CertificateType: Token-Decrypting-Urgent:$true

    VM Location: http://connect.microsoft.com/site642/Downloads/DownloadDetails.aspx?DownloadID=29506

     Rahul Gangwar


  • Patterns for Great Architecture

    ADFS 2.0; The Service is Unavailable Error



    I just came to understand that while using the Step by step guides for ADFS 2.0 alongwith combination of the Virtual machines that are present at connect Microsoft website, you might receive "Service Unavailable" errors while trying to navigate to ADFS asmx services or FederationMetadata.xml file.

    The virtual Machines present at Microsoft connect location (below) have the Token-signing certificate and token-decrypting certificates on ContosoDC server expired on April 22 2011. Because of this ADFS is not able to build the certificate chain for these operations and as a result metadata endpoints are not exposed. An end user trying to work with these labs simply doesn’t see federationmetadata.xml file.

    Temporary Resolution:

    While it would take time to get the VMs updated, you can work ahead by renewing the certificates manually. In order to do so folow below steps:

    1. Log in as Administrator on ContosoDC

    2. Open Powershell in administrative mode and enter the command: "Add-PSSnapin Microsoft.Adfs.Powershell"

    3. Enter following commands one by one. After hitting enter, you need to wait till the command prompt returns back.

    Update-ADFSCertificate -CertificateType: Token-Signing -Urgent:$true

    Update-ADFSCertificate -CertificateType: Token-Decrypting-Urgent:$true

    VM Location: http://connect.microsoft.com/site642/Downloads/DownloadDetails.aspx?DownloadID=29506

     Rahul Gangwar


  • Patterns for Great Architecture

    MVVM with RIA Services


    MVVM without RIA:

    ·         MVVM relies on the concept that View (XAML) will not contact your Model directly.

    ·         View will contact Model via ViewModel classes.

    ·         ViewModel talks with View using Data Binding and Commanding in XAML based applications.

    ·         ViewModel talks with Model via web service communication in case of Silverlight when we don’t use RIA Service.





    MVVM with RIA:


    ·         RIA Service abstracts the web service layer from developer’s perspective and presents on client an identical set of classes that we have in model on server.

    ·         This simply means that under normal MVVM scenario we just change the communication between ViewModel and Model to go via the auto generated model classes on client side instead of explicitly calling web services (though internally it is still WCF communication).

    ·         In my opinion, this should not pose any problems to the MVVM pattern given that we keep the communication link same as above.






    ·         We should never attach the auto generated client model classes of RIA with View directly. This link of View to Model should happen via ViewModel only. In a nutshell, these auto-generated model classes should never be considered as ViewModel.


    ·         We also have the availability of some RIA controls like DomainDataSource (in XAML). In case of MVVM we should NEVER use any such RIA controls in XAML since they break the very pillar of MVVM by contacting auto generated client side model of RIA directly (which is same as above point).


    Rahul Gangwar



  • Patterns for Great Architecture

    Silverlight Graphics and Hardware Acceleration



    Today is May 20th 2010 and I am writing this post to discuss the working mechanism of Silverlight’s capability of utilizing GPU of your machine.

    There are two kinds of graphics popular today: 

    1. Bitmap Graphics:

    Here the graphics is a file that stores the display information in the form of pixel information. Basically the term itself means map of bits. Each pixel stores information about how to light it up, in the data structure: Color Depth. This structure (color depth) can be of various size like 8bit, 16 bit, 32 bit etc. and larger the data structure the more information about the pixel can be stored. This determines the bit depth. So when you buy a scanner/camera kind of device you can relate what bit depth mean and what mega pixel means. Two devices having same megapixel but different bit depth will capture different quality of images.  

    Now larger the number of pixels per inch you have for your image, lesser would be distortion if you try expanding the image. But as we store larger number of pixels, the size of image also grows large. This means for all practical purpose there is a limit to which you can expand your image to a larger size. Theoretically no matter how many pixels an image contains, expanding to larger size will definitely distort its quality. It is just that it won’t be visible to human eye if the expansion and number of pixels are maintained to a balanced level. Not only that the image quality decreases even if you shrink it beyond the allowed limit. That is why we use less pixel images if the display required is small (not primarily because of size, but for quality). So a high resolution image/video will not look good on a small device. You need to re-encode it!


    2. Vector Graphics 

    Here the graphics is store as mathematical formulas. Like lines, formulas for different shapes like curves or even custom algorithm is created to interpret the shape. This means that no matter how big/small display we need, the information will simply remain the same. Size of vector images will remain always same irrespective of how large/small it is displayed. The size changes with the complexity of information stored. If you want to display simple shapes like triangle, circle or even combination of such shapes, vector images are best. But if you ever try to convert a photograph that you clicked on that famous hill station, your resulting vector image would be larger in size and also in worse condition because vector graphics techniques are not so sophisticated to convert the graphics created by God into man made. No lens in the world could be better than your eye and no speaker in the world could be better than your ear! You might call that as a limiting factor or something mysterious :).  

    Now coming back to Silverlight, the graphics that you display with Silverlight can be rendered using two ways: 

    1. Using Graphics Card (GPU) (primarily when the graphics is bitmap).

    2. Using Software that is run by CPU (primarily when the graphics is vector based).  

    The XAML that you write in your SL application are displayed using CPU. They are all formulas that are calculated and rendered by CPU. When you instruct SL to use GPU we call that as GPU acceleration. Why “acceleration”, I don’t know. Probably “assisted” would have been better term.  


    It is simple. You specify the UI elements that you want to display using GPU by adding the attribute CacheMode=”Bitmap”. Now all the UI elements are represented in form of object tree in Silverlight. Silverlight does the calculation to create the object tree and forms a bitmap for that object tree. Then, Silverlight gives this bitmap to GPU to display it and keeps it cached. Now, when the application screen refreshes, that part of the screen that you specified to be displayed by GPU is simply rendered by GPU and the rest part of the UI is rendered by SL using CPU. As a result, we are using less CPU and improving performance. Even if you are performing some animations like scaling, changing opacity etc. these are done by GPU and not by CPU.  

    Why not Always?
    This seems interesting so why not do it always using GPU? Few reasons (not in order of any priority):

    ·         There are some animations that GPU cannot assist (like changing perspective (3D), animating on each pixel such as pixel shader effects). There is no point in using GPU acceleration in such scenarios.  

    ·         If the GPU has less memory and if the memory required by Silverlight is large, Silverlight falls back to CPU mode and you will hit performance. Also there is no way to query memory available in GPU from Silverlight code. However you can query other details like GPU vendor, driver version etc. through Analytics.GpuCollection structure.  

    ·         Cache Invalidations: If your UI element is changing a lot and that change cannot be assisted by GPU (another example is changing text of a UI element lie textbox), then cache invalidation happens and we hit on performance.  



    You also need to enable the GPU acceleration at plug-in level by setting a parameter as shown below:

    <param name="enableGPUAcceleration" value="true" />

    Also note that on Mac OSX GPU acceleration works only in full screen mode. Users need to have a DirectX9 (windows) or OpenGL2 (Mac) compatible video card. 


    You can test the GPU acceleration by adding another parameter to the plug-in as shown below:

    <param name="enableCacheVisualization" value="true" />

    This parameter shows all GPU accelerated graphics in natural colors while all non GPU accelerated graphics in Red color for testing purpose.

    When you test CPU percentage, make sure you use lesser number of cores and more complex graphics to actually see the difference in performance.  

    Beware Animation:  

    Please make sure that the UI elements for which you have used GPU acceleration are using below animations only and not anything else: 

    Scale, Rotate, Changing Opacity, Clipping (only if clip is rectangular). For all other types of animations like 3D, moving, pixel shaders, do not use GPU acceleration.  

    Poor Joke:
    If you like the post, you can pay pal me here.






  • Patterns for Great Architecture

    WCF Data Service; Server Driven Paging (SDP); Silverlight



    Today is January 5th 2010 and I am writing this post to explain the mechanism of the Server Driven Paging feature of WCF Data Service and also a code guide on how to use this from Silverlight 4.0.

    The Pattern:

    Server Driven paging is one of the new features that have been added in WCF Data Service Version 1.5 CTP2 (formerly called as ADO.NET Data Service). The feature in simple terms means transmitting data in pages. This feature is important from performance point of view and also an important pattern used in applications where UI layer and data access layer are separated by cloud. I view this as a pattern used many places; the most common of being “chunking” approach used in large data transfer. Take example for WCF data transfer of a large file. We typically implement and recommend to use “chunking” pattern to transmit large data from one place to another because of many reasons. One reason could be to support “resume” functionality. So if data transfer is interrupted at 50%, we don’t need to start all over again next time. We can resume from 50%. Another reason could be to allow large data transfers where network policies put an upper bound on data transfers. One more possibility I can think of is unreliable network (which internet is) where packet loss is very common and would otherwise lead to long time to get a perfect data flow in one go. Server driven paging is the same pattern/algorithm applied to transfer large set of records between WCF Data Service and client so that it doesn’t put pressure on client (e.g, to render millions of records with small scroller in a single data grid, or storing large records in client memory after he waited for good 10 minutes or so). In short, paging makes the overall process faster on average. Server driven paging gives more performance since we save time in transmitting very large number of records over the wire and improve the end user experience.

    The Mechanism:

    The mechanism is simple and worth understanding to make decision about whether to use this feature or not:

    Step 1: Client sends request to fetch data by firing query.


    Step 2: Server Fetches result from Data Base for a given page. This step needs a little elaboration since here comes the important factor that might decide how effective SDP of WCF Data service is for your scenario:


    When a user fires a query from client, the data service fetches the result based on page size and the page on which currently the user is. If we want to implement paging in our application from scratch, we usually follow one of the two approaches to achieve this:

    •  Session based approach: We store the page on which a user currently is in the session. So we always know which page the user wants when he clicks next/previous. Pros: Sometimes in order to improve speed we also cache result set in session and simply return the required page and in fact for such cases, this is the only approach we use. We cannot afford to go to DB every time the request comes.
      Another benefit of this approach is that the client code is very simple, we simply have to tell whether we need next or previous page. On server side, we fetch records from DB that should be present on a particular page.
      Cons: Drawback is that the moment session expires, the page tracking is lost and it also requires us to scale server. Not a preferred approach for most of business scenarios that I have encountered.

    • No session approach: Here we establish some unique key that can be used to calculate which records should be present on a particular page. An example would be a combination of “page number” and “page size” would give the records that should be present on a given page. Every time a new page is requested, we go to database to fetch the records for that page and return them. This approach is used where scalability is more important than speed. We don’t put pressure of storing calculated results in memory at server.

    WCF Data Service uses “no session” approach and goes to DB every time a new page is fetched. It uses primary key as the counter to reach to a particular page. So when a next page request comes, it contains the primary key of the last record of the last page. WCF Data Service goes to Database, orders the query by primary key, fetches records of page size starting after the provided argument by client and returns result. The benefit of this approach is that you don’t depend on session for paging. This approach is also ideal for clients (especially thick clients like Silverlight) that might not talk to server for long time. Another benefit is that we don’t need to scale the server by not storing large set of data in session. The drawback of this approach is that it is not performance effective since it goes to DB for each request. Not only that, it uses orderby in query fired to DB which is even slower than an equivalent query not using “orderby”.

    Step 3: WCF Data Service returns the data for the requested page. Along with the Data it also returns a URL that contains the URL to fetch next page.


    Sample Code with Silverlight:

    The following Sample Code has been developed using Visual Studio 2010 and Silverlight 4, but it will work fine with SL 3.0 as well since Data Service is all based on REST. Just that right client APIs need to be used and correct version of Data Service is needed at server side to enable Server Driven Paging. Contact me if you need a working sample code and if anything below did not worked.

    • Create a Silverlight 4.0 Application
    • Add Database to Server. Add Employee Table with FirstName, LastName and eMail (Primary Key). This is the table that we are going to page via server upon client request. Fill it with some data… maybe 5 records.
    • Add DataLayer at server side using Entity Framework.
    • Add ADO.NET Data Service at server side. Attach it to DataLayer, set its Access Rule, turn on ServerDriven Paging with Page size and Protocol version to be used by adding following statements in InitializeService() method of ADO.NET Data Service:

    config.SetEntitySetAccessRule("*", EntitySetRights.All);

    //enable server driven paging with PageSize

    config.SetEntitySetPageSize("Employees", 2);

    //This is required for server driven paging to work and maintain backword compatibility since SDP doesn't work with version 1 of data service

    config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;

    • Add DataGrid and next button on Silverlight client. Remember to autogenerate columns to true for datagrid.
    • Fix port address for website and add service reference to WCF data service in SL client.
    • On Silverlight client code behind, add proxy to data service and client type that will hold link to next URI on every request fired to data service. The code is similar to below:

    //add proxy to wcf data service where we can fire queries

    public DatabaseEntities myDB = new DatabaseEntities(new

    Uri(@"http://localhost:53951/WebDataService1.svc/", UriKind.RelativeOrAbsolute));

    //add the client type that will hold the link to next uri (next page of records) after every request

    public DataServiceQueryContinuation<Employee> token = null;

    • On PageLoad, execute query to get all records from server. Code is same as any WCF Data Service code and shown below:

    public MainPage()



    //execute query to get all records from table

    var q = from e in myDB.Employees

    select e;

    var eq = q as DataServiceQuery<Employee>;

    eq.BeginExecute(new AsyncCallback(MyCallback), eq);


    • On Callback, write code to collect response for two different conditions. The entire code behind is shown below. I have not included XAML since its trivial having a blank Datagrid with autogenerate columns to true and a button for next page:

    public partial class MainPage : UserControl


    //add proxy to wcf data service where we can fire queries

    public DatabaseEntities myDB = new DatabaseEntities(new

    Uri(@"http://localhost:53951/WebDataService1.svc/", UriKind.RelativeOrAbsolute));

    //add the client type that will hold the link to next uri (next page of records) after every request

    public DataServiceQueryContinuation<Employee> token = null;

    public MainPage()



    //execute query to get all records from table

    var q = from e in myDB.Employees

    select e;

    var eq = q as DataServiceQuery<Employee>;

    //this is called once and the caller is DataServiceQuery<Employee>;

    //so in callback it has to be collected in the same datatype which is what is written

    //in when next querytoken is null (since we are still to collect it).

    eq.BeginExecute(new AsyncCallback(MyCallback), eq);


    public void MyCallback(IAsyncResult result)


    //slot to hold returned result

    IEnumerable<Employee> response = null;

    if (token != null) //this is executed everytime we have clicked on next button


    //since in next button click we used DatabaseEntities to fire query , we need to typecast it to the same type

    var eq = result.AsyncState as DatabaseEntities;

    //collect the response

    response = eq.EndExecute<Employee>(result);


    else//this "else" is executed only for the first call to WCF data service; we get token only after first call


    //since the first call was made by DataServiceQuery<Employee>, we are typecasting it to the same type on first callback

    var eq = result.AsyncState as DataServiceQuery<Employee>;

    //collect the response

    response = eq.EndExecute(result);


    //bind the data to datagrid (but do it before typecasting response to get token

    //or collect data first in separate instance and then typecast the returned result to get next token)

    dataGrid1.ItemsSource = response.ToList();

    //collect next page uri by typecasting the returned response

    var qor = response as QueryOperationResponse<Employee>;

    token = qor.GetContinuation();


    private void button1_Click(object sender, RoutedEventArgs e)


    //if there are more pages

    if (token != null)

    //fire the next page query to proxy of WCF Data Service

    myDB.BeginExecute(token, new AsyncCallback(MyCallback), myDB);



    Comments/error always welcome. Contact me if you need a working sample code and if anything below did not worked.

  • Patterns for Great Architecture

    Why Task Based Programming (Parallel Extensions)



    Today is October 15th 2009 and I am writing this post to express my understanding about parallel programming using Parallel Extensions (or Task based parallel programming); specifically to express why should we go for this approach.

    Since the Parallel Extension announcement, I was excited to get my hands dirty with it but after working with Parallel Extensions, I had few initial questions that I thought everyone would have and there is less content available online that targets against benefits of Parallel extensions. Below are a couple of questions that I am targeting in this post.

    • Is Threading going to die? Are Parallel Extensions replacements to Threading?
    • Sometimes Threading gives me better performance than Task based programming. Why?

    Before we come to these questions, it would be worth understanding options other than parallel extensions that we used to follow for parallel programming. Before Parallel Extensions, we would use primarily any of the below two modes for parallel programming:

    Directly Creating Threads:
    Here we create Threads directly in our code something similar to the following code:

    Thread t = new Thread(DoSomeWorkMethod);

    The benefit of this approach is that each of the thread will run at all times. .NET Framework’s scheduling algorithm will make sure that all the threads that we created will execute in similar to RoundRobin algorithm (though actual algorithm is different). This approach is especially useful in cases where I want to provide justice to all threads.
    Another benefit associated with this approach is the fact that we have control over these created threads like aborting a thread in the middle.

    The drawback of this approach comes into picture when we have more threads than the available CPUs (cores). As an example, if we have two cores and I am running 10 threads, context switching, invalidation of each thread’s cache etc. So we need to keep a balance between how many CPUs we have and how many Threads should we create for optimum performance.

    Using Thread Pool:
    .NET Framework has a thread pool (configurable) which limits the number of Threads that can be run at a time. Now when we want to execute something in a separate thread, we simply queue our work on a thread of thread pool instead of creating our own separate thread. Code looks something similar to below:

    DoSomeWorkMethod, someInputValue);

    Thread Pool size for .NET Framework can be set by doing a simple calculation using number of CPUs the machine has. After this, Thread Pool makes sure to run only those number of threads that are optimal for a particular machine. This prevents any context switching and other overheads. So we get best performance.

    The drawback of this approach is that developer doesn’t have much control over the Queued tasks over Thread Pool. In other words, once we have queued a work item, we do not get a reference to it and there is no explicit support for knowing when it completed, or for blocking until it completes or for cancelling it, or for identifying it in the debugger via some sort of ID etc. Plus, you are not guaranteed when your Task will start to run, so progress is unknown.

    Using Parallel Extensions:
    This approach is the one that we are here to discuss. Task based programming combines the benefit of both the approaches with some new additions. Using task Based programming, you create Tasks instead of Threads and start executing them. Programmatically they are almost identical to Thread Based programming with additions features. Here is a code snippet that uses the new Task class:

    Task t = Task.Create(DoSomeWorkMethod,


    • With Task Based programming, by default, you can be sure that we are not actually decreasing the performance by doing excessive multitasking since Task Manager (central class in Parallel Extensions) takes care of this issue by queuing any additional tasks if all the existing CPUs are busy.
    • With Task based programming you have complete control over the Task. You can pause, abort, wait, join etc.
    • The scheduling algorithm is improved over Thread Pool approach with new features like “Task Stealing”.
    • You get additional constructs to easily work with, like Parallel LINQ, Parallel For Loop etc.


    Personally I see the only drawback is that you are not guaranteed when the Task will execute. So if you want to guarantee of starting your Task execution as soon as it is created, use plain old Threading Way. UI multitasking as an example should be done with Threading and not Thread Pool or Task based programming.

    I will also share one interesting incident that happened while I was developing a Proof Of Concept. I was running the samples on a 4 Core Machines and created 2 Threads to do a Task and then 2 Tasks to do the same task and I could not see any improvement. Then I realized that if I am creating Threads which are less in number than available free CPUs I might actually get some greater performance sometimes, but that is not a guarantee. With task based programming, I can guarantee that either all my Tasks are running in parallel or all CPUs are busy executing my tasks.

    I will try to cover benefits approach in future posts but for now, it is done.

    I hope this post was useful. If you are a Microsoft Partner, please feel free to involve our team for all your development consultation needs by contacting us here. Comments, corrections; I Love them.

    Rahul Gangwar

  • Patterns for Great Architecture

    Microsoft Codename "Gemini"




    Today is July 28th 2009 and I am writing this small post to share my understanding on Gemini and Business Intelligence arena and specifically around Analysis Service.

    Talk about data in Microsoft and the very first thing that comes in mind is SQL Server. Microsoft hardly released SQL 2008 and we are looking forward for next version of SQL Server. That is very ambitious.

    When it comes to Business Intelligence problem, things like Reporting, Analysis, data-mining come automatically forms part of the solution. But there are few patterns that should be considered for developing an efficient BI solution. Before talking about Analysis Service, let's understand how operations are different in OLTP vs OLAP databases.

    In OLTP databases, data is written in form of rows (relational databases). So for example, if I want to store two records of two column structure having (name and email as columns) the storage will go in a fashion:
    "Rahul" "rahul@contoso.com", "Sabiha" "sabiha@contoso.com".
    Eeffectively, data is written over disk one row at a time. This makes typical OLTP transactions efficient  since I can quickly read/write a all columns of ONE record somewhere in middle of 50 million rows. But when it comes to reading ALL records (let's say email of all 50 Million records), relational databases fail miserably in terms of performance.

    In OLAP databases, data is stored in a different fashion. It is stored in the form of columns rather that rows. So if I want to store the same set of records in columnar fashion, it would be stored this ways:
    "Rahul" "Sabiha", "rahul@contoso.com" "sabiha@contoso.com".
    The benefit and drawbacks are here.
    If I want to read columns of ONE record somewhere in middle of 50 million rows, it will fail miserably in comparison to relational databases. But when it comes to reading a column of ALL records, this will be surprisingly fast (and this requirement is very common in Analysis: to read large number of records rather than one record).

    So one thing is clear that columnar databases are better than relational databases for data analysis.

    Another thing that need to be considered while designing analysis solution is the storage medium. Databases that use disk as the storage medium will perform very slow for performing analysis algorithms than the databases that use in memory medium as storage. Storing data in memory is fast because of the medium speed as well as saving CPU cycles. Although the durability property of ACID is lost in this process (since memory is volatile) but this still holds a good candidate for analysis solutions.  Even hybrid storage mechanisms can be considered depending on the requirement scenario.

    Now comes Microsoft's project codename "Gemini". Gemini is next version of Analysis Service based on the concepts of using columnar database and in memory storage, making analysis of data hell fast. Gemini will be shipped in the next version of SQL Server (2008 R2/Kilimanjaro).
    The good thing is that Excel users can use its capability by installing a plug-in. Hence the term "self service analysis". Every user in an organization will be able to analyse data and take decisions using simple Excel. More here:

    I hope this gives a peek about what Gemini is and how it will work.

    Comments/feedbacks welcome.

    Rahul Gangwar

  • Patterns for Great Architecture

    Physics Based Games in Silverlight



    Today is September 29 - 2008 and I am writing this short post (though big, but short for writing anything over Farseer/SL) to give an idea about one of the immense possibilities with Silverlight - Game Development. I will focus on simplicity to target the beginners and specifically cover possibilities with Farseer Engine. The Farseer Physics Engine is an easy to use 2D physics engine designed for Microsoft’s XNA and Silverlight platforms.

    I do not know why it is named Farseer but possibly because it refers to followers of Eldar; one of the most ancient and advanced races in the universe's history. Their armies usually have the advantages of speed and technology. Possibly Jeff Weber can tell.

    Physics Engine:

    There are several games that simulates their motion like a real world entity. For example, if you have used Windows Live SkyDrive, while uploading a file, you are presented with a screen having a Baal. You can push this ball and it collides with walls, bounces back pretty much the same way as if it was a real ball. The ball has all the properties of a real world - it always feels a gravitational pull downwards, on experiencing a force i would follow a parabolic path similar to a real world ball. Now, no one has written the logic on how a ball should move when hit. such simulation is done by associating certain features to the ball (like mass, friction etc.), rest all is handled by the underlying Physics Engine. Physics based games are more quality games and closer to real world experience and form a different group of games altogether.

    In short statement, any piece of logic that could provide simulation of Newtonian Physics is a physics engine. It doesn't necessarily has to be a game engine, though the most common application is game industry, it is also used in various other fields such as research, advertisements etc. Primarily there are two kinds of Physics Engines - "Real Time" and "High Precision". Both the engines aim to simulate the laws of physics (like friction, moment of inertia, force, gravity (which is again a force) etc.) but since there is less availability of computation over most of the end users machines, "Real Time" engines eat up some of the details and simulate the bare minimum required to provide a real world feeling. On the other hand, "high precision" engines take into account of almost all the factors that exist in our world like air speed etc. and are commonly used in scientific projects like defence, space etc. and are run over high end machines like super computers.

    most of the machines out there depend entirely on CPU alone for most of the computational needs. In areas where high graphics intensive tasks are required, we can install specialized GPU (Graphics Processing Units)- we also know them by common name - graphics card. Similarly, in high end applications which use Physics Engines, we can also install PPU (Physics Processing Unit) for shedding off the load from CPU giving a better performance.

    In case of Farseer, it is not an easy task to write a Physics Engine for browser based scenarios since you have to think 100 times before planning for using resources. Physics based calculations cannot be done easily when the machines are poor. That is the reason why you might find some glitches in terms of performance on the physics games over Silverlight and Farseer will improve continually over time - sometimes to cope up with new releases of Silverlight, at other time to add new demanded features and performance restrictions. Hats off to the people out there who are contributing to the developer community by creating such product. Now I can start my post after paying respect to those great people.

    Silverlight as gaming platform:

    Games have few components. Let us examine these and see what Silverlight has to offer w.r.t. these requirements.

    1. A nice light and programmable UI - Silverlight supports these features through various concepts like Path, Vector shapes etc. and most importantly, they are based on XAML so you can talk to your graphics during runtime which you might not be able to do with a normal graphics (like jpeg etc.).

    2. A way to provide simulation to the UI elements - Silverlight supports these through various subtle concepts like animations. More importantly, since we can work in C# we can use all the good features and usability of the same. You don't have to learn any new tool/language.

    3. A way to interact with end user in sufficient real time fashion - Silverlight makes this possible through its optimised CLR (called AgCLR; and do you know that CLR of SL is almost same as that of a the .NET Framework). Also adding to the fact that entire processing will be done at client side, we can expect SL based games to be sufficiently real time. And if you don't see it, probably its time to upgrade the machine.

    4. Game Logic - Now don't expect SL to write game logic, otherwise it would have been a Product rather than a Platform.

    5. Game Engine - Game engine is kind of another framework sitting on top of the underlying framework and it exposes functionality to your game to ease the development effort. It is similar to the fact that .NET APIs, provides a framework where we build other applications on top of them. However it has to be noticed that a game engine is not something that is generic. One game engine works only with a particular genere of games. It is similar to the concept of Software Factories which could be applied to a specific kind of applications only. Silverlight doesn't provide any. You have to write your own (if required) or depend on others available engines in market.

    6. Generalized Game Engines - Physics engine is a good candidate to fit in this scenario. These engines help us to add certain features rather acting as a specific game engine. For example, Silverlight doesn't provide any API to simulate Physics, so we have to Physics Engines to add those features to our games/apps. Another example could be 3D rendering engine. Again, Silverlight doesn't support natively any 3D based APIs, so we have to take help of 3D rendering engines like Kit3D. By using Kit3D, we can develop 3D games over Silverlight. Combining Kit3D with Farseer could give us 3D, physics based games on Silverlight. Kit3D is my next topic of interest that I will take up.

    Exploring Farseer:

    I will not go into details of Silverlight/any other details and assume that people reading from here onwards have sufficient knowledge about concepts like Silverlight, Storyboards in Silverlight, C# etc. instead, we will straight away look at some common capabilities of Farseer engine to get started since there are lots of material over SL but very few on Farseer.

    Out of the many capabilities of Farseer, I would explain only few capabilities like, PhysicsSimulator class, movement using "Body" class, Collision using "Geom" class. Apart from this I will also cover on how to achieve a Game Loop using Storyboards (although a better method exist in RC0 onwards of SL which is using CompositionTarget) and an organized way to Handling Keyboard Inputs. First we will discuss three classes and then we will move towards Logic of how to apply them.

    PhysicsSimulator Class:

    This is the main class that developer calls to apply the Physics he wants to apply. Physics Simulator class does many things but for now I could say that you can use it to specify Gravity to the world. It takes values through which you could specify how much gravity any object should experience in X/Y direction. Let's say we have Gravity of 5000 on Y direction and the scene is to represent a constant air flow in X direction, we can apply 100 units of force in X direction. So that ways whenever you will simulate any object, apart from the forces that you have applied, this gravitation and Air will also be applied. This can be specified in its constructor similar to this: PhysicsSimulator physicsSimulator = new PhysicsSimulator(new Vector2(0, 5000));

    Movement (Body Class):

    Body class is the central class that applies different Physics concepts like Force, torque, etc. It doesn't have any UI. We tie the Body object to a specific XAML object (or sprite) by writing a code which makes changes to different properties of the XAML code based on changes in the Body object. So, for example, if we want to give a physics motion to a rectangle XAML element, we will create a rectangle Body object and write code (lets say a method called Update) whose main job will be to keep the properties of Body object in sync with properties of Rectangle (like Canvas.Left etc.). So the concept is: Farseer world never does any changes directly to XAML (neither body object nor PhysicsSimulator). Physics Simulator only simulates the body object, it is we who make changes to the actual sprite based on the changes that Physics Simulator has done to body. So, Physics simulator will move the Body object according to the laws of physics and our code will merely keep our sprite in sync with Body object. This is important because now we can decide which properties to change based on the changes in Body object and not necessarily tied upon the Farseer Engines default functionality.

    This feature essentially provides a loose coupling between the Engine and application making it reusable in different ways. Jeff could have written logic to change specific properties depending on the type of object in question. But then, we would have problems like:

    • It would have been impossible to use Farseer with the newly added classes/objects/third party XAMLs etc.
    • It would have become more time consuming for Jeff to build Farseer and also the engine could have become more heavy (which is a No Go for SL based applications). there are endless list of problems that could have got associated but thanks to Farseer designers for making it very generic and usable in different scenarios.

    Collision (Geom Class):

    We enrich a body further by providing it a geometry.

    Collision is achieved through Geom (or Geometry) class. I don't know why Farseer implementers changed its name from Geometry to Geom so I will use this name interchangeably. Geom class is responsible to decide how a body should react when someone comes in its path. There are several factors in Physics which decide the result of collision. Like Friction (if its not a head on collision), Coefficient of Restitution etc. which we can set and then associate it with a Body.

    In its creation process(basically using factory pattern) itself, we can specify the Physics Simulator to be used and the Body on which it has to act. You don't have to write any logic to handle collision. Based on the properties set to the Geometry of a Body, Physics simulator decides what should happen to other bodies 9which should also be controlled by the Physics Simulator) when they collide. The collision result will be calculated automatically. As said earlier, we have to keep our sprite in sync with Body object in the game loop since Body is the central place where Physics Simulator keeps its end result of any computation. We will see this shortly in a code segment. I will try to upload the code as soon as I finish this post but I might get busy so, if someone needs code sample for the explained concepts, please feel free to contact me by clicking on the email link on my home page of blog/share email via a comment (though that will be visible to everyone).

    Game loop:

    Game Loop is something that keep things going. It is kind of a timer mechanism. faster the game loop, the more real time the end user experience will be but at same time, a fast game loop will consume more memory (and sometimes its unnecessary). You have to decide the game loop frequency on your personal need basis.

    In Silverlight, you could achieve the effect of time through DispatcherTimer but that is not a recommended approach, so people usually go for Empty Storyboard approach. But with release of Silverlight 2 RC, we have a better approach. Though I will show here the empty storyboard mechanism.

    In empty storyboard approach, we create a storyboard in the code, set its duration, associate a event handler for completion and begin the storyboard. The code looks similar to below.

    void GameLoop()
                timer = new Storyboard();
                timer.Duration = TimeSpan.FromMilliseconds(15);
                timer.Completed += new EventHandler(timer_Completed);

    When storyboard completes, it reached its completed event handler as show below and there we do all that we want to do with the game and at the end re-begin the storyboard.

    void timer_Completed(object sender, EventArgs e)
                //Write all logic that you need to execute every 15 milliseconds (in this case).



    Handling Keyboard:

    Keyboard inputs can be handled directly by handling key down, key up events of the Canvas of our XAML. But it is not advised to handle it that way (you will do it if you want a real real realtime response). I am telling this because SL based applications has to be least resource intensive so it is good to share good practice. otherwise you could ignore this section and write canvas event handlers manually.

    In a more optimised approach, we handle key inputs of canvas, and instead of responding to it directly, store the key stroke details in a shared array. When game loops itself (maybe 15 milliseconds in this case) we will simply fetch the key press activities from the array and react to it. this will allow us to move at our own pace rather than getting things hanged/behave abnormally because of an impatient end user hitting arrow keys more like a madman. Attached is a class file that is used to wrap this logic in the form of a class. the class file is present in the getting started samples of Farseer available from Codeplex web site. So, instead of reinventing the wheel, I am representing it as is. see the attached KeyHandler.zip file which has classes associated with it.

    Farseer Sample Logic:

    Okay, so enough of theories, here is one logic that you can apply to create a simple rectangle and circle and then apply motion to it along with handling collisions.

    You need the sprites. In our case these would be XAML files for rectangle and circle having associated code behind files. The code behind files would contain nothing else than an object of type Body and a public function called Update (remember? This is the function which will keep different properties of rectangle/circle in sync with the contained Body object). here is a sample XAML and code behind class structure:

    Rectangle XAML and Code Behind:

    <UserControl x:Class="PhysicsSLGettingStarted.MyRectangle"
        Width="400" Height="300">
                   <Canvas IsHitTestVisible="False">
                    <Rectangle x:Name="rectangle" Fill="Yellow" Stroke="Black" StrokeThickness="2" Canvas.Left="0" Canvas.Top="0">
                                <RotateTransform x:Name="rotateTransform" Angle="0"/>
                                <TranslateTransform x:Name="translateTransform" X="0" Y="0"/>


    public partial class MyRectangle : UserControl
            //We could have implemented like a property but for simplicity I am not making it very complex.
            //This is the RectangleBody object that is changed by Physics simulator.
            //Our update method syncs the coordinates of rectangle with properties of this RectangleBody
            public Body RectangleBody;
            float x, y, rotation;

            public MyRectangle()


            //Exposing size as a property so that size of rectangle can be set from external code
            public Vector2 Size
                    //Vector class is just a structure of packaging things. Its upto us to use it the way we like

                    //We are seeting the width and height of rectangle and the user control
                    //equal to the value supplied by external code
                    this.Width = value.X;
                    rectangle.Width = this.Width;
                    this.Height = value.Y;
                    rectangle.Height = this.Height;

                    //Required to match the coordinate systems
                    translateTransform.X = -this.Width / 2;
                    translateTransform.Y = -this.Height / 2;
                    rotateTransform.CenterX = this.Width / 2;
                    rotateTransform.CenterY = this.Height / 2;
                    return new Vector2((float)this.Width, (float)this.Height);

            //Method that keeps properties of the rectangle in sync with the RectangleBody properties
            //Look how we are simply mapping the properties of Canvas with body properties
            public void Update()
                if (RectangleBody == null) return;

                if (x != RectangleBody.Position.X)
                    x = RectangleBody.Position.X;
                    this.SetValue(Canvas.LeftProperty, Convert.ToDouble(x));

                if (y != RectangleBody.Position.Y)
                    y = RectangleBody.Position.Y;
                    this.SetValue(Canvas.TopProperty, Convert.ToDouble(y));

                if (RectangleBody.Rotation != rotation)
                    rotation = RectangleBody.Rotation;
                    rotateTransform.Angle = (rotation * 360) / (2 * Math.PI);

    Circle XAML and Code Behind:

    XAML and code behind of circle is almost similar to rectangle above. Please see the attached code for details.

    Main Page Code Behind (Page.xaml.cs)

    Important details added:

    public partial class Page : UserControl
            PhysicsSimulator physicsSimulator;
            MyRectangle rect;
            Body rectBody;
            Geom rectGeom;
            Body circleBody;
            MyCircle circle;
            Geom circleGeom;

            Storyboard timer;
            Vector2 force;

            public Page()
                //Create Physics Simulator
                physicsSimulator = new PhysicsSimulator(new Vector2(0, 5000));
                //Create Body. Note the constructor uses physics simulator
                rectBody = BodyFactory.Instance.CreateRectangleBody(physicsSimulator, 128, 128, 1);

                //Rectangle will not move
                rectBody.IsStatic = true;
                circleBody = BodyFactory.Instance.CreateCircleBody(physicsSimulator, 64, 1);//fix

                //Create Geometry and set different physics factors
                rectGeom = GeomFactory.Instance.CreateRectangleGeom(physicsSimulator, rectBody, (float)this.Width, (float)this.Height / 5);
                rectGeom.RestitutionCoefficient = .5f;
                rectGeom.FrictionCoefficient = .2f;
                circleGeom = GeomFactory.Instance.CreateCircleGeom(physicsSimulator, circleBody, 64, 20);
                circleGeom.RestitutionCoefficient = .5f;
                circleGeom.FrictionCoefficient = .2f;

                //Add these bodies to canvas

                //Attach the bodies to physics simulator so that they can be managed by it.

                //Call their update method once during load tiem so that their initial position and size are set

             //This method adds the XAML (sprites) to the main page's XAML
            void AddtoCanvas()
                //Create a rectangle sprite and give it a body, initial size etc.
                rect = new MyRectangle();
                rect.RectangleBody = rectBody;
                rect.RectangleBody.Position = new Vector2((float)this.Width / 2 + 400, (float)this.Height / 2 + 200);
                rect.Size = new Vector2((float)this.Width, (float)this.Height / 5);
                //Create a circle sprite and give it a body, size, position etc.
                circle = new MyCircle();
                circle.Radius = 60;
                circle.CircleBody = circleBody;
                circle.CircleBody.Position = new Vector2((float)this.Width / 2 +500, (float)this.Height / 2 - 100);
                //Add these sprites to main canvas


            void GameLoop()
                timer = new Storyboard();
                timer.Duration = TimeSpan.FromMilliseconds(15);
                timer.Completed += new EventHandler(timer_Completed);
            void timer_Completed(object sender, EventArgs e)
                Button_Click(null, null);

            private void Button_Click(object sender, RoutedEventArgs e)
                //Apply force
                force = Vector2.Zero;
                force.Y = -force.Y;
                force += new Vector2(0, 5000);
                //apply force in y direction
                //Rotate the ball so that it falls down
               //update simulation
                //update the effect of physics simulator by calling update methods of sprites
                //Start the game loop

    Please see the attached code for more details.

    I hope this would give you a quick start for moving ahead with Physics based animations and Games.

    If you are an ISV and want our team to be engaged for training/development projects, please feel free to connect with me here.

    Happy Gaming!

    Rahul Gangwar

  • Patterns for Great Architecture

    Silverlight and The Application Design


    Today is August 5th 2008 and I am writing this short post just to convey how Silverlight changes the overall design of a conventional browser based application. Current public version of Silverlight is 2 Beta2.

    There are many reasons to use Silverlight;

    • To enhance the UI of existing browser based applications; When we install Silverlight plug-in, browser can understand more set of tags (XAML) apart from HTML tags. Moreover we can use the capabilities of the presentation engine of Silverlight to draw better UI.
    • To scale the application; Since now most of the end user interaction and processing will happen at client side itself, there will be less number of hits on the server. So now the application could support more number of concurrent users.
    • To do things beyond JavaScript; Not many things can be done with JavaScript which can be achieved with Silverlight. one example could be rendering HD video, which requires us to use graphics card capabilities to render it.
    • To use existing skill set of developers; Silverlight supports multiple languages (like C#, VB.NET, IronRuby, IronPython etc.) we could reuse the existing capabilities of developers to create better applications over web rather than learning new technologies to enhance the UI of browser based applications.

    Apart from these and many other features, one thing that attract me most is the way application deployment design changes with Silverlight development.

    Almost every business application has three layers: The presentation/UI layer, Business Logic layer and data Layer. If we talk in context to browser based applications, the presentation layer sits over Browser, while the rest of layers (Business + data) sits at server side. Whenever there is a requirement to do some business processing or access data we go back to the server and this gives a bad end user experience.

    With Silverlight, this design changes a little bit, but that little makes huge difference. With Silverlight, Presentation and Business Logic Layer can sit at Browser side while Data Layer can sit at server side. This means that now we have to go back to server ONLY when we need to get some data, otherwise we can sit happily at client side and attend them in real time fashion; just like windows applications do. On top of that, whenever we want to talk to server for data, we do so asynchronously which means end user doesn't come to know about whether server interaction is happening or not.

    The above idea is conveyed in figure given below.



    Any comments/suggestions/questions more than welcome.


  • Patterns for Great Architecture

    ADO.NET Sync Services: Services Based Architecture's Step-by-Step Demo


    Today is June 3rd 2008 and I am writing this blog post to demonstrate a step by step guide to developing an ADO.NET Sync Services Sample Application that is using Service Based Architecture. Though I planned to first write a post on ADO.NET Sync Services as a whole from architecture perspective and then provide a step by step instructions for the sample code, but now, when I have completed the step by step instructions before the architecture document, I am posting the same.

    In almost every practical case of disconnected applications architecture, the server side central database is exposed via http end points like web services. Moreover with the release of technologies like ADO.NET Data Services, it is a matter of few clicks to expose the entire data model to the external world using web services.

    So I thought of writing down the steps that I took in Visual Studio 2008 to create a client windows application which will be talking to the central database server via a http WCF Service. If you are interested in the sample code as well, please feel free to get in touch with me.

    1. Open Visual Studio and create a new windows application which will store all the client specific files.
    2. Add a new class library project to the solution which will hold all the server side code.
    3. Add a third web application project to the solution which will expose a web service. The web service will be used as a broker between the client windows application and the server side class library. So also add a new WCF service to the web application project. You can delete the Default.aspx page which is created by default by Visual Studio 2008. Also since this will be Cassini based, right click the project and select properties. On the properties page, select “Web*” tab and select “Specific part” radio button. This will fix the port on which the web server will run and we can have a fixed URL to the web service. Click on save.
    4. In the windows application, add the SQL Server Table that you want to cache. You can do this by adding a LocalDatabaseCache file from the project templates. This will present you with a wizard where you will specify the SQL Server Tables. In the first form of wizard, specify the Server Connection to tell the Visual studio about the SQL Server with which you want to do the synchronization. Clicking on the drop down will present you with already known connections, if you want to specify a new connection, click on “New Connection” button adjacent to the Server Connection drop down and fill the required details.
    5. Once you specify the Server Connection details, it will automatically suggest you with a Client Connection name. This basically specifies the location and name of SQL CE database file to be created at client side.
    6. Once you specify the Client Connection details, you can see “Add” button enabled. This button allows you to select the tables on the SQL Server to be cached on the SQL CE database that you just specified in above steps. Going a little deep, this process creates a SQLCE database at client side and simply copies the schema from SQL Server to the CE database. So click on Add button, select tables that you want to cache/synchronize and click OK. Please explore other options shown if needed.
    7. The moment you finish this wizard, you will be prompted with another wizard of creating a DataSet for the SQL CE database just created. If you wish to write your own ADO.NET code skip it, otherwise choose the Tables for which you want Visual studio designer to write the ADO.NET code. For first demo, I would recommend to go through the dataset wizard rather than writing your own ADO.NET code.
    8. Now, we write the server side synchronization code into the class library created. We will do this by using the wizard again. Though we could have done it in previous steps but just to make it simple, I am doing it in this step. Double-click .sync file in the windows application. This will open the wizard that we have seen earlier. Here click on the “Advanced” button to see more options on the same page. Then on the “Server Project location” drop down select the class library project and click OK. This will transfer all the server side code from the client application to the Class Library which will be used by the web service.
    9. Add a reference of the ClassLibrary project into the web application project because web service will use it. Also add references to Microsoft.Synchronization.Data and Microsoft.Synchronization.Data.Server assemblies into the web application project.
    10. On the windows application add “service reference” and add the web service reference to the windows application.
    11. On the class library, you will see a .SyncContract.cs file which has code for an Interface and class. We will have to duplicate the same structure on the WCF files. So copy the method declarations of the interface of .SyncContract.cs file (along with [OperationContract()] declarations) and paste them to the Interface file of WCF Service (in the web application project you will see a file starting with “I” if you have not added any additional files starting with “I” letter). You can delete the default method called “DoWork()” provided by Visual Studio. Copy also the “using” declarations so that there are no build errors.
    12. Similarly copy all the methods and the private variable of the class from .SyncContract.cs file into the .svc.cs file of web service. Copy also the “using” declarations. The code also contains a constructor method. Just change the constructor name in .svc.cs file to the class name of the .svc.cs file. You can remove [System.Diagnostics.DebuggerNonUserCodeAttribute()] declarations (if present) from the methods.
    13. In the same .svc.cs file add another using statement for the class library project (e.g., using ClassLibrary1;). Try to compile and see if there are any errors. Remove any errors if I have missed any logical step.
    14. On the windows application, right click the .sync file and click “view Code”. Here, you will see one partial class already created with an empty OnInitialized() method. Create another class in same file (let us call it RemoteProviderClass) implementing the ServerSyncProvider class. Add following “using” declarations on the top of file:
    15. using Microsoft.Synchronization;
      using Microsoft.Synchronization.Data;
      using System.Data;
      using System.Collections.ObjectModel;

      And the class declaration looks similar to this:
      public partial class RemoteProviderClass:ServerSyncProvider

    16. After copying the using statements, copy only the methods (not private variable) of the .svc.cs file of web application to this file inside the new class created (RemoteProviderClass). Again, one of the methods will be a constructor, so change its name appropriately. Notice that each of the method is calling methods from a private variable whose name starts with an underscore. Declare a private variable inside the RemoteProviderClass corresponding to the proxy of the web service. The variable name should be same as the one that each method is using and starting with an underscore. Also, inside the constructor method, delete existing code and write a line to instantiate this proxy variable.
    17. Try to build the solution and you will see errors inside GetSchema() method. This is because Collection<string> is transferred as string[] array. So write code to convert the Collection into string array. Also notice the override keyword; this should be used with all methods instead of virtual keyword. The code should be similar to this:
    18. public override SyncSchema GetSchema(Collection<string> tableNames, SyncSession syncSession)
      string[] tables = new string[tableNames.Count];
      tableNames.CopyTo(tables, 0);
      return this._serverSyncProvider.GetSchema(tables, syncSession);

    19. Everything should compile well now.
    20. On the OnInitialized method write code to create an instance of this new class and assign to the RemoteProvider property of the sync agent. Also specify the sync direction of the tables. Here I have one table in my SQLCE database called Employee. The code looks similar to this
    21. partial void OnInitialized()
      this.RemoteProvider = new ProxyForWebServiceCall();
      this.Employee.SyncDirection = SyncDirection.Snapshot;

    Now we will create a UI on the windows form and invoke the synchronization process on timer tick event. Calling synchronization is straightforward but calling inside a timer that is invoked every 1 second is a little bit tricky. Practically, web service might return response in 1 second, and also till web service returns back, the UI thread will be unresponsive, so we will invoke the synchronization code asynchronously and only when one web service call has returned. I will illustrate this in following steps and also show the sample code to make things more comprehensive.

    1. Open Form1 design and from data menu click on “Show Data Sources”. Drag and drop the table on the Form designer surface (in my case Employee table). Visual Studio will automatically create UI for end user editing and also write ADO.NET code for performing those operations on the local SQLCE database. Also drag and drop a timer control and set its enabled property to true and set its interval to 1000 (1 second). On the code behind of Form1 add using Microsoft.Synchronization.Data; declaration.
    2. On the code behind of Form1, declare three private variables belonging to the class of the Form. A Boolean variable to see whether we have to call the synchronization code or not. A delegate to the synchronize method of the Sync Agent class. A variable of the sync agent class. To see the class name of your sync agent navigate to the .cs file of the .sync file and see the class name of the OnInitialized() method. The declarations look similar to this:
    3. public partial class Form1 : Form
      bool IsContinue = true;
      private delegate SyncStatistics SynchronizeMethod();
      LocalDataCache1SyncAgent syncAgent = new LocalDataCache1SyncAgent();

    4. Now we will write code in the timer tick event handler to invoke synchronization. Here on the very first line we will check whether IsContinue is true or not, if it is not true, this means another web service call is pending and we have to wait, so we will return. But if IsTrue allowed us to enter further into synchronization code, let us block all other ticks of timer to enter. So the following code:
    5. if (!IsContinue)
      IsContinue = false;

    6. Then we will specify the synchronization direction of our global sync agent variable. I have set it to Bidirectional here. So this code:
    7. syncAgent.Employee.SyncDirection = Microsoft.Synchronization.Data.SyncDirection.Bidirectional;

    8. Next we will instantiate the delegate with the sync agent’s synchronize() method so that we can invoke it asynchronously. Then invoke it asynchronously. Notice that in the second line, we have passes a method name, MyCallBack, (which we will create in coming steps) which will be invoked once web service returns response. MyCallBack will simply set the IsContinue to true so that now timer can start for a fresh synchronization execution. So the code below:
    9. SynchronizeMethod syncMethod = new SynchronizeMethod(syncAgent.Synchronize);
      IAsyncResult result = call.BeginInvoke(new AsyncCallback(MyCallBack), null);

    10. After invoking the webs ervice, we can collect the resulty of web service by using EndInvoke. The result will be used to see if the DataGrid should be refreshed or not. If there were changes between two databases, then only form should be reloaded. So the code below:
    11. Microsoft.Synchronization.Data.SyncStatistics syncStats = syncMethod.EndInvoke(result);
      if (syncStats.TotalChangesDownloaded > 0)
      Form1_Load(null, null);

    12. And finally the callback method:
    13. private void MyCallBack(IAsyncResult result)
      IsContinue = true;

      Compile and run the application by setting windows application as startup project. Make changes from windows application and you can see them at SQL server and similarly make changes in SQL Server and they will be visible in Windows Application datagrid. Make sure that you do not change primary key field. This is just a start, go ahead and implement conflict resolution and try to make changes from multiple machines at same time.

    I would love to see your comments/suggestions/corrections if any.

    Rahul Gangwar

  • Patterns for Great Architecture

    High Scalability Architecture For Data Types Like Image


    Today is April 3rd 2008 and this small article focus on the approach of implementing a scalable solution which involve serving large amount of data such as Images. This is an infrastructure pattern.

    Ever wondered how a large web site serve content such as Images to end user? Take for example you have a photo sharing web site. Generally we use SQL Server as the back end storage solution for any application architecture. However, not all data types can be stored in the SQL Server. When it comes to data types such as images, SQL Server stores them in the form of BLOB. But BLOB in SQL Server has its own inherent problems. If you do read or write to BLOB, the whole of the BLOB being transferred over the wire and stored in memory. This can cause network congestion and memory pressure, particularly when there is a considerable load of concurrent users.

    If you need to read or write BLOB data such as images, you should first consider the options of storing them directly on a hard disk and storing the physical path or the URL in the database. This reduces load on the database. This basically requires two components. One, a File Server (for storing actual data) and then the Database management System to store references of the same. Microsoft provides solution for both the components in the form of Windows Storage Server and SQL Server. The following diagram explains this solution in a better way:





    Now, when we understand that dedicated File Server is the solution for storing datat types such as Images, how do we typically implement it? Such a solution is implemented in form of Storage Area Network (SAN) where Network Attached Storage (NAS) Devices are used to store the large data and act as a File Server. In such a solution even SQL Server resides on SAN.

    However, if you still want to store BLOB data in the database (maybe because you don’t want to have complex infrastructure of Storage Server), consider the following options to reduce the performance cost:

    --> Use chunking to reduce the amount of data transferred over the wire. Chunking involves more round trips, but it places comparatively less load on the server and consumes less network bandwidth. You can use the DataReader.GetBytes (from Application code) to read the data in chunks or use SQL Server-specific commands, such as READTEXT and UPDATEDTEXT, to perform such chunking operations.

    --> Avoid moving the BLOB repeatedly because the cost of moving them around can be significant in terms of server and network resources. Consider caching the BLOB on the client side after a read operation.


    Comments and suggestions, as always, more than welcome.

    Rahul Gangwar

  • Patterns for Great Architecture

    Patterns used for navigating between pages in ASP.NET:


    ASP.NET offers different APIs and ways (like Server.Transfer, Response.Redirect etc.) through which you can navigate between pages in your web applications. But before choosing any of these ways, we must have an understanding about the pattern that we want to follow. Depending on the pattern used, our choice about selecting an API changes.

    Broadly, there are two patterns used for navigating between pages depending on the requirement:

    1. Loosely Coupled Architecture:
    In this pattern, each page has its own processing unit. So if you have a page that has some input controls, its code behind will have all the logic to form SQL Queries based on input and get the result also from database. Then when it comes to displaying the result, another page (the results page) is invoked and the data of previous page are passed to the new page. This is called loosely coupled architecture since each page is given a responsibility and it performs its work without being dependent on any other parts of the system. So one page is given the responsibility to gather user input and prepare the result and another page is given the responsibility to display data. The picture given below illustrates this architecture in a better way:


    When to use this pattern:

    Now when we know that what this architecture means, we can appreciate the scenarios where we can use it. The scenarios are too wide, and one possibility that I can think of is the scenario where you want a decentralized approach. Whenever we have scenarios where processing logic is different for each page, we go for this approach since there is no point in delegating the processing logic to results page. If we delegate the processing logic to results page, we will have many results page so to avoid this, we choose to keep the processing logic in the input gathering page itself.

    How to implement:

    The default model of ASP.NET provides this facility where each page posts to itself until we specifically configure the controls to post to another page. The steps involved are:

    a) Implement the input page with all the data preparation logic in its code behind. The controls on this page will post to itself.
    b) Invoke another page when the processing is done and pass the processed data to the target page. This second step can be achieved in many ways depending on the requirements. Below I show few techniques that can be used.

    a) On the first page’s code behind (having input controls), when all the processing is over, add the data to be transferred to next page into the “Context” collection (Context.Items.Add() method can be used for adding into Context collection) and then invoke the target page using Server.Transfer() . On the target page (the results page) retrieve the data from the Context Collection simply by using Context.Items[] collection and typecast it if needed. Context collection remains into scope this ways on the target page as well (remember we cannot use response.redirect() method if we want to use Context collection to store our data since context dies the moment the request thread leaves the server).

    b) Another API to achieve this is to use Request property. This basically gives you reference to the current context and then using Request.Form we can access the source page.

    c) If you want to use the data between multiple requests, you can store data into Session collection instead of context collection. Everything else remains same (and yes we can use response.redirect() method if we are using session to store our data).

    d) A third way could be to get the reference of the source page on the target page. Whenever we use Server.Transfer() method to navigate to another page, we can access the source page from Context.Handler property. So all we need to do is to create storage on the first page (for example by creating a property called data on source page) and store the result in that property. Then, on the target page access the properties (in fact everything that is public on the first page including controls) by using Context.Handler. This approach is used in cases where we have many discrete data elements to be transferred to the target page. So instead of adding them to Context Collection on first page, we add into the page itself by creating properties. Then we can access all the properties of source page on the target page. Another good thing in this approach is that we do not need to typecast in contrast to Context/Session approaches. More can be found here:

    2. Coupled Architecture (Centralized approach):
    In this pattern, we designate a central page for processing the input and displaying the results. We form separate pages for gathering input. The input pages gather data from end user and post it to central processing unit (the results page). This is the approach where you post the source page directly to another page (the cross post scenarios). The following figure illustrates this:


    When to use this pattern:

    This approach is used whenever we want to reuse the code. If the logic can be reused across multiple pages, then there is no point in writing the same logic on multiple pages and making code unmanageable. In that case we create a central processing unit (the results page) and all other pages just collect data from end users and post themselves to the results page.

    How to implement:
    We can use PostbackURL property of controls to specify the target page. In this approach, on the source page we specify the target page URL in the PostbackURL property of controls (for example a submit button). This ways, when end user clicks on the submit button all the data of the input page is sent to the target page directly. So no need to write any code behind on source page. A very clean way to implement this pattern.

    Yet another model can be followed which involves, first posting to the source page itself and then source page transferring to the target page but no processing is done at source page. We can do this by using Response.Redirect() and then passing the input data using query strings.


  • Patterns for Great Architecture

    ADO.NET Entity Framework


    ADO.NET Entity Framework is Microsoft's step towards its Data Platform Vision. In this blog, I am going to talk about the architecture of the ADO.NET Entity Framework and the way it will change application development style. Though it is in Beta Phase right now but is expected to be part of Visual Studio 2008 during mid of year 2008.

    The Vision:

    If we closely look at the new development initiatives from Microsoft, we can clearly see where Microsoft Development Platform is heading towards. Be it the new "language features" in C# 3.0 like LINQ or new "components" in framework like Communication Foundation, Workflow Foundation or the new features in "products" like Biztalk, SQL Server etc., one thing is common in all of them: "Abstraction". By the term abstraction I mean hiding all the complex details and exposing simple interfaces to achieve one goal of allowing the developers to concentrate on "what" do they need rather than "how" do they need.

    Take example of LINQ. Simply put, the primary aim of LINQ is to develop a common querying language within the programming language so that developer does not have to learn new querying languages (like SQL, XPath etc.) varying on the kind of data source they are dealing with. Today with LINQ, you don't need to know SQL or XPath to search data from a database or XML data source. You just need to know LINQ and search every data source with it.

    Now look at the structure of a typical query expression in LINQ. You will find the structure similar to SQL having keywords like "Select", "From", "Where" etc. along with a fleet of operators to work with. It is typically the smell of any 4GL language. SQL is a 4th Generation Language, similarly LINQ is a 4th Generation Language. And what is a 4th Generation language in anyway? Any construct that offers me the ability to specify "what" without asking me "how" is a 4th Generation language.

    Take another example of WCF. You can now create generic services in WCF within no time that can work on REST principle (REST is like backbone of today's web) and that can also work on a completely proprietary protocols with very little efforts. In WCF we call them "Service" which is as generic as we have "object" type in any OO Programming Language. The same service can behave as a web service, can use message queues, can use TCP with just simple changes in configuration. Developer does not have to worry about intrinsic details. In fact whenever I personally design an architecture for Microsoft Partners, I either think of WCF or Biztalk whenever communication comes into picture. These two technologies are key to enable communication framework within software components as well as enabling integration between different disparate systems.

    The examples can go on and on since the technologies offered by Microsoft are many. But the point that I am trying to make is that Microsoft has this vision to introduce abstraction to such an extent that things become extremely simple and we can produce great software within no time, no cost.

    With ADO.NET Entity Framework, the Microsoft vision remains the same: To abstract data access with programmers so that they spend their time in writing "what" rather than "how". Imagine if the 4GL concept spans across all layers, application development will be over in the requirement gathering phase itself since the moment we specified 'what" do we need from application, application is ready. ADO.NET Entity Framework is a little step towards that vision.


    A typical application has those 3 layers: Presentation, Business and Data Access. ADO.NET Entity framework will change the way Data Access layer is written. One way of looking at it is to consider as an ORM (Object Relational Modelling) framework. Any ORM framework aims to bridge the gap between the way data is represented in database (or better data source) and the way it is represented inside the application. An ORM framework does that by providing a framework that does the mapping between your classes and properties within it with database tables and their columns. This framework is an extra layer between the data access layer and data source. So developer do not have to write code against database, fetch data in some format, convert it to fit with objects in application and then use the data. Instead, developers do all their modifications on objects only and the changes are translated by the ORM framework into appropriate database calls. ADO.NET Entity Framework is also an ORM framework that provides this functionality. ORM frameworks are not new concepts, we have plenty of them (like nHibernate) in market. But what is different with ADO.NET Entity Framework is that it is much more than simply an ORM framework. Not only does it bridges gap between representation of data within application and database, but also it aims to improve upon performance (unlike other ORM frameworks that hits to performance) and more importantly it integrates with the programming language itself (we have LINQ to entities for the same reason) making this integration more flawless and easy to use.


    I would take the picture (shown below) from one of the "data points" column of MSDN Magazine that clearly shows the high level developer architecture of ADO.NET Entity Framework. Let us dissect each element to appreciate the architecture better.



    Entity Framework Layers (Logical + Mapping + Conceptual):

    The three entity framework layers shown in above diagram are the core of whole functionality. These layers are also called as EDM (Entity Data Model). Entity framework engine basically takes decisions on the basis of the EDM. The first step that we have to do in order to use the entity framework is to generate (optionally edit based on requirements) these application specific layers. But do not worry, generating these layers (EDM) is not more than a matter of clicks. You can generate an EDM using a database as a starting point. You can then modify the XML manually (or possibly using a modelling tool that may be available in a future release of Visual Studio). When you add an ADO.NET EDM to your project, the wizard walks you through the process of creating the EDM.

    The Logical layer contains the entire database schema in XML format. This represents the story at database side.

    The Conceptual Layer is again an XML file that defines the entities (your custom business objects) and the relationships (the custom ones) that your application knows. So if you say that each Customer can have many associated orders, that relationship is defined in the conceptual layer even if this relationship does not exist at database side.

    The Mapping layer is also an XML file that maps the entities and relationships defined at conceptual layer with the actual relationships and tables defined at logical layer. This is the core of entire Entity Framework that bridges the gap.

    Within application we always interact with Conceptual layer. To talk to Conceptual Layer we can use

    • A new language called Entity SQL directly (actually Entity SQL queries are fired using Entity Client provider which is very similar to ADO.NET model).
    • LINQ to Entities, which is again another language integrated querying facility.
    • Using Object Services which in turn uses an ObjectQuery Object.

    Whenever we commit changes (like invoking SaveChanges() method) to databases from application, appropriate queries are fired (it also takes care of the relationships). We don’t have to think about relationships at application level since all the mapping is done at EDM layers. So, Entity Framework handles Relationships quiet neatly.

    Wrapping UP:

    The best wrap up that I can write is what John Papa gave in his article:

    "The Entity Framework allows developers to focus on the data through an object model instead of the logical/relational data model. Once the EDM is designed and mapped to a relational store, the objects can be interacted with using a variety of techniques including EntityClient, ObjectServices, and LINQ.

    While the traditional objects such as the DataSet, DataAdapter, DbConnection, and DbCommand are still supported in the upcoming release of ADO.NET available in Visual Studio "Orcas," the Entity Framework brings major additions that open ADO.NET to new and exciting possibilities."

    There are other Microsoft initiatives in the data platform vision like having a framework to expose data as service and tools for developing for an occasionally connected architecture. More on those initiative in my future blog entries once I get my hands dirty with them as I have done with Entity Framework :).



  • Patterns for Great Architecture

    Microsoft Vision for development platform - Codename Oslo


    If someone asks me Microsoft's strategy on application development, I would simply say SOA spanning across the servers, clients and the cloud. Microsoft is hitting on its strategy of SOA by releasing new products and platforms at a rapid rate. The next big thing from Microsoft towards this goal will be "Oslo".

    Today in any typical application development scenario, there are various experts involved at different levels. Each if these experts try to develop a model of the area in which they have expertise. For example, business analyst might model the software requirements, architects model the high-level designs, administrative staff might model the deployment and maintenance requirements etc. Then developers develop the code corresponding to the model.

    The approach is perfect where each person is doing his job in the field the he is expert but at the same time there are some impedance mismatch between these different sets of people. The models developed by an analyst is developed using entirely different set of tools than the tools used by an architect or developer. Not only this, if someone wants to be an end-to-end expert of the software lifecycle he will have to learn a vast range of tools, modeling languages etc. Even bigger than this is the problem when it comes to development. Developers have very less usage of the models that other experts have created as far as the development efforts are concerned. Models just remain a source of reference and this is the reason why modeling is still only at the surface of the software lifecycle.

    To bridge all sorts of mismatch problems, Microsoft plans to bring an integrated environment for everyone. Analysts, Architects, IT professionals, Developers can all work together, and not only work together but also benefit from the model created by others practically. Microsoft is making it possible by building a general-purpose modeling language that everyone can use, common tools to automate the tasks and a repository where all the models will reside.

    "Oslo" is the codename for all the technologies that will make this possible. With Oslo we can imagine a development environment where users will design a model for an application that then becomes the application itself. This will shorten the time from design to development and bridge the different phases of software lifecycle to make it a continuous process. The "Connected Tools Team" in Microsoft is responsible for the new modeling tool that will make this process a reality.

    Oslo includes:

    --> Future release of .NET Framework 4. So we can expect that .NET framework 4 will have further enhancements to the WCF Workflow Foundation Technologies.

    --> Biztalk Server family will evolve for enabling a highly scalable SOA and BPM solutions. The Biztalk Server is expected to be built on top of WCF and WF and will also have the capabilities to develop, deploy and manage composite applications.

    --> We will also see the release of Biztalk Services that will allow the development of cross organizational composite applications.

    --> We will see a wide variety of tools that allows the dream of model driven development to turn into reality. So this is basically the extension of Visual Studio Team System capabilities which will be the unified environment for all class of experts bridging each phase's output with input to next phase.

    --> Better repository system for metadata.






  • Patterns for Great Architecture



    Conceptual Silver Light:

    I am writing this entry because I feel Silver Light to be the sword of developers in Web 2.0/S+S world.

    I remember my first introduction to HTML programming in my school days. One of my seniors was doing all the HTML stuff in an attempt to design an attractive web page. When I saw the way he wrote html tags that was a surprise to me (more because it seemed quite terse to me). But then he told me that its actually very easy and intuitive if you know the elements that you have to use.

    That said, I started learning HTML and came to know about various buzzwords like DHTML, JavaScript, DOM Object Model etc.

    I request your patience for reading the HTML stuff because this will suddenly and finally open up the doors off Silver light.

    One beautiful concept that I like about HTML programming is the Object Model. In HTML programming there are several HTML elements like <html>, <head>, <Table> and list goes on.

    The basics of HTML programming lies in the fact that there are fixed (though large) number of elements that a standard browser understands. And these elements are nothing more than a way to tell the browser that “Hey! Organize the UI in this way”. So in effect, HTML can also be considered as a formatting mechanism used to format the UI on browser. I always wondered what if I want to have a UI that is not represented by any of the available elements… then the basics of composition (nesting in HTML) solved the purpose. So, if we want to create a UI that is not represented by any standard element in HTML, we can create the required UI by combining the basic HTML elements in appropriate form.

    The browsers evolved in terms of the capabilities, they started understanding DHTML elements also for some moving UI, then browsers also supported scripting languages like JavaScript for both making UI flexible as well as interacting with end user and manipulating HTML elements. Then came more advanced capabilities in browser like asynchronous client script callbacks, XMLHTTP etc which led to cropping up of many new technologies similar to AJAX.

    Still, I feel a bit limited in the way I program windows application and a web application. The complexity becomes even more complex when we have to do the multimedia, animation etc. IMHO, the primary reasons to why web programming is still complex are following:

    1.     There are two sides of the web programming framework, client and server. Both talk to each other using http. HTTP has inherent problems like being stateless etc. and we came up with different solutions (rather patches like session state etc.).

    2.     Handling end user interactions involve either roundtrips (which has another set of drawbacks) or through client side languages like JavaScript etc. This personally makes me hesitant in doing more things at client side and instead delegating it to server side. We can do stuff like asynchronous callbacks etc. but they again require lots of plumbing.

    3.     The UI controls of web programming are less rich than the windows programming because we have to get limited by the capabilities that browser provides. We really cannot do anything beyond what browser permits even if .NET or any server side platform has immense capabilities.

    These and many other factors make web programming a little poor.

    For me, most of the problems will be solved if we have the same capabilities on client side that we have at server side. So that said, if we have .NET on client side as well, we just need to program the application the way we used to and the application will run at server side similar to a windows application.

    In this attempt Microsoft started off with smart client applications (click once) with added benefits like automatic updates etc. Great! But they did not pick success in developer world since not all clients are expected to have a .NET framework installed.

    To overcome this functionality, Microsoft designed a new .NET framework which they called as Silver Light and this is a small size, mini version of .NET framework that can be installed on client side as a plug-in to most of the modern browsers. This mini .NET framework was geared towards making UI at client side more attractive and is currently oriented to provide the multimedia and animation capabilities on browsers similar to technologies like flash.

    However I see this from a different angle because this makes me program it in a better way and appreciate the vision.

    When I install Silver Light on a browser apart from considering as if I have the power of .NET at client side, I also look it as increasing the capability of browser all of a sudden. I install Silver Light and Boom! – My browser can now support more tags than ever before. Apart from understanding <HTML> <HEAD> <TABLE> etc. it will now also understand <CANVAS> <BRUSH><TEXTBLOCK> and many more. Now I don’t have to peek around for getting things done for my complex requirements but instead, now I have increased the element set of browser. Not only the element set; now I can also use either Java Script or my favorite C#. Isn’t this great? Well for me this is like a phenomenon.


    The following diagram illustrates this more clearly:




    We can see that Inside the DOM we have another DOM like object model which is the Silver Light Object Model. The tags that we write in DOM are called HTML while the tags that we write inside the Silver Light DOM are called XAML. Simple! Now just learn XAML and the way to access XAML and HTML elements using C# and we are set to program Silver Light.


    Though Silver Light holds lots of promises for developers in its current stage it’s fighting with some initial hiccups like, right now we cannot use C# but instead we will have to still use JavaScript in the 1.0 version. However in Alpha version we can write in .NET compliant languages like C#, etc.

    Moreover, at present, Silver Light is geared for Multimedia but we can see more controls in Silver Light class library to have most of the capabilities of any windows application. To me Silver Light holds promise for next generation Mash ups in web 2.0 or S+S world since it holds most capabilities and flexibility out of all the available development technologies from Microsoft.


  • Patterns for Great Architecture

    SaaS-Value Proposition



    Is the Total Cost of Ownership of a SaaS application really low than a traditional application for an end user?

    SaaS is buzzword in today’s delivery model of software. Many ISVs are hastily looking for converting their existing traditional applications on to the SaaS strategy. Once the product finally moves onto SaaS delivery model, then starts the process of attracting customers/end-users. The primary reason that customers (especially SMBs) are told to move to the SaaS product is that the total cost of ownership is low for them. The tenants don’t have to invest on the hardware, maintenance, employing IT staff etc. Moreover, customers are charged on per month/pay as use etc. model. SaaS offers so much of flexibility in terms of acquiring the software that customers fall prey of SaaS product immediately. The value proposition of TCO is so much tempting. That’s a good thing but at the same time this value proposition is basically confusing the customers about the other value propositions about SaaS. Not only this, at few instance the value proposition of TCO is actually wrong. There has to be a good understanding among both ISVs as well as customers about the concept and philosophy of SaaS in order to fully appreciate the SaaS model.

    Let’s talk about the TCO of a SaaS application for a tenant. When a tenant buys a SaaS application, definitely, the TCO is low as compared to its traditional approach version. But how far? If we take an example where the customer is going to use the software for typically longer period it is quite possible (not necessary) that SaaS application will cost the customer much more than the traditional version. It will actually be cheaper for the end user to buy the software, build infrastructure and maintain it rather than paying on a monthly basis. Yes that is true.

    I can give an analogy to the real world to understand the concept. When I go out to another city (maybe for vacation) I will need a vehicle to move around the city. I have two options there, buy a new car for me or hire a cab. Of course since I am there for a less period, the cab will intuitively be a better option. But what if I use cab even in the city where I live?  Definitely as time moves from one year to another and so on, I will realize that buying a Car was a better option. Similar is the concept of SaaS application. In software field, some things change and are not same as the real world of Cabs and Cars. Those differences are the real value proposition for SaaS according to me.

    For example, in software world, our needs/business requirements changes and we will need a framework to accommodate these changes into our existing infrastructure without investing heavily. Of course, it will not be a good idea to get a new version of application according to our business changes. SaaS application is built from ground up keeping these business changes in mind. A SaaS application provides the flexibility to customers to change the business rules.

    Among other features that make SaaS approach more valuable include ease of sharing Data to third parties, availability of backup data centers for disaster recovery and, especially for smaller businesses, having access to the enterprise level application services that would be near impossible to achieve in-house.

    Among other benefits of moving to SaaS, if I start counting I see the list endless. Things like achieving on-demand scalability just by a little extra, turning variable cost into fixed costs by delegating the variable costs on to the hosting provider, Integration with other LOB applications… list is endless.

    The idea that I want to convey here is that for end users cost is not the sole value proposition, but when we look at the total value that SaaS can deliver, SaaS becomes even more compelling.


  • Patterns for Great Architecture

    How to SaaS


    This is my first blog entry and I am happy to get this blog meant only for Microsoft Full Time Employees.

    Okay coming on to the title for this blog post. Today I am going to write down on Sofware as a Service (SaaS) architecture strategy.

    I will un-cover the technical aspects involved in building a SaaS application and very high level overview of the technical aspects of a SaaS solution.

    These concepts can be applied to any Application which is a good candidate for a SaaS application.


    1.     SaaS- Introduction

    2.     SaaS -Challenges

    3.     SaaS -Solution to challenges



    A SaaS application can be defines as any “Software deployed as a service and accessed using internet technologies”. In order to realize a SaaS solution (in fact any solution) we need to do two things:

    1.  Build our application which can be used as a service over internet by different clients. This could range from a programmatic service (accessed programmatically by other softwares) to a stand-alone application used by different clients. This step is involves us to follow certain steps that are different than developing a normal “On-Premise” or simply called a non-SaaS application.

    Typically when we develop an application (a non-SaaS application) then we do assume two things:

    ·         Application will be installed at client side (e.g., a desktop application) be it through a CD or downloading via internet. This also implies that the client will have to do the maintenance (like upgrading to new version, apply patches, maintain databases etc.) of application even after he has bought the application (not true for Application Service Provider model). This fact is not true in terms of a web application. So we can say that any web application forms a different category of application since it is not installed at client side. But still we cannot call ANY web application a SaaS application.


    ·         Application will be served to requirement of a particular client. If another client needs some changes in the application, we will make changes in the application source code and then run another instance of that application for the new client. So in nutshell, we assume that one application is meant for one client only. This assumption is true even for a web application.


    Here comes the difference! The core of a SaaS application is based on the opposites of these two (above mentioned) assumptions. The following are assumptions that are true for a SaaS application.

    ·         Application will be typically deployed and maintained by a hosting provider. So, clients don’t have to invest in terms of buying hardware resources and employing the IT staff to manage the application. This will be done by the hosting provider.

    ·         Application will be accessed by clients using internet. In special cases where a SaaS application is hosted inside the enterprise itself this is not true but still the clients accessing the hosted application will still use internet technologies to access the application.

    ·         A single SaaS application (in fact a single database too) serves more than one client having different needs. So the application will be designed in such a way that only a single application instance will be able to provide different functionalities to different clients. This model is more popularly known as Multi-tenant model.


    Among these assumptions, the last assumption related to multi-tenancy requires a SaaS application to be architected in a special way keeping in mind about certain aspects of a multi-tenant application. These aspects form the challenges of building a SaaS application and also our next topic of discussion.

    2.  SaaS world does not end just by building the application that satisfies SaaS characteristics. Deploying a SaaS application forms another set of challenges. Typically a SaaS application is deployed by a SaaS hosting provider and the hosting provider is responsible of maintaining the application. So, not only clients get rid of maintaining the application but also ISVs get rid of that aspect.

    Microsoft provides end to end resources in developing and deploying a SaaS application. It provides resources ranging from development frameworks and technical resources used to build and design the application to hosting options that assist in deploying the application.




    As we have seen that one of the important characteristics of a SaaS application is multi-tenancy, this aspect implies further challenges in building a SaaS application. Let’s see those challenges after trying to define the multi tenancy.

    Multi Tenant Architecture:

    A single SaaS application serves more than one client having different needs. This means that you run only one instance of the application and that single instance will be accessed by all clients. For example, you built a CRM SaaS application and hosted it over a SaaS hosting provider. Now many organizations have different CRM needs depending on the nature of business they do. We will not build any separate application for all the companies but our single application will have the option using which each client will customize the SaaS application according to his need. Not only that a single database can be used (not always) for storing data of different clients. So security is also a major concern. Let’s see all the concerns and challenges that we need to consider while building a SaaS application.



    Here are some important challenges that we need to consider while building a SaaS application. It is important to note that not all challenges are required in a single SaaS application. These challenges depend on your application’s requirement. Here only the challenges are presented. Next section will deal about ways to address those challenges.

    Business Agility:

    Different organizations can have (in fact they do have) different business needs. For example process of approving a loan in two banks may be different in a way that Bank-X requires a Loan Manager to approve the loan request document before sanctioning the loan while Bank-Y requires both the Loan Manager and Bank Manager to approve the loan request. Now if we are building a SaaS application to target these different business requirements, then, we must provide a way through which different clients can configure the SaaS application according to their business requirement. This is what we mean by Business Agility (we can call it as ability of an application to configure its business rules).

    User Interface Configurability:

    Each client of our SaaS application might have the requirement to change the UI of the application according to their need. This could range from configuring color, font, Company Logo etc. to positioning , ordering of UI elements (for example web parts) etc. incorporating this capability of configuring the UI of a SaaS application according to client’s requirements form another requirement and thus challenge to building a SaaS application.

    Database configurability:

    Since typically a SaaS application uses a single database (not always) for every client, all the data of different clients can be stored in a single database. This approach makes a SaaS application more efficient since only a single database is used but this also poses challenges on how to design the database so that different data model needs can be accommodated in a single database approach. This approach is commonly known as Multi-tenant database.


    Since a single application instance caters to requests of all clients, it is important (in fact a must) that we have some mechanism to make sure that data of each client remains secured and users of a particular client can access data meant for them alone.


    A single application can be used by different clients and each client can have a number of users accessing the SaaS application. This puts a constraint that the application must be able to handle increased number of users. Scalability is a challenge that has to be handles at 3 different levels: The SaaS application (your code) must take care of scalability issues, the SaaS application server (generally done by hosting providers) must be scalable and then finally the database server must be scalable. Scalability at all these three different levels is achieved using different techniques that we will see later on.


    SaaS-Solution to Challenges


    We will take all the challenges, discussed in previous section, one at a time and propose techniques used to address these challenges. To some techniques we will provide the direct links to already existing resources instead of dwelling those techniques in detail out here.

    Business Agility:

    This aspect involves designing your SaaS application architecture in such a way that clients of the application can modify the business logic written in your application. Now, how do we provide the modification of business logic of application without writing new code or even modifying existing code?

    The most commonly used approach is Metadata driven architecture. Let’s see how it work and what ways (maybe tools etc.) do Microsoft helps in designing such applications.

    Concept of metadata driven architecture is quite simple and complex aspects are handled by Microsoft technologies and tools. In a metadata driven system there are three major steps involved:

    1. Metadata: Whatever features of an application are to be configured during runtime, we define those features of the application in the form of Metadata (it could be an XML file stored on hard drive or a backend database). This metadata can be extended by writing more data to it.
    2. Metadata Services: A set of programmatically accessible services are designed that provides an interface to access/modify the metadata. This can be thought of as an interface layer or façade layer. We provide these services so that metadata can be exposed (maybe over internet) to any application/tool while keeping the integrity, validity etc. of the metadata intact. Imagine a situation where an application (by mistake) modified the metadata that violates some core semantics of metadata. Providing an interface layer to any data store is a common principle in building a distributed application and any SaaS application is on first place a distributed application.
    3. An Engine: The business layer of our SaaS application will be nothing more than an engine that runs the business processes defined in the metadata store and accessed using metadata services. These steps could be a bit complex depending on the configurability requirement of your SaaS application therefore Microsoft provides a number of options to build such systems.

    Most commonly used technology is using BizTalk Server. BizTalk server has two main components: 

      • A process execution engine that manages the steps, applies the business logic, and invokes the supporting applications of a complex process and/or transaction set.
      • A multi-transport messaging hub for transforming and routing information to and from applications and process steps.

    Another technology seen by developers in their most loved development environment (Visual Studio) is Windows Workflow Foundation (Part of .NET Framework 3.0) used to model business processes as workflows.

    Yet another technology from Microsoft is Microsoft Web Services used to build Service oriented applications which in turn is essential to deliver the business agility and IT .

    For more details on these technologies you can refer to following resources:







    User Interface Configurability:

    The process involved in allowing end users to modify the user interface according to their need is similar to achieving Business Agility- Using Metadata Driven Architecture. End-user preferences are stored in form of Metadata and UI is composed using this metadata. User interface metadata defines connections into user interface elements, such as menus, Windows Forms controls, and HTML.






    Database Configurability:

    For a SaaS application we can use different databases for each client and also use single database for every client depending on the need of the application. This asks us to design a Multi-Tenant Data Architecture (where tenant is equivalent to client).

    This involves choosing an approach to in a given scenario and also how to realize that approach. There is a very good article dedicated to developing a Multi-Tenant database system. I would recommend going through that article while going to develop the data model for your application.




    Security: (Most of the content of this heading is from the famous Catching the Long Tail article)

    As we discussed earlier, security is an important aspect of a SaaS application (in fact any application). Security requires special attention for a SaaS application because a single application instance is serving different clients. This poses two major challenges:

    1. How will the administrative tasks related to security handles? For example, creating new users, security policies, and permissions etc. for every client is obviously different.
    2. How to keep client specific request secured? This also involves keeping data for clients secured even if we have data for different clients in same table.

    There are different techniques and patterns such as filtering, encryption etc. Let’s see an overview of how are security requirements handles by a SaaS application.

    Common approach is to create security services for the application. So security is also applied in the form of services (sole reason is to incorporate the distinct requirement of SaaS application). SaaS application is built on the foundations of SOA (service oriented architecture) and services are building blocks of any SOA application.  Typically we divide security services into two components:

    • Authentication Services
    • Authorization Services

    The SaaS provider typically delegates to each tenant the responsibility for creating and maintaining its own user accounts, a process known as delegated administration. Delegated administration creates a situation in which the customer is responsible for creating individual user accounts, but the vendor has to authenticate them. To accommodate this delegated-administration model, SaaS designers use two general approaches for handling authentication: a centralized authentication system, or a decentralized authentication system. The approach that you choose will have ramifications for the complexity of your architecture and the way end users experience the application, and you should consider what your business model says about the needs of the application, customers, and end users when making a decision.

    In a centralized authentication system, the provider manages a central user account database that serves all of the application's tenants. Each tenant's administrator is granted permission to create, manage, and delete user accounts for that tenant in the user account directory. A user signing on to the application provides his or her credentials to the application, which authenticates the credentials against the central directory and grants the user access if the credentials are valid.

    This approach requires a relatively simple authentication infrastructure that is comparatively easy to design and implement, and that does not require any changes to the tenant's own user infrastructure. An important disadvantage to this approach is that a centralized authentication system makes it much more difficult to implement single sign-on, in which the application accepts the credentials that the user has already entered to gain access to his or her corporate network. Without single sign-on, users are frequently presented with an inconvenient login prompt when logging in to the application, and they must enter their credentials manually.

    In a decentralized authentication system, the tenant deploys a federation service that interfaces with the tenant's own user directory service. When an end user attempts to access the application, the federation service authenticates the user locally and issues a security token, which the SaaS provider's authentication system accepts and allows the user to access the application


    This is an ideal approach when single sign-on is important, because authentication is handled behind the scenes, and it doesn't require the user to remember and enter a special set of credentials. The decentralized approach is more complex than the centralized approach, however, and a SaaS application with thousands of customers will require individual trust relationships with each of the thousands of tenant federation services.

    In many cases, the SaaS provider may want to consider a hybrid approach—using the centralized approach to authenticate and manage users of smaller tenants, and the federated approach for larger enterprises that demand, and will pay for, the single sign-on experience.

    To understand more about security concepts and how to apply them you can refer following resources:



    To configure security at database level use guidelines given in following resource:




    Scalability refers to the capability of an application to handle large concurrent requests. There are two approaches generally used to scale an application:

    Scale Up: Scaling-up is process of upgrading the existing computer. For example, you can upgrade your Server computer from a 4 processor machine to an 8 processor.

    Scale Out: Scaling-out is the process of adding additional computers. This ways if a server reaches its maximum number of requests, another server takes up further requests. SaaS solutions generally use Scale out. A Common techniques used for scaling out is clustering.

    Scaling an application can typically be done at 3 levels:

    • Scaling the backend database
    • Scaling the Web Server on which the web application is running (scale up and scale out)
    • Scaling the application itself

    Please refer to following resources to learn about different scaling techniques:


    Scaling SQL Server: 


     Techniques used for scaling application:


     Architecture and design review of application for scalability




    Another good resource on scaling



    Scaling the Web Server:

    Good resource on Scaling a Microsoft® Windows® Server 2003 running IIS 6 (also included is a case study of scaling an asp.net website on IIS 6.0)



    I will keep posting new entries on SaaS in future.

    Please let me know your comments/questions.




Page 1 of 1 (20 items)