Solutions Monkey : Web Service

Building Pub-sub applications using MSMQ

anandraj — Tue, 20 Sep 2005 06:47:00 GMT

MSMQ version 3.0 supports multicasting through the PGM (Pragmatic Multicasting) protocol in WinXP and W2K3.
You can build simple publisher-subscriber applications using this feature.
Queues can be associated with a Multicast IPAddress. Subscriber applications can listen on these queues for new topics.
Publisher applications can multicast the topics to the multicast IPAddress. All the queues that are associated with the multicast IPAddress will receive the topic.
Subscriber apps can receive the topics from their queues and process them.

Advantages:
1. Both MSMQ and PGM are part of the Operating system. They are a big bang for your buck!.
2. PGM protocol has better reliabiliy than UDP.
3. You can use MSMQ trigger feature if you want to filter topics based on message properties (label for example) without having to write a single line of code.

Disadvantages:
1. The fact that pub-sub is justapoxed with queue means you cannot publish topics synchronously. This means you might have some disadvantages in some scenarios where you might want
the subscribers have to process the topic and return results back to the publisher.
2. If the publisher and subcribers are on different sub-nets the router should support PGM capabilities.
3. MSMQ doesn't support multicasting transactional messages. This means there is no exactly-once in-order delivery guarantees for these messages. There is a possibility of message loss.

For more information on the MSMQ multicasting feature read the following article on MSDN.
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msmq/msmq_ref_properties_queue_0nub.asp

The MSMQ multicasting feature is available through the MsmqIntegration channel in WCF. This allows WCF client applications to multicast messages to a Multicast IPAddress. WCF receiver applications can listen on the queues and process the messages.
For more information on the MsmqIntegration channel please review the WinFX SDK.
http://winfx.msdn.microsoft.com/library/default.asp?url=/library/en-us/indigo_con/html/b8757992-ffce-40ad-9e9b-3243f6d0fce1.asp

WCF doesn't ship with a pub-sub feature in V1, however you can be rest assured that the brightest minds in the WCF team are thinking about it seriously for V2.
This solution will not be one that is juxtoposed with queues for Queueing and Pub-sub are two important but orthogonal pieces of an enterprise software.

-AnandRaj

Microsoft Message Queue - An Introduction

anandraj — Sat, 06 Aug 2005 10:00:00 GMT

In this series I would like to introduce you to MSMQ history, key concepts, dwell into painpoints, suggest workarounds, and give an insigth into to future.

What is MSMQ?

MSMQ enables Asychronous communicaiton between two different applications. Application A(Sender) sends a message to a queue hosted in MSMQ, Application B(receiver) receives and processes the message from the queue MSMQ is usually used in the following scenarios a. Where there is an impedence mismatch between the sending and receiving applicaiton b. Load-levelling scenarios c. Sending and Receiving applications need to be running at different times d. Sending and Receiving applications need to be running across network or even across the internet.

MSMQ enables you to send messages over a variety of protocols such as TCP, IPX (if you care anymore :-) ) , HTTP, PGM (Pragmatic Multicasting) MSMQ provides out of the box security and routing of messages.

The History

MSMQ was designed and developed by a Microsoft developement team in Haifa, Israel.

MSMQ version 1.0 was shipped with the NT Option pack. In this version MSMQ was integrated with SQL Server. The programming model was easy but the Installation and management was cumbersome.

MSMQ version 2.0 shipped as an optional windows component with Windows 2000. Active Directory was born, MSMQ was quick to grab the opportunity to replace SQl-server dependency with AD.

MSMQ vesrion 3.0 shipped as an optional windows component with WinXP and Windows 2003. Major new features like HTTP protocol support, Multicasting support, Distribution lists, etc were introduced. Security and reliability of the product was greatly enhanced in version 3.0

Key Conepts

MSMQ can be installed in two modes

Domain Mode - Integrated with Active Directory For MSMQ to be installed in domain mode the computer on which MSMQ is installed should be joined to a domain. The benefits of AD integratio are the following.
- You can create and use both public and private queues: Public queues are one whose properties(metadeta) are published in the Active directory. Public queues can be discovered by querying the AD. All publics in an enterpirse can be managed through the AD Management console. This acts as a single point of management Note: The messages in the public queues are not stored in AD, only the properties are stored in AD. The messages are stored on the computer on which the queue is create. Private queues are one whose properties(metadata) and messages(data) are published on the computer on which the queue was created. The name "Private" is rather misleading. You can send and receive message to a private queue from anywhere on the network using the Direct format name addressing scheme. Infact this is the only addressing scheme available for pivate queues.
- Integrated securit: MSMQ should be installed in domain mode if you require out of the box authentication and encryption of messages. This is because for authentication MSMQ signs the message using the user certificate registered in AD. For encryption the message is signed using the machine certificate registered in AD.
- Routin: If you need to route messages between different sites then you need MSMQ to be installed in domain mode. All routing configurations are stored in AD.
Workgroup Mode - Does not require Active Directory If the computer is joined to a domain MSMQ will be installed in workgroup mode. In this mode you have the capability to create and use only private queues. You don't have the out of the box authentication and encryption features. Howevere MSMQ allows you to attach any standard X.509 certificate to the message, but the receiver application need to authenticate the message itself using crypto APIs. The routing feature is also not available if MSMQ is installed in workgroup mode. If you dont require out of the box security or routing workgroup mode is your friend because it is easy to install and manage and doesn't have any external dependencies (AD).

APIs

You can program MSMQ using 4 APIs

C++ API: Coding is a really tedious and cumbersome, but obviously the most performant b.
COM API: COM wrapper over the C++ API. Better coding experience. Can use VB or C++ to program.
System.Messaging: .NET wrapper over the C++ API (uses PInvoke to call into the C++ APIs). Considerably imporved programming experience over C++ and COM APIs. Great increase in productivity. Performance is equal to or better than the COM APIs.
Indigo Queuing programming model: The next generation distributed programming model. Built on top of C++ APIs. Service oriented programming model. Performance is equal to or better than System.Messaging. Best programming experience. Microsoft will make all its investments in Indigo going forward.

Queues

Queues are nothing but First-In-First-Out(FIFO) datastructures.Queues can be created using the MSMQ management tool inside Computer management console MMC or programatically. Queues can be created and used dynamically. However you need to be aware of issues that might be caued by replication delays in the case of public queues. Private queues does not have such issues There is no physical limit on the number of queues that you can create on a given computer

Messages

Queues hold messages. Messages contain several properties and body. Some properties such as priority, correlationID, messageID etc are used to specify the contract of the message. Some properties like Label, body etc can contain the payload or data. The total size of the message including properties and body is limited to 4MB. The body of the message can contain pretty much anyting. You can ship COM objects that supports IPersistsStream interface, .NET Data types that can be serialized through binary or XML serializer. You need to be aware of the fact that message occupy both virtual memory and page pool. A fixed number of page pool bytes are consumed per message. If you have a lot of messages (like 1 million) on the disk you will start putting pressure on the page pool. In normal situaltions where sender and receiver applications are working smoothly you don't end up with a lot of messages in the queue. However in scenarios where there is a huge impedence mismatch between the sender and receiver or the receiver applicatio is down you might end up queuing a lot of messages.

MSMQ supports specifying storage quotas at the machine level and queue level. This ensures that you get warnings in the form of Negative acknowledgements when you hit the quota. This can help you prevent situations you have too many messages in the queue which will put pressure on system memory and page pool.

MSMQ support two types of messages

Express: Express messages are not stored on the disk and hence will be lost if the computer restarts/crashes. Express messages as the name suggests are very fast. They are used in scenarios where you dont really care about message loss. For example a stock applications sends stock quotes every second and it doesnt care if one of the quote was lost, it can catch up int the next second. However the application wants the message to reach the destination as fast as possible.
Recoverable: Recoverable messages are stored on the disk and hence can withstand computer crashes/restarts. Sending and receiving messages

You can send messages to both local and remote queues. You can send messages using several protocols such as TCP, IPX, HTTP, PGM (Multicasting). Receive is always done over RPC. You can receive messages from both local and remote queues.

Messages can be send and received as a part of a transaction.

MSMQ supports two types of transactions

Internal transaction: You can send and receive messages to MSMQ as a part of a transaction
MSDTC co-ordinated transaction You can send and receive messages as a part of a transaction where other resource managers such as SQL-Server can participate.

You can send and receive transactional messages to both local and remote queues.

High Availability and Scalability

MSMQ provides high availability by the use of Microsoft Clustering service. MSMQ supports both Active-Active and Active-Passive clusering. MSMQ provides scalability in two scenarios.

Load-balancing

In this scenario you can have a farm of MSMQ servers behind a hardware or software load balancer. Clients can send messages to the MSMQ servers through the load balancer.

If you are using the TCP protocol to send messages you are limited to only non-transactional sends.

If you are using the HTTP protocol you can send messages both transactionaly and non-transactionlly.

Load-levelling

In this scenario the queues are hosted on a central server (clustered for high availability). Client applications send messages to the queues. The reveiver applications are hosted on a farm of MSMQ servers. They receive and process messages from the central queues. Messages in the central queues are levelled across the server farm.

Security

MSMQ supports authentication and encryption of messages. When you send a message you set certain properties of the message to specify whether you want authentication and encryption or not. You can also select the has algorithm and encryption algorithm that msmq uses to sign and encrypt the message.

You can set the "requires authentication" property on the queue thereby resticting the applications to be able to send authenticated messages to the queue.

You can also set the encryption property of the queue to accept only encrypted messages.

Note: Messages are encrypted only on the wire, once they reach the destination queue messages are decrypted and stored as plain text.

-Anandraj

Integrating Indigo with COM applications

anandraj — Mon, 25 Jul 2005 04:36:00 GMT

In this blog I want to give you some insight into how Indigo is integrated with COM/COM+ applications. COM/COM+ has been at the heart of Microsoft's application server strategy for quite a while.VB COM applications have been one of the most famous breed of applications in any Microsoft shop before the advent of .NET. Indigo architects have not been blind to these facts, from the word go a lot of thought and energy has been put into action to smoothen migration of these applications to Indigo. Indigo integrates with COM/COM+ applications in the following ways

Exposing COM+ applications as Indigo end points

Indigo ships with a command line tool called COMSVCConfig. This tool can be used to expose existing COM+ applications as Indigo services. For example to expose the IBookFinance interface of a BookSample COM+ application you can run the following command.

ComSvcConfig /application:BookSample /interface:BookLogic.BookSales,IBookFinance /hosting:complus

The /hosting commandline parameter can be used to specify if you want to host the Indigo service inside the complus (dllhost.exe) or WAS (Windows Activation service). Once a COM interface is exposed as an Indigo service any Indigo client application can perform a secure, reliable, transacted (SRT) call to the service over a variety of protocols that Indigo supports like TCP, HTTP etc. What goes on the wire is SOAP.

Existing COM client applications can communicate with the new Indigo service

Indigo ships with a moniker that enables existing COM lient applications to talk to Indigo services. The moniker acts a broker in between the COM client application and Indigo service. What goes over the wire is pure SOAP. The COM client application can also choose the protocol used by the moniker to communicate with the service (Http, TCP etc). Here is an example of invoking the moniker.

Dim BookFinanceService As IBookFinance
Set BookFinanceService = _GetObject ("service:address=net.tcp://INTLIVE/{4a226b7a-aa77-419d-99ff-5b25f94078f3}/{e4e15887- b180-3e96-aedd-a6fd7b011b53}/{5d9a0467-affe-3250-988d-24b439686656},binding=customBinding,bindingconfig=TcpBinding")

What does this mean to you?

First of all you can start migrating you COM applications to Indigo in a phased manner. During the migration phyase you can have a combination of both Indigo client and COM Servers or Indigo Services and COM client applications. Adding Indigo support to COM applications enables sceanrios where the COM client applications can communicate with the service across firewalls and over internet. COM components today have this drawback where you have to go through a lot of hoops to enable clients to communicate with the service over firewalls.

What are the disadvantages?

Well I know what you are thinking Performance!!. In COM world you are used to pretty good performance because the DCOM packets are pretty compact when compared to SOAP on the wire. Several performance benchmarks have been done in the recent past to compare Indigo's performance with System.EnterpriseServices (.NET API for COM+)and it has been inferred that Indigo's performance is very much comparable if not better to System.EnterpriseServices. These results will be published in tune with Indigo relases and I will keep you all posted regarding the same.

-AnandRaj

All things queued - Integrating Indigo queued applications with Microsoft Message Queue

anandraj — Sat, 09 Jul 2005 04:57:00 GMT

In my last post I delved in depth about how Indigo Integrates with all other existing distributed stacks and how it enables interop with systems from other vendors. I will now start a series of posts starting with this digging deeper into each of those integration and interop points.

In this post I am going to talk about Indigo queuing and how it is integrated with Microsoft Message Queue (MSMQ).

Note: I assume that you have some basic knowledge on the workings of MSMQ. If you don't please feel free to reply to the post with your questions and I will clarify.

Microsoft Message Queue(MSMQ) is the enterprise queuing infrastructure from Microsoft. Indigo is the next generation SOA based distributed programming model from Microsoft. So the marriage between these two technologies is natural.

Indigo supports integration with MSMQ using two transport channel bindings

msmqIntegration binding
netProfileMsmq binding

MsmqIntegration channel

The msmqIntegration binding should be used in scenarios where you have an existing MSMQ application(sender or receiver) that you want to use to send/receive messages to a new Indigo queued application. When used in an Indigo queued application the MsmqIntegration channel binding performs the transformation between an Indigo message and an Msmq message to enable integration between the two applications.

For example if you have an MSMQ sender app that sends messages to a queue and you write a new Indigo queued service that receives and processes the message using the MsmqIntegration channel, the channel will convert the MSMQ message into an Indigo message before passing it to the Indigo pipeline for dispatching.

What goes on the wire is an MSMQ message that is digestible by older MSMQ applications.

NetProfileMsmq channel

The NetprofileMsmq binding should be used in scenarios where you have Indigo client and service applications and you want to use a queue as conduit for the communication between them. In this case the Indigo SOAP message is stuffed into an MSMQ message and transported over MSMQ infrastructure from the client to a queue. When the Indigo service receives the message from a queue the netProfileMsmq channel extracts the Indigo SOAP message from the MSMQ message and passes it to the Indigo pipeline for dispatching.

You can think of the Indigo Queued programming model as a new API for MSMQ. If you are familiar with COM+ Queued components(QC) this should be very similar (but much more robust).

The Indigo Queued programming model has the following advantages over existing MSMQ APIs.

You don't have to write any MSMQ specific plumbing code in your client or service application. On the service side you have to declare your service contract, implement it, select one of the above channels depending on your scenario and tell the service what queue to listen/send to by specifying the queue address in the configuration file of the service and that's it. The Channels will take care of MSMQ plumbing. You concentrate more on your business logic. You dont have to write any complex listener code for listening on the queues and dispatching the message. The channel acts as a listener and a proxy for you.
Indigo Queuing programming model has the poison message handling feature. A message is deemed poisonous after several attempts to process the message have failed due to some reason or the other (like malformed message). The failure might be permanent or transient. The poison message will prevent processing of other messages in the queue. There are settings in the channel which allows you to control how many times you want to try processing the message before moving it to a retry queue, after how long you want to bring the message from the retry queue back to the main queue to attempt processing again and what is the maximum number of retries after which the message is dumped into a final resting queue (poison queue). This feature does not exist in MSMQ today. Poison message is a very common problem and I bet you are writing some kind of code to handle this today in your application.
MSMQ provides Encryption and Authentication security features at the transport level. However MSMQ does not provide any Application to Application authentication or encryption feature. Moreover MSMQ security is dependant on Active directory which can be painful and will not work if the computer is not joined to an Active directory domain. Indigo queuing on the other hand layers SOAP security on top of MSMQ. Indigo Queued programming model provides app to app authentication and encryption. It does not require Active directory. Indigo queued programming model also supports enabling the MSMQ transport level security through the configuration settings in the channel.

The bottom line is MSMQ is getting better and brighter with the addition of Indigo queuing.

If you are an SQL Server 2005 SQL Service Broker (SSB) fan hold your breath. There will be an Indigo SSB Queued channel soon that will enable you to write Indigo queued applications that can use SSB queues. The good news is your Client and service code will have minimal or no changes when you switch from the MSMQ transport channels to the SSB channel. All you should have to do is to select the Indigo SSB channel in your configuration file instead of the MSMQ channel.

That is the beauty of Indigo! It provides a programming model that is truly protocol and transport agnostic.

-AnandRaj

Indigo and Integrating

anandraj — Sat, 02 Jul 2005 05:56:00 GMT

Mey has made a great statement about why Integration should not be an after thought, I am pretty sure you are convinced by now it is not. Now let’s try to understand how Indigo the next generation distributed programming model from Microsoft enables integration between different systems. The following are the key pillars of Indigo

Unification: Unification is about how Indigo provides a single composable programming model for all the existing distributed stacks.
Integration: Integration is about how different Microsoft technologies work together
Interop: Interop is about how different Microsoft technologies work with systems from other vendors such as IBM, BEA etc

Today you have different programming models to accomplish different tasks. For example if you want to do queuing you need to use System.Messaging, If you want to do transactions you need to use System.EnterpriseServices, if you need to do security you need to use WSE , it goes on and on. As a programmer you are constantly forced to context switch between these different programming models. Indigo does a great job of providing a unified programming model where in you can compose all these different functionalities into your application without having to do a lot of context switching. Going forward you have to just learn one programming model i.e. Indigo. If you want queuing you just add an attribute to your Indigo Service contract that makes it queued, if you want to secure the communication with your Indigo service you just add the appropriate security attributes for authentication and privacy, if you want transaction you just add the transaction attribute. This new programming paradigm should help you concentrate more on what you care most about; your business logic.

Ok, now that I have got you exited about Indigo I also realize that you cannot just throw away all the investments that you made on developing the applications that you have today using System.Messaging, System.EnterpriseServices, WSE , ASMX etc. If I were you I will have the following questions.

While I develop my new applications with Indigo will it work with my existing applications? For example Will my new Indigo queued application work with my existing queued application built using System.Messaging application?
Can I migrate my applications to Indigo “one at a time” instead of “everything at once”

Fortunately the answers to the both the questions above are an astounding “Yes”. Firstly we can derive great solace from the fact that Indigo team is made of the same developers who built System.Messaging, System.EnterpriseServices, WSE ASMX, .NET Remoting etc. This means integration with existing systems was on the minds of every Indigo developer from the word go. The result is a great integration story.

You can run any of your existing applications side-by-side with Indigo applications.
Existing MSMQ applications seamlessly integrate with new Indigo queued applications
WSE 3.0 applications are wire compatible with Indigo.
ASP.NET Webservice (ASMX) applications are wire compatible with Indigo.

The only caveat is .NET Remoting. If you have an application that uses .NET Remoting you are out of luck. Indigo apps are not wire compatible with your .NET remoting applications.

Now let’s move on to Interoperability. Enterpise infrastructure is comprised of systems deployed over time by various vendors; these diverse systems have to work with each other. How does Indigo help you to interop with systems built on top of non-microsoft software?

Indigo achieves interoperability through the use of different standard WebService protocols like WS-I basic profile, WS-Security, WS-AutomicTransactions, WS-ReliableMessaging etc. Microsoft is working closely with IBM, BEA and other key players in the industry to drive these protocols to standardization. The adoption of these protocol standards by different vendors mean increased interoperability between your diverse systems. This essentially boils down to this; your Indigo app that runs on Windows 2003 or Longhorn web server can do reliable messaging with a Java application running on a IBM mainframe.

I am excited about this because this will make the software world a heaven where diverse systems can co-exist peacefully. Today we are forced to build bridges that bring these systems together. We spend a lot of time, money and energy in building those huge monolithic bridges. Most of the time these bridges become a bottleneck or a single point of failure.I have great hopes that the emerging WebService standards will bring a lasting solution for interoperability.